
guaardvark
Local image, video, music-video and Film Crew generation, voice and music, upscaling, LoRA training, Hugging Face model onboarding, agent swarms, local knowledge, code intelligence and supervised outreach, driven from Claude Code through the guaardvark MCP server.
- GitHub stars
- 327
Install
/plugin install guaardvark@guaardvarkRun it inside Claude Code after adding the marketplace with /plugin marketplace add guaardvark/guaardvark.
Contents
Discovered from sourceSkills 16
Cast Library and LoRA training
Read setup first; training needs the lora_trainer plugin (CUDA, bf16) and an
installed train-ready base (Z-Image, SDXL or FLUX family). B=${GUAARDVARK_URL:-http://localhost:5000}.
The pipeline
- Create the subject
curl -s -X POST $B/api/cast-library/subjects -H 'Content-Type: application/json' -d '{ "kind": "character", "name": "Mara", "description": "late 30s, short grey hair, freckles", "trigger_word": "mara_v1", "voice_id": null }'kindischaracter,environmentorprop.GET $B/api/cast-librarylists subjects; the numericidis whatgenerate_imageand batch routes take assubject_ids. - Upload reference images (5 to 20 clear shots, varied angles, same subject):
curl -s -X POST $B/api/cast-library/subjects/$ID/upload-refs -F files=@1.jpg -F files=@2.jpgAsk first whether the person consented; do not train on someone who has not. - Vision bible from the refs:
POST $B/api/cast-library/subjects/$ID/bible/from-refs(the vision model writes the identity description that every prompt inherits). - Plan and generate training samples:
POST .../$ID/planthenPOST .../$ID/generate(cancel with.../generate/cancel). ReviewGET .../$ID/samples; view one withGET .../samples/<sample_id>/image; drop bad ones withDELETE .../samples/<sample_id>orPOST .../samples/<sample_id>/regenerate. - Approve samples:
POST .../$ID/samples/approve. - Train:
POST $B/api/cast-library/subjects/$ID/trainwith optional{"training_settings": {...}}. A 409already_trainingmeans wait; atrain_base_not_readyerror names the base model to install first. Cancel:.../train/cancel. Progress:GET $B/api/cast-library/subjects/$ID(status, base model, LoRA path when done). - Use it:
generate_imagewithsubject_ids=[ID]; batch image/video routes withsubject_ids; the Film Crew casts it; the music-video Director locks it per cut.
Rules
- Training is a GPU job of tens of minutes; say so and check
inspect_gpufor conflicts first. - The trigger word is applied by the system when
subject_idsis passed; the user does not need to type it. - Everything stays local: refs, samples and the LoRA file under
data/.
Code intelligence with Guaardvark (MCP tools)
| tool | use |
|---|---|
list_code_repositories | which folders are marked as repositories (gives the folder id) |
search_codebase | search Guaardvark's own source: by meaning once a code search index is built (System Map page), otherwise by words and names |
search_code | case-insensitive regex across files |
list_code_files | directory listing to orient |
read_code | a file, or a line range of it (start_line, end_line); large files come back in pages |
read_ast_node | one class or function by name from a Python file |
get_repository_map | PageRank-ranked architectural map for a folder id |
get_dependency_graph | file-level import graph for a folder id |
analyze_code | structure, patterns, improvement notes for a file |
codegen | a complete modified version of a file from instructions (writes a new file) |
verify_change | confirm text now exists in a file after an edit |
map_codebase | run the System Mapper: the most severe findings, then the numbers |
system_map_query | ask the System Mapper about Guaardvark's code: where <feature>, module <path>, symbol <name> (callers, callees), route <path>, tool <name>, subsystem <name>, impact <path> (dependents, tests to run), findings <scope> |
self_improvement_status | can the self-improvement engine run (lock, flag, already running) |
swarm_status | the coding swarm (see the swarm skill) |
Pattern
list_code_repositories→ folder id.get_repository_mapfor the shape,search_codebasefor the question.- Read the exact code (
read_code/read_ast_node) before answering; quote file and line. - Edits: make them with your own editor tools in the user's checkout;
verify_changeafterwards.
Rules
- These tools read Guaardvark's index, which can lag the working tree; when it matters, read the file from disk too.
map_codebase,system_map_queryandself_improvement_statusare read-only here. Dispatching a fix to the self-improvement engine is done in the Studio, on purpose.
Film Crew with Guaardvark
Read setup first. Needs comfyui; casting with trained characters needs
lora_trainer (see the cast skill). Renders take minutes per shot.
Start: MCP start_film_crew
script_text: a screenplay or a plain scene list. A logline works; the screenwriter expands it.name(defaults to the first line),video_model(defaults to the active scene model,minimax-h3-int8on a typical box;wan22-5bfor silent i2v).- The screenwriter starts at once. Casting, storyboards and GPU renders wait for the user in Studio (Agents > Film Crew) or through the routes below.
Stages and gates (REST)
B=${GUAARDVARK_URL:-http://localhost:5000}, P = production id from the tool.
| stage | call | note |
|---|---|---|
| status | GET $B/api/production/$P | stage, subjects, shots, errors |
| subjects found by the writer | GET $B/api/production/$P/subjects | characters, environments, props |
| cast one subject | POST $B/api/production/$P/cast/<subject_id> with {"action": ...} | link to an existing Cast Library subject or plan a new one |
| confirm casting | POST $B/api/production/$P/casting/confirm | user gate: every subject needs a cast plan first |
| storyboard shot image | GET $B/api/production/$P/storyboard/shot/<shot_id>/image | review keyframes |
| redo one shot | POST $B/api/production/$P/storyboard/shot/<shot_id>/regenerate | |
| approve storyboard | POST $B/api/production/$P/storyboard/approve | user gate: releases the renders. A 409 with flagged_shots lists frames the curator flagged; show them, and resend with {"confirm_flagged": true} only if the user says to render them anyway |
| retry a failed stage | POST $B/api/production/$P/retry | |
| list / delete | GET $B/api/production, DELETE $B/api/production/$P |
Without the MCP tool: POST $B/api/production with {"name", "script_text", "settings": {"video_model": "..."}}.
How to run it well
- Show the screenwriter's output (scenes, shots, subjects) before touching casting.
- Consistent faces come from trained LoRAs. If a subject has no Cast entry, offer to create and train one (the cast skill) or cast it "as described" and warn identity may drift.
- State the render cost before the storyboard approval: shots × seconds × model speed.
- The editor assembles the shots in the Video Editor; the result appears in the media library.
- Nothing posts or uploads. The film stays on the machine unless the user moves it.
Images with Guaardvark
Read setup first if the backend or the comfyui plugin state is unknown.
One image: MCP generate_image
promptis scene, pose, lighting, setting. Plain prose. Do not paste JSON or tag soup; the default model (Z-Image Turbo) reads prompts as language, and SD-era tag lists hurt it.modeldefaultautopicks the best downloaded model. Only override when the user names one:zimage-turbo,krea2-turbo,krea2-raw,flux-dev,sd-xl,sdxl-turbo,realistic-vision,epic-realism.width/height: 512, 768 or 1024.style: realistic, artistic, anime, photographic, digital-art.- Consistent character: pass
subject_ids=[<cast id>]as its own array. Never put the trigger word alone in the prompt and expect the LoRA to load. Find ids withGET /api/cast-library(see the cast skill). - On-image text: quote the exact words in double quotes inside the prompt.
- The tool returns the image URL (
/api/outputs/generated_images/<file>.png, relative to the backend), the model that ran, steps, seed and whether a Cast LoRA was applied. Show the URL and the prompt you used. Measured: 768x768 on Z-Image Turbo in ~20 s on a free 16 GB card. - Over MCP the call queues by default (
wait_for_resultdefaults to false there) and returnsImage queued as batch ImageBatch_...at once. Pollget_generation_status(batch_id=...)every few seconds untilcompleted; it returns the file URL. Passwait_for_result: trueto block for the render instead (allowed up to 30 minutes). A call that exceeds the server's timeout answers with an error that says the render is still running; it is not lost. - A failed call carries the backend's reason (plugin off, out of memory, bad model). Read it
and act on it;
inspect_gpuandGET /api/plugins/statusare the two checks that resolve most.
Photo edits over MCP run as jobs
edit_image, outpaint_image and remove_background answer at once over
MCP with ... queued as job tooljob_...: the work runs in the Guaardvark backend and keeps
going if the client disconnects. Poll get_generation_status(batch_id="tooljob_...") every few
seconds until done (it gives the file URL) or failed (it gives the error). Pass
wait_for_result: true to wait up to 60 s (half the server's MCP timeout when that is
shorter) for the result; a longer edit still comes back as the job id. A Qwen-Image-Edit takes
about three minutes on a 16 GB card, and a job waits its turn while another render holds the GPU.
Edit an existing image: MCP edit_image
instructionis the change ("put a cowboy hat on him", "make the shirt red"). The image the user just attached is used automatically; otherwise passimageas a path or URL. Optionalreference_image_2/reference_image_3(Qwen-Image-Edit only) for extra people or style; the call is refused when a reference cannot be read or the edit would run on another backend.modelautouses Qwen-Image-Edit when installed, else FLUX.1 Kontext. With neither installed the call is refused and names the pack; relay that instead of retrying. Override withqwen-image-editorkontext. Install those packs from Manage Image Models → Image editing (qwen-image-edit,flux-kontext-dev) — do not Install unless the user asked. Naming another downloaded image model runs a light img2img pass that keeps most of the picture.- Same canvas, same pose. For a brand-new picture use
generate_image. For a new scene that keeps a face usegenerate_identity.
New scene from a face: MCP generate_identity (off by default)
- Not exposed unless the server runs with
GUAARDVARK_IDENTITY_TOOL=1: the likeness it keeps has not passed verification yet. If the tool is absent, say so and offeredit_imageinstead. - Attach a likeness the user has the right to use (their photo or a Cast subject they uploaded).
consentedmust betrue. Refuse if they have not confirmed that. promptis the new scene. Needs the PuLID identity pack (pulid-flux; Manage Image Models → Image editing installs it with its face files, EVA02-CLIP andflux-dev). Comfy must have been restarted after the PuLID-Flux custom node was added.- This is not a face swap onto an existing poster, and not an instruction edit of the same photo.
Background remove: MCP remove_background
- Cuts the subject out of the attached photo (transparent PNG). ONNX matting, no diffusion; needs a background-removal model from Manage Image Models → Image editing.
- For a new background, remove first then
edit_image/generate_identity, or describe the new scene inedit_imageif Qwen-Image-Edit is installed.
Change part of a photo / outpaint
edit_image: change or remove something ("remove the coffee cup"). Runs on Qwen-Image-Edit or FLUX.1 Kontext; refused when neither is installed.outpaint_image: extend the canvas (left/right/top/bottompixels) and fill. Prefers Qwen.
Many images: REST batch
B=${GUAARDVARK_URL:-http://localhost:5000}
curl -s -X POST $B/api/batch-image/generate/prompts -H 'Content-Type: application/json' -d '{
"prompts": ["prompt one", "prompt two"],
"model": "auto",
"subject_ids": []
}'
promptsmay be strings or{"prompt": "..."}objects. There is a per-batch maximum; if the server answers 400 "Too many prompts", split the list.- Optional
adapters(user LoRAs from the models skill) andsubject_ids(Cast Library). - The response is
data.batch_id(ImageBatch_<date>_<n>). PollGET $B/api/batch-image/status/<batch_id>?include_results=true:statusgoes running → completed, withcompleted_images/total_images,output_dir, and oneresults[]entry per prompt (success,image_path,thumbnail_path,generation_time,metadata.model_used). A contact sheet:GET $B/api/batch-image/preview/<batch_id>; one file:GET $B/api/batch-image/image/<batch_id>/<image_name>(the basename ofimage_path). Cancel withPOST $B/api/batch-image/cancel/<batch_id>. Measured: one 1024x1024 prompt completed in ~30 s. - Helpers:
POST /api/batch-image/enhance-prompt,/analyze-prompt,/expand-concept(JSON body with the prompt) when the user wants prompt help before spending GPU time.
Rules
- Say which model actually ran (the response names it). Do not promise a model that is not installed.
- Generation time depends on the GPU; a first image after Ollama held the card can take longer because the orchestrator swaps models. That is normal.
- Never upload the user's images anywhere. Everything here is local.
Knowledge with Guaardvark (MCP tools)
Everything here is the guaardvark MCP server. All retrieval is local (Ollama embeddings,
hybrid search, cross-encoder reranking on the user's GPU).
Find and read
| tool | use |
|---|---|
search_knowledge_base | the question in plain words; returns passages with sources. Cite them. |
summarize_corpus | what the whole knowledge base covers, before searching blind |
list_documents | which documents are indexed and how many passages each has |
get_document_outline | sections or pages of one document, in order |
read_document_section | the actual text of one section or page, no search |
process_file | extract content from a PDF, DOCX, CSV, Excel, image or other file the user names |
index_document | add a file in Guaardvark's uploads, outputs or install folder to the knowledge base, so the tools above find it |
fetch_url | title, description and main text of one web page; pass query (the user's question) to get the passage about it instead of the top of the page |
web_search | web results (titles, snippets, URLs) when the answer is not local; source names the search engine that answered, and no_results says when it found nothing |
Pattern: summarize_corpus or list_documents → search_knowledge_base → read the exact
section with get_document_outline + read_document_section before quoting.
Remember
save_memory: a fact, preference or instruction to keep across sessions. Use when the user says "remember", or states a durable preference. One fact per call.search_memory: check before assuming; recalled memories are what was true when saved.update_memory: correct a saved fact, preference or note in place, by the IDsearch_memoryprints, instead of saving a second version next to it.delete_memory: forget one entry by its ID when the user asks. It cannot be undone.- Over MCP these two change only fact, preference and note entries; lessons and Guaardvark's own entries are managed on the Agent Memory page.
Rules
- Quote the source (document name and section) for any claim that came from retrieval.
- If retrieval returns nothing, say so; do not fill the gap from general knowledge without labelling it.
index_documentadds one file at a time and waits up towait_seconds(default 45) for the indexer; a long PDF may still be indexing when it answers, andlist_documentsshows it once done. Whole folders are indexed from the Studio's Library page.
Models and LoRAs with Guaardvark
B=${GUAARDVARK_URL:-http://localhost:5000}. Nothing downloads without an explicit Install;
the product never phones home on its own. Always confirm the size and the licence with the user first.
What is there
- Video:
GET $B/api/batch-video/models(registry + user catalog, withis_downloaded/is_ready,missing_files,capabilities). - Image:
GET $B/api/batch-image/models. - Download a registry model:
POST $B/api/batch-video/models/download {"model_id": "wan22-14b"}/POST $B/api/batch-image/models/download {"model_path": "<id>"}; progress atGET .../models/download-status. - Chat photo-tool packs install from Manage Image Models → Image editing:
qwen-image-edit(~28 GB with its encoder and VAE),flux-kontext-dev,pulid-flux(with its face files, EVA02-CLIP andflux-dev),bgremove-birefnet/bgremove-u2net. The rows come back aseditinginGET $B/api/batch-image/models; Install withPOST $B/api/batch-image/models/download {"model_path": "comfy:<pack id>"}. Confirm size and licence; never start those downloads unless the user asked.
Add from a Hugging Face URL (the "paste a link" path)
- Inspect:
POST $B/api/batch-video/models/from-hf {"url": "https://huggingface.co/<org>/<repo>"}(or thebatch-imagetwin). The server reads the repo and returns what it found: files, revision, the likelyrole(checkpoint, LoRA, VAE, text encoder),family, whether amodel_indexexists, and a proposed catalog entry. - Show that to the user: which file(s), role, family, size, licence.
- Register (and optionally install) with
POST $B/api/batch-video/models/user(orbatch-image) sending the entry back.roleandfamily(image) orlike(video) decide where the Studio offers it. The server re-inspects the repo; do not sendhas_model_indexfrom the client. Gated repos needHF_TOKEN; confirm size and licence from the Look-up response first. - Remove:
DELETE .../models/user/<model_id>with{"delete_files": true|false}.
Using a user LoRA
- Batch image:
adapters: [{"id": "<user model id>", "scale": 0.8}]on/generate/prompts. - Batch video:
lora_name+lora_strength, oradapterson/generate/text. - A trained Cast LoRA is different: it rides on
subject_ids(the cast skill).
Rules
- A direct
.safetensorsURL works when it lives on huggingface.co; the inspector needs the repo to read the file list. For other hosts, ask the user to download the file and drop it in the Studio's model folder instead. - Declared limits (min steps, max frames) come from the registry entry; a user model inherits
them from
like/family. Do not invent numbers for a model the registry does not know.
Music video with Guaardvark
Read setup first. Needs the comfyui plugin and an installed i2v model
(wan22-5b by default; minimax-h3-int8 for frame-matched continuity).
Start: MCP generate_music_video
song: document id or a filesystem path to mp3/wav/flac/ogg. A path is uploaded for the user.style_prompt: mood, palette, movement, era ("grainy 16 mm, neon rain, slow dolly, 1984").name(defaults to the song file name),i2v_model(defaults to the active i2v model).- The tool analyses the song (tempo, beats, sections), writes one unique prompt per cut and stops at the approval gate. It does not spend GPU rendering clips. Tell the user that.
The gate and the rest of the pipeline (REST)
B=${GUAARDVARK_URL:-http://localhost:5000}, ID = the project id the tool returned.
| step | call | what it does |
|---|---|---|
| read the plan | GET $B/api/music-video/$ID | cuts, prompts, status, storyboard state |
| edit prompts | POST $B/api/music-video/$ID/plan with the edited cut prompts (and optional global style) | operator edits before render |
| regenerate the plan | POST $B/api/music-video/$ID/regenerate-plan or /replan | new prompts from the Director |
| storyboards first | POST $B/api/music-video/$ID/generate-storyboards | one keyframe still per cut, thumbnails-first review |
| redo one keyframe | POST $B/api/music-video/$ID/regen-storyboard/<idx> | keeps the others |
| approve | POST $B/api/music-video/$ID/approve | releases per-clip generation. Only on the user's explicit yes |
| re-analyse | POST $B/api/music-video/$ID/analyze | restart song analysis |
| cancel | POST $B/api/music-video/$ID/cancel | |
| list all | GET $B/api/music-video (?project_id=<id> or ?project_id=none to filter) | |
| progress | GET $B/api/music-video/$ID/progress | stages, the clip in flight (sampler step/steps), ETA. Report this instead of guessing |
| rename / file under a project | PATCH $B/api/music-video/$ID with {name?, project_id?, notes?, tags?} | the finished video joins that project's Documents |
| edit one cut | POST $B/api/music-video/$ID/cut/<idx> with {prompt?, seed?, locked?} | at the approval gate it only changes the plan; while generating it re-renders that cut: ask first |
| reopen a finished video | POST $B/api/music-video/$ID/revise | back to the approval gate, rendered clips kept; on approve only changed cuts render. /replan re-renders all |
| copy | POST $B/api/music-video/$ID/duplicate with {name?, project_id?, keep_clips?} | new video at the approval gate |
| versions | GET/POST $B/api/music-video/$ID/revisions, POST .../revisions/<rid>/restore | saved on approve, reassemble, revise, restore; restore lands at the approval gate |
| a project's videos | GET $B/api/projects/<id>/videos | music videos and Film Crew filed under it |
Creating a project without the MCP tool: POST $B/api/music-video with
{"name", "song_document_id", "style_prompt", "settings": {"i2v_model": "..."}, "user_treatment": "optional short story", "project_id": optional}.
The MCP tool takes project (a Project id or exact name).
user_treatment becomes the creative source the Director writes cuts from.
How to run it well
- Start the project, show the cut list and the beat count.
- Offer storyboards before approval: they are cheap and catch a wrong style early.
- Ask for approval in plain words with the cost: number of cuts times the clip time for the
chosen model. Approve only after a clear yes, and only once
GET $B/api/music-video/$IDshowscurrent_stageawaiting_approvalwith cuts: before analysis finishes the approve route answers 409. - Character in the video: create a Cast subject and train a LoRA first (the cast skill) so the Director can lock identity across cuts.
- The finished cut is assembled to the detected beat grid. The Video Editor page can trim or overlay text afterwards.
Music and sound effects with Guaardvark
Read setup first; needs the audio_foundry plugin. B=${GUAARDVARK_URL:-http://localhost:5000}.
ACE-Step takes ~10 GB VRAM; the orchestrator evicts other models while it runs.
A song
Over MCP, call generate_music (style, optional lyrics, seconds up to 240,
instrumental, seed, model); it returns a job_id for get_generation_status, which reports
the file and a download link when the song is done. Without MCP, use REST:
curl -s -X POST $B/api/audio-foundry/generate/music -H 'Content-Type: application/json' -d '{
"style_prompt": "indie folk, fingerpicked acoustic guitar, warm male vocal, 96 bpm, intimate",
"lyrics": "[verse]\nLines here\n[chorus]\nHook here",
"negative_prompt": "distorted, muddy, synth",
"duration_s": 90, "instrumental_only": false,
"output_format": "wav", "seed": 1234, "async": true
}'
model: omit (or"ace-step") for ACE-Step v1;"ace-step-1.5"for ACE-Step 1.5 (MIT, 48 kHz, turbo model plus a planner) once it is installed in Audio Studio → Manage models. Not installed →400withneeds_install; nothing is downloaded.- For v1,
style_promptuses ACE-Step's tag vocabulary: genre, instruments, mood, tempo, vocal type. Vague words ("professional", "futuristic") drift to the model's prior; be concrete.POST $B/api/audio-foundry/rewrite-music-promptturns plain English into that tag form. For 1.5, write a sentence. The loaded 1.7B planner expands it into a caption, lyrics, tempo, key and meter before the render, and keeps a bpm or key written in the sentence.POST $B/api/audio-foundry/plan-musicwith the same body as generate returns that arrangement and does not render.negative_promptis accepted but neither ACE-Step model applies it (the result saysnegative_prompt_applied: false). lyricswith[verse]/[chorus]/[bridge]markers; omit or setinstrumental_only: true.duration_sup to 240. Always send"async": truefor anything over a few seconds.- Response
202 {"job_id"}→ pollGET $B/api/audio-foundry/jobs/<job_id>; done whenstatusisdone, withpath,document_id,duration_s,seed. Reuse the seed to reproduce a take.
A sound effect or ambience
curl -s -X POST $B/api/audio-foundry/generate/fx -H 'Content-Type: application/json' -d '{
"prompt": "rain on a tin roof, distant thunder, no music", "duration_s": 20, "output_format": "wav", "async": true
}'
duration_s up to 47. Same job polling.
Then
- A finished song can go straight into the music skill-video as the
song. - Jobs:
GET $B/api/audio-foundry/jobslists,DELETEclears finished ones. - Attribution comes back in the job result; keep it with the file if the user publishes.
Operating Guaardvark
B=${GUAARDVARK_URL:-http://localhost:5000}. MCP tools: inspect_gpu, read_logs, swarm_status, self_improvement_status.
GPU and plugins
inspect_gpu(MCP): nvidia-smi, the exclusive lock (Ollama vs video), orchestrator slots, running plugins.GET $B/api/plugins/statusandGET $B/api/plugins/<id>/health;GET $B/api/plugins/stats/gpu.POST $B/api/plugins/<id>/start | stop | restart | enable | disableforollama,comfyui,audio_foundry,upscaling,lora_trainer,swarm,vision_pipeline,discord,gpu_embedding,video_editor. Start services through these routes, never with systemctl; the orchestrator must know what is resident.GET $B/api/plugins/<id>/logs?lines=200for a plugin's own log.- Memory orchestrator:
GET $B/api/gpu/memory/status,GET/POST $B/api/gpu/memory/tier.POST .../evict {"slot_id"}frees a slot; use only when the user asks to clear the card.
Logs and tasks
read_logs(MCP) tails a file underlogs/(backend, celery, plugin logs).GET $B/api/celery/tasksshows active Celery tasks;GET $B/api/meta/...job routes list and retry jobs (POST $B/api/meta/retry_job/<job_id>,POST $B/api/meta/cancel_job/<job_id>).- A "stuck" job is usually the GPU lock held by another plugin: check
inspect_gpubefore retrying.
Interconnector (other machines sync from this one)
GET $B/api/interconnector/status,GET .../nodes,GET .../sync/history?limit=20.POST $B/api/interconnector/sync/manualwith{"direction": "push", "entities": [...], "sync_files": true, "profile_name": "..."};POST .../nodes/<node_id>/testchecks a link.- Other boxes never
git pull; they sync through this. A frontend change needs a bundle rebuild there afterwards.
Autoresearch (overnight RAG tuning)
POST $B/api/autoresearch/start {"mode": "...", "budget_hours": 6}; GET .../status,
GET .../runs, GET .../runs/<run_id>/ledger; POST .../stop. Promotions of a better config are
explicit: POST .../promotions/<config_id>/activate, POST .../promotions/revert.
A run executes on its own Celery worker (research@<host>, log celery_research.log), started
with the run and stopped about two minutes after it; the main worker stays free. Its model calls wait
while a render or training job holds the GPU. POST .../stop ends it at its next model call.
Infographic
curl -s -X POST $B/api/infographic/generate -H 'Content-Type: application/json' -d '{
"title": "Three numbers", "scene": "clean flat design, navy and white",
"callouts": ["42% faster", "0 cloud calls", "1 GPU"], "hashtags": ["#localai"], "footer": "guaardvark.com",
"style": "flat_vector", "aspect": "1:1", "seed": 3
}'
Styles: editorial, flat_vector, photo_real, comic; aspects 16:9, 1:1, 9:16, 4:5, 3:2.
The call waits for the GPU and returns when the PNG is saved (image_url, also filed in the Library
under Images › Infographics). model is optional: Krea 2 Turbo when installed, else Z-Image Turbo;
qwen-image-21 is non-commercial. GET $B/api/infographic/status lists the models and which is installed.
Rules
- Restarting a plugin kills its jobs. Check the queues and say so before restarting.
- Never restart the backend or Celery from here; tell the user what to run in the checkout.
Outreach with Guaardvark (MCP tools, supervised)
Nothing here posts. Approval happens in the Studio's Approvals page, by design, and a
Reddit, YouTube or Facebook draft posts only when the user presses Post now there with the
Agent Screen open. X and LinkedIn are not offered for outreach; the user's own LinkedIn
account takes posts through request_publish instead.
| tool | use |
|---|---|
outreach_status | enabled / cadence / per-platform posting modes |
outreach_list_queue | drafts, default status='drafted' (pending review) |
outreach_draft_post | draft a comment or share post for a platform and thread URL; grounded in the thread and the user's pitch sheet, graded, then queued |
outreach_reject_draft | reject a draft so it will not post; fails and says why once the post is being submitted or is already out |
request_publish | ask to post the user's own announcement to one of their social connections; it waits on the Approvals page |
Pattern
outreach_statusto confirm the loop is on.outreach_draft_postwith the platform and the thread. Show the draft and its grade.- The user edits, approves and starts the post in the Studio. You can reject on their word; you cannot approve or post.
Rules
- Do not bulk-draft across many threads. The system is for keeping up with engagement on the user's own products and topics, not for volume.
- Say clearly that a draft is queued, not posted.
Guaardvark: connect and check
Guaardvark runs on the user's own machine. Everything below is local; nothing leaves the box.
Where it is
- Backend URL:
${GUAARDVARK_URL:-http://localhost:5000}. The macOS default port is 5055 (AirPlay owns 5000). The web UI is on port 5173 in dev, or the same port as the backend in Docker. - Health: !
curl -sf ${GUAARDVARK_URL:-http://localhost:5000}/api/health || curl -sf http://localhost:5055/api/health || echo "backend not reachable on 5000 or 5055" - Plugins: !
curl -sf ${GUAARDVARK_URL:-http://localhost:5000}/api/plugins/status || echo "plugin status unavailable"
If the backend is not reachable, tell the user to start it from the Guaardvark checkout
(./start.sh, or docker compose up), then retry. Do not try to start it yourself.
Two ways to drive it
- MCP tools (preferred when present): the
guaardvarkMCP server exposes RAG, memory, code intelligence, file processing, web fetch, image/video/animation/music-video/film-crew generation, songs and speech (generate_music,generate_speech),get_generation_statusfor any queued batch or song, outreach drafting and GPU/log inspection. Generation tools queue by default over MCP and return a batch id. In Claude Code they appear asmcp__guaardvark__<tool>afterpython -m backend.mcp install, or asmcp__plugin_guaardvark_guaardvark__<tool>when the plugin was installed from the marketplace. Install once from the checkout:
Restart the client after installing so it re-reads its MCP config.python -m backend.mcp install # writes the server entry into every detected client: Claude Code, Codex, # Cursor, Grok, Antigravity, opencode, Claude Desktop, Zed, Gemini python -m backend.mcp doctor # self-test + stale-config scan python -m backend.mcp list-tools # what is exposed right now - REST for everything the MCP policy does not expose (voice cloning, sound effects, upscaling, batches,
cast/LoRA training, swarm launch, interconnector, plugins). Use
curlagainst the backend URL. Responses are wrapped as{"success": bool, "data": {...}, "message": str}on most routes; a few older routes return the bare object. Readdatawhen it is present.
Before generating anything
- GPU services are plugins. Check
GET /api/plugins/status; a generation route answers 503 when its plugin is not running. Start one withPOST /api/plugins/<id>/startwhere<id>iscomfyui(image + video),audio_foundry(voice, music, FX),upscaling,lora_trainer,swarm. Only one heavy model owns the GPU at a time; the orchestrator evicts Ollama for video and vice versa, so a first call after a switch is slow.inspect_gpu(MCP) shows who holds it. - Installed models:
GET /api/batch-video/modelsandGET /api/batch-image/modelslist every registry entry with capabilities; checkis_readybefore naming a model. Nothing downloads without an explicit Install, so if a model is missing say so and offerPOST /api/batch-video/models/download {"model_id": "..."}(or the image route with{"model_path": "..."}). - Outputs land under
data/outputs/in the checkout and are also served read-only over MCP asguaardvark://outputs/...resources.
Skills in this pack
| Skill | Use for |
|---|---|
| image | one image, edits, cast characters, batch image runs |
| video | one clip, image-to-video, first/last frame, batch video runs, MiniMax H3 with sound |
| music-video | a song in, a beat-cut music video out, with the approval gate |
| film-crew | screenplay to finished short: writer, casting, storyboards, render, edit |
| voice | narration, TTS, consent-gated voice cloning |
| music | full songs with lyrics, instrumentals, sound effects |
| upscale | 2x to 8K upscaling of images and video |
| video-editor | edit existing clips: cut to a song's beat, trim dead air, titles, Shotcut projects |
| cast | Cast Library subjects and LoRA training for consistent characters |
| models | add any Hugging Face model or LoRA from a URL |
| swarm | parallel coding agents in git worktrees from a plan file |
| knowledge | the user's indexed documents, memory, web fetch |
| code | code search, repository map, self-improvement status |
| outreach | supervised social drafts (never posts) |
| ops | GPU, logs, Celery, plugins, Interconnector sync, autoresearch, infographics |
Swarm with Guaardvark
Read setup first; needs the swarm plugin (POST /api/plugins/swarm/start).
B=${GUAARDVARK_URL:-http://localhost:5000}. Status is also the MCP tool swarm_status.
1. Write the plan file
A markdown file in the target repo. Each ## heading is one task. Structured fields are optional;
the parser infers file scope from paths in the prose when they are absent:
# Plan: split the settings page
## Extract the GPU panel
- files: frontend/src/pages/SettingsPage.jsx, frontend/src/components/settings/GpuPanel.jsx
- backend: claude
Move the GPU section into its own component with the same props.
## Add a test for the panel
- files: frontend/src/__tests__/GpuPanel.test.jsx
- depends_on: Extract the GPU panel
backend / assign to picks the agent backend (Claude Code, or a local Cline/OpenClaw agent on
Ollama in Flight Mode). Tasks that touch the same files should depend on each other; the parser
warns about conflicts.
2. Launch
curl -s -X POST $B/api/swarm/launch -H 'Content-Type: application/json' -d '{
"plan_path": "/abs/path/repo/docs/plan.md", "repo_path": "/abs/path/repo",
"max_agents": 4, "auto_merge": false, "flight_mode": false, "dry_run": false,
"acknowledge_dirty_tree": false
}'
- A dirty working tree is refused unless
acknowledge_dirty_tree: true; tell the user what is uncommitted first. self_code: truetargets Guaardvark's own checkout and is only allowed on the configured root.dry_run: trueparses the plan and reports tasks and conflicts without starting agents.
3. Watch, merge, clean up
| call | |
|---|---|
GET $B/api/swarm/status and GET $B/api/swarm/status/<swarm_id> | tasks, states, worktrees |
GET $B/api/swarm/<swarm_id>/logs/<task_id>?lines=100 | one agent's log |
GET $B/api/swarm/<swarm_id>/diff/<task_id> | its diff before merge |
POST $B/api/swarm/<swarm_id>/bus/broadcast | message every agent |
POST $B/api/swarm/merge {"swarm_id": "...", "repo_path": "..."} | merge finished branches in dependency order |
POST $B/api/swarm/cancel | stop |
POST $B/api/swarm/cleanup {"swarm_id": "...", "delete_branches": false} | remove worktrees |
GET $B/api/swarm/templates | saved plan templates |
Rules
- Review diffs before
mergeunless the user asked forauto_merge. - Keep
max_agentsat or below the machine's capacity; each Claude agent is a separate session and each local agent competes for the GPU with generation jobs. - Report per task: state, files touched, whether it merged. Never say "done" on a swarm that still has running or failed tasks.
Upscaling with Guaardvark
Read setup first; needs the upscaling plugin (POST /api/plugins/upscaling/start).
B=${GUAARDVARK_URL:-http://localhost:5000}. Models: GET $B/api/upscaling/models (default HAT-L_SRx4);
install one with POST $B/api/upscaling/models/download {"model": "<id>"} only after the user says so.
One image (synchronous, seconds)
curl -s -X POST $B/api/upscaling/upscale/image \
-F file=@/abs/path/in.png -F model=HAT-L_SRx4 -F scale=4 -F sharpen=0.2 -F denoise_strength=0.1
Returns the finished file (output_path, served under GET $B/api/upscaling/output/image/<filename>).
Form options: model, scale, sharpen, denoise_strength.
Many images (queued)
curl -s -X POST $B/api/upscaling/upscale/images -F files=@a.png -F files=@b.png -F model=HAT-L_SRx4 -F scale=2
Returns a job id; poll GET $B/api/upscaling/jobs/<job_id>; GET $B/api/upscaling/jobs lists all.
Video (queued, frame by frame)
curl -s -X POST $B/api/upscaling/upscale/video -H 'Content-Type: application/json' -d '{
"input_path": "/abs/path/clip.mp4", "model": "realesrgan-x2", "scale": 2, "two_pass": false, "suffix": "_4k"
}'
output_path is optional. Same job polling. Video upscaling is slow: minutes per hundred frames
at 1080p on a 16 GB card. Say so before queuing.
Rules
- Preview first when in doubt:
POST $B/api/upscaling/preview(multipart, same form fields) runs a crop. - Pick the model for the content: HAT-L for photos, Real-ESRGAN x2 for video, anime models for line art. The models list carries descriptions.
- The GPU is exclusive; a running video upscale blocks image and video generation until it finishes.
Editing video with Guaardvark
Read setup first. Needs the video_editor plugin (POST $B/api/plugins/video_editor/start).
B=${GUAARDVARK_URL:-http://localhost:5000}. Edits render on the CPU with melt and
auto-editor, so they work without an NVIDIA card; GET $B/api/video-editor/status says whether
melt, auto-editor, ffmpeg and ffprobe were found, with an install hint for each.
Paths are absolute paths on the Guaardvark machine. Most routes also take a Documents id instead
(noted per route). Results land under data/outputs/videos/ (mlt-projects/, editor-renders/)
and are added to Documents unless the call sends "register": false.
A video with no other instruction can mean this skill or upscale: ask which.
Cut a bin of clips to a song (what the Video Editor page does)
- Plan: the bin and the song become an arrangement. Queued; an Ollama vision model looks at sampled frames of each clip, so say it takes a while for a large bin.
curl -s -X POST $B/api/video-editor/plan -H 'Content-Type: application/json' -d '{
"bin_clips": [{"clip_id": "a", "source_path": "/abs/a.mp4"}, {"document_id": 812}],
"song_path": "/abs/song.mp3", "style_recipe_name": "Music Video", "scan_mode": "both-and", "seed": 0
}'
song_document_id replaces song_path. GET $B/api/video-editor/recipes lists the style
recipes (a stock install has Cinematic, Dark, Default, Grunge, Music Video). scan_mode
(audio, motion, both-or, both-and) decides how dead stretches of a clip are found, with
audio_threshold, motion_threshold and margin. Returns job_id; poll
GET $B/api/video-editor/jobs/<job_id> until status is done or failed. result.arrangement
is the timeline and result.song.duration_seconds the song length.
- Render it (synchronous; minutes for a long song):
curl -s -X POST $B/api/video-editor/shotcut/compose-arrangement -H 'Content-Type: application/json' -d '{
"arrangement": <result.arrangement>, "song_document_id": 77, "song_duration_seconds": 184.2,
"width": 1920, "height": 1080, "render_mp4": true
}'
audio_path replaces song_document_id. Returns mlt_path, rendered_mp4 and documents.
Quick beat-sync from a pool of clips (queued)
curl -s -X POST $B/api/video-editor/beat-sync/render -H 'Content-Type: application/json' -d '{
"audio_path": "/abs/song.mp3", "video_paths": ["/abs/a.mp4", "/abs/b.mp4"],
"subdivision": 2, "min_clip_seconds": 1.2, "use_onsets": false, "render_mp4": true
}'
audio_document_id and video_document_ids replace the paths. Cuts land on beats with no
vision pass. Poll the job as above; the result carries tempo_bpm, cut_count, mlt_path and,
with render_mp4, rendered_mp4. POST $B/api/video-editor/analyze {"audio_path": ...} returns
a song's tempo, beat times and sections without editing anything.
Trim silence and dead air (synchronous)
curl -s -X POST $B/api/video-editor/auto-editor/trim -H 'Content-Type: application/json' -d '{
"input_path": "/abs/recording.mp4", "threshold": 0.04, "margin": "0.2sec", "mode": "mp4"
}'
document_id replaces input_path. Returns the kept clips (start and end seconds) and
output_path; "mode": "kdenlive" writes a Kdenlive project instead of a trimmed MP4.
One clip with a soundtrack and titles (queued)
curl -s -X POST $B/api/video-overlay/render-timeline -H 'Content-Type: application/json' -d '{
"video_document_id": 812, "video_trim_start": 2.0, "video_trim_end": 14.5,
"audio_document_id": 77, "audio_volume": 0.8,
"text_elements": [{"text": "Opening night", "fontSize": 64, "fontColor": "#ffffff",
"x": 120, "y": 900, "startSeconds": 0, "endSeconds": 4}]
}'
Document ids only. x and y are pixels from the top left of the frame; fontColor is a colour
name or #rrggbb; titles are drawn level (no rotation). Returns job_id; poll
GET $B/api/video-overlay/render-status/<job_id> until it carries document_id, the finished MP4.
Rules
- Never overwrite or delete the user's source clips; every route writes new files.
POST $B/api/video-editor/open-in-shotcut {"mlt_path": ...}opens Shotcut on the desktop of the machine running Guaardvark. Ask before using it; offer the.mltpath otherwise.GET $B/api/video-editor/projectslists the projects saved on the Video Editor page. Read them; leave changing them to the page.- Filters and transitions an arrangement may name:
GET $B/api/video-editor/catalog/filtersandGET $B/api/video-editor/catalog/transitions.
Video with Guaardvark
Read setup first if the backend or the comfyui plugin state is unknown.
Video needs a 16 GB-class NVIDIA card. Clips take minutes, so every route is queued.
Pick the model
GET ${GUAARDVARK_URL:-http://localhost:5000}/api/batch-video/models lists the registry with
is_downloaded / is_ready (and missing_files), capabilities (modes t2v/i2v, max_frames, native_fps,
aspect_ratios, min_steps, speed_profiles, audio_out), vram_mb, size_gb, license. Installed on a typical box:
| id | what it is | notes |
|---|---|---|
wan22-5b | Wan 2.2 TI2V-5B, t2v + i2v, 24 fps, up to 121 frames | the everyday default |
wan22-14b / wan22-14b-i2v | Wan 2.2 14B MoE, 16 fps, 81 frames | best quality; speed_profile: lightx2v-4 for 4-step Lightning |
ltx23-distilled-fp8 | LTX-2.3 distilled, 8 steps, 161 frames | fastest long clips |
minimax-h3-int8 | MiniMax H3, text / first / last / first+last frame, generates its own stereo soundtrack and spoken lines | pass audio: true; ~6.5 min for 5 s on a 16 GB card |
cogvideox-5b / -i2v | CogVideoX, 8 fps, 49 frames | legacy |
The active default is GET /api/settings/active_video_model (resolved.t2v, resolved.i2v, resolved.scene).
Never pass a step count below the model's min_steps; the server raises it and the result would be smeared anyway.
One clip: MCP generate_video
prompt: scene, subject, motion, style, and any spoken lines (H3 speaks them).model,duration_s(clamped to the model),aspect_ratio(one the model declares),style(cinematic, realistic, anime, 3d_animation, ...),num_inference_steps,speed_profile.audio: trueforces a soundtrack-capable model (H3) and fails on a silent family.first_image/last_image: document id or path; last frame needs a first+last mode model.reference_images(up to 9),reference_clips(up to 3, 2-15 s, their own sound left out) andreference_audio(up to 3): lock a person, look, motion or voice on the reference build (minimax-h3-ref2va-int8, picked when no model is named). The prompt names them<Picture N>,<Video N>,<Audio N>; to keep a clip's sound, pass its track asreference_audio. In the Studio this is the References input on Video Gen.wait_for_resultdefault false: the tool returns a batch id and a Studio deep link at once. Give the user the link; pollget_generation_status(batch_id=...)(MCP) orGET /api/batch-video/status/<batch_id>if they ask you to wait.wait_for_result: trueblocks for the clip, up to 30 minutes.
Looping animation: MCP generate_animation
Frame-morph GIF/MP4 via img2img: prompt, motion, frames 2-24, strength 0.1-0.5, format gif|mp4|both.
Use for short loops and stickers, not for cinema clips. Over MCP it answers at once with a job id
(tooljob_...); poll get_generation_status for the GIF and MP4 URLs, or pass
wait_for_result: true to wait up to 60 s for them.
Batch: REST
B=${GUAARDVARK_URL:-http://localhost:5000}
# text to video, one clip per prompt
curl -s -X POST $B/api/batch-video/generate/text -H 'Content-Type: application/json' -d '{
"prompts": ["a red kite over a grey sea", "the same kite at dusk"],
"model": "wan22-5b", "prompt_style": "cinematic", "enhance_prompt": true, "seed": 42
}'
# image to video
curl -s -X POST $B/api/batch-video/generate/image -H 'Content-Type: application/json' -d '{
"image_paths": ["/abs/path/frame.png"], "prompt": "slow push in, wind in the grass", "model": "wan22-5b"
}'
Optional keys the server honours: negative_prompt, guidance_scale, motion_strength,
interpolation_multiplier (RIFE frame interpolation), combine_frames, lora_name +
lora_strength, adapters, guides (per-prompt audio/image anchors on models that declare them),
last_frame_paths (image route), storyboard_concept / storyboard_shots.
Poll GET $B/api/batch-video/status/<batch_id>; files via
GET $B/api/batch-video/video/<batch_id>/<video_name>; cancel POST $B/api/batch-video/batch/<batch_id>/cancel;
retry POST $B/api/batch-video/retry/<batch_id>.
Rules
- Quote the model and the queued batch id back to the user. Never claim a clip is done until the status route says so.
- The GPU is exclusive: while video renders, chat models are evicted. Warn before queuing a long batch.
- The Studio page (Video Gen) shows the same queue; the user may prefer to watch it there.
Voice with Guaardvark
Read setup first. Expressive voices need the audio_foundry plugin running;
Piper works without it. B=${GUAARDVARK_URL:-http://localhost:5000}.
Which engine
| engine | route | when |
|---|---|---|
| Chatterbox | Audio Foundry backend: "chatterbox" | expressive, emotion presets, cloning |
| Kokoro | Audio Foundry backend: "kokoro" | fast, clean, 53 built-in voices in 9 languages (af_heart default); runs on the CPU without a GPU |
| Piper | /api/voice/text-to-speech | offline fallback, no GPU |
GET $B/api/audio-foundry/voices lists what is installed. GET $B/api/voice/voices lists Piper voices.
Speak a line or a script
Over MCP, call generate_speech (text up to 3000 characters, optional voice such as
af_heart, engine auto | kokoro | chatterbox); it waits and returns the file with a download
link. Cloning is not offered over MCP. Without MCP, or for the Chatterbox knobs, use REST:
curl -s -X POST $B/api/audio-foundry/generate/voice -H 'Content-Type: application/json' -d '{
"text": "The line to speak.",
"backend": "auto", # auto | chatterbox | kokoro
"voice_id": "af_heart", # Kokoro voice, or omit
"emotion": "calm", # Chatterbox preset, or omit
"exaggeration": 0.5, "cfg_weight": 0.5, "temperature": 0.8, # Chatterbox knobs, optional
"seed": 7, "output_format": "wav", "async": true
}'
- Short text returns the file directly (
path,document_id). With"async": trueor long text you get202 {"job_id"}: pollGET $B/api/audio-foundry/jobs/<job_id>untilstatusisdone; the result haspathanddocument_id. Cancel:POST .../jobs/<job_id>/cancel. - Multi-section narration with pauses:
POST $B/api/voice/narrate{"script": "...", "engine": "kokoro", "voice": "...", "pause_between_sections": 0.6, "output_format": "wav"}. - Piper only:
POST $B/api/voice/text-to-speech {"text", "voice": "libritts"}returnsaudio_url.
Clone a voice (consent-gated)
- The reference must go through the upload route; that is what records consent. Arbitrary
file paths are refused with 403.
The response gives the stored path.curl -s -X POST $B/api/audio-foundry/voice-clips/upload -F file=@/abs/path/ref.wav -F name="Narrator sample"GET $B/api/audio-foundry/voice-clipslists clips. - Generate with
"backend": "chatterbox", "reference_clip_path": "<that path>". - Before uploading, ask whether the voice belongs to the user or someone who consented. Do not clone a public figure or anyone who has not agreed. Refuse politely if unclear.
Rules
- 10 to 20 seconds of clean speech is enough for a clone; more is not better.
- Say which engine ran (the response reports it);
autofalls back to Kokoro on a Chatterbox error. - Audio files are local under
data/outputs/; they also appear in the Audio library page.
MCP servers 1
Sponsors
Details
- Marketplace
- guaardvark/guaardvark
- Category
- Productivity
- Keywords
- local-ai, self-hosted, image, video, music, voice, comfyui, ollama