Local image, video, music-video and Film Crew generation, voice and music, upscaling, LoRA training, Hugging Face model onboarding, agent swarms, local knowledge, code intelligence and supervised outreach, driven from Claude Code through the guaardvark MCP server.

GitHub stars
327

Install

/plugin install guaardvark@guaardvark

Run it inside Claude Code after adding the marketplace with /plugin marketplace add guaardvark/guaardvark.

Contents

Discovered from source

Skills 16

Cast Library and LoRA training

Read setup first; training needs the lora_trainer plugin (CUDA, bf16) and an installed train-ready base (Z-Image, SDXL or FLUX family). B=${GUAARDVARK_URL:-http://localhost:5000}.

The pipeline

  1. Create the subject
    curl -s -X POST $B/api/cast-library/subjects -H 'Content-Type: application/json' -d '{
      "kind": "character", "name": "Mara", "description": "late 30s, short grey hair, freckles",
      "trigger_word": "mara_v1", "voice_id": null
    }'
    
    kind is character, environment or prop. GET $B/api/cast-library lists subjects; the numeric id is what generate_image and batch routes take as subject_ids.
  2. Upload reference images (5 to 20 clear shots, varied angles, same subject): curl -s -X POST $B/api/cast-library/subjects/$ID/upload-refs -F files=@1.jpg -F files=@2.jpg Ask first whether the person consented; do not train on someone who has not.
  3. Vision bible from the refs: POST $B/api/cast-library/subjects/$ID/bible/from-refs (the vision model writes the identity description that every prompt inherits).
  4. Plan and generate training samples: POST .../$ID/plan then POST .../$ID/generate (cancel with .../generate/cancel). Review GET .../$ID/samples; view one with GET .../samples/<sample_id>/image; drop bad ones with DELETE .../samples/<sample_id> or POST .../samples/<sample_id>/regenerate.
  5. Approve samples: POST .../$ID/samples/approve.
  6. Train: POST $B/api/cast-library/subjects/$ID/train with optional {"training_settings": {...}}. A 409 already_training means wait; a train_base_not_ready error names the base model to install first. Cancel: .../train/cancel. Progress: GET $B/api/cast-library/subjects/$ID (status, base model, LoRA path when done).
  7. Use it: generate_image with subject_ids=[ID]; batch image/video routes with subject_ids; the Film Crew casts it; the music-video Director locks it per cut.

Rules

  • Training is a GPU job of tens of minutes; say so and check inspect_gpu for conflicts first.
  • The trigger word is applied by the system when subject_ids is passed; the user does not need to type it.
  • Everything stays local: refs, samples and the LoRA file under data/.

Code intelligence with Guaardvark (MCP tools)

tooluse
list_code_repositorieswhich folders are marked as repositories (gives the folder id)
search_codebasesearch Guaardvark's own source: by meaning once a code search index is built (System Map page), otherwise by words and names
search_codecase-insensitive regex across files
list_code_filesdirectory listing to orient
read_codea file, or a line range of it (start_line, end_line); large files come back in pages
read_ast_nodeone class or function by name from a Python file
get_repository_mapPageRank-ranked architectural map for a folder id
get_dependency_graphfile-level import graph for a folder id
analyze_codestructure, patterns, improvement notes for a file
codegena complete modified version of a file from instructions (writes a new file)
verify_changeconfirm text now exists in a file after an edit
map_codebaserun the System Mapper: the most severe findings, then the numbers
system_map_queryask the System Mapper about Guaardvark's code: where <feature>, module <path>, symbol <name> (callers, callees), route <path>, tool <name>, subsystem <name>, impact <path> (dependents, tests to run), findings <scope>
self_improvement_statuscan the self-improvement engine run (lock, flag, already running)
swarm_statusthe coding swarm (see the swarm skill)

Pattern

  1. list_code_repositories → folder id.
  2. get_repository_map for the shape, search_codebase for the question.
  3. Read the exact code (read_code / read_ast_node) before answering; quote file and line.
  4. Edits: make them with your own editor tools in the user's checkout; verify_change afterwards.

Rules

  • These tools read Guaardvark's index, which can lag the working tree; when it matters, read the file from disk too.
  • map_codebase, system_map_query and self_improvement_status are read-only here. Dispatching a fix to the self-improvement engine is done in the Studio, on purpose.

Film Crew with Guaardvark

Read setup first. Needs comfyui; casting with trained characters needs lora_trainer (see the cast skill). Renders take minutes per shot.

Start: MCP start_film_crew

  • script_text: a screenplay or a plain scene list. A logline works; the screenwriter expands it.
  • name (defaults to the first line), video_model (defaults to the active scene model, minimax-h3-int8 on a typical box; wan22-5b for silent i2v).
  • The screenwriter starts at once. Casting, storyboards and GPU renders wait for the user in Studio (Agents > Film Crew) or through the routes below.

Stages and gates (REST)

B=${GUAARDVARK_URL:-http://localhost:5000}, P = production id from the tool.

stagecallnote
statusGET $B/api/production/$Pstage, subjects, shots, errors
subjects found by the writerGET $B/api/production/$P/subjectscharacters, environments, props
cast one subjectPOST $B/api/production/$P/cast/<subject_id> with {"action": ...}link to an existing Cast Library subject or plan a new one
confirm castingPOST $B/api/production/$P/casting/confirmuser gate: every subject needs a cast plan first
storyboard shot imageGET $B/api/production/$P/storyboard/shot/<shot_id>/imagereview keyframes
redo one shotPOST $B/api/production/$P/storyboard/shot/<shot_id>/regenerate
approve storyboardPOST $B/api/production/$P/storyboard/approveuser gate: releases the renders. A 409 with flagged_shots lists frames the curator flagged; show them, and resend with {"confirm_flagged": true} only if the user says to render them anyway
retry a failed stagePOST $B/api/production/$P/retry
list / deleteGET $B/api/production, DELETE $B/api/production/$P

Without the MCP tool: POST $B/api/production with {"name", "script_text", "settings": {"video_model": "..."}}.

How to run it well

  1. Show the screenwriter's output (scenes, shots, subjects) before touching casting.
  2. Consistent faces come from trained LoRAs. If a subject has no Cast entry, offer to create and train one (the cast skill) or cast it "as described" and warn identity may drift.
  3. State the render cost before the storyboard approval: shots × seconds × model speed.
  4. The editor assembles the shots in the Video Editor; the result appears in the media library.
  5. Nothing posts or uploads. The film stays on the machine unless the user moves it.

Images with Guaardvark

Read setup first if the backend or the comfyui plugin state is unknown.

One image: MCP generate_image

  • prompt is scene, pose, lighting, setting. Plain prose. Do not paste JSON or tag soup; the default model (Z-Image Turbo) reads prompts as language, and SD-era tag lists hurt it.
  • model default auto picks the best downloaded model. Only override when the user names one: zimage-turbo, krea2-turbo, krea2-raw, flux-dev, sd-xl, sdxl-turbo, realistic-vision, epic-realism.
  • width / height: 512, 768 or 1024. style: realistic, artistic, anime, photographic, digital-art.
  • Consistent character: pass subject_ids=[<cast id>] as its own array. Never put the trigger word alone in the prompt and expect the LoRA to load. Find ids with GET /api/cast-library (see the cast skill).
  • On-image text: quote the exact words in double quotes inside the prompt.
  • The tool returns the image URL (/api/outputs/generated_images/<file>.png, relative to the backend), the model that ran, steps, seed and whether a Cast LoRA was applied. Show the URL and the prompt you used. Measured: 768x768 on Z-Image Turbo in ~20 s on a free 16 GB card.
  • Over MCP the call queues by default (wait_for_result defaults to false there) and returns Image queued as batch ImageBatch_... at once. Poll get_generation_status(batch_id=...) every few seconds until completed; it returns the file URL. Pass wait_for_result: true to block for the render instead (allowed up to 30 minutes). A call that exceeds the server's timeout answers with an error that says the render is still running; it is not lost.
  • A failed call carries the backend's reason (plugin off, out of memory, bad model). Read it and act on it; inspect_gpu and GET /api/plugins/status are the two checks that resolve most.

Photo edits over MCP run as jobs

edit_image, outpaint_image and remove_background answer at once over MCP with ... queued as job tooljob_...: the work runs in the Guaardvark backend and keeps going if the client disconnects. Poll get_generation_status(batch_id="tooljob_...") every few seconds until done (it gives the file URL) or failed (it gives the error). Pass wait_for_result: true to wait up to 60 s (half the server's MCP timeout when that is shorter) for the result; a longer edit still comes back as the job id. A Qwen-Image-Edit takes about three minutes on a 16 GB card, and a job waits its turn while another render holds the GPU.

Edit an existing image: MCP edit_image

  • instruction is the change ("put a cowboy hat on him", "make the shirt red"). The image the user just attached is used automatically; otherwise pass image as a path or URL. Optional reference_image_2 / reference_image_3 (Qwen-Image-Edit only) for extra people or style; the call is refused when a reference cannot be read or the edit would run on another backend.
  • model auto uses Qwen-Image-Edit when installed, else FLUX.1 Kontext. With neither installed the call is refused and names the pack; relay that instead of retrying. Override with qwen-image-edit or kontext. Install those packs from Manage Image Models → Image editing (qwen-image-edit, flux-kontext-dev) — do not Install unless the user asked. Naming another downloaded image model runs a light img2img pass that keeps most of the picture.
  • Same canvas, same pose. For a brand-new picture use generate_image. For a new scene that keeps a face use generate_identity.

New scene from a face: MCP generate_identity (off by default)

  • Not exposed unless the server runs with GUAARDVARK_IDENTITY_TOOL=1: the likeness it keeps has not passed verification yet. If the tool is absent, say so and offer edit_image instead.
  • Attach a likeness the user has the right to use (their photo or a Cast subject they uploaded). consented must be true. Refuse if they have not confirmed that.
  • prompt is the new scene. Needs the PuLID identity pack (pulid-flux; Manage Image Models → Image editing installs it with its face files, EVA02-CLIP and flux-dev). Comfy must have been restarted after the PuLID-Flux custom node was added.
  • This is not a face swap onto an existing poster, and not an instruction edit of the same photo.

Background remove: MCP remove_background

  • Cuts the subject out of the attached photo (transparent PNG). ONNX matting, no diffusion; needs a background-removal model from Manage Image Models → Image editing.
  • For a new background, remove first then edit_image / generate_identity, or describe the new scene in edit_image if Qwen-Image-Edit is installed.

Change part of a photo / outpaint

  • edit_image: change or remove something ("remove the coffee cup"). Runs on Qwen-Image-Edit or FLUX.1 Kontext; refused when neither is installed.
  • outpaint_image: extend the canvas (left/right/top/bottom pixels) and fill. Prefers Qwen.

Many images: REST batch

B=${GUAARDVARK_URL:-http://localhost:5000}
curl -s -X POST $B/api/batch-image/generate/prompts -H 'Content-Type: application/json' -d '{
  "prompts": ["prompt one", "prompt two"],
  "model": "auto",
  "subject_ids": []
}'
  • prompts may be strings or {"prompt": "..."} objects. There is a per-batch maximum; if the server answers 400 "Too many prompts", split the list.
  • Optional adapters (user LoRAs from the models skill) and subject_ids (Cast Library).
  • The response is data.batch_id (ImageBatch_<date>_<n>). Poll GET $B/api/batch-image/status/<batch_id>?include_results=true: status goes running → completed, with completed_images / total_images, output_dir, and one results[] entry per prompt (success, image_path, thumbnail_path, generation_time, metadata.model_used). A contact sheet: GET $B/api/batch-image/preview/<batch_id>; one file: GET $B/api/batch-image/image/<batch_id>/<image_name> (the basename of image_path). Cancel with POST $B/api/batch-image/cancel/<batch_id>. Measured: one 1024x1024 prompt completed in ~30 s.
  • Helpers: POST /api/batch-image/enhance-prompt, /analyze-prompt, /expand-concept (JSON body with the prompt) when the user wants prompt help before spending GPU time.

Rules

  • Say which model actually ran (the response names it). Do not promise a model that is not installed.
  • Generation time depends on the GPU; a first image after Ollama held the card can take longer because the orchestrator swaps models. That is normal.
  • Never upload the user's images anywhere. Everything here is local.

Knowledge with Guaardvark (MCP tools)

Everything here is the guaardvark MCP server. All retrieval is local (Ollama embeddings, hybrid search, cross-encoder reranking on the user's GPU).

Find and read

tooluse
search_knowledge_basethe question in plain words; returns passages with sources. Cite them.
summarize_corpuswhat the whole knowledge base covers, before searching blind
list_documentswhich documents are indexed and how many passages each has
get_document_outlinesections or pages of one document, in order
read_document_sectionthe actual text of one section or page, no search
process_fileextract content from a PDF, DOCX, CSV, Excel, image or other file the user names
index_documentadd a file in Guaardvark's uploads, outputs or install folder to the knowledge base, so the tools above find it
fetch_urltitle, description and main text of one web page; pass query (the user's question) to get the passage about it instead of the top of the page
web_searchweb results (titles, snippets, URLs) when the answer is not local; source names the search engine that answered, and no_results says when it found nothing

Pattern: summarize_corpus or list_documents → search_knowledge_base → read the exact section with get_document_outline + read_document_section before quoting.

Remember

  • save_memory: a fact, preference or instruction to keep across sessions. Use when the user says "remember", or states a durable preference. One fact per call.
  • search_memory: check before assuming; recalled memories are what was true when saved.
  • update_memory: correct a saved fact, preference or note in place, by the ID search_memory prints, instead of saving a second version next to it.
  • delete_memory: forget one entry by its ID when the user asks. It cannot be undone.
  • Over MCP these two change only fact, preference and note entries; lessons and Guaardvark's own entries are managed on the Agent Memory page.

Rules

  • Quote the source (document name and section) for any claim that came from retrieval.
  • If retrieval returns nothing, say so; do not fill the gap from general knowledge without labelling it.
  • index_document adds one file at a time and waits up to wait_seconds (default 45) for the indexer; a long PDF may still be indexing when it answers, and list_documents shows it once done. Whole folders are indexed from the Studio's Library page.

Models and LoRAs with Guaardvark

B=${GUAARDVARK_URL:-http://localhost:5000}. Nothing downloads without an explicit Install; the product never phones home on its own. Always confirm the size and the licence with the user first.

What is there

  • Video: GET $B/api/batch-video/models (registry + user catalog, with is_downloaded / is_ready, missing_files, capabilities).
  • Image: GET $B/api/batch-image/models.
  • Download a registry model: POST $B/api/batch-video/models/download {"model_id": "wan22-14b"} / POST $B/api/batch-image/models/download {"model_path": "<id>"}; progress at GET .../models/download-status.
  • Chat photo-tool packs install from Manage Image Models → Image editing: qwen-image-edit (~28 GB with its encoder and VAE), flux-kontext-dev, pulid-flux (with its face files, EVA02-CLIP and flux-dev), bgremove-birefnet / bgremove-u2net. The rows come back as editing in GET $B/api/batch-image/models; Install with POST $B/api/batch-image/models/download {"model_path": "comfy:<pack id>"}. Confirm size and licence; never start those downloads unless the user asked.

Add from a Hugging Face URL (the "paste a link" path)

  1. Inspect: POST $B/api/batch-video/models/from-hf {"url": "https://huggingface.co/<org>/<repo>"} (or the batch-image twin). The server reads the repo and returns what it found: files, revision, the likely role (checkpoint, LoRA, VAE, text encoder), family, whether a model_index exists, and a proposed catalog entry.
  2. Show that to the user: which file(s), role, family, size, licence.
  3. Register (and optionally install) with POST $B/api/batch-video/models/user (or batch-image) sending the entry back. role and family (image) or like (video) decide where the Studio offers it. The server re-inspects the repo; do not send has_model_index from the client. Gated repos need HF_TOKEN; confirm size and licence from the Look-up response first.
  4. Remove: DELETE .../models/user/<model_id> with {"delete_files": true|false}.

Using a user LoRA

  • Batch image: adapters: [{"id": "<user model id>", "scale": 0.8}] on /generate/prompts.
  • Batch video: lora_name + lora_strength, or adapters on /generate/text.
  • A trained Cast LoRA is different: it rides on subject_ids (the cast skill).

Rules

  • A direct .safetensors URL works when it lives on huggingface.co; the inspector needs the repo to read the file list. For other hosts, ask the user to download the file and drop it in the Studio's model folder instead.
  • Declared limits (min steps, max frames) come from the registry entry; a user model inherits them from like/family. Do not invent numbers for a model the registry does not know.

Music video with Guaardvark

Read setup first. Needs the comfyui plugin and an installed i2v model (wan22-5b by default; minimax-h3-int8 for frame-matched continuity).

Start: MCP generate_music_video

  • song: document id or a filesystem path to mp3/wav/flac/ogg. A path is uploaded for the user.
  • style_prompt: mood, palette, movement, era ("grainy 16 mm, neon rain, slow dolly, 1984").
  • name (defaults to the song file name), i2v_model (defaults to the active i2v model).
  • The tool analyses the song (tempo, beats, sections), writes one unique prompt per cut and stops at the approval gate. It does not spend GPU rendering clips. Tell the user that.

The gate and the rest of the pipeline (REST)

B=${GUAARDVARK_URL:-http://localhost:5000}, ID = the project id the tool returned.

stepcallwhat it does
read the planGET $B/api/music-video/$IDcuts, prompts, status, storyboard state
edit promptsPOST $B/api/music-video/$ID/plan with the edited cut prompts (and optional global style)operator edits before render
regenerate the planPOST $B/api/music-video/$ID/regenerate-plan or /replannew prompts from the Director
storyboards firstPOST $B/api/music-video/$ID/generate-storyboardsone keyframe still per cut, thumbnails-first review
redo one keyframePOST $B/api/music-video/$ID/regen-storyboard/<idx>keeps the others
approvePOST $B/api/music-video/$ID/approvereleases per-clip generation. Only on the user's explicit yes
re-analysePOST $B/api/music-video/$ID/analyzerestart song analysis
cancelPOST $B/api/music-video/$ID/cancel
list allGET $B/api/music-video (?project_id=<id> or ?project_id=none to filter)
progressGET $B/api/music-video/$ID/progressstages, the clip in flight (sampler step/steps), ETA. Report this instead of guessing
rename / file under a projectPATCH $B/api/music-video/$ID with {name?, project_id?, notes?, tags?}the finished video joins that project's Documents
edit one cutPOST $B/api/music-video/$ID/cut/<idx> with {prompt?, seed?, locked?}at the approval gate it only changes the plan; while generating it re-renders that cut: ask first
reopen a finished videoPOST $B/api/music-video/$ID/reviseback to the approval gate, rendered clips kept; on approve only changed cuts render. /replan re-renders all
copyPOST $B/api/music-video/$ID/duplicate with {name?, project_id?, keep_clips?}new video at the approval gate
versionsGET/POST $B/api/music-video/$ID/revisions, POST .../revisions/<rid>/restoresaved on approve, reassemble, revise, restore; restore lands at the approval gate
a project's videosGET $B/api/projects/<id>/videosmusic videos and Film Crew filed under it

Creating a project without the MCP tool: POST $B/api/music-video with {"name", "song_document_id", "style_prompt", "settings": {"i2v_model": "..."}, "user_treatment": "optional short story", "project_id": optional}. The MCP tool takes project (a Project id or exact name). user_treatment becomes the creative source the Director writes cuts from.

How to run it well

  1. Start the project, show the cut list and the beat count.
  2. Offer storyboards before approval: they are cheap and catch a wrong style early.
  3. Ask for approval in plain words with the cost: number of cuts times the clip time for the chosen model. Approve only after a clear yes, and only once GET $B/api/music-video/$ID shows current_stage awaiting_approval with cuts: before analysis finishes the approve route answers 409.
  4. Character in the video: create a Cast subject and train a LoRA first (the cast skill) so the Director can lock identity across cuts.
  5. The finished cut is assembled to the detected beat grid. The Video Editor page can trim or overlay text afterwards.

Music and sound effects with Guaardvark

Read setup first; needs the audio_foundry plugin. B=${GUAARDVARK_URL:-http://localhost:5000}. ACE-Step takes ~10 GB VRAM; the orchestrator evicts other models while it runs.

A song

Over MCP, call generate_music (style, optional lyrics, seconds up to 240, instrumental, seed, model); it returns a job_id for get_generation_status, which reports the file and a download link when the song is done. Without MCP, use REST:

curl -s -X POST $B/api/audio-foundry/generate/music -H 'Content-Type: application/json' -d '{
  "style_prompt": "indie folk, fingerpicked acoustic guitar, warm male vocal, 96 bpm, intimate",
  "lyrics": "[verse]\nLines here\n[chorus]\nHook here",
  "negative_prompt": "distorted, muddy, synth",
  "duration_s": 90, "instrumental_only": false,
  "output_format": "wav", "seed": 1234, "async": true
}'
  • model: omit (or "ace-step") for ACE-Step v1; "ace-step-1.5" for ACE-Step 1.5 (MIT, 48 kHz, turbo model plus a planner) once it is installed in Audio Studio → Manage models. Not installed → 400 with needs_install; nothing is downloaded.
  • For v1, style_prompt uses ACE-Step's tag vocabulary: genre, instruments, mood, tempo, vocal type. Vague words ("professional", "futuristic") drift to the model's prior; be concrete. POST $B/api/audio-foundry/rewrite-music-prompt turns plain English into that tag form. For 1.5, write a sentence. The loaded 1.7B planner expands it into a caption, lyrics, tempo, key and meter before the render, and keeps a bpm or key written in the sentence. POST $B/api/audio-foundry/plan-music with the same body as generate returns that arrangement and does not render. negative_prompt is accepted but neither ACE-Step model applies it (the result says negative_prompt_applied: false).
  • lyrics with [verse] / [chorus] / [bridge] markers; omit or set instrumental_only: true.
  • duration_s up to 240. Always send "async": true for anything over a few seconds.
  • Response 202 {"job_id"} → poll GET $B/api/audio-foundry/jobs/<job_id>; done when status is done, with path, document_id, duration_s, seed. Reuse the seed to reproduce a take.

A sound effect or ambience

curl -s -X POST $B/api/audio-foundry/generate/fx -H 'Content-Type: application/json' -d '{
  "prompt": "rain on a tin roof, distant thunder, no music", "duration_s": 20, "output_format": "wav", "async": true
}'

duration_s up to 47. Same job polling.

Then

  • A finished song can go straight into the music skill-video as the song.
  • Jobs: GET $B/api/audio-foundry/jobs lists, DELETE clears finished ones.
  • Attribution comes back in the job result; keep it with the file if the user publishes.

Operating Guaardvark

B=${GUAARDVARK_URL:-http://localhost:5000}. MCP tools: inspect_gpu, read_logs, swarm_status, self_improvement_status.

GPU and plugins

  • inspect_gpu (MCP): nvidia-smi, the exclusive lock (Ollama vs video), orchestrator slots, running plugins.
  • GET $B/api/plugins/status and GET $B/api/plugins/<id>/health; GET $B/api/plugins/stats/gpu.
  • POST $B/api/plugins/<id>/start | stop | restart | enable | disable for ollama, comfyui, audio_foundry, upscaling, lora_trainer, swarm, vision_pipeline, discord, gpu_embedding, video_editor. Start services through these routes, never with systemctl; the orchestrator must know what is resident.
  • GET $B/api/plugins/<id>/logs?lines=200 for a plugin's own log.
  • Memory orchestrator: GET $B/api/gpu/memory/status, GET/POST $B/api/gpu/memory/tier. POST .../evict {"slot_id"} frees a slot; use only when the user asks to clear the card.

Logs and tasks

  • read_logs (MCP) tails a file under logs/ (backend, celery, plugin logs).
  • GET $B/api/celery/tasks shows active Celery tasks; GET $B/api/meta/... job routes list and retry jobs (POST $B/api/meta/retry_job/<job_id>, POST $B/api/meta/cancel_job/<job_id>).
  • A "stuck" job is usually the GPU lock held by another plugin: check inspect_gpu before retrying.

Interconnector (other machines sync from this one)

  • GET $B/api/interconnector/status, GET .../nodes, GET .../sync/history?limit=20.
  • POST $B/api/interconnector/sync/manual with {"direction": "push", "entities": [...], "sync_files": true, "profile_name": "..."}; POST .../nodes/<node_id>/test checks a link.
  • Other boxes never git pull; they sync through this. A frontend change needs a bundle rebuild there afterwards.

Autoresearch (overnight RAG tuning)

POST $B/api/autoresearch/start {"mode": "...", "budget_hours": 6}; GET .../status, GET .../runs, GET .../runs/<run_id>/ledger; POST .../stop. Promotions of a better config are explicit: POST .../promotions/<config_id>/activate, POST .../promotions/revert. A run executes on its own Celery worker (research@<host>, log celery_research.log), started with the run and stopped about two minutes after it; the main worker stays free. Its model calls wait while a render or training job holds the GPU. POST .../stop ends it at its next model call.

Infographic

curl -s -X POST $B/api/infographic/generate -H 'Content-Type: application/json' -d '{
  "title": "Three numbers", "scene": "clean flat design, navy and white",
  "callouts": ["42% faster", "0 cloud calls", "1 GPU"], "hashtags": ["#localai"], "footer": "guaardvark.com",
  "style": "flat_vector", "aspect": "1:1", "seed": 3
}'

Styles: editorial, flat_vector, photo_real, comic; aspects 16:9, 1:1, 9:16, 4:5, 3:2. The call waits for the GPU and returns when the PNG is saved (image_url, also filed in the Library under Images › Infographics). model is optional: Krea 2 Turbo when installed, else Z-Image Turbo; qwen-image-21 is non-commercial. GET $B/api/infographic/status lists the models and which is installed.

Rules

  • Restarting a plugin kills its jobs. Check the queues and say so before restarting.
  • Never restart the backend or Celery from here; tell the user what to run in the checkout.

Outreach with Guaardvark (MCP tools, supervised)

Nothing here posts. Approval happens in the Studio's Approvals page, by design, and a Reddit, YouTube or Facebook draft posts only when the user presses Post now there with the Agent Screen open. X and LinkedIn are not offered for outreach; the user's own LinkedIn account takes posts through request_publish instead.

tooluse
outreach_statusenabled / cadence / per-platform posting modes
outreach_list_queuedrafts, default status='drafted' (pending review)
outreach_draft_postdraft a comment or share post for a platform and thread URL; grounded in the thread and the user's pitch sheet, graded, then queued
outreach_reject_draftreject a draft so it will not post; fails and says why once the post is being submitted or is already out
request_publishask to post the user's own announcement to one of their social connections; it waits on the Approvals page

Pattern

  1. outreach_status to confirm the loop is on.
  2. outreach_draft_post with the platform and the thread. Show the draft and its grade.
  3. The user edits, approves and starts the post in the Studio. You can reject on their word; you cannot approve or post.

Rules

  • Do not bulk-draft across many threads. The system is for keeping up with engagement on the user's own products and topics, not for volume.
  • Say clearly that a draft is queued, not posted.

Guaardvark: connect and check

Guaardvark runs on the user's own machine. Everything below is local; nothing leaves the box.

Where it is

  • Backend URL: ${GUAARDVARK_URL:-http://localhost:5000}. The macOS default port is 5055 (AirPlay owns 5000). The web UI is on port 5173 in dev, or the same port as the backend in Docker.
  • Health: !curl -sf ${GUAARDVARK_URL:-http://localhost:5000}/api/health || curl -sf http://localhost:5055/api/health || echo "backend not reachable on 5000 or 5055"
  • Plugins: !curl -sf ${GUAARDVARK_URL:-http://localhost:5000}/api/plugins/status || echo "plugin status unavailable"

If the backend is not reachable, tell the user to start it from the Guaardvark checkout (./start.sh, or docker compose up), then retry. Do not try to start it yourself.

Two ways to drive it

  1. MCP tools (preferred when present): the guaardvark MCP server exposes RAG, memory, code intelligence, file processing, web fetch, image/video/animation/music-video/film-crew generation, songs and speech (generate_music, generate_speech), get_generation_status for any queued batch or song, outreach drafting and GPU/log inspection. Generation tools queue by default over MCP and return a batch id. In Claude Code they appear as mcp__guaardvark__<tool> after python -m backend.mcp install, or as mcp__plugin_guaardvark_guaardvark__<tool> when the plugin was installed from the marketplace. Install once from the checkout:
    python -m backend.mcp install      # writes the server entry into every detected client: Claude Code, Codex,
                                       # Cursor, Grok, Antigravity, opencode, Claude Desktop, Zed, Gemini
    python -m backend.mcp doctor       # self-test + stale-config scan
    python -m backend.mcp list-tools   # what is exposed right now
    
    Restart the client after installing so it re-reads its MCP config.
  2. REST for everything the MCP policy does not expose (voice cloning, sound effects, upscaling, batches, cast/LoRA training, swarm launch, interconnector, plugins). Use curl against the backend URL. Responses are wrapped as {"success": bool, "data": {...}, "message": str} on most routes; a few older routes return the bare object. Read data when it is present.

Before generating anything

  • GPU services are plugins. Check GET /api/plugins/status; a generation route answers 503 when its plugin is not running. Start one with POST /api/plugins/<id>/start where <id> is comfyui (image + video), audio_foundry (voice, music, FX), upscaling, lora_trainer, swarm. Only one heavy model owns the GPU at a time; the orchestrator evicts Ollama for video and vice versa, so a first call after a switch is slow. inspect_gpu (MCP) shows who holds it.
  • Installed models: GET /api/batch-video/models and GET /api/batch-image/models list every registry entry with capabilities; check is_ready before naming a model. Nothing downloads without an explicit Install, so if a model is missing say so and offer POST /api/batch-video/models/download {"model_id": "..."} (or the image route with {"model_path": "..."}).
  • Outputs land under data/outputs/ in the checkout and are also served read-only over MCP as guaardvark://outputs/... resources.

Skills in this pack

SkillUse for
imageone image, edits, cast characters, batch image runs
videoone clip, image-to-video, first/last frame, batch video runs, MiniMax H3 with sound
music-videoa song in, a beat-cut music video out, with the approval gate
film-crewscreenplay to finished short: writer, casting, storyboards, render, edit
voicenarration, TTS, consent-gated voice cloning
musicfull songs with lyrics, instrumentals, sound effects
upscale2x to 8K upscaling of images and video
video-editoredit existing clips: cut to a song's beat, trim dead air, titles, Shotcut projects
castCast Library subjects and LoRA training for consistent characters
modelsadd any Hugging Face model or LoRA from a URL
swarmparallel coding agents in git worktrees from a plan file
knowledgethe user's indexed documents, memory, web fetch
codecode search, repository map, self-improvement status
outreachsupervised social drafts (never posts)
opsGPU, logs, Celery, plugins, Interconnector sync, autoresearch, infographics

Swarm with Guaardvark

Read setup first; needs the swarm plugin (POST /api/plugins/swarm/start). B=${GUAARDVARK_URL:-http://localhost:5000}. Status is also the MCP tool swarm_status.

1. Write the plan file

A markdown file in the target repo. Each ## heading is one task. Structured fields are optional; the parser infers file scope from paths in the prose when they are absent:

# Plan: split the settings page

## Extract the GPU panel
- files: frontend/src/pages/SettingsPage.jsx, frontend/src/components/settings/GpuPanel.jsx
- backend: claude
Move the GPU section into its own component with the same props.

## Add a test for the panel
- files: frontend/src/__tests__/GpuPanel.test.jsx
- depends_on: Extract the GPU panel

backend / assign to picks the agent backend (Claude Code, or a local Cline/OpenClaw agent on Ollama in Flight Mode). Tasks that touch the same files should depend on each other; the parser warns about conflicts.

2. Launch

curl -s -X POST $B/api/swarm/launch -H 'Content-Type: application/json' -d '{
  "plan_path": "/abs/path/repo/docs/plan.md", "repo_path": "/abs/path/repo",
  "max_agents": 4, "auto_merge": false, "flight_mode": false, "dry_run": false,
  "acknowledge_dirty_tree": false
}'
  • A dirty working tree is refused unless acknowledge_dirty_tree: true; tell the user what is uncommitted first.
  • self_code: true targets Guaardvark's own checkout and is only allowed on the configured root.
  • dry_run: true parses the plan and reports tasks and conflicts without starting agents.

3. Watch, merge, clean up

call
GET $B/api/swarm/status and GET $B/api/swarm/status/<swarm_id>tasks, states, worktrees
GET $B/api/swarm/<swarm_id>/logs/<task_id>?lines=100one agent's log
GET $B/api/swarm/<swarm_id>/diff/<task_id>its diff before merge
POST $B/api/swarm/<swarm_id>/bus/broadcastmessage every agent
POST $B/api/swarm/merge {"swarm_id": "...", "repo_path": "..."}merge finished branches in dependency order
POST $B/api/swarm/cancelstop
POST $B/api/swarm/cleanup {"swarm_id": "...", "delete_branches": false}remove worktrees
GET $B/api/swarm/templatessaved plan templates

Rules

  • Review diffs before merge unless the user asked for auto_merge.
  • Keep max_agents at or below the machine's capacity; each Claude agent is a separate session and each local agent competes for the GPU with generation jobs.
  • Report per task: state, files touched, whether it merged. Never say "done" on a swarm that still has running or failed tasks.

Upscaling with Guaardvark

Read setup first; needs the upscaling plugin (POST /api/plugins/upscaling/start). B=${GUAARDVARK_URL:-http://localhost:5000}. Models: GET $B/api/upscaling/models (default HAT-L_SRx4); install one with POST $B/api/upscaling/models/download {"model": "<id>"} only after the user says so.

One image (synchronous, seconds)

curl -s -X POST $B/api/upscaling/upscale/image \
  -F file=@/abs/path/in.png -F model=HAT-L_SRx4 -F scale=4 -F sharpen=0.2 -F denoise_strength=0.1

Returns the finished file (output_path, served under GET $B/api/upscaling/output/image/<filename>). Form options: model, scale, sharpen, denoise_strength.

Many images (queued)

curl -s -X POST $B/api/upscaling/upscale/images -F files=@a.png -F files=@b.png -F model=HAT-L_SRx4 -F scale=2

Returns a job id; poll GET $B/api/upscaling/jobs/<job_id>; GET $B/api/upscaling/jobs lists all.

Video (queued, frame by frame)

curl -s -X POST $B/api/upscaling/upscale/video -H 'Content-Type: application/json' -d '{
  "input_path": "/abs/path/clip.mp4", "model": "realesrgan-x2", "scale": 2, "two_pass": false, "suffix": "_4k"
}'

output_path is optional. Same job polling. Video upscaling is slow: minutes per hundred frames at 1080p on a 16 GB card. Say so before queuing.

Rules

  • Preview first when in doubt: POST $B/api/upscaling/preview (multipart, same form fields) runs a crop.
  • Pick the model for the content: HAT-L for photos, Real-ESRGAN x2 for video, anime models for line art. The models list carries descriptions.
  • The GPU is exclusive; a running video upscale blocks image and video generation until it finishes.

Editing video with Guaardvark

Read setup first. Needs the video_editor plugin (POST $B/api/plugins/video_editor/start). B=${GUAARDVARK_URL:-http://localhost:5000}. Edits render on the CPU with melt and auto-editor, so they work without an NVIDIA card; GET $B/api/video-editor/status says whether melt, auto-editor, ffmpeg and ffprobe were found, with an install hint for each.

Paths are absolute paths on the Guaardvark machine. Most routes also take a Documents id instead (noted per route). Results land under data/outputs/videos/ (mlt-projects/, editor-renders/) and are added to Documents unless the call sends "register": false.

A video with no other instruction can mean this skill or upscale: ask which.

Cut a bin of clips to a song (what the Video Editor page does)

  1. Plan: the bin and the song become an arrangement. Queued; an Ollama vision model looks at sampled frames of each clip, so say it takes a while for a large bin.
curl -s -X POST $B/api/video-editor/plan -H 'Content-Type: application/json' -d '{
  "bin_clips": [{"clip_id": "a", "source_path": "/abs/a.mp4"}, {"document_id": 812}],
  "song_path": "/abs/song.mp3", "style_recipe_name": "Music Video", "scan_mode": "both-and", "seed": 0
}'

song_document_id replaces song_path. GET $B/api/video-editor/recipes lists the style recipes (a stock install has Cinematic, Dark, Default, Grunge, Music Video). scan_mode (audio, motion, both-or, both-and) decides how dead stretches of a clip are found, with audio_threshold, motion_threshold and margin. Returns job_id; poll GET $B/api/video-editor/jobs/<job_id> until status is done or failed. result.arrangement is the timeline and result.song.duration_seconds the song length.

  1. Render it (synchronous; minutes for a long song):
curl -s -X POST $B/api/video-editor/shotcut/compose-arrangement -H 'Content-Type: application/json' -d '{
  "arrangement": <result.arrangement>, "song_document_id": 77, "song_duration_seconds": 184.2,
  "width": 1920, "height": 1080, "render_mp4": true
}'

audio_path replaces song_document_id. Returns mlt_path, rendered_mp4 and documents.

Quick beat-sync from a pool of clips (queued)

curl -s -X POST $B/api/video-editor/beat-sync/render -H 'Content-Type: application/json' -d '{
  "audio_path": "/abs/song.mp3", "video_paths": ["/abs/a.mp4", "/abs/b.mp4"],
  "subdivision": 2, "min_clip_seconds": 1.2, "use_onsets": false, "render_mp4": true
}'

audio_document_id and video_document_ids replace the paths. Cuts land on beats with no vision pass. Poll the job as above; the result carries tempo_bpm, cut_count, mlt_path and, with render_mp4, rendered_mp4. POST $B/api/video-editor/analyze {"audio_path": ...} returns a song's tempo, beat times and sections without editing anything.

Trim silence and dead air (synchronous)

curl -s -X POST $B/api/video-editor/auto-editor/trim -H 'Content-Type: application/json' -d '{
  "input_path": "/abs/recording.mp4", "threshold": 0.04, "margin": "0.2sec", "mode": "mp4"
}'

document_id replaces input_path. Returns the kept clips (start and end seconds) and output_path; "mode": "kdenlive" writes a Kdenlive project instead of a trimmed MP4.

One clip with a soundtrack and titles (queued)

curl -s -X POST $B/api/video-overlay/render-timeline -H 'Content-Type: application/json' -d '{
  "video_document_id": 812, "video_trim_start": 2.0, "video_trim_end": 14.5,
  "audio_document_id": 77, "audio_volume": 0.8,
  "text_elements": [{"text": "Opening night", "fontSize": 64, "fontColor": "#ffffff",
                     "x": 120, "y": 900, "startSeconds": 0, "endSeconds": 4}]
}'

Document ids only. x and y are pixels from the top left of the frame; fontColor is a colour name or #rrggbb; titles are drawn level (no rotation). Returns job_id; poll GET $B/api/video-overlay/render-status/<job_id> until it carries document_id, the finished MP4.

Rules

  • Never overwrite or delete the user's source clips; every route writes new files.
  • POST $B/api/video-editor/open-in-shotcut {"mlt_path": ...} opens Shotcut on the desktop of the machine running Guaardvark. Ask before using it; offer the .mlt path otherwise.
  • GET $B/api/video-editor/projects lists the projects saved on the Video Editor page. Read them; leave changing them to the page.
  • Filters and transitions an arrangement may name: GET $B/api/video-editor/catalog/filters and GET $B/api/video-editor/catalog/transitions.

Video with Guaardvark

Read setup first if the backend or the comfyui plugin state is unknown. Video needs a 16 GB-class NVIDIA card. Clips take minutes, so every route is queued.

Pick the model

GET ${GUAARDVARK_URL:-http://localhost:5000}/api/batch-video/models lists the registry with is_downloaded / is_ready (and missing_files), capabilities (modes t2v/i2v, max_frames, native_fps, aspect_ratios, min_steps, speed_profiles, audio_out), vram_mb, size_gb, license. Installed on a typical box:

idwhat it isnotes
wan22-5bWan 2.2 TI2V-5B, t2v + i2v, 24 fps, up to 121 framesthe everyday default
wan22-14b / wan22-14b-i2vWan 2.2 14B MoE, 16 fps, 81 framesbest quality; speed_profile: lightx2v-4 for 4-step Lightning
ltx23-distilled-fp8LTX-2.3 distilled, 8 steps, 161 framesfastest long clips
minimax-h3-int8MiniMax H3, text / first / last / first+last frame, generates its own stereo soundtrack and spoken linespass audio: true; ~6.5 min for 5 s on a 16 GB card
cogvideox-5b / -i2vCogVideoX, 8 fps, 49 frameslegacy

The active default is GET /api/settings/active_video_model (resolved.t2v, resolved.i2v, resolved.scene). Never pass a step count below the model's min_steps; the server raises it and the result would be smeared anyway.

One clip: MCP generate_video

  • prompt: scene, subject, motion, style, and any spoken lines (H3 speaks them).
  • model, duration_s (clamped to the model), aspect_ratio (one the model declares), style (cinematic, realistic, anime, 3d_animation, ...), num_inference_steps, speed_profile.
  • audio: true forces a soundtrack-capable model (H3) and fails on a silent family.
  • first_image / last_image: document id or path; last frame needs a first+last mode model.
  • reference_images (up to 9), reference_clips (up to 3, 2-15 s, their own sound left out) and reference_audio (up to 3): lock a person, look, motion or voice on the reference build (minimax-h3-ref2va-int8, picked when no model is named). The prompt names them <Picture N>, <Video N>, <Audio N>; to keep a clip's sound, pass its track as reference_audio. In the Studio this is the References input on Video Gen.
  • wait_for_result default false: the tool returns a batch id and a Studio deep link at once. Give the user the link; poll get_generation_status(batch_id=...) (MCP) or GET /api/batch-video/status/<batch_id> if they ask you to wait. wait_for_result: true blocks for the clip, up to 30 minutes.

Looping animation: MCP generate_animation

Frame-morph GIF/MP4 via img2img: prompt, motion, frames 2-24, strength 0.1-0.5, format gif|mp4|both. Use for short loops and stickers, not for cinema clips. Over MCP it answers at once with a job id (tooljob_...); poll get_generation_status for the GIF and MP4 URLs, or pass wait_for_result: true to wait up to 60 s for them.

Batch: REST

B=${GUAARDVARK_URL:-http://localhost:5000}
# text to video, one clip per prompt
curl -s -X POST $B/api/batch-video/generate/text -H 'Content-Type: application/json' -d '{
  "prompts": ["a red kite over a grey sea", "the same kite at dusk"],
  "model": "wan22-5b", "prompt_style": "cinematic", "enhance_prompt": true, "seed": 42
}'
# image to video
curl -s -X POST $B/api/batch-video/generate/image -H 'Content-Type: application/json' -d '{
  "image_paths": ["/abs/path/frame.png"], "prompt": "slow push in, wind in the grass", "model": "wan22-5b"
}'

Optional keys the server honours: negative_prompt, guidance_scale, motion_strength, interpolation_multiplier (RIFE frame interpolation), combine_frames, lora_name + lora_strength, adapters, guides (per-prompt audio/image anchors on models that declare them), last_frame_paths (image route), storyboard_concept / storyboard_shots. Poll GET $B/api/batch-video/status/<batch_id>; files via GET $B/api/batch-video/video/<batch_id>/<video_name>; cancel POST $B/api/batch-video/batch/<batch_id>/cancel; retry POST $B/api/batch-video/retry/<batch_id>.

Rules

  • Quote the model and the queued batch id back to the user. Never claim a clip is done until the status route says so.
  • The GPU is exclusive: while video renders, chat models are evicted. Warn before queuing a long batch.
  • The Studio page (Video Gen) shows the same queue; the user may prefer to watch it there.

Voice with Guaardvark

Read setup first. Expressive voices need the audio_foundry plugin running; Piper works without it. B=${GUAARDVARK_URL:-http://localhost:5000}.

Which engine

engineroutewhen
ChatterboxAudio Foundry backend: "chatterbox"expressive, emotion presets, cloning
KokoroAudio Foundry backend: "kokoro"fast, clean, 53 built-in voices in 9 languages (af_heart default); runs on the CPU without a GPU
Piper/api/voice/text-to-speechoffline fallback, no GPU

GET $B/api/audio-foundry/voices lists what is installed. GET $B/api/voice/voices lists Piper voices.

Speak a line or a script

Over MCP, call generate_speech (text up to 3000 characters, optional voice such as af_heart, engine auto | kokoro | chatterbox); it waits and returns the file with a download link. Cloning is not offered over MCP. Without MCP, or for the Chatterbox knobs, use REST:

curl -s -X POST $B/api/audio-foundry/generate/voice -H 'Content-Type: application/json' -d '{
  "text": "The line to speak.",
  "backend": "auto",            # auto | chatterbox | kokoro
  "voice_id": "af_heart",       # Kokoro voice, or omit
  "emotion": "calm",            # Chatterbox preset, or omit
  "exaggeration": 0.5, "cfg_weight": 0.5, "temperature": 0.8,   # Chatterbox knobs, optional
  "seed": 7, "output_format": "wav", "async": true
}'
  • Short text returns the file directly (path, document_id). With "async": true or long text you get 202 {"job_id"}: poll GET $B/api/audio-foundry/jobs/<job_id> until status is done; the result has path and document_id. Cancel: POST .../jobs/<job_id>/cancel.
  • Multi-section narration with pauses: POST $B/api/voice/narrate {"script": "...", "engine": "kokoro", "voice": "...", "pause_between_sections": 0.6, "output_format": "wav"}.
  • Piper only: POST $B/api/voice/text-to-speech {"text", "voice": "libritts"} returns audio_url.

Clone a voice (consent-gated)

  1. The reference must go through the upload route; that is what records consent. Arbitrary file paths are refused with 403.
    curl -s -X POST $B/api/audio-foundry/voice-clips/upload -F file=@/abs/path/ref.wav -F name="Narrator sample"
    
    The response gives the stored path. GET $B/api/audio-foundry/voice-clips lists clips.
  2. Generate with "backend": "chatterbox", "reference_clip_path": "<that path>".
  3. Before uploading, ask whether the voice belongs to the user or someone who consented. Do not clone a public figure or anyone who has not agreed. Refuse politely if unclear.

Rules

  • 10 to 20 seconds of clean speech is enough for a clone; more is not better.
  • Say which engine ran (the response reports it); auto falls back to Kokoro on a Chatterbox error.
  • Audio files are local under data/outputs/; they also appear in the Audio library page.

MCP servers 1

Sponsors

Details

Marketplace
guaardvark/guaardvark
Category
Productivity
Keywords
local-ai, self-hosted, image, video, music, voice, comfyui, ollama