
pai-pro
Filmmaking skills for Claude Code: scripts, characters, locations, storyboards, clips, voices, and a local canvas to assemble them.
- version
- 0.1.0
- GitHub stars
- 357
Install
/plugin install pai-pro@pai-proRun it inside Claude Code after adding the marketplace with /plugin marketplace add Utopai-Research/pai-code.
Contents
Discovered from sourceSkills 6
When to propose
- 3+ nodes share a clear semantic tie (same scene, same character, same beat)
- The relationship would be obvious to a reader within 2 seconds of scanning the canvas
- You can write a ≤ 30-character title that names the tie
Skip if fewer than 3 members or the tie is just generation order.
Contract
- Use
canvas_layout.jsto write memberpositionsandgroupFramesin one atomic sidecar update. - Never write or edit
workflow.json; never putx/yinto node data. - A layout change may move member nodes when that makes the canvas clearer.
- The frame's geometry (
x,y,width,height) is a bounding box computed from the final member positions. - A node may appear in at most one frame; evict it from old frames in the same update.
- No nested frames.
frameId:frame_<unix_ms>. Titles ≤30 chars recommended.
Patterns
Pick the one that fits. Grouping is current canvas state; read workflow.json per the project PROJECT_AGENT.md § "Choosing context" to verify ids.
1. Scene grouping
- Triggers: 3+ prompt/note/image/video nodes around one location, beat, or plot point.
- Title:
Scene <N> — <location or beat>. - Typical size: 3–8 members.
- Members: prompt/shot/image_result/video_result plus scene-scoped notes.
- Layout: place the script/shot note on the left, then images/videos in reading order to the right. Put attached voice/audio below the source card.
2. Character-reference set
A character card + its reference images.
- Triggers: ≥2 images of the same character.
- Title:
<Character name> — references. - Typical size: 2–6 images.
- Members: any
image_resultnodes depicting the same character. - Layout: hero/reference card first, variations in a compact grid, attached voice node below.
3. Act / beat grouping
- Triggers: the user framed the session at act/beat granularity ("everything for act 2", "the whole chase sequence", "opening titles").
- Title:
Act <N>or beat name. - Typical size: 8–15 members. If larger, prefer splitting into scene subgroups instead.
- Members: all nodes that belong to that act/beat, spanning multiple scenes.
- Layout: arrange scene clusters left-to-right in story order, with enough gutter that frames do not overlap.
4. Production-state grouping (opt-in)
- Use only when user explicitly sorts by quality/status: approved, draft, rejected, WIP, final.
- Title: one status word.
- Typical size: open-ended.
Recipe
-
Read
./workflow.json+./canvas_positions.json. workflow.json gives you node ids + labels + subtypes; canvas_positions.json gives you each node'sx/yAND the existinggroupFramesmap. Reads are unrestricted; writes go throughcanvas_layout.js. -
Pick members. Identify which nodes belong in the proposed frame by looking at their ids, labels, prompts, and subtypes. Keep only ids that actually exist in
nodes. -
Plan positions. Preserve existing positions when they already read well. Otherwise move the selected nodes into a compact layout for the chosen pattern. Use these default gaps:
- horizontal card gap: 40 px
- vertical row gap: 36 px
- frame padding: 24 px
-
Evict existing frame members in the same layout JSON:
- If the old frame would still have ≥ 2 members after eviction: include it under
groupFrames.upsertwithmemberIdsminus the evictee. - If the old frame would have < 2 members: include its id under
groupFrames.delete.
- If the old frame would still have ≥ 2 members after eviction: include it under
-
Compute frame bboxes from final member positions with 24px padding. Fallback sizes:
note: 280 × 420 (width hardcoded; height =NOTE_CARD_FALLBACK_HEIGHTfor first paint)image_result: 290 × 220 (16:9 default; ifdata.metadata.aspect_ratiois present, scale accordingly)video_result: 290 × 220 (same caveat; checkdata.aspectordata.metadata.aspect_ratio)audio_result: 240 × 64pending/pending_generation/pending_attachment: 260 × 200
If measured heights appear in
canvas_positions.json, prefer them.minX = min(node.x for each member) minY = min(node.y for each member) maxX = max(node.x + node.w for each member) maxY = max(node.y + node.h for each member) x = minX - 24 y = minY - 24 width = (maxX - minX) + 48 height = (maxY - minY) + 48 -
Decide title + hue. Default hue 200 if you have no signal.
-
Apply one layout update with all node moves and frame changes:
node "$PAI_REPO_ROOT/server/cli/canvas_layout.js" \ --layout-json '{"positions":{"note_2":{"x":120,"y":80},"image_3":{"x":440,"y":80},"video_1":{"x":760,"y":80}},"groupFrames":{"upsert":{"frame_1716579123456":{"memberIds":["note_2","image_3","video_1"],"x":96,"y":56,"width":978,"height":468,"hue":200,"title":"Scene 1 — Causeway"}},"delete":[]}}' -
Extending an existing frame — same CLI/frameId, full new
memberIds, recomputed bbox. -
Confirm to the user in one sentence. Example: "Grouped the three Morris reference shots under their own frame."
What not to do
- Don't propose groupings proactively when there's no clear semantic tie — wait until grouping earns the frame.
- Don't use grouping as a generic tidy operation when there is no semantic tie.
- Don't write
canvas_positions.jsondirectly. Usecanvas_layout.jsso positions and frames apply together. - Don't call removed workflow group ops (
addGroup,updateGroup,deleteGroup). - Don't nest (put one frame's id inside another frame's
memberIds). - Don't assign a node to two frames. Include any eviction in the same layout update.
CLI shape
Standard tier:
node "$PAI_REPO_ROOT/server/cli/generate_image.js" --prompt "..." [--aspect-ratio 16:9] [--image-size 2K] [--label "..."] [--subtype <character|location|edit|reference|split|storyboard>] [--name "..."] [--role "..."] [--description "..."] [--source-node-id <id>] [--ref-source-id <id> ...]
Pro tier for storyboard mosaics and video-bound character sheets:
node "$PAI_REPO_ROOT/server/cli/generate_image_pro.js" --prompt "..." --size 2560x1440 [--label "..."] [--subtype <character|location|edit|reference|split|storyboard>] [--name "..."] [--role "..."] [--description "..."] [--source-node-id <id>] [--ref-source-id <id> ...]
Pro accepts --size only; no --aspect-ratio / --image-size. Common sizes: 1024x1024, 1280x720, 720x1280, 1920x1920, 2560x1440, 1440x2560, 3840x2160, 2160x3840.
--label defaults to the truncated prompt (≤30 chars) if omitted; pass an explicit one when you have a better caption.
Use @Image1, @Image2, … in --ref-source-id order. The CLI emits one derived edge per ref.
Mirror external URLs first with mirror_url.js --url <URL>, then pass the returned node_id via --ref-source-id.
If a note authored the image, pass --source-node-id <note_id>.
Do not attempt to invent images via ASCII art or markdown embedding — call the CLI.
First-use image mode
For the ask-once flow and per-mode prices, see the project PROJECT_AGENT.md § "First-use generation choices".
Mode mapping: Standard 2K -> generate_image.js --image-size 2K; Pro 2K -> pro exact 2K; Max quality -> pro exact 4K.
Patterns
Pick the one that fits. For source lookup, follow the project PROJECT_AGENT.md § "Choosing context"; this skill only owns image-specific prompt and CLI shape.
Character pre-flight. First ask: will this character appear in downstream video (video, clip, promo, 宣传片, 短片, 连续剧, film, scene, 拍片, shot, short film)?
- Read
workflow.jsonfor uploaded refs (subtype:"reference",metadata.source:"user_upload", not archived). - Video-bound -> Pattern 7, not Pattern 1. Use ≥3 actor refs when available; with 0-2 refs, generate text-only 4-panel sheet.
- Announce one line before firing; allow redirect to Pattern 1.
- One-off static art/poster/portrait -> Pattern 1.
This pre-flight is non-negotiable. Pattern 1's single front portrait gives the video model an anchor that's too narrow; identity drifts shot-to-shot. Skipping straight to Pattern 1 for video work is the single most-common mistake.
Story/script anchor defaults. When script-compose or story-to-video-workflow routes a breakdown here:
- One base 4-panel sheet per material character.
- Extra sheets for material character variants: age, wardrobe/uniform/disguise, injury/dirty/wet/bloodied state, transformation, or continuity-significant looks.
- Detailed location anchors for settings that affect shots.
- Same-location variants for framing/scale/time/weather/light/dressing/story state/close-detail coverage changes.
- Prefer reference-to-clip after anchors. Storyboard only if requested, hard to control, or needed for diagnosis.
- Do not drop material variants for budget/speed; caller should adjust video resolution/runtime first.
1. Character portrait (one-off static stills only)
Triggers: character portrait/headshot/hero/villain/lead only for one-off static stills. Video-bound -> Pattern 7.
node "$PAI_REPO_ROOT/server/cli/generate_image.js" --prompt "..." --aspect-ratio 9:16 --image-size 2K --subtype character --name "Detective Morris" --role "..." --description "..."— no refs. A character is an identity anchor, not a derivative.- Prompt template:
[style] character portrait of [NAME], [role]. [age, build, wardrobe, distinguishing features]. Front-facing medium close-up, eye-level, looking directly at camera, neutral expression. Plain neutral background, soft even lighting. No dramatic shadows, no stylized lighting, no side profile, no multiple views. - Inherit project style or default to realistic. Name unnamed characters.
- No edges — characters are roots, so no
--ref-source-id.
2. Location establishing still
Triggers: establish/design/picture a location, or approval of a script-compose location offer.
node "$PAI_REPO_ROOT/server/cli/generate_image.js" --prompt "..." --aspect-ratio 16:9 --image-size 2K --subtype location --name "Causeway" --description "..." [--source-node-id <script_or_shot_note_id>]— no refs. A location is a setting anchor, not a derivative.- Prompt template:
[style] establishing still of [LOCATION NAME]. [visual brief — architecture, lighting, atmosphere]. Wide shot, eye-level, no characters present. - Keep frame empty of characters. Include architecture/layout, surfaces, dressing, era, weather, time, light, and story state when relevant.
- Same-location variants preserve place identity while changing wide/close scale, day/night, weather, dressing, damage, or detail coverage.
- No ref edges by default. If the location was derived from a script or shot note, use
--source-node-idso the authorship edge lands. - Follow-on: after the last scripted location, recommend the next reference-to-clip render. Mention storyboard only if requested/needed.
3. Edit / variation / turnaround of an existing image
Triggers: change/edit/swap/replace/add/remove/tweak/what-if/variation on an existing image.
-
Identify the source node (usually the most recent
image_result, or one the user named). Grabsource.idandsource.metadata.aspect_ratio. -
node "$PAI_REPO_ROOT/server/cli/generate_image.js" --prompt "..." --aspect-ratio <source ratio> --image-size <source size or 2K> --subtype edit --source-node-id <source.id> --ref-source-id <source.id>. -
Prompt as a transformation, not a full re-description:
<concrete change>. Preserve everything else.✅ "Change the rain to falling snow. Keep the detective, wardrobe, and camera framing unchanged." ✅ "Render as a full 3D turnaround sheet of the same character. Preserve face, wardrobe, and proportions." ❌ "A detective in a snowy alley at night wearing a trench coat…" — over-specifies, identity drifts.
-
The CLI emits the derived edge from
<source.id>based on--ref-source-id. -
Multi-step chains use one edge/call per step; do not flatten A -> C.
4. Scene featuring existing characters
Triggers: put character in setting / shot of X and Y / character action in location.
- Identify each character involved — any
image_resultof that person (up to 16). Collect each one'sid. node "$PAI_REPO_ROOT/server/cli/generate_image.js" --prompt "..." --aspect-ratio <fit the shot> --image-size 2K --ref-source-id <char1.id> --ref-source-id <char2.id> ....- Prompt: the full scene description. Name each character by their role so the generator binds identity to role. Refer to them in the prompt as
@Image1,@Image2, … in--ref-source-idorder. - No
--subtype— a scene is neither a character nor an edit. CLI emits one derived edge per--ref-source-id.
5. Standalone still
Triggers: a fresh image unrelated to existing canvas content ("generate a mountain at dusk", "a noir alley — just the setting").
- Plain
node "$PAI_REPO_ROOT/server/cli/generate_image.js" --prompt "..."with sensible defaults (16:9, 2K unless the user asks otherwise). No subtype, no refs.
6. Storyboard mosaic — one composite per clip / shot note
Triggers: storyboard, mosaic, grid, shot list, coverage, keyframe sheet, shot planning, image previs. Output is ONE composite per clip/<=15s shot note, not one image per panel.
- Tool/call: one
generate_image_pro.js --size 2560x1440 --subtype storyboard --label "Storyboard — Shot <N>" --source-node-id <shot_note_id>per mosaic. - Size: default
2560x1440; grid shape is cell layout, not canvas shape. Override only for explicit portrait/square/vertical/ratio:- "2x2 storyboard" →
2560x1440(default) - "2x2 square storyboard" →
1920x1920 - "2x2 portrait storyboard" →
1440x2560 - "vertical 2x4 mosaic" →
1440x2560
- "2x2 storyboard" →
- Grid/refs: default 2×2. Pass relevant character/location refs (≤32). Warn before 3×3+; recommend smaller sheets.
For the canvas pre-flight, per-shot-note iteration logic, missing-anchor nudge, verbatim prompt template, and default panel coverage when no script slice exists: see references/storyboard-mosaic.md.
Node fields the CLI sets
- ONE
image_resultnode PER mosaic. - Set
data.subtype = "storyboard"by passing--subtype storyboard.
7. Character reference sheet (default for ANY video-bound character work — with or without actor refs)
Use proactively for any character that will appear in downstream video, regardless of actor refs.
Mode A — ≥3 uploaded actor refs. Pass photos as --ref-source-id; add --source-node-id <note_id> if a script/shot authored the design.
Mode B — 0-2 refs / from scratch. Omit actor refs; describe age/build/wardrobe/distinguishing features explicitly. Add --source-node-id <note_id> when authored by a note.
Also fires on explicit asks: "design a character sheet / turnaround / reference sheet / character design for [character]", "make a 4-panel character design", "generate a production reference sheet for downstream video work".
Output: one Front-full / Profile-full / Back-full / Closeup-bust sheet, passed directly to video as --ref-source-id.
- Pre-flight: identify uploaded reference nodes (
subtype:"reference", ideally ≥3 photos). Confirm ref count in one line. node "$PAI_REPO_ROOT/server/cli/generate_image_pro.js" --prompt "..." --size 2560x1440 --subtype character --name "<character_name>" --role "..." --ref-source-id <ref1> --ref-source-id <ref2> --ref-source-id <ref3> [--source-node-id <script_or_shot_note_id>]— pro tier is the default for character sheets because panel layout, text suppression, and identity consistency are load-bearing. Do not pass--aspect-ratioor--image-size. Never fire Mode A with fewer than 3 refs (model overfits to the one angle it has).- With 0-2 actor refs, use the Mode B text-only command from
references/character-sheet.md: same pro command, but omit every actor-photo--ref-source-id; include--source-node-idonly when a script or shot note authored the design. - One call/sheet per base character or material variant. Generate base before variants. For identity-preserving variants, pass base sheet as ref and describe only persistent wardrobe/state change.
- Use the sheet as
--ref-source-id <sheet_id>for downstream shots; no normal cropping needed.
For the verbatim 4-panel prompt template, optional per-angle crops, and gotchas (no-text rule, photo-priority, exact panel counts): see references/character-sheet.md.
Node fields the CLI sets
- ONE
image_resultnode withdata.subtype = "character". data.name,data.role,data.descriptionfrom the CLI flags.- One
derivededge per--ref-source-id(so the sheet is provenance-linked to each actor photo it triangulated from).
After the CLI returns
For draft-stage JSON, one sentence with the price/status — see the project PROJECT_AGENT.md § "Draft gate". For terminal results, run the alignment check before the next-step recommendation — see the project PROJECT_AGENT.md § "Prompt alignment check".
Run only on explicit user intent, never on file drop. Dropped text/PDF already exists as a note (data.body) and mirror (./assets/notes/<note_id>.md).
Defaults: a 30s beat is one moment; match the input language; with characters, prefer meaningful dialogue and let narration support rather than carry the scene, but let wordless action carry a beat instead of packing every one with dialogue.
Stop at script capture, shot notes, and anchor extraction. Route multi-stage work back to story-to-video-workflow.
Capture target duration when observable:
- Explicit user duration wins ("30 seconds", "2-minute short").
- Timestamp blocks come next; sum them.
- Otherwise estimate roughly and mark it as an estimate.
Store target_duration_sec and duration_basis when known. If implied runtime is >~3 minutes, call out scope before shot/video planning.
1. Triage → Capture
Classify the input, then capture as in §2. Never skip straight to §3.
- Screenplay (INT./EXT. + ALL-CAPS cues + dialogue) → use verbatim. For dropped text/PDF, read
workflow.jsondata.bodyor./assets/notes/<note_id>.md. Pick a 2–5 word title; identify duration basis; do not rewrite to fit. - Story / concept (prose, pitch, logline) → sketch ONE paragraph back (setting, characters, conflict, target duration) and ask if it's the shape. Iterate. On "yes/go", rewrite using the rules below, then capture.
- Neither → don't run; defer to
image-compose/video-compose.
Torn between screenplay and story? Prefer screenplay — safer than rewriting.
Rewrite rules (story → screenplay):
- Format:
INT./EXT. LOCATION - TIMEslug, present-tense action, ALL-CAPS cue + dialogue. No scene numbering. No camera directions (that'svideo-compose). - Preserve user-quoted dialogue verbatim.
- With characters, include dialogue that reveals motive/conflict/relationship; avoid narration-only exposition unless VO-driven.
- Pace speech at ~2.2-2.5 words/sec plus reaction/action room.
- Duration: match if stated; default 30–45s. Don't overshoot.
- Short input, longer target? Keep verbatim and ask "reads as ~Ns; extend?" — don't silently pad.
2. Capture — canvas note + title
ONE note. No split. Canvas writes go through the mutator, never direct workflow.json.
read./workflow.json(read-only inspection — see iftitleis already set).- Append the script note via the mutator with
subtype: "script":
Omitnode "$PAI_REPO_ROOT/server/cli/canvas_mutate.js" \ --op addNode \ --payload-json '{"node":{"type":"note","data":{"subtype":"script","label":"Script: <title>","body":"<full screenplay verbatim>","metadata":{"author":"agent","timestamp":"<ISO>","target_duration_sec":45,"duration_basis":"estimated from script length"}}}}'target_duration_sec/duration_basisonly when there is no defensible signal. Stdout returnsassigned.node_id— keep it for §3 (shots derive from this id). - Set the workflow title if empty:
node "$PAI_REPO_ROOT/server/cli/canvas_mutate.js" --op setTitle --payload-json '{"title":"<title>"}' - Confirm with
Captured., then offer the next step as a choice rendered per the projectPROJECT_AGENT.md§ "Recommendation and choice shape". Recommended option: "Split it into <=15s shots and extract characters/locations/voices." Plus an escape to do something else.
STOP. Do NOT proceed to §3 without an explicit user command.
3. Analyze — on explicit user command
Triggers (judge intent): "split into shots / clips", "break this up", "pull the characters / locations", "who's in this", "analyze this script", "design the characters from this script". Not triggers: "what's in this", "summarize", "tell me about it" — those are read-and-reply.
When triggered:
- Slug — kebab-case of the working title. Collision → suffix
-2,-3. - Shot splits (≤15s each): use
metadata.target_duration_secor estimate. Split on natural beats (slug/dialogue/location/time/appearance changes). For >15s material, keep resulting shots as close to 15s as natural (default ≈ceil(total_seconds / 15)shots); split shorter only for hard cuts, dialogue turns, continuity shifts, or strong beats — don't over-fragment just because the script's own time markers say so. Pace speech at ~2.2-2.5 words/sec plus reaction/action room; silent action ~3–5s. If dialogue cannot fit naturally, split it; reduce only when the user asked for compression. Never rewrite — shot bodies are verbatim slices. Each shot note hassubtype: "shot". Build oneaddBatchwith N shot notes + N derived edges:node "$PAI_REPO_ROOT/server/cli/canvas_mutate.js" \ --op addBatch \ --payload-json '{ "nodes": [ {"type":"note","data":{"subtype":"shot","label":"Shot 1 (0–15s)","body":"<slice>","metadata":{"author":"agent","timestamp":"<ISO>"}}}, {"type":"note","data":{"subtype":"shot","label":"Shot 2 (15–30s)","body":"<slice>","metadata":{"author":"agent","timestamp":"<ISO>"}}} ], "edges": [ {"from":"<script_note_id>","to":"$0","kind":"derived"}, {"from":"<script_note_id>","to":"$1","kind":"derived"} ] }'$Nplaceholders are 0-indexed positions innodes; the mutator resolves them to the assigned ids after running. Reply'sassigned.node_idsis the array of shot ids in the same order. - Anchor extraction — from the shot bodies, extract only downstream needs:
- Characters: recurring/visually important people/entities. Include one-line base visuals only when given.
- Variants: same character with materially different on-screen look by scene/shot: age jump, costume change, injury, disguise, transformation, wet/dirty/bloodied state if it must persist across shots. Do not create variants for transient expressions or tiny props.
- Locations: distinct settings plus same-setting variants when framing/scale, time, weather, lighting, dressing, story state, or close/detail coverage matters.
- Voices: every speaking character and narration/V.O.; preserve speaker labels and dialogue language.
- Missing anchors: first character, variant, location, or voice that blocks rendering Shot 1.
- Parse offer — ONE compact planning line plus a soft next step:
Plan check: ~<seconds>s, <shots> shots, <N> character(s), <V> variant(s), <M> location(s), <S> voice need(s). Missing: <first blocker>.If N>0, V>0, M>0, or S>0, offer next step with project choice shape. Recommended: "Design the character/location anchors, then voices." Anchors include base/variant character sheets, detailed location/location variants, and speaker/VO voice anchors. On approval, route toimage-composefirst (base character sheets, needed character variants, and location stills) with--source-node-id <script_note_id>so the new nodes wire back to the script. After image anchors land, route speaking/narration needs tovoice-compose. Don't generate insidescript-compose. Skip the offer if every count is 0.
If the user's command was narrower ("just the shots", "only characters"), do only that sub-step and skip the offer.
4. Revisions
Surgical (title still fits): update script-note body + affected shot bodies in place. Use the mutator's updateNode op (one call per node, or batched via updateBatch).
Structural (title no longer fits): new script note (addNode); old→new edge addEdge with kind:"derived"; new shot family via addBatch against the new script note. Leave old shots; delete only if asked (deleteNode cascades edges for you).
Story-to-video workflow
Contract
- This skill wakes first for story/script/promo-to-video work.
- Own sequencing only. Before execution, load the matching capability skill; do not call
generate_*here. - Recommendations are not consent. Stop before paid generation or pipeline changes unless the user explicitly approved an autonomous workflow.
Default arc
Use this ladder unless the user skips, reorders, supplies refs, or asks for a rough direct render:
- Clarify only blockers.
- Raw idea/story ->
script-composeproduction script; existing screenplay -> capture/adapt. script-composesplits <=15s dialogue-aware shots and extracts characters, material variants, detailed locations/location variants, and speaker/VO needs.image-composecreates useful visual anchors: base/variant character sheets and detailed location/detail anchors.voice-composecreates reusable anchors for every speaker and VO/narrator.- Confirm shot count, durations, continuity needs, and first blocker.
- Default render path: straight-to-video from refs. Storyboard only if requested, hard to control, or needed for diagnosis.
- Default dispatch: hybrid. Chain continuous dependent shots; render independent scenes/shots in parallel.
- Render clips, assign Timeline
shot_idwhen sequence order is unambiguous, then hand off to Timeline.
Plan ahead internally, but only ask the next meaningful user-facing choice; the Consent and gates ladder fixes when render path and dispatch become askable.
Skill routing
| Need | Load next |
|---|---|
| Script capture, rewrite, split, or analysis | script-compose |
| Character, location, storyboard, starting frame, or visual anchor | image-compose |
| Narration, dialogue read, character voice, or audio node | voice-compose |
| Clip render, continuation, audio refs, storyboard animation, or video prompt | video-compose |
| Scene/ref grouping or canvas layout frames | groups-compose |
Capability skills own CLI flags, node grammar, refs, and recovery hints. PROJECT_AGENT.md owns shared failure handling.
Consent and gates
- Draft-only, failed, and cancelled generations do not advance the pipeline.
- One-off generation outside the story pipeline routes directly to the capability skill.
- Honor explicit rough-direct/skip choices.
- Gate ladder: script -> shot notes -> anchors/user refs/rough-direct -> real clip plan -> render path -> dispatch. Ask each rung only after the prior one is real; stop after render-path unless the user already names dispatch too.
VO and dialogue invariants
- Script/shot notes carry dialogue/VO until final audio exists.
audio_result.data.textis source of truth only for approved final narration/line reads.video-composeincludes spoken text verbatim and treats voice samples as timbre anchors.
Recommendation shape
Follow the project PROJECT_AGENT.md § "Recommendation and choice shape". Recommend one concrete next step. Add a second option only when there is a real tradeoff.
Planning checkpoint
Before recommending refs/video, inspect workflow.json when needed and summarize only:
- Target duration from user duration, timestamps, or a rough estimate.
- Planned <=15s shot count.
- Characters, material variants, detailed locations/location variants, close/detail needs, speakers/VO.
- First missing anchor blocking the next clip.
If the story implies more than roughly 3 minutes, recommend narrowing scope before clip planning.
After shot notes, missing video-bound character/location/voice anchors are the default next step; include a rough-direct skip when speed matters. Once anchors/user refs/rough-direct are settled, offer only a short ref review or clip-plan confirmation if ambiguity remains.
Render path
Ask only after the script/shot plan is settled and anchors, usable refs, rough-direct, or a simple single-clip case make rendering real. If anchors are still missing, return to Planning checkpoint.
Use project choice shape:
- header:
Render - question:
Choose render path. - options:
- label:
Straight to video (Recommended)description:Fastest path to motion. - label:
Storyboard firstdescription:Generate storyboard images first for composition control.
- label:
For storyboard-first, load image-compose Pattern 6: one composite mosaic per clip/<=15s shot note, subtype storyboard.
Dispatch for multiple clips
Ask only after render path is picked and a multi-clip plan exists. Skip for one clip. Use project choice shape:
- header:
Dispatch - question:
Choose clip dispatch. - options: order these by the observable story signals below; suffix the first label with
(Recommended).- label:
Hybriddescription:Chain within continuous scenes; render separate scenes independently. - label:
Paralleldescription:Render all clips independently. - label:
Sequentialdescription:Each clip continues from the previous one; boundaries default to a hard cut to a new angle (avoids the same-shot seam) — keep a boundary same-shot only for an unbroken oner.
- label:
Signals: continuous scene/state -> sequential (hard-cut handoffs between clips); a single unbroken action the viewer must read as ONE motion -> one ≤15s clip, else sequential with a same-shot handoff; separate scenes/time jumps/wardrobe changes/montage -> parallel; continuous clusters separated by hard cuts -> hybrid. Do not chain video refs across location, time, wardrobe/state, dream/reality, or montage breaks.
After media results
After terminal generate_*:
- If it is only draft-stage JSON, report the price/status and stop.
- If
ok:false, follow project failure handling and do not advance the pipeline. - If
ok:true, identify the landed node id from the result or canvas state. - Read
workflow.jsonif shots, refs, voices, clips, or reel order affect the next decision. - Recommend exactly one next useful filmmaking move.
Typical priority:
- Script note landed -> recommend splitting into <=15s shot notes and extracting anchors.
- Shot notes exist but anchors are missing -> recommend the first missing character/location anchor, with a rough-direct skip option. Do not ask render path or dispatch yet.
- Character/location ref landed -> finish remaining anchors; then ref review, clip-plan confirmation, or straight-to-video. Mention storyboard only if requested/useful.
- Voice landed -> recommend using it with the matching visual ref in the next dialogue/narration clip.
- Storyboard landed -> recommend review or animating the matching clip.
- Video clip landed -> recommend the next clip, or Timeline handoff when all planned clips are ready.
Final handoff
Timeline owns reel order. Numeric video_result.data.shot_id means a clip is in the reel. When all planned story clips are ready and order is unambiguous, assign shot_id = 1..N with one updateBatch before handoff:
node "$PAI_REPO_ROOT/server/cli/canvas_mutate.js" \
--op updateBatch \
--payload-json '{"updates":[{"id":"<video_1>","patch":{"shot_id":1}},{"id":"<video_2>","patch":{"shot_id":2}}]}'
Do not use generate_video.js --shot-id for speculative/partial ordering. Assign after clips land. Local export uses reel_stitch.js only on explicit request. Then tell the user to open Timeline to inspect and preview.
Intent dispatcher. Patterns name trigger, call, edges, and prompt reference.
Hard defaults
- Stage by default per
PROJECT_AGENT.md. - Audio on by default; pass
--no-audioonly for explicit silent/no-audio requests. Trailer/portrait/cinematic framing is NOT a trigger; audio is the baseline, not optional polish. - Reference-to-clip default: use available character/variant/location/voice refs directly. Storyboard only if requested, hard to control, or needed for diagnosis.
- Preserve scripted dialogue/VO exactly unless the user asks for rewrite.
First-use video mode
For the ask-once flow and per-mode prices, see the project PROJECT_AGENT.md § "First-use generation choices". Pass --resolution only for 480p Draft or 1080p Final.
CLI shape
node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." [--version <2.0|2.5>] [--duration <seconds>]
[--aspect-ratio 16:9] [--resolution <480p|720p|1080p>] [--no-audio]
[--label "..."] [--ref-source-id <id> ...] [--ref-audio-source-id <audio_id> ...]
[--source-node-id <id>] [--shot-id <N>]
Calls go via --stage — see the project PROJECT_AGENT.md § "Draft gate".
--label defaults to truncated prompt. Use --ref-source-id for image/video refs, --ref-audio-source-id for audio refs, and --source-node-id for the authoring note. Mirror external URLs first. Do not set --shot-id during speculative/partial generation unless user asks for a reel position; story sequences assign Timeline order after planned clips land.
Match stated single-clip duration with --duration; omit for 15s default. Split or chain >15s totals.
Each clip costs real money even after staging — only stage after the user has explicitly asked for a video.
Model version (--version)
Omit it for 2.0 (today's model): output 4-15s, priced per output second by resolution.
--version 2.5 renders PAI Video 2.5. Differences that change how you call it:
--durationis 5-30 (not 4-15).-1(follow-the-input edit mode) is rejected.--resolutionmust be480p,720p, or1080p;--aspect-ratiomust be one of16:9 | 9:16 | 4:3 | 3:4 | 1:1 | 21:9 | adaptive.- One flat price per billed-duration tier, where billed = output seconds + every
reference-video second (each clip measured locally with ffprobe and rounded up).
Hard cap 60 billed seconds; over that the CLI refuses before staging and names both
halves. Tier prices are in the project
PROJECT_AGENT.md§ "Media CLIs (server/cli/)" — quote them from there, never from memory. - Refused inside an Auto run (the run's budget estimate is computed from 2.0 rates). Stage 2.5 clips outside Auto.
- A 2.5 job carrying video references needs
ffprobeon the host; without it the CLI fails withinfrarather than guessing a price.
On failure the limits blob is model-scoped: read min_output_sec / max_output_sec
and max_billed_sec off the returned JSON rather than assuming 2.0's numbers.
Reference caps
2.0 (video-generation) — ≤9 image refs, ≤3 audio refs, ≤3 video refs; video refs cap at 15s aggregate.
2.5 — ≤30 image refs, ≤10 audio refs, ≤10 video refs; reference video caps at 30s aggregate, and every reference second also counts toward the 60s billed-duration cap.
Both: audio/video refs must be 1.8s-15.2s each (the shared reference-upload step's window), and audio refs need an image or video anchor — on 2.5 that one is checked only after credits are frozen, so the CLI refuses it locally. Read durations from workflow.json; on failure use the returned limits + sent, which are model-scoped.
Reference roles — vocabulary
Prompt wording binds each ref role:
| Role | Flag | Wording in prompt |
|---|---|---|
| Character identity | --ref-source-id (image) | "the character in @Image1" |
| Location / setting | --ref-source-id (image) | "the location shown in @Image1" |
| Opening frame | --ref-source-id (image) | "opening frame @Image1, …" |
| Closing frame | --ref-source-id (image) | "closing on the frame from @Image1" |
| Source clip — continue (next clip in a chain) | --ref-source-id (video) | Default = hard cut: "Hard cut from @Video1: open on a NEW camera angle; do not match its final frame." Same-shot ("Continue from @Video1 … maintain camera position") only for an authored held beat / oner / explicit user request — see references/video-extension.md |
| Source clip — transform | --ref-source-id (video) | "Re-render @Video1 in …" |
| Camera-move source | --ref-source-id (video) | "camera moves match @Video1" |
| Action source | --ref-source-id (video) | "action choreography matches @Video1" |
| VFX template | --ref-source-id (video) | "use the visual-effects template from @Video1" |
| Voice / timbre anchor | --ref-audio-source-id | "Use @Audio1 as voice/timbre reference. Speak once, no echo." |
Prompt-language conventions
- Ref syntax:
@Image1/@Video1/@Audio1, positional by flag order. Every@ImageN/@VideoN/@AudioNMUST have a matching--ref-source-id/--ref-audio-source-idflag — the CLI rejects a mismatch (bad_args) before generating. Mentioning the same ref many times is fine; only the highest index per kind needs a flag. - Spoken text: include script/shot/user dialogue/VO verbatim; do not summarize, translate, shorten, polish, or invent.
- Dialogue scenes: keep the shot/script dialogue in the prompt; use one approved voice sample per speaker as a timbre anchor. Bind each quoted line to the intended character and the matching
@AudioNreference. Do not generate per-line audio refs unless the user explicitly wants separate final audio. - Final audio exception: if an audio node is the approved narration/line read, use
audio_result.data.textverbatim. If it is just a character voice sample, do not replace the shot dialogue with the sample text. - Add dialogue guards for model-spoken lines: "each line spoken exactly once, no echo, no repeated reads." Add phonetic spelling for names or words likely to slur.
- One camera move, one action speed, concrete sound/music (
No Musicif none). Use exact terms:locked off,handheld, subtle,slow dolly in,slow orbit,whip pan,speed ramp. - Avoid conflicts ("static camera" + "orbit shot").
- For brand / MV / ad work, end the prompt with a negative line: "no captions, watermarks, distortion, stretching."
- For polish on a single-shot clip: see
references/video-single-shot.md.
Patterns
Pick the one that fits. Source lookup follows PROJECT_AGENT.md.
Storyboard guard: storyboard images route to Pattern 7 / references/video-multi-shot.md, never generic I2V/opening-frame wording.
1. Standalone T2V
Triggers: fresh clip unrelated to canvas content.
Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..."; omitted flags default to 15s, 16:9, 720p, audio on. Add --resolution 480p or --resolution 1080p only if the chosen video mode requires it.
Edges: none.
For the bracket scaffold and slot-by-slot construction when the user wants polish: see references/video-single-shot.md.
2. Animate a canvas image (I2V)
Triggers: animate/make video/put motion on a specific canvas image_result.
Source: named image_result; storyboard mosaics route through Storyboard guard.
Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-source-id <image.id>.
Edges: { from: <source.id>, to: video_<N>, kind: "derived" } — emitted by the CLI.
Anchor wording: opening-frame default (opening frame @Image1) or closing-frame (closing on the frame from @Image1); both use --ref-source-id.
For slot-by-slot construction and the opening- vs closing-frame phrasing: see references/video-single-shot.md.
3. Compose with canvas characters / locations
Triggers: video of character, character in setting, character action in location.
Source: character / location image_result nodes (cap from §Reference caps).
Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-source-id <char1.id> --ref-source-id <char2.id> ....
Edges: { from: <char.id>, to: video_<N>, kind: "derived" } — one per --ref-source-id.
- Prefer exact shot refs: current wardrobe/state character sheet, detailed location, same-location variant.
For single-shot composition and adjacent-role wording: see
references/video-single-shot.md. For ≥2 internal shots in one render: seereferences/video-multi-shot.md.
4. Extend a canvas clip
Triggers: continue/extend/what happens after/scene follows existing video_result.
Source: any canvas video_result node — agent-generated or user-uploaded (data.metadata.source is "pai" for generated and "user_upload" for dropped).
Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-source-id <source_video.id>.
Edges: { from: <source_video.id>, to: video_<N>, kind: "derived" }.
Boundary defaults to a HARD CUT (clip 2 opens on a new angle — this avoids the same-shot seam morph). If the whole sequence fits ≤15s, render ONE multi-shot clip (Pattern 7) instead of chaining. Same-shot continuation is the exception (authored held beat / story-required oner / explicit user request).
For the hard-cut + same-shot prefixes, the ≤15s guard, the sub-intent decision tree, and sequencing across linked calls: see references/video-extension.md.
5. Edit a canvas clip
Triggers: re-render/restyle/add/remove/swap/change/rewrite existing video_result. Creative edits use generate_video.js; ffmpeg is for mechanical ops.
Source: any canvas video_result node — agent-generated or user-uploaded.
Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-source-id <source_video.id>.
Edges: { from: <source_video.id>, to: video_<N>, kind: "derived" }.
For the Restyle / Partial / Replace / Re-plot decision tree and per-mode templates: see references/video-editing.md.
6. Voice-driven clip
Triggers: have character say/narrate, use character voice.
Source: any canvas audio_result node — agent-generated or user-uploaded.
Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." --ref-audio-source-id <audio_id>. Often combined with character image refs for face + voice — pass both --ref-source-id <character_id> (for the character image) and --ref-audio-source-id <audio_id> (for the voice).
Prompt:
- Character voice sample:
The character in @Image1 says exactly: "...". Use @Audio1 as the voice/timbre reference only. Speak the quoted line exactly once, no echo, no repeated reads. - Multiple speakers: bind each visual ref and audio ref explicitly, e.g.
The character in @Image1 uses @Audio1 and says exactly: "..."; the character in @Image2 uses @Audio2 and replies exactly: "...". Each line is spoken exactly once, no echo, no repeated reads. - Approved final narration/line read: use
audio_result.data.textexactly and bind it withUse @Audio1 for timing, cadence, and voice. Keep the words unchanged. - No audio node: preserve requested dialogue verbatim as
[Character] says exactly: "..."; add the once/no-echo guard and phonetic spellings for risky words. - Audio uploads without
data.text: sound/timing only; do not invent transcript. - Images identify speakers; spoken words come from audio text, script/shot note, or user dialogue.
Edges: depends on which character refs attach (one kind: "derived" per ref).
7. Multi-shot / brand / ad / MV
Triggers: ≥2 shots inside one render, ad/MV/brand framing, or ≥10s with multiple movements.
Call: node "$PAI_REPO_ROOT/server/cli/generate_video.js" --prompt "..." [--ref-source-id <image|video.id> ...] [--ref-audio-source-id <audio.id> ...].
Edges: as per the underlying pattern (3, 4, 5) for any refs attached.
For the 4-section scaffold (timeline / effects inventory / density map / energy arc) and how to populate the timeline from canvas script shot notes or storyboard mosaic panels: see references/video-multi-shot.md.
Common combinations
Cross-pattern asks route to one primary reference:
| Combo | Primary reference | Extra refs to attach |
|---|---|---|
| Character + voice-over | (Pattern 6 inline) | character image |
| Music video with characters | video-multi-shot.md | character images + audio (Pattern 6 wording) |
| Restyle preserving identity | video-editing.md (Restyle) | source video + character image |
| Multi-clip chained sequence | video-extension.md | source video for each link |
| Compose with camera-move from reference | video-single-shot.md | character images + camera-move video ref |
| Render one script shot from canvas | Pattern 1, 2, or 3 by shot content (no dispatch — translate the shot note body to slot rules; preserve dialogue/VO verbatim) | character / variant refs + location / variant refs + voice anchors if the shot involves them |
| Render a continuous script span (>15s total) as a dependent sequence | video-extension.md (script-driven chain; hard-cut handoffs by default — keep a link same-shot only for an unbroken oner the viewer must read as one motion) | source video per link + character refs (mandatory under hard cut) for identity |
| Render a short script (≤15s total) as one piece | video-multi-shot.md (cross-skill source) | character image refs locked across shots |
| Render a storyboard mosaic as one 15s video (every panel becomes a shot block) | video-multi-shot.md (storyboard cross-skill source; required for image_result.subtype === "storyboard") | mosaic image + character / location image refs that authored the mosaic |
Sequence dispatch guidance
For multiple clips, prefer hybrid dispatch: independent for separate scenes/time jumps/wardrobe-state changes/montage; chain only same continuous action/location/lighting/wardrobe-state/emotional beat. Do not chain across breaks where continuity is undesirable.
If a budget-aware caller needs savings, lower video resolution before shortening runtime. Do not suggest dropping material character variants, location variants, or voice anchors that preserve continuity.
After the CLI returns
For draft-stage JSON, one sentence with the price/status — see the project PROJECT_AGENT.md § "Draft gate". For terminal results, run the alignment check (frames via extract_frames.js) before the next-step recommendation — see the project PROJECT_AGENT.md § "Prompt alignment check". --ref-source-id flags drive provenance edges; they're captured in the draft argv and materialize on the real video_result after the user fires.
Failure hints
Video-specific message hints:
asset_rejectedwith "DownloadFailed" —failed_urlwas unreachable; swap.asset_rejectedwith "DurationTooLong" / "DurationTooShort" —failed_url's duration is outside 1.8s–15.2s. Swap it or trim with ffmpeg.bad_argswith "reference_audio cannot be the only reference input" — add an image or video ref alongside the audio.bad_argswith "invalid video duration, exceeds 15s" — sum of video refs breachedlimits.max_total_video_sec. Read eachvideo_result.data.durationfrom canvas and drop refs until the sum is ≤15s.
Default to one short reusable timbre sample per speaking character and one VO/narrator sample when narration exists. video-compose keeps actual shot dialogue/VO in the video prompt. Treat audio_result.data.text as downstream speech only for approved final narration/line reads.
Patterns
Follow PROJECT_AGENT.md for context/staging. This skill owns voice prompt + CLI shape.
1. Character voice sample
Triggers: "give / design a voice for [character]", "what does [character] sound like", "voices for all the characters on the canvas".
-
Target: any
image_resultfor the person; don't gate on subtype. Readdata.local_pathbefore prompt; layername/role/descriptionon top. -
Call:
node "$PAI_REPO_ROOT/server/cli/generate_voice.js" \ --text "<line>" \ --prompt "<voice design brief>" \ --source-node-id <character.id> -
Prompt describes the voice, not the character:
[age bracket] [gender], [timbre], [register], [pace], [accent if relevant]. [optional emotional color].✅ "Mid-50s man, gravelly baritone, measured pace, slight rasp from decades of smoking, weary but steady." ✅ "Young woman, bright mezzo, warm, quick and percussive. Slight Southern lilt." ❌ "Detective Morris's voice." — names the character, not the voice. The model needs sound qualities.
-
text: 1-3 sentence in-character sample (≤200 chars), not every script line. -
Script breakdowns: one staged call per speaker; preserve labels. Add separate VO/narrator via Pattern 2.
2. Narrator / VO voice sample or final line audio
Triggers: narrator voice, voice-over, "a voice that says X" without character, narration track, or explicit final line-read audio.
- Omit
--source-node-id:node "$PAI_REPO_ROOT/server/cli/generate_voice.js" \ --text "<the narration line>" \ --prompt "<voice design brief>" - Same prompt convention as Pattern 1.
- Reusable VO/narrator anchor: short sample line in narrator style, not full script narration.
- Final narration/line-read: copy approved text exactly into
--text; thendata.textis source of truth.
Sponsors
Details
- Marketplace
- Utopai-Research/pai-code
- Category
- Community