oaustegard avatar

media-processing

Image manipulation, video/audio processing, file format conversion, SVG conversion, and computer vision augmentation.

version
2.3.0
GitHub stars
404

Install

/plugin install media-processing@oaustegard-claude-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add oaustegard/claude-skills.

Contents

Discovered from source

Skills 17

Animating with code

You write the drawing; a headless browser (or PIL, cairo, ffmpeg) renders it. The method below comes from the ClaudeAnimationBase kit's guide and holds for any renderer. The kit itself is the fastest route to a hand-painted character short.

Before building, name the route and why (Oskar's standing steer, memory 06886186): the kit, a CSS/HTML page captured frame by frame, a pure ffmpeg filtergraph, or PIL/cairo frames.

The contract: frames are pure functions of t

  • A shot is fn(t). Same t, same pixels. No counters, no state carried between frames, no integrated physics, no unseeded random. Then frames can render out of order, split across parallel workers, and resume after a crash.
  • Stable per-object values come from a hash of an index. Deliberate wobble ("boil") comes from a seed that changes on a fixed clock (the kit uses 12 Hz, each drawing held two frames).
  • Reseed per element. A moving element consumes a variable amount of randomness; without its own seed it shifts the stream for everything drawn after it and the still parts jitter every frame.

Procedure

  1. Storyboard in writing, before any scene code. One logline (wants ___, but ___, so ___), a small palette with a colour arc, a motif that pays off, and the emotion keys across the whole film. Then per shot: time span, transition in, the event, the reaction, the camera, and the reads.
  2. Write the reads. A read is one thing the viewer must understand, with a start and end time. Each needs time for the eye to find it, to understand it, and to register before the next starts. No two important reads overlap. If they do not fit the shot, lengthen the shot or cut a read. Timing is where generated animation fails most: everything at one brisk speed, events stacked, moments gone before they land.
  3. Check the storyboard: an event in every shot, a transition at every seam (including in and out of the film), no text, the ending rhymes with the opening. If a person is waiting, show them the storyboard before building.
  4. Block key poses as stills at each shot's storytelling moments. If the poses do not read as stills, motion will not save them.
  5. Add motion between poses, one shot at a time, in order: anticipation before big moves, slow in/out instead of raw lerp, arcs, overlapping action (eyes lead, body follows, appendages settle last), no twinning (offset left and right, offset characters' phases), exaggeration.
  6. Review loop (below) on every shot. Fix, look again.
  7. Render the full film; verify with ffprobe that duration and frame count match the config.

Review loop

You cannot see motion by reading code. Render images and open them with Read.

viewwhat it showscatches
contact sheetfirst, middle, last frame of every shotmissing events, small or lost characters, dead stretches
fixed-step sheet (every 0.1–0.15 s)one shot, read in ordertiming: where is the eye now, does the viewer get it yet
frame stripevery frame of a 0.5 s stretchpops, snaps, missing anticipation, boil jitter on still parts
cropfull-res detailfaces, contacts (feet on ground, props in hands), muddy glows
crop following a world pointthe same detail through a moving cameracontacts on camera moves

Count frames per read (24 = 1 s). A read that gets a few frames, or shares its frames with another read, will be missed. Budget: one sheet per shot, one strip per key motion and per transition, one crop per face that carries the story.

Rules from the kit's guide

Defaults, overridden by whatever the person asks for.

  • No text. No captions, labels, signs, or words on screens. Reactions are painted marks (!, ?, sweat, hearts, a bulb). A sign that restates the story is the classic failure.
  • Flat 2D, no projected 3D. Turns go through drawn key views (front, 3/4, side, back 3/4, back) with smears. Depth from overlap, scale and paler, bluer colour.
  • Nothing still. Idle motion per emotion, drifting cameras, swaying grass. A frozen frame reads as a bug.
  • Faces act, never snap. Mood changes go squint, swap under the squint, take, overshoot, settle.
  • One beat. A single BPM drives idles and bounces even with no music.
  • No pure black or white. Additive light for glows: pigment-mixing brushes turn yellow-over-blue green.

The ClaudeAnimationBase route

https://github.com/JohnHeibel/ClaudeAnimationBase, MIT, verified at commit 0ac8bf2 (2026-09-24). Its ANIMATION_GUIDE.md (36 KB) is the full API: read it whole before drawing. render.mjs provides the review views as flags (--sheet=, --strip=a:b, --crop=x,y,w,h, --crop-at=, --stills=, --clip, --frames --workers=N + --encode).

Recipe for a CCotw container, verified 2026-10-25 (4 CPUs, no GPU, Chromium at /opt/pw-browsers/chromium-*/chrome-linux64/chrome):

SCRATCH=/path/outside/the/checkout   # the session scratchpad
CHROME=$(ls -d /opt/pw-browsers/chromium-*/chrome-linux64/chrome | tail -1)
git clone --depth 1 https://github.com/JohnHeibel/ClaudeAnimationBase /tmp/jail/cab
# deps from the registry, outside the checkout, no install scripts; versions from package-lock.json
mkdir -p "$SCRATCH/cabdeps" && cd "$SCRATCH/cabdeps" && npm init -y >/dev/null \
  && npm install --ignore-scripts p5@2.3.3 p5.brush@2.2.3 puppeteer-core@25.11.0
cp -r "$SCRATCH/cabdeps/node_modules" /tmp/jail/cab/
sed -i '/fonts.googleapis.com/d' /tmp/jail/cab/studio.html          # only letter() uses it
sed -i "s/{ waitUntil: 'networkidle0' }/{ waitUntil: 'networkidle0', timeout: 0 }/; s/{ timeout: 60000 }/{ timeout: 0 }/" /tmp/jail/cab/render.mjs
chown -R nobody:nogroup /tmp/jail/cab
sh scripts/jail.sh -C /tmp/jail/cab -- node render.mjs --soft-gl \
  --chrome="$CHROME" \
  --sheet=0.5,2,4,6,8,10 --cols=3 --w=480 --out=out/sheet.jpg

In claude-workspace, run the kit only inside scripts/jail.sh (hub rule for code nobody here wrote). Elsewhere, drop the jail prefix.

Cost. Under SwiftShader the demo's six-frame sheet took 0.3, 0.2, 36, 47, 40 and 42 s per frame. At ~40 s, the 11 s demo (264 frames) is about 3 hours on one worker. Before a full render, either ask the scene to avoid p5.brush watercolour fill (the kit's README suggests this for machines without a GPU), split across --frames --workers=N and run it as a background job, or render on a GPU host (--gpu-angle=gl-egl|vulkan). The review loop stays cheap: sheets and strips render a handful of frames.

Failure signals

signalcausefix
connect ENETUNREACH 127.0.0.1:<port> from puppeteernetwork namespace with lo downclaude-workspace scripts/jail.sh raises lo since 711fac1; pull the hub
Navigation timeout of 30000 ms exceeded, with ERR_INTERNET_DISCONNECTEDpage waits on Google Fonts with no networkdelete the fonts link
same timeout, no failed requestp5.brush init takes ~20 s under SwiftShadertimeout: 0 on goto and waitForFunction
Failed to construct 'OffscreenCanvas': Value is not of type 'unsigned long'a NaN in a point list; the stack points at the scene, not the NaNguard degenerate geometry (Math.acos of a ratio > 1)
an outline collapses to a dot at zoom ≥ 2stroke drawn far from the origindraw long edges as inkLines no bigger than the canvas
still parts flicker in a stripshared random stream behind a moving elementboilSeed(key) before each element
five WebGL: INVALID_OPERATION ... not from the associated program warningsp5.brush, once per pageharmless; any other page error is real

Verification

Open every sheet and strip with Read and look before calling a shot done. For the final file, this must match the config's duration and fps:

ffprobe -v error -show_entries format=duration:stream=nb_frames,r_frame_rate -of default=nw=1 out/video.mp4

Report the route used, the render time per frame, and anything the review loop found that you did not fix.

converting-files

Four conversion engines are already in this container — pandoc, libreoffice (headless), ImageMagick's convert, and ffmpeg. The hard part isn't running them; it's picking the right one. pandoc mangles a real .docx into PDF; LibreOffice is the tool for that. pandoc is better than LibreOffice for md → docx. Neither touches mp4 → gif. This skill encodes that routing so the choice isn't re-derived (often wrongly) each time.

scripts/convert.py is a dispatcher over those four binaries, not a new engine. Use it for the common cases; drop to the raw engine when you need flags it doesn't pass through.

Is this the right skill?

You want to…Use instead
Change a file's format (ext A → ext B)this skill
Edit/author a Word/PPT/Excel/PDF's contentdocx / pptx / xlsx / pdf skills
Read a file already shown in contextjust read it — no conversion
Extract text/tables from a PDFpdf-reading skill
Fill or merge PDFspdf skill

Format change = here. Content change = a content skill. A docx → pdf is a format change; "fix the grammar in this docx" is not.

Engine routing (the actual content of this skill)

Source family → target family decides the engine. The dispatcher applies this; the table is here so the reasoning is auditable and so you can call the engine directly when needed.

From → ToEngineWhy
markup → markup (md, html, rst, latex, org, epub, rtf, ipynb…)pandocwhat it's built for
markup → Office (md → docx/pptx/odt)pandoccleaner than LibreOffice from markup
Office → markup (docx/odt/pptx → md/html…)pandocreads modern OOXML well
Office → PDF (docx/pptx/xlsx → pdf)LibreOfficepreserves layout; pandoc mangles it
Office ↔ Office, or legacy (.doc/.ppt/.xls) anythingLibreOfficepandoc can't read legacy binary formats at all
markup → PDF (md → pdf)pandoc (+pdflatex/xelatex)lighter than LibreOffice for plain markup
image ↔ image (png, jpg, gif, webp, tiff, heic, svg…)ImageMagickconvert in out
image → PDFImageMagick
audio ↔ audio (mp3, wav, flac, ogg, aac, m4a, opus…)ffmpeg
video ↔ video, mp4 → gif, gif → mp4ffmpeggif-from-video is an ffmpeg job, not ImageMagick

gif lives in both image and video worlds: png → gif is ImageMagick (still image), mp4 → gif is ffmpeg (animation). The dispatcher handles the split by the source family.

Usage

# Plan only — print engine + reason + exact command, run nothing. Do this first
# on an unfamiliar pair to confirm the route before committing.
python3 /mnt/skills/user/converting-files/scripts/convert.py --plan in.docx out.pdf

# Convert.
python3 /mnt/skills/user/converting-files/scripts/convert.py in.docx out.pdf
python3 /mnt/skills/user/converting-files/scripts/convert.py notes.md notes.docx
python3 /mnt/skills/user/converting-files/scripts/convert.py clip.mp4 clip.gif

# Pass engine flags through (everything after OUTPUT goes to the engine):
python3 .../convert.py photo.png photo.jpg -quality 85         # ImageMagick
python3 .../convert.py song.wav song.mp3 -b:a 192k             # ffmpeg
python3 .../convert.py paper.md paper.pdf --pdf-engine=xelatex # pandoc

# Force a specific engine if you disagree with the route:
python3 .../convert.py weird.docx weird.md --engine pandoc

Exit codes: 0 ok, 2 no route exists, 3 engine failed (stderr tail printed), 4 bad args / missing input.

Batch

The dispatcher is one file at a time by design — batch is a shell loop so each file's failure is visible:

for f in *.png; do
  python3 /mnt/skills/user/converting-files/scripts/convert.py "$f" "${f%.png}.webp" || echo "FAILED: $f"
done

Gotchas (the things that cost a re-run)

  • LibreOffice names the output itself — it writes <input-stem>.<ext> into --outdir, ignoring your chosen filename. The dispatcher renames to your OUTPUT after. If you call libreoffice raw, expect the input-stem name.
  • LibreOffice headless is single-instance. Two concurrent --headless calls collide on the user profile and one silently does nothing. Run them serially (the batch loop above is serial — fine).
  • markup → pdf needs a LaTeX engine. pdflatex/xelatex are present, but exotic Unicode wants --pdf-engine=xelatex. If LaTeX chokes on the content, route through docx first: md → docx (pandoc) then docx → pdf (LibreOffice).
  • ImageMagick here is IM6 (convert, not magick). Policy may block some formats; if convert refuses a PDF/PS op, that's the /etc/ImageMagick-6/policy.xml security policy, not a missing codec.
  • markup → jira emits heading anchors. md → jira works (both are pandoc formats), but headings come out as h1. {anchor:slug}Title — pandoc materializing the auto-generated heading ID. Suppress with --engine pandoc ... -f markdown-auto_identifiers (pass the format-with-extension yourself, since the dispatcher infers a bare markdown).
  • --plan lies about nothing but runs nothing — it can't tell you the content will survive (e.g. a pptx → md drops all layout). Plan checks the route; only a real run checks the result.

Why not VERT (or any web converter) as the skill

VERT (vert.sh) is a Svelte/WASM browser app wrapping these same engines (libvips, ffmpeg.wasm, Pandoc-wasm) for a human at a tab who wants local privacy. There's no library or CLI to import — the app is the UI. In-container we already have the native binaries with full filesystem access and no browser memory ceiling, so the WASM wrappers would be strictly slower and weaker. VERT is a good recommendation to a person; it's the wrong shape for a skill. This skill is the in-container equivalent of what VERT does in a tab.

Creating Video

Claude cannot render video itself, but it can direct a generator. This skill drives a multi-clip pipeline — script → per-scene generation → review → assembly — with Claude as the editing agent overseeing continuity and cut.

Requires ffmpeg/ffprobe and Cloudflare AI Gateway creds (/mnt/project/proxy.env).

Default model: Gemini Omni Flash (gemini-omni-flash-preview, Interactions API) — Google's own guidance as of 2026-07. It beats Veo on character consistency, reference-image control, text rendering, and conversational editing, and generates faster (~30–90 s/clip vs 1–3 min). Fall back to Veo 3.1 (scripts/veo_generate.py, kept intact) only for scene extension, first+last-frame interpolation, or legacy pipelines — Omni supports neither.

The five stages

  1. Shot list — break the story into beats, up to 10 s each, one prompt per shot.
  2. Generate — scripts/omni_generate.py, run detached.
  3. Review — parsing-video on each clip and on the assembled cut.
  4. Assemble — scripts/assemble.py, run detached.
  5. Iterate — re-review, then edit failing scenes conversationally (cheaper and more stable than regenerating from scratch).

Stage 2 — Generation (Omni via Cloudflare AI Gateway)

Auth is CF AI Gateway BYOK: proxy.env supplies CF_ACCOUNT_ID, CF_GATEWAY_ID, CF_API_TOKEN; the Google key lives inside the gateway. Both the interactions call and the files/{id}:download retrieval route through it (verified 2026-07-20).

Run detached. The Interactions API is synchronous — the HTTP call holds open for the whole generation (~30–90 s/clip). bash_tool caps at ~50 s. Launch and adaptive-wait on the DONE sentinel:

# prompts.json = {"1": "...", "2": {"text": "...", "images": ["ref0.png"]}, ...}
set -a; . /mnt/project/proxy.env; set +a
(setsid python3 scripts/omni_generate.py prompts.json --out omni/ \
    --negative "on-screen watermark" &)
# then, in a separate call:
timeout 45 sh -c 'while [ ! -f omni/DONE ]; do sleep 3; done'; cat omni/generate.log

Gotchas (all verified 2026-07-20 — the script already handles them):

  • No dry run exists. Even input:"" returns 200 and bills a full generation (~58k video output tokens). Every request costs money; never "validate" with a throwaway call.
  • No negativePrompt parameter (Veo had one; Omni 400s on unsupported params). --negative folds into the prompt as "Do not include: X."
  • Videos >4MB need delivery:"uri" → poll files/{id} to ACTIVE → download via the gateway. The script always uses uri delivery, inline base64 as fallback.
  • Output is up to 10 s 1280×720 mp4 with synchronized audio + SynthID watermark. No video extension, no first↔last-frame interpolation, no multi-video referencing — those are Veo territory.
  • The egress proxy 503s on cold start → retry with backoff.

Omni defaults to multi-shot. Left alone it invents its own cuts inside a clip, which fights a shot-list pipeline. Every per-scene prompt should say "single continuous shot" / "no scene cuts" unless the beat wants internal cuts.

Stage 3 — Review (use the parsing-video skill)

Contact-sheet each clip and the assembled cut. Per-clip sheets miss cross-clip continuity; the full-cut sheet is where character drift, prop jumps, and logic breaks show up in one read (Oskar, 2026-07-18: review the whole assembly, not just scenes). Scan every sheet against the continuity checklist:

  • Character — same face/hair/wardrobe across every shot they appear in.
  • Prop — same identity, size, color, and attachment point shot to shot.
  • Physical logic — is the world coherent (a window must be open before a bird lands on the sill)?
  • Action completeness — is the key action actually shown, not cut around?

Stage 4 — Assembly (scripts/assemble.py)

Trims each clip to its beat, crossfades, drops audio by default, optionally burns an overlay word, and holds the final frame so the ending lands.

(setsid python3 scripts/assemble.py omni/scene_1.mp4 ... omni/scene_6.mp4 \
    --out film.mp4 --tail-hold 1.3 &)

Run detached too — six trims plus a 30 s stitch exceed the bash ceiling on the single-core container. Diagnosed: abrupt ending → --tail-hold; jarring cuts between independently generated ambiences → audio dropped by default (--keep-audio to acrossfade instead).

Stage 5 — Iterate by editing, not regenerating

results.json stores each scene's interaction_id. A failed detail (wrong color, unwanted object, lighting) is a one-line conversational edit that preserves everything else:

python3 scripts/omni_generate.py --edit <interaction_id> \
    "Make the scarf red. Keep everything else the same." --out omni/scene_3_v2.mp4

Simple edit prompts work best; append "Keep everything else the same." Reserve full regeneration for scenes whose composition or action is wrong. (Editing requires store=true, the script's default.)

Prompt craft — continuity is the hard part

The failure modes below came from real crits on 2026-07-18 (Veo era). Omni narrows several of them but the disciplines still pay on the first pass.

  • Pin characters and props with reference images — the first-class fix for the drift that text locks never fully solved under Veo. Generate one canonical still per character/prop, put it in the scene's images list, and bind it in the prompt: the woman <IMAGE_REF_0> is holding <IMAGE_REF_1> (refs start at 0; <FIRST_FRAME> pins a starting frame instead).
  • Still lock a character sheet in text. One verbatim appearance/wardrobe block, reused in every shot with that character — references constrain, text directs.
  • Lock the setting. One room/location description across co-located scenes.
  • Name the actor in every shot ("the same woman's hands") — never leave it to the model to infer who is on screen.
  • Spell out scene logic — physical coherence, the correct order of beats, and show the payoff action rather than cutting away from it.
  • Timing is promptable. [0-3s] ... [3-6s] ... timecode syntax or natural language ("after 3 seconds, ...") controls beats inside a clip.
  • On-screen text now renders reliably — spell out exactly what any visible text should say (signs, labels, title cards). Burning text in assembly (assemble.py --overlay-last) remains an option for cross-clip title cards.
  • Audio is promptable. Describe the track ("no dialogue", "calm background music") or Omni invents one per clip — which still won't match across clips; assembly drops audio by default for that reason.

Scope

  • Transcode / trim / merge / GIF / subtitles → processing-video.
  • Reading, summarizing, or QA of existing video content → parsing-video (also the review step here).
  • Scene extension / first+last-frame control → the retained Veo path (scripts/veo_generate.py --list to enumerate models; strings drift).

Detecting TIPS Zones

Zero-shot zone detection: text prompts → patch-grid cosine heatmaps → bboxes. Companion to svg-portrait-mode — replaces manual focus_targets / focus_edges annotation with a TIPSv2 B/14 forward pass.

Quick Start

from tips_zones import detect_zones
from portrait_mode import portrait_mode

focus_targets, focus_edges = detect_zones(
    "photo.jpg",
    targets=["dog face"],
    edges=["dog paws", "dog ears", "dog body"],
    distractors=["wooden floor", "carpet rug", "shoes", "wall"],
    ckpt_dir="/path/to/tips/checkpoints",
    tips_root="/path/to/tips",
)

svg, stats = portrait_mode(
    "photo.jpg",
    focus_targets=focus_targets,
    focus_edges=focus_edges,
    style_transforms={"background": "desaturate:0.7"},
)

Amortise model load across multiple images:

from tips_zones import load_models, detect_zones

models = load_models(ckpt_dir, tips_root, device="cpu")
for img in images:
    ft, fe = detect_zones(img, targets=[...], edges=[...], distractors=[...],
                          ckpt_dir=ckpt_dir, tips_root=tips_root, models=models)
    ...

How It Works

image → B/14 vision encoder (MaskCLIP values trick on last block)
     → (32×32 patch grid at 448, or 64×64 at 896) × 768-d patch features
text labels → prompt ensemble (9 TCL templates) → B/14 text encoder
     → per-label mean feature → L2-normalise
per-label heatmap = cos(patch feature, label feature)  # raw, no softmax
bbox = top-k% patches → largest connected component → scaled + padded to image coords

Why no softmax over labels

Naïve softmax assumes labels are mutually exclusive. dog face, dog ears, and dog body are all true of the same pixels, so softmax collapses to near-uniform and every heatmap covers the whole subject. Raw cosines + per-label top-k threshold works much better — at the cost of requiring distractor labels to anchor the relative scale. Always pass some distractors (floor, wall, props — whatever is in the scene but not the subject).

Parameters

detect_zones(
    image,                     # path | PIL Image
    targets,                   # ["main subject label", ...]
    edges=(),                  # ["sub-region label", ...]
    distractors=(),            # scene elements to anchor against — pass these!
    *,
    ckpt_dir,                  # has tips_v2_oss_b14_{vision,text}.npz + tokenizer.model
    tips_root,                 # local clone of google-deepmind/tips
    input_size=448,            # 448 → 32×32 grid, 896 → 64×64 (~12× slower on CPU)
    target_top_frac=0.04,      # fraction of patches kept per target label
    edge_top_frac=0.06,        # fraction of patches kept per edge label
    pad_frac=0.02,             # bbox padding as fraction of image dim
    device="cpu",
    models=None,               # optional pre-loaded (img_model, text_model, tokenizer)
)

Returns (focus_targets, focus_edges) — both lists of {'bbox': (x1,y1,x2,y2), 'label': str}.

Performance (CPU, 16 cores)

StepTime
load_models (warm)~3.5s
load_models (cold, over 9p)~50s
Text encoding (9 templates × N labels)~0.1s
Vision forward @ 4480.3–0.6s
Vision forward @ 896~6–7s

Inference is negligible next to portrait_mode() on large images.

Capability Notes

Subject / background split: strong. B/14 separates subject from scene reliably — typical split ~30/70 subject:background on single-subject photos.

Sub-part discrimination: weak at B/14 + 448. "dog face" vs "dog paws" vs "dog ears" tend to fire on the same region. The 32×32 patch grid is not the bottleneck (64×64 at 896 barely helps); B/14's patch features just don't encode fine sub-part semantics strongly. If you need per-part zones:

  1. Sharpen prompts — "close-up of dog's furry face" > "dog face" (try first)
  2. L/14 or SO/14 model (richer features, larger download)
  3. Sliding-window inference (tile crops, stitch heatmaps)

For coarse target/edge zoning (the portrait_mode use case), B/14 at 448 is enough.

Requirements

Python deps:

pip install torch torchvision tensorflow tensorflow-text scipy pillow numpy --break-system-packages -q

Upstream TIPS repo (for the tips.pytorch image/text encoder modules):

git clone https://github.com/google-deepmind/tips /path/to/tips

B/14 checkpoints (~500MB total) go in a directory passed as ckpt_dir:

  • tips_v2_oss_b14_vision.npz
  • tips_v2_oss_b14_text.npz
  • tokenizer.model

Download links are in the TIPS repo README.

Prompt Engineering Tips

  • Always include distractors. Without them, top-k thresholding has no relative scale. 3–7 distractors covering scene elements (floor, wall, background objects) is the sweet spot.
  • Use concrete nouns over abstract ones. "carpet rug" > "textured floor".
  • Top_frac tuning. If a target bbox is too small, raise target_top_frac (0.04 → 0.08). Too big / bleeds into scene: lower it.
  • Pad modestly. pad_frac=0.02 works for most photos; raise to 0.05 for subjects near frame edges.

EXIF Caveat

portrait_mode (via OpenCV) honours EXIF rotation. PIL (this skill's preprocessing) does not. For correctly-oriented source images they agree; for EXIF-rotated phone photos the detected bboxes will be in the raw pixel orientation. Either:

  • Re-save the source with EXIF baked in: Image.open(p).rotate(0, expand=True).save(p)
  • Or call ImageOps.exif_transpose(pil) before passing to detect_zones.

drawing-video

Footage in, narrated comic strip out. Five stages: probe → ground → narrate → draw → QA/compose. The hard part is not the drawing; it is not lying about what the footage shows. One stage is a gate.

Requires: processing-video (ffmpeg, present), invoking-gemini (image + video models), proxy.env (CF gateway creds — auto-read by the gemini client).


Stage 0 — Probe & extract frames

ffprobe -v quiet -show_entries format=duration -of csv=p=0 in.mov
mkdir -p frames
ffmpeg -v error -i in.mov -vf "fps=1,scale=640:-1" frames/f_%03d.png   # 1/sec, downscaled

Keep full-resolution stills too — you will feed them to the image model as reference. For a subject hidden in clutter, pull a few full-res frames at specific timestamps: ffmpeg -v error -ss 11 -i in.mov -frames:v 1 hr.png.


Stage 1 — GROUND THE STORYLINE (the gate)

Never narrate a scene whose subjects and action you have not confirmed from the footage. Thumbnail-sized frames read poorly: one pass built an entire "indoor cyclist in a garage" storyline for a video that was actually a leashed dog watching a deer — no bicycle in any frame, the leash visible in four — then fed the fiction to the image model, which dutifully drew it. A wrong storyline launders a hallucination into finished art. The absence of a clear read is not license to invent a vivid one.

Two ways to ground. Prefer Mode B — it is the more reliable and reads the whole clip, not sampled stills.

Mode B (recommended): native video understanding via gemini-4-hydrogen

gemini-4-hydrogen parses video directly (verified 2026-11-20 on a 4 s test clip). Compress first, send as inline data.

ffmpeg -v error -i in.mov -vf "scale=480:-2,fps=8" -c:v libx264 -crf 30 -preset veryfast -an small.mp4
import sys, base64; sys.path.append('/mnt/skills/user/invoking-gemini/scripts')
from gemini_client import get_cf_credentials, _cf_request
from pathlib import Path
c = get_cf_credentials()
b = base64.b64encode(Path("small.mp4").read_bytes()).decode()
prompt = ("Watch this video and describe the actual storyline in 3-4 sentences: which "
          "subjects appear, what happens, the arc. Be literal — only what is visibly present.")
contents = [{"parts": [{"inlineData": {"mimeType": "video/mp4", "data": b}}, {"text": prompt}]}]
resp = _cf_request("gemini-4-hydrogen", contents,
                   {"temperature": 0.3, "thinkingConfig": {"thinkingLevel": "low"},
                    "maxOutputTokens": 600}, c)
print("".join(p.get("text","") for p in resp["candidates"][0]["content"]["parts"]))

Inline works for small files (a 22s 4K clip compressed to 2.3 MB worked with the earlier gemini-3.5-flash; Hydrogen was checked on a 4 s clip). For long clips, sample frames (below) or use the File API. gemini-2.5-flash-image is not a parser — this is gemini-4-hydrogen (text/multimodal). _cf_request sends the config exactly as written, so thinkingLevel must be low or higher: Gemini 4 returns HTTP 400 for minimal. (invoke_gemini would downgrade minimal to low for you; this lower-level call does not.)

Mode A: read the frames yourself

Build a contact sheet (PIL grid of frames/*.png), then view enough full-resolution frames to name every subject and the action out loud before writing a word of narration. If you cannot tell what a shape is, extract a higher-res crop at that timestamp and look again. Do not proceed on a guess.

Either way, end Stage 1 with one plain paragraph: subjects, setting, arc. If the user has already told you the storyline, that paragraph is authority — use it verbatim and skip re-derivation.


Stage 2 — Beats & narration

Let the video decide the panel count. Break the clip into its natural beats — the distinct moments the story actually has — and make one panel per beat. Do NOT default to four or to a 2×2 grid. A 12-second perch-and-glide is three beats (sentinel → launch → glide); a busier clip may be five or six. Fewer, larger panels for a slow arc; more, tighter panels for rapid action. A hardcoded 2×2 four-panel default is wrong — layout must fit the footage.

Write one short caption per beat, in the requested voice (David Attenborough hushed-drama is the common ask; dial the stakes above what the mundane footage deserves). Captions must be short enough to render legibly in a box — one or two sentences.

Keep a character/setting bible in one line: the exact look of each recurring subject (breed/colour, leash colour, the woods, the light). You will paste it into every panel prompt so the panels stay consistent.


Stage 3 — Design the page, then draw the panels

Division of labour (the default): you are weak at drawing but decent at layout, so you design the page and outsource only the drawing. Design a bespoke layout that fits the story — panel count from Stage 2, panel sizes and arrangement chosen to serve the arc (a splash panel for the climax; a tall panel for a vertical action; a downward stack for a descent; growing panels to build drama). Assign each panel a target aspect ratio. Then generate each panel with Gemini at that aspect ratio, and composite the drawn panels into the layout you designed (PIL). This beats both the rigid uniform grid and the one-shot page.

  • Request the aspect in the prompt ("Draw ONE comic panel, 16:7 wide landscape" / "4:3 large" / "3:4 tall"). Gemini 3 honours it closely (measured 2026-07-18: asked 16:7/16:6/4:3, got 2.33/2.72/1.34).
  • Composite with cover-fit (resize to fill the box, centre-crop) so a slightly-off aspect still fills its box cleanly. Design boxes at each panel's target aspect to minimise cropping.

Two alternatives, by request:

  • One-shot full page — Gemini 3 can render an entire multi-panel comic in ONE image call; describe the whole layout (panel count, arrangement, gutters, per-panel scene + caption, title) in the prompt. Use when the user wants Gemini's freeform layout and you don't need pixel control. Trade-off: no per-panel regeneration, less border control.
  • Uniform grid — the simplest fallback; a fixed 2×N. Use only when the story genuinely has equal-weight beats.

Model for all paths: gemini-3-pro-image (GA; was gemini-3-pro-image-preview). There is no Gemini 3.5 image model — 3.5 is text-only; the current image line is gemini-3-pro-image and gemini-3.1-flash-image. Do not use gemini-2.5-flash-image. (gemini-3.5-flash is the video parser in Stage 1, not an image model.)

Four rules for per-panel drawing, each guarding against a defect observed 2026-07-18:

  1. Anchor every panel to real stills. Pass TWO images per panel: a single shared anchor still (canonical setting + subject) and the panel's own still. Instruct: "First reference = canonical setting+subject; second = this panel's framing; redraw as a comic panel." This is what keeps the drawn scene faithful instead of inventing.
  2. State continuity explicitly. "All N panels are ONE comic: identical setting, identical design ()." Without it the woods and the animal drift panel to panel.
  3. Constrain creatures with negatives. The image model turned an adult doe into a spotted fawn. Say "adult doe, uniform tan, NO white spots, NOT a fawn" — spell out what it is not.
  4. Lock the caption box. "Place ONE caption box at the top containing EXACTLY this text and NOTHING else — no extra words, no stray letters, no style notes: ''. Do not write any other text anywhere." Stray style words leaked into a caption box until this was added.

Run panels detached and adaptive-wait — bash_tool times out at ~50s and image-pro calls are slow. Launch all panels in parallel threads in one detached script writing a DONE sentinel; poll with timeout 45 sh -c 'while [ ! -f DONE ]; do sleep 3; done'. Build the request with _cf_request(model, contents, {"temperature":0.65,"responseModalities":["IMAGE","TEXT"]}, creds); extract the image from the part whose key is inlineData/inline_data.


Stage 4 — QA, then compose

QA before shipping — do not present unseen panels. If the local image viewer is available, look at every panel. If it is not (it can go dark mid-session), QA through Gemini vision instead:

from gemini_client import invoke_gemini
invoke_gemini(prompt=("QA this comic panel. JSON only: setting, subject, deer, "
   "deer_spotted_fawn (true only if white baby spots), caption (verbatim), palette."),
   model="flash", image_path="panel_1.png", max_output_tokens=500, thinking_level="low")  # flash = gemini-4-hydrogen

Check per panel: subjects match the grounded storyline, setting/subject consistent across panels, captions verbatim with no stray text and no doubled words (Gemini has repeated a word, e.g. "and and"), creatures correct (adult vs fawn). Regenerate only the panels that fail — single-panel regen is cheap. For a one-shot page, QA the whole page the same way. (Note: the vision QA reports your own title/subtitle as "stray text" — that is expected, not a defect.)

Compose with PIL into the layout you designed in Stage 3. Strip any baked border first — image-model panels arrive with inconsistent baked frames (one dark, others white), which reads as a mismatched-border defect: crop a uniform ~8–10px inset off every panel, then draw one identical border on all. Cover-fit each panel into its designed box, add a title band, save to /mnt/user-data/outputs/, then present_files. (The one-shot page needs none of this — it arrives pre-composed; just QA and present.)


Pitfall summary (observed 2026-07-18)

  • Confabulated the storyline from thumbnails → Stage 1 gate; prefer Mode B.
  • Wrong prompt made the image model draw the fiction → anchor to real stills.
  • Woods/subject drifted between panels → shared anchor + explicit continuity.
  • Doe rendered as a spotted fawn → negative constraints on creatures.
  • Style words leaked into a caption box → lock the caption text.
  • Panel-1 border differed → strip baked borders, apply one uniform frame.
  • Guessed "Gemini 3.5 image" → no such model; use gemini-3-pro-image, bin 2.5-flash-image.
  • Local image viewer unavailable → QA via Gemini vision, never ship unseen.
  • Hardcoded a 2×2 four-panel layout → derive panel count from the beats; design the layout to fit the story.
  • Gemini doubled a word in a caption ("and and") → QA captions for doubled words, not just stray text.

Drawing with pycairo

Cairo paints paths with solid colours, gradients, mesh gradients and images, and composites them with 20-odd operators. It writes PNG, PDF, SVG and PostScript. It does not shape text, filter pixels, combine paths, outline strokes, or do depth. scripts/cairokit.py fills each gap; scripts/selftest.py exercises all of it. Both were verified on pycairo 1.29.1 / cairo 1.18.0 / Python 3.11.

Setup

pip install --break-system-packages pycairo numpy uharfbuzz fonttools skia-pathops
python3 /mnt/skills/user/drawing-with-pycairo/scripts/selftest.py /tmp/cairo-selftest

The self-test prints the cairo version, whether Pango is usable, per-panel timings, and the path of a 2×2 contact sheet. Read that sheet: panel A shows joined Arabic in green next to broken toy-API Arabic in red, B shows four boolean results and an outlined wave, C shows glow and a soft shadow, D shows a smooth-shaded torus. Then:

import sys; sys.path.insert(0, "/mnt/skills/user/drawing-with-pycairo/scripts")
from cairokit import *

In the environment where this was built, pycairo was not preinstalled. Pango was also absent: the system PyGObject was compiled for Python 3.12 while python3 was 3.11, and there were no Pango typelibs. pango() reports which case you are in.

Render, then look

Every render ends with a Read of the PNG before the next edit or before reporting. Exit codes and timings don't show visual bugs. In the session that produced this skill, looking caught five defects that ran without error:

  • a painter sort drawn near-to-far
  • inward normals combined with a light pointing away from the camera, which left a black knot
  • a scene scaled in absolute pixels, cropped once the frame size changed
  • glyphs crowding on a squashed spiral, because arc length was measured on the unsquashed curve
  • a stray hook where a polyline's first move_to wasn't its first sample

For several outputs, contact_sheet() tiles them so one Read covers all of them. For animation, extract two or three frames with ffmpeg -ss T -i out.mp4 -frames:v 1 f.png and read those.

What cairokit provides

needcall
surface + context with backgroundnew_image(w, h, bg)
ASCII caption on a boxlabel(c, text, x, y)
pixels as numpy (BGRA, premultiplied, live view)pixels(s), from_array(a), from_rgb_float(rgb)
blur, drop shadow, neon glowgauss_blur(a, sigma), blurred(s, sigma), glow(sharp)
shaped text for any script, as outlinesShaper(font_path).text_path(c, text, x, y, size)
text along a curveShaper.text_on_curve(c, text, polyline_walker(points), size)
union / intersection / difference / xorboolean(c, build_a, build_b, op)
outline of a strokestroke_to_path(c, width, cap, join)
perspective, depth sort, smooth shadingproject(P), painter_order(z), gouraud_mesh(quads, colors, order)
frames to MP4, many PNGs to one sheetframes_to_mp4(pattern, out), contact_sheet(files, out)
Pango availabilitypango() → (Pango, PangoCairo) or None

Find a font for a script with fc-match -f '%{file}' 'DejaVu Sans:lang=ar' (or fc-list :lang=hi family file). DejaVu Sans covers Arabic and Hebrew; FreeSans covers Devanagari.

Measured behaviour (800×600 unless noted)

thingresult
alpha circles, antialiased~58k/s (no-AA: ~139k/s)
one 200k-segment polyline stroke~1.0 s
ImageSurface max side32767; 32768 raises cairo.Error
glow(), two blur radii~240 ms
10k quads, flat fill_preserve+stroke~160 ms
10k quads, one gouraud_mesh()~125 ms (faster, and smooth)
animation frame, 10k-quad mesh at 640×480~130 ms
mesh gradient → PDFkept as vector (ShadingType 6/7)
mesh gradient → SVGrasterised to an embedded <image> (SVG has no mesh gradient)
10k flat quads → SVG vs PDF3.5 MB vs 0.36 MB

Cairo renders on one CPU core. For large batches or long animations, spread frames across processes with multiprocessing rather than threads.

Behaviour that trips people

Text. show_text() / text_path() map codepoints to glyphs one by one, left to right, with no joining, reordering, ligatures or kerning. They are fine for ASCII labels. For anything else, use Pango if pango() returns modules, otherwise Shaper. Shaper does no line wrapping, no bidi across mixed-direction runs, and no font fallback: one font per call, and you split runs yourself.

Paths and the CTM. Path coordinates are transformed when they are appended, and the path is not part of the saved state. save(); scale(); <append>; restore(); fill() fills the transformed shape without distorting anything later. Line width, however, is transformed at stroke() time. Stroking inside a non-uniform scale() gives calligraphic, uneven lines, so restore first.

Pixels. ARGB32 memory is B,G,R,A per pixel, alpha-premultiplied, with rows get_stride() bytes apart. Call flush() before reading and mark_dirty() after writing. pixels() and from_array() handle this. Blurring premultiplied data is correct; un-premultiplying first gives dark fringes.

Seams. Two antialiased fills sharing an edge leave a faint line of background between them (conflation). Fixes: fill_preserve() plus a 0.5–0.7 px stroke in the same colour, or a single MeshPattern for the whole surface.

MeshPattern. Neighbouring Coons patches must use the identical curve for their shared edge. Generate each edge's control points in a canonical direction and reverse them for the other patch, or seams appear. Patches paint in insertion order, which is what makes gouraud_mesh() a painter's-algorithm renderer.

3D. There is no depth buffer. Sorting by mean depth fails when polygons interpenetrate or are long and thin across depth; subdividing helps. Use two-sided normals (flip toward the camera) unless the mesh orientation is known. Screen y points down, so a light "from the top left, toward the camera" is (-x, -y, -z) with camera-space +z pointing away.

skia-pathops. Round caps and joins produce conic segments, and simplify() rejects them; stroke_to_path() converts them to quadratics first. Any other pathops output passed to append_pathops() needs convertConicsToQuads() too.

Output. SVGSurface / PDFSurface must be finish()ed, or the file is truncated. RecordingSurface records once and replays at any scale or to any backend. Operators such as ADD, SCREEN, MULTIPLY and DIFFERENCE are on the context (set_operator). Float formats FORMAT_RGB96F / FORMAT_RGBA128F exist from cairo 1.18 for HDR-ish accumulation.

Route elsewhere

taskuse
data chartscharting (matplotlib/seaborn)
photo edits, resizing, format conversionprocessing-images
raster image to SVGimage-to-svg
illustrations or portraits from a promptinvoking-gemini
real 3D (occlusion, textures, >50k polygons)a rasteriser or ray tracer; cairo's painter approach stops being practical
polygon analysis (areas, buffers, spatial predicates)shapely; use skia-pathops when the result goes back into cairo as curves

Image to SVG Reproduction

Convert raster images into faithful SVG reproductions using data-driven color quantization and contour extraction. Never hand-draw shapes from visual interpretation — always extract geometry from the actual pixel data.

Core Principle

Trust the data, not your imagination. Claude's visual interpretation of images is unreliable for precise color matching, shape positioning, and spatial relationships. Every shape, color, and position must come from computational analysis of the source pixels.

Quick Start

pip install opencv-python-headless scikit-image scipy scikit-learn --break-system-packages -q
apt-get install -y librsvg2-bin -qq
import sys
sys.path.insert(0, '/mnt/skills/user/image-to-svg/scripts')
from pipeline import image_to_svg

svg, flow = image_to_svg("source.jpg", mode="painting")

with open("output.svg", "w") as f:
    f.write(svg)

flow.summary()  # timing + status per step

Mode Selection

Look at the image and ask: "Does this have smooth gradients or hard edges?" Gradients → higher K. Hard edges → lower K.

ModeKBest forDark shape gating
"graphic"28Logos, icons, Kandinsky, flat designLoose (keeps thin lines)
"illustration"40Comics, editorial, digital artModerate
"painting"56Renaissance, Impressionist, watercolorStandard
"photo"64Portraits, landscapes, still lifeStandard (prevents woodcut artifacts)

Default is "painting". When uncertain, start there.

Tradeoffs: K=64 produces ~2300 shapes (~1.2MB SVG) vs K=28's ~1000 shapes (~550KB), measured before v1.9.0's compact output, which is about 2.6x smaller (see Output Format). Processing time roughly doubles with K. The quality gain in tonal gradation is substantial for photos but wasted on graphic art.

All mode defaults (K, dark_lum, compactness_min, etc.) can be overridden via **kwargs:

svg, flow = image_to_svg("source.jpg", mode="graphic", K=12, min_area=20)

Compositional Pipeline (Line Art)

For images dominated by lines, strokes, and geometric shapes (Kandinsky, architectural drawings, technical illustrations, comic ink work), the standard fill-only pipeline produces jagged filled polygons instead of clean strokes. The compositional pipeline solves this with two passes:

Pass 1 — Line Extraction: Isolate thin features via morphological erosion → skeletonize to 1px centerlines → Hough line detection → merge collinear fragments → measure stroke width → sample color. Emits SVG <line> elements with stroke-width.

Pass 2 — Fill Extraction: Suppress line regions from image (replace with local background estimate via median blur) → run standard K-means quantization on the cleaned image → contour extraction → <path> fills.

Composition: Fills render behind strokes in layered <g> groups.

# Auto-detect: classifies input and routes automatically
svg, flow = image_to_svg("kandinsky.jpg", mode="graphic")

# Force compositional pipeline
svg, flow = image_to_svg("technical_drawing.png", mode="graphic", pipeline="compositional")

# Force fill-only (previous default behavior)
svg, flow = image_to_svg("photo.jpg", mode="painting", pipeline="fill")

Pipeline selection (pipeline parameter):

ValueBehavior
"auto" (default)Classify input via edge density + luminance bimodality + Hough line count. Route to compositional for graphic art, fill-only for photos.
"fill"Force fill-only pipeline. Use for photos, paintings, or when compositional produces unwanted results.
"compositional"Force two-pass pipeline. Use for line art, technical drawings, or ink work where you know lines are present.

Auto-classification heuristics: An image is classified as graphic when it has high edge density (>5% edge pixels) combined with bimodal luminance distribution (>0.35 bimodality coefficient), or high straight-line density (>3 Hough lines per 10k pixels).

SVG output structure (compositional):

<svg ...>
  <rect ... />        <!-- background -->
  <g id="fills" stroke-width="1" stroke-linejoin="round">  <!-- filled regions (painter's algorithm) -->
    <path class="c3" d="m12 40l5-2..z"/>
  </g>
  <g id="strokes">    <!-- line strokes (on top) -->
    <line x1="..." y1="..." x2="..." y2="..." stroke="#000" stroke-width="2.5" stroke-linecap="round"/>
  </g>
</svg>

Stroke width control: Measured perpendicular to each detected line, then scaled by 0.65x and capped at 4.5 SVG units. This prevents thick features from rendering as bloated strokes while keeping thin lines crisp.

Current limitation — straight lines only: Hough transform detects straight segments. Curved strokes (arcs, spirals) are not yet extracted as strokes — they fall through to the fill pass. Future work: cv2.fitEllipse or spline fitting on skeleton branches.

Palette Remapping (Warhol Effects)

Separate structure from color: K-means finds regions, palette remapping assigns bold colors. This produces screen-print / pop art effects.

# Named preset
svg, flow = image_to_svg("photo.jpg", mode="graphic", K=4, palette="pop")

# Custom hex list (darkest → lightest mapping order)
svg, flow = image_to_svg("photo.jpg", mode="graphic", K=8,
    palette=["#000", "#dc143c", "#ff69b4", "#ffd700", "#32cd32", "#00bfff", "#ff8c00", "#f5f5f5"])

# Override background separately
svg, flow = image_to_svg("photo.jpg", mode="graphic", K=4, palette="ocean", bg_color="#000000")

Built-in presets: bw, mono3, mono4, pop, pop2, neon, warhol4, warhol6, warhol8, sunset, ocean

How it works: Unique shape colors are sorted by luminance. Palette entries are mapped proportionally — palette[0] replaces the darkest cluster, palette[-1] replaces the lightest. Background defaults to the lightest palette entry unless bg_color is set. Palette length doesn't need to match K exactly; colors are binned proportionally.

Portraits: Use K=16-24 even with bold palettes. Facial features (glasses, beard, brow) need tonal range that low K eliminates. A good rule of thumb: palette length ≈ K/3 for clean luminance binning. At K=8 with a 4-color palette, a face becomes an undifferentiated blob.

Contrast preprocessing warning: External contrast boosting (contrast-stretch, sigmoidal-contrast) can confuse background detection. The pipeline's edge-contact heuristic assumes untouched luminance distributions — aggressive tone-mapping pushes subject tones into background-adjacent bins, causing misclassification (e.g., dark jacket regions classified as background and mapped to the lightest palette color). If you see subject regions tearing to the background color, try without preprocessing first. The pipeline's own bilateral blur + optional kuwahara/oilpaint handles tonal separation.

Background Detection Override (bg_clusters)

Control which clusters are treated as background:

# Auto-detect (default) — edge-contact heuristic
svg, flow = image_to_svg("photo.jpg", mode="illustration", K=20, palette="warhol6")

# Disable — no clusters removed, no background rect color override
svg, flow = image_to_svg("photo.jpg", mode="illustration", K=20, palette="warhol6", bg_clusters=0)

# Force specific cluster indices (from quantize step's sorted_clusters output)
svg, flow = image_to_svg("photo.jpg", mode="illustration", K=20, palette="warhol6", bg_clusters=[2, 5])

Use bg_clusters=0 when palette remapping already controls all colors explicitly and background detection is getting in the way. Use bg_clusters=[list] when you know which clusters are background but the heuristic misidentifies them.

Portrait Pop-Art Recipe (Warhol Style)

# Key: enough K for facial features, palette length ~K/3, modest smoothing
# Do NOT apply contrast preprocessing — it breaks background detection.
results = image_to_svg_batch("portrait.jpg", [
    {"name": "hot",   "mode": "illustration", "K": 20, "smooth": "kuwahara:6",
     "palette": ["#000", "#D4145A", "#FF6B9D", "#FF85C0", "#FFD700", "#FFEF82", "#FFF8DC"]},
    {"name": "cool",  "mode": "illustration", "K": 20, "smooth": "kuwahara:6",
     "palette": ["#0D0035", "#4A00E0", "#7B68EE", "#00D4FF", "#7FFFD4", "#B0FFE0", "#E0FFFF"]},
    {"name": "earth", "mode": "illustration", "K": 20, "smooth": "kuwahara:6",
     "palette": ["#1a0a00", "#8B4513", "#CD853F", "#DEB887", "#F5DEB3", "#FAEBD7", "#FFF8DC"]},
    {"name": "neon",  "mode": "illustration", "K": 20, "smooth": "kuwahara:6",
     "palette": ["#0d0d0d", "#ff00ff", "#00ff00", "#ffff00", "#00ffff", "#ff69b4", "#f5f5f5"]},
], svg_width=700)

Why this works: K=20 preserves enough tonal clusters for facial structure (glasses, beard, brow ridge). 7-color palettes give ~K/3 luminance bins — enough variation to separate features without muddying. kuwahara:6 smooths texture without dissolving edges (:12 erases glasses). Raw source → pipeline smoothing only; no external contrast manipulation.

ImageMagick Preprocessing (smooth)

Reduce shape count and SVG file size by 20-45% using ImageMagick edge-preserving filters before quantization. Requires ImageMagick on PATH (pre-installed on Claude.ai containers).

# Oilpaint: bold, painterly smoothing (default strength=8)
svg, flow = image_to_svg("photo.jpg", mode="photo", smooth="oilpaint")

# Stronger smoothing = fewer shapes, more stylized
svg, flow = image_to_svg("photo.jpg", mode="illustration", K=32, smooth="oilpaint:12")

# Kuwahara: subtler, preserves more structure (default strength=5)
svg, flow = image_to_svg("photo.jpg", mode="painting", smooth="kuwahara:7")

# Works with batch API too
results = image_to_svg_batch("photo.jpg", [
    {"name": "raw",      "mode": "photo"},
    {"name": "smooth",   "mode": "photo", "smooth": "oilpaint"},
    {"name": "stylized", "mode": "illustration", "K": 32, "smooth": "oilpaint:12", "palette": "pop"},
])

Available filters: oilpaint (ImageMagick -paint), kuwahara (ImageMagick -kuwahara). Append :N for custom strength.

How it works: The IM filter runs before the pipeline's bilateral+Gaussian blur. Both are edge-preserving smoothers at different scales — IM handles coarse texture, bilateral handles fine detail. The result is cleaner K-means regions with fewer fragmented shapes.

Measured impact (1206×1597 photo, K=32):

smoothShapesSVG sizeReduction
none33811868KB—
oilpaint (8)23851329KB-29%
oilpaint:1218421065KB-43%
kuwahara (5)27191453KB-22%
kuwahara:720001152KB-38%

Pipeline Architecture

Uses the flowing DAG runner. Steps with independent inputs run in parallel.

Fill-only pipeline (pipeline="fill")

preprocess → quantize → ┬─ detect_background ─┬─ extract_contours → assemble_svg
                        └─ edge_map           ─┘

Compositional pipeline (pipeline="compositional")

classify_input ──→ extract_lines ──→ suppress_line_regions ──→ [fill pipeline on cleaned image]
                        │                                              │
                        └──────────── lines ───────────────────→ assemble_compositional ←── fills

Steps (fill-only):

  1. preprocess — Bilateral + Gaussian blur (edge-preserving texture removal)
  2. quantize — K-means color quantization at chosen K
  3. detect_background — Identifies background clusters by edge contact (parallel with edge_map)
  4. edge_map — Sobel edge detection via cv2.Sobel (parallel with detect_background)
  5. extract_contours — Per-cluster contour extraction with dark territory awareness and woodcut prevention (d=1 dilation; stroke handles gaps)
  6. assemble_svg — Z-ordered painter's algorithm assembly with stroke=fill gap coverage

Additional steps (compositional):

  1. classify_input — Edge density + bimodality + Hough line count analysis
  2. extract_lines — Morphological thin-feature isolation → skeletonize → Hough → merge collinear → measure stroke width → sample color
  3. suppress_line_regions — Replace line pixels with median-blur background estimate
  4. assemble_compositional — Layer fills behind strokes in grouped SVG

Resume on failure

svg, flow = image_to_svg("source.jpg", mode="photo")
# If extract_contours failed:
flow.override(extract_contours, corrected_shapes)
flow.resume()  # quantize, detect_background, edge_map stay cached

Batch API

Generate multiple variants from one image, sharing computation across runs with the same K:

from pipeline import image_to_svg_batch

results = image_to_svg_batch("photo.jpg", [
    {"name": "photo",   "mode": "photo"},
    {"name": "warhol",  "mode": "graphic", "K": 12, "palette": "warhol4"},
    {"name": "neon",    "mode": "graphic", "K": 12, "palette": "neon"},
    {"name": "sunset",  "mode": "graphic", "K": 12, "palette": "sunset"},
    {"name": "bw",      "mode": "graphic", "K": 16, "palette": "bw"},
], svg_width=1400)

for name, svg in results.items():
    with open(f"{name}.svg", "w") as f:
        f.write(svg)

Variants sharing the same K run the pipeline (preprocess → quantize → edge_map → extract_contours) once, then fan out at assembly for palette remapping. This guarantees structural identity across palette variants (same shapes, same paths) and saves ~20-60s per shared K group.

Verification still applies in batch mode. The turnkey feel of batch processing makes it easy to skip the side-by-side comparison — don't. Render at least one variant per K group and verify before delivering. Background detection failures and palette mapping issues are invisible without rendering.

Verification Protocol

After EVERY run, render and visually compare side-by-side. This is non-negotiable.

import subprocess
from PIL import Image

subprocess.run(['rsvg-convert', '-w', '1400', 'output.svg', '-o', 'output.png'])

orig = Image.open('source.jpg')
rendered = Image.open('output.png')
target_h = 800
orig_r = orig.resize((int(orig.width * target_h / orig.height), target_h))
rend_r = rendered.resize((int(rendered.width * target_h / rendered.height), target_h))
gap = 20
comp = Image.new('RGB', (orig_r.width + rend_r.width + gap, target_h), (255,255,255))
comp.paste(orig_r, (0, 0))
comp.paste(rend_r, (orig_r.width + gap, 0))
comp.save('comparison.png')
# LOOK AT comparison.png BEFORE claiming success

Manual Post-Processing

Handling Subtle Color Differences

When two regions have similar luminance but different hue/saturation, K-means in RGB space merges them. Use HSV multispectral analysis:

hsv = cv2.cvtColor(rgb, cv2.COLOR_RGB2HSV)
h_ch, s_ch, v_ch = hsv[:,:,0], hsv[:,:,1], hsv[:,:,2]

# Separate gray (low saturation) from red (high saturation) at similar brightness
red_mask = ((h_ch < 12) | (h_ch > 168)) & (s_ch > 120) & (v_ch > 80)
gray_mask = (s_ch < 80) & (v_ch > 40) & (v_ch < 120) & spatial_constraint

Saturation is the key discriminator for colors that look similar in grayscale but are visually distinct.

Positioning Overlays

When adding shapes not captured by quantization, derive coordinates from the SVG render, not the source image. The extraction pipeline shifts positions due to contour simplification.

# WRONG: extract from source, insert into SVG (coordinate mismatch)
# RIGHT: render SVG → detect gap in render → create shape in render coords → insert
svg_render = cv2.imread('rendered_svg.png')

Output Format

Since v1.9.0 fills are written compactly. On Starry Night the painting output went from 3.64 MB to 1.35 MB (825 KB → 293 KB gzipped), rendering identically in Chromium and librsvg:

<svg xmlns="http://www.w3.org/2000/svg" id="s1a2b3c" viewBox="0 0 1000 625">
  <style>#s1a2b3c .c0{fill:#3e85bb;stroke:#3e85bb}#s1a2b3c .c1{...}</style>
  <rect width="1000" height="625" fill="#16151e"/>
  <g stroke-width="1" stroke-linejoin="round">
    <path class="c0" d="m793 0l-10.8 2.9-1 14.7..z"/>
  </g>
</svg>
  • One CSS class per palette color, scoped under a per-document id so several SVGs can be inlined in one page without their classes colliding.
  • Consecutive same-color shapes merge into one <path> (draw order kept; evenodd shapes stay separate), so the element count is below the shape count.
  • Path data uses relative m/l commands on a grid of 10^-precision units. precision=1 (default) matches the contour grid and is lossless; precision=0 rounds to whole viewBox units for about 20% less, at a mean pixel difference of ~1%.
  • Shapes keep their absolute "M x,y L x,y Z" path strings internally (flow.value(extract_contours)["shapes"]); only the emitted SVG is compact.
  • To animate a color, target its class: @keyframes k{50%{fill:#f0f;stroke:#f0f}} #s1a2b3c .c0{animation:k 4s infinite}.

Gap Coverage: stroke=fill

Every fill <path> is stroked with its own fill color, at stroke-width="{gap_stroke}" with round joins (set once on the fills group). This bleeds each shape outward with its own fill color, covering inter-cluster gaps with the locally correct color.

Auto-scaling: gap_stroke is computed as max(1.0, round(svg_width / source_width)). A 500px source at svg_width=1000 gets gap_stroke=2; a 1000px source gets gap_stroke=1. This prevents the "snow" artifact where high-shape-count SVGs show visible light halos at zoom-out from excessive stroke bleed. Override explicitly: image_to_svg("img.jpg", gap_stroke=3).

Why stroke beats dilation for gaps: Dilation operates on binary masks before contour simplification — it blurs detail. Stroke operates on final polygons after approxPolyDP — it catches all gaps including those introduced by simplification. Pure vector; since v1.9.0 the stroke costs only the stroke declaration in each color's class.

Background fallback: When detect_background finds no clusters, the bg rect uses #000000 (black) instead of white. Black reads as shadow; white reads as absence.

Dilation is reduced to iterations=1 — just enough for morphological noise cleanup. Gap coverage is fully handled by stroke.

Anti-Patterns

  1. Never hand-draw shapes from visual interpretation. Use CV extraction.
  2. Never claim a fix works without rendering and comparing. A rendered comparison is the only verification.
  3. Never use geometric primitives (circles, rectangles) to approximate extracted contours.
  4. Never extract coordinates from the source image and insert into the SVG without verifying alignment.
  5. Never boost saturation globally. Do targeted per-color adjustments based on measured ΔE.
  6. Never aggressively merge near-background colors. Only merge colors <10 RGB distance from background AND heavily touching edges.
  7. Don't use bezier smoothing unless requested. Simple L polygons produce smaller SVGs.
  8. Don't use a dilation kernel larger than 3×3. Use iterations=1 on a 3×3 kernel — stroke=fill handles gap coverage in vector space, so dilation only needs to close noise holes.

Known Limitations

  • Thin linework (fill-only): The dark shape gating that prevents woodcut artifacts in photos can filter deliberate thin lines in graphic art. The "graphic" mode loosens this, but very fine crosshatching may still degrade. Use pipeline="compositional" for line-art inputs — it extracts thin features as SVG strokes instead.
  • Curved lines (compositional): Hough transform only detects straight line segments. Curved strokes (arcs, spirals, freehand curves) fall through to the fill pass and render as filled polygons. Future work: cv2.fitEllipse or spline fitting on skeleton branches.
  • Ring/arc structures: Large dark rings (like Kandinsky's outer circle) fragment across multiple K-means clusters. Each cluster's contours are independent, so the ring doesn't form one smooth shape. A dark-cluster-merging step would help.
  • Gradient transitions: At any K, smooth gradients produce staircase banding. Higher K reduces this but never eliminates it.
  • Parallel line groups (compositional): Dense hatching or ruled lines may merge incorrectly if the perpendicular distance between adjacent lines is below the merge threshold (6px). The merge step currently doesn't detect parallel-but-offset lines as distinct strokes.

Dependencies

pip install opencv-python-headless scikit-image scipy scikit-learn --break-system-packages
apt-get install -y librsvg2-bin  # for rsvg-convert

Compiled acceleration: nn_assign.c is auto-compiled on first use if gcc is available (27x faster label assignment). Falls back to numpy if unavailable.

Cross-skill dependencies (resolved automatically by pipeline.py):

Listening to music

Claude hears nothing, but it can render audio in the real engine and measure it. Set up the loop render → look → measure → change one thing → re-render, and keep a table of the numbers per version. A version is better when a number you chose beforehand moves by more than the take-to-take noise, and the spectrogram shows the same thing.

Requirements: Node with playwright and Chromium (preinstalled in Claude Code on the web), Python with librosa soundfile scipy matplotlib pillow, and for sheet music music21 verovio mir_eval cairosvg (pip install --break-system-packages librosa music21 verovio mir_eval cairosvg). The first render installs @strudel/web into ~/.cache/listening-to-music/. All scripts live in scripts/; run them with --help or read the docstring for options.

1. Render

S=/path/to/listening-to-music/scripts
node $S/render_strudel.mjs loop.js --out v1.wav --seconds 40 --warmup 4        # Strudel code
node $S/record_page.mjs page.html --click "#power" --out v1.wav --seconds 40 --warmup 8 \
     --query "station=house" --eval "window.__fm.setSeed(7)"                   # any Web Audio page

Both print peak level and flag clipping. A peak above 1.0 means the listener hears hard clipping; fix levels before judging anything else.

For a before/after comparison, fix the randomness (a seed hook via --eval) so both versions play the same material, and record each version twice.

2. Look

python3 $S/spectrogram.py v1.wav --out v1.png --from-onset --pyin-floor G3 --label v1

Then open v1.png with the image viewer. The top panel is a note-axis CQT spectrogram with the pYIN lead line, and the bottom panel is chroma. Compare it with the reference picture or the previous version side by side. The .npz next to it holds the numbers for step 3.

3. Measure

QuestionToolReads
Do simultaneous notes rub? (discordant chords)render_strudel.mjs --events → clashes.pyminor 2nd/9th overlap per cycle, by layer pair; worst moments
Do they rub in the recording?rubs.py, --pair for replicated takesshare of tonal peak energy in semitone/minor-9th pairs
Is it rougher or smoother overall?roughness.py, --pair for replicated takesSethares roughness median/p90
Does it match a reference image?reference.py ticks/decode/compareper-semitone profile, melody agreement, overtone offsets, chroma agreement
Is the melody audible over the mix?spectrogram.py pYIN voiced %, reference.py compare melody %

The clash scan is exact for the notes as written. rubs.py confirms them in the audio, where release tails and reverb add overlaps the note data does not show. Record harmony checks with percussion and noise textures muted in both versions: drum partials form their own peaks and hold a full mix at a rub share near 0.28 whatever the chords do. Use --warmup, 40 s or more, and two takes per version; --pair prints the take-to-take spread and says when a change is inside it.

Summed roughness (roughness.py) is a coarse overall measure. On Strudel FM (2026-09-24) it could not tell the harmony fix apart from take-to-take noise in sleep and house. On the same takes, the rub meter measured sleep −51%, ambient −33%, lofi −20% and house −13%, with spreads of 0.006–0.015. A loud bass drone dominates the roughness normalisation and hides a pad's rubs. Don't treat an unmoved roughness figure as proof that nothing changed.

Sheet music: notation out, notation in

Notation is the other way to hear. A score shows voicings, clusters (noteheads pushed sideways are seconds), register and rhythm at a glance, and Claude reads clean engraving accurately. All these scripts share one events JSON (layer, t0, t1, midi; times in bars).

node $S/render_strudel.mjs loop.js --events ev.json --cycles 8          # notes as written
python3 $S/score.py ev.json --out score.png --layers LEAD,KEYS,BASS --harmony   # engrave + analyse
python3 $S/transcribe.py take.wav --out mel.json --bpm 115              # notes as heard (melody)
python3 $S/compare_notes.py ev.json mel.json --ref-layer LEAD --cand-layer melody --align 1
python3 $S/read_score.py tune.musicxml --out ref.json                   # .mxl .mid .abc .krn, corpus:
python3 $S/to_strudel.py ref.json --out tune.js                         # score -> Strudel code

score.py --harmony prints each chord change with its pitches, music21's chord name, a Roman numeral in the detected key, and a rub flag. On Strudel FM the same four lofi bars went from 15 of 28 chord changes with a rub to 0 of 28 after the voicing fix, and the engraving showed why: the old Fmaj7 had its E and F side by side.

Measured on 2026-09-25 (compare_notes.py, onset within 0.05 bar and pitch within 50 cents):

PathTestF1
read_score → to_strudel → render eventsBach BWV 66.6, 4 parts, 163 notes100% every part
transcribe.py, one linesame chorale, soprano alone, piano, not used for tuning87%
transcribe.py --no-splitsynth lead with delay echoes82% (65% with splitting)
transcribe.pyfull four-part chorale0%: monophonic only
OMR (oemer 0.1.8)clean engraved soprano line6%; key read as flats, quarters as whole notes
Claude reading the image → ABCunseen chorale, soprano + bass, 38 notes100% (one clean engraving)

So:

  • Score images: read them yourself. Open the page and zoom into dense staves by cropping a 3× render or scan. Write ABC (K:, M:, L:, one V: per staff), convert it with read_score.py, re-engrave with score.py, and compare the two pictures bar by bar before using the notes. Use oemer only as a rough first pass, and never trust it unchecked.
  • Transcription is monophonic. For chords and inner voices, read the generating code's events or the score. Splitting at re-attacks is on by default, because a re-struck piano note has no gap; switch it off with --no-split for leads with delay or echo.
  • Unknown downbeat: a recording that starts mid-loop needs compare_notes.py --align N. It searches shifts up to N bars and prints the one used.

4. Change one thing, re-render, re-measure

Change one parameter or rule at a time, and after each change render and rerun the same measurements. A metric can reward the wrong thing, so check it against the spectrogram each round.

Known traps (each cost a round on 2026-09-24/25)

  • Mini-notation strings are patterns, not values. .delaytime("3/8") means "3, over 8 cycles", so the delay was 3 s (clamped to 1 s). Pass a number: .delaytime(0.39). .gain("0.9, 0.3") on a stacked pattern is itself a stack, so every hit fired twice and the drums peaked at 2.9× full scale. Give each layer its own scalar gain.
  • AudioWorklet synths (supersaw, …) need a secure context. Pages served from http:// get no audioWorklet, and those synths are silent while everything else plays. The scripts serve local pages from https://local.test/ for this reason. initStrudel also loads the worklets only on the first document click, so the scripts click.
  • pYIN floor. With a C3 floor, a loud bass made pYIN pick the bass's upper partials an octave below the tune. Tracking fell from 60% to 25% even though the melody was unchanged. Set the floor just under the melody's lowest note.
  • Lines above a melody are often its overtones. In a reference, lines at the melody's pitch +12, +19 and +24 semitones are usually its harmonics, not a second part. Test it with reference.py compare (the overtone rows against the control rows) before voicing a pad from them. A misread gave a Gmaj7 that the source never played.
  • Measure in the engine. An offline numpy model of Strudel's FM mispredicted the rendered harmonic spectrum. The low-passed saw it favoured came out darker in the engine than FM.
  • A metric can be confounded. A "lead prominence" score rewarded a pad that doubled the melody's notes. Before trusting a metric, ask what else could raise it.
  • Check reproducibility once. Render the same version twice and confirm the numbers repeat, so differences between versions can be trusted.
  • music21 respelling. chordify() discards pitch spelling, and a Key's inferred tonic transposes to the wrong letter (a leading tone of F instead of E♯ in F♯ minor). notes.spell re-spells after every chordify from the key's own scale plus the raised 6th and 7th. Without that, the score reads as wrong notes.
  • Quantise to a beat subdivision, not a bar fraction. A 16-step bar in 3/4 produced 64th notes and triple dots; the grid is now 16ths in any metre.
  • Decoding reference images. Bin centres sit on the tick rows. Get exact ticks with reference.py ticks rather than by eye; a half-bin error splits every note across two semitones.

Making videos with Kinescript

Kinescript renders a JSON scene to video with Skia and ffmpeg. Layers start and end on cues ("at": "title.out+0.2") rather than on hand-summed seconds, and every frame is a pure function of the scene and the frame number. The kine CLI also lints the scene (check), draws labelled contact sheets (look) and speaks narration (voice). Its author is Dimitri (GitHub drknowhow). Oskar Austegard, who maintains this skill collection, contributes as a collaborator. Muninn is the name of the Claude agent that works for him.

This file was last checked against Kinescript 2.5.0 (2026-11-12), and upstream releases often. kine docs is the authoritative reference for the installed version. Read it before writing a scene, and don't rely on feature lists remembered from earlier sessions, this file included.

When NOT to use this skill

SituationUse instead
Real web UI in a phone frame, arbitrary DOM/CSS layoutmaking-remotion-videos
AI-generated footage (Veo, Gemini video)creating-video
Trim, transcode or caption existing footageprocessing-video
Hand-drawn or p5.js character animationanimating-with-code

Setup in a Linux container

sudo apt-get install -y ffmpeg libegl1 libgl1 libfontconfig1   # skia needs EGL even on the CPU backend
pip install kinescript "skia-python>=144"
kine doctor
  • Pin skia-python ≥ 144. pyproject.toml allows ≥ 87, but 138.0 crashes.
  • With no GPU, KINE_BACKEND=cpu is used. Kinescript 2.5.0 rendered a 2:12 1080p explainer (3,968 frames of text, shapes and captions) in 41 s at -j 4 on 4 CPUs, 98 fps (oaustegard/experiments/colbert-pooling-explainer). Versions before 2.x managed 3–7 fps, so budget from a short --from/--to render rather than from memory.
  • In oaustegard/claude-workspace sessions, that repo's rules require third-party code to run inside its sandbox, scripts/jail.sh (no network, no secrets). Install from PyPI outside it and run kine through it.

Kokoro for kine voice

kine voice needs a voice plugin. Kokoro's, kinescript-kokoro, is not on PyPI: it lives in the Kinescript repo at plugins/kinescript-kokoro. Kokoro 0.9.4 and misaki 0.9.4, the latest releases (April 2025), require Python 3.10–3.12; on 3.13 pip falls back to kokoro 0.7.16. So give Kokoro its own 3.12 venv and point KINE_VOICE_CMD at it; kine stays in the system Python:

git clone --depth 1 https://github.com/drknowhow/kinescript ~/kinescript
uv python install 3.12 && uv venv -p 3.12 ~/kokoro-env
VIRTUAL_ENV=~/kokoro-env uv pip install --index-strategy unsafe-best-match \
  --extra-index-url https://download.pytorch.org/whl/cpu torch "kokoro==0.9.4" "misaki[en]==0.9.4" \
  https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
export KINE_VOICE_CMD="env PYTHONPATH=$HOME/kinescript/plugins/kinescript-kokoro $HOME/kokoro-env/bin/python -m kinescript_kokoro"

The CPU torch index avoids a multi-GB CUDA download; the venv comes to 1.4 GB. Installing spaCy's model up front saves misaki's first-use download. The first kine voice run downloads the model (327 MB). With no network at voice time (a sandbox), fetch hexgrad/Kokoro-82M (config.json, kokoro-v1_0.pth, voices/*.pt) with huggingface_hub.snapshot_download beforehand and pass HF_HOME and HF_HUB_OFFLINE=1 inside KINE_VOICE_CMD.

Procedure

  1. Read kine docs for layer types, cue syntax, motions and the sound rules, and kine docs 3d as well when the scene uses depth, a camera or models. kine new writes a starter scene.
  2. Storyboard in writing: scenes, their order, and the one thing the viewer must take from each. For anything longer than a few layers, generate the scene from a Python builder that holds one timeline array (the games promo's kine/build_scene.py is an example). Don't hand-edit 500 lines of JSON.
  3. Narration: in the scene's audio list, give each voice sound a text, a speaker and a rate, then kine voice scene.json. It re-speaks only sounds whose text, speaker or rate changed. It writes the audio and a words file of word timings. Mark a word with [[go]] before it to cue a layer from it ("at": "vo1:go"). Before speaking anything, read references/kokoro-speech.md, which covers names, numbers, stress, pauses, pace and blends.
  4. kine check scene.json: fix every error, read every warning, run it again. The warnings for tiny text and reading time are worth acting on: they pushed the games promo's type to ≥ 44 px and its sublines to five words or fewer.
  5. kine look scene.json --boxes (and --audio once there is sound): read the contact sheet image before rendering. kine still renders one frame. kine probe --at T gives every layer's resolved box.
  6. Poster frame: set canvas.poster to a moment that shows the title, e.g. "poster": "title.in+0.5" (seconds, a marker, a cue or a word). Feeds, players and a <video> without a poster show frame 0 before play, and a scene that fades in from its background shows up there as a blank rectangle. With poster set, the render draws that frame as frame 0 (one frame, 1/fps s, before the opening; audio unchanged). kine check warns blank-first-frame when frame 0 is empty and no poster is set. For a page, kine still scene.json --at poster -o poster.jpg gives the <video poster> image. --no-poster keeps the real frame 0 for a master.
  7. Render: kine render scene.json -o out.mp4 -q high -j 4. Long renders belong in a background job. The result's "poster" names the frame used.
  8. Verify the file: ffprobe for size, fps and duration, a frame strip (ffmpeg -vf "select='not(mod(n\,90))',scale=270:-1,tile=8x2") whose first tile is the poster, and a listen to the voice track.

Brands

When the person has a brand, or makes several videos that should look alike, use a Kinescript brand (1.9.0) instead of copying colours and fonts into each scene. kine docs brands has the full format.

Muninn's videos use the Muninn brand: brand/kinescript/ in oaustegard/muninn.austegard.com (paper and night palettes from the site, Fraunces, Source Serif 4, DM Mono, the raven mark, Muninn's voice). Copy that folder into the project as brand/ and set "brand": "brand".

  • Find or make one. kine brand list shows the library (~/.kinescript/brands). kine brand init acme makes one from the default Kinescript brand. kine brand import acme logos/mark.svg proposes dark and light palettes from a logo, each passing check's contrast; show the proposal to the person, then rerun the command with --apply. Over MCP, the kine_brand tool does each of these as an action.
  • Use it in the project. kine brand use acme copies it into brand/. The scene says "brand": "brand" and, if it needs one, "variant": "light". A later change to the library brand reaches the project only when you run use again, so old videos keep rendering the same.
  • Write scenes in the brand's names, not literals: "$accent", "$text", "textStyle": "heading", "enter": "$motion.enter". The built-ins (titleCard, lowerThird, charts, logo, bug, sting, endCard) already draw in it. "intro": true, "outro": true and "music": true add the brand's sting, end card and music bed. Voices take the brand's speaker unless a sound names its own.
  • Change it through the tools, not by editing brand.json by hand: kine brand set acme palettes.dark.accent '"#ff633f"', add-file, or kine brand wizard for the person. Each write is checked; pass --rev (base_rev) from your last read so a change made meanwhile is not overwritten.
  • kine check reports brand problems: brand-locked when the scene changes a part the brand locks (remove the override, don't remove the lock), and contrast and logo size issues for the brand's colours and placements.

Avatars

An avatar (2.0) is a character that talks: its mouth follows a voice line's words, and it blinks, breathes and glances on its own. kine docs avatars is the reference.

  • Start from the built-in one. "avatar": "kine:host" is a plain, friendly figure in the scene's palette with all nine mouth shapes and Kokoro's af_heart voice. kine avatar new FOLDER copies it as SVG parts to redraw. A brand can name avatars ("avatar": "brand:host").
  • Make it talk with say, which takes a line ({"text": ...}, spoken by kine voice), a sound id, or a list of them: {"id": "host", "use": "avatar", "avatar": "kine:host", "x": "30%", "h": 600, "say": {"text": "..."}}. Cue other layers from its words, e.g. "at": "host-say:render".
  • Time the acting as lists on the layer: expressions ({"name": "smile", "at": ..., "dur": ...}), gestures (nod, wave, ...) and gaze (camera, left, #chart, ...). Kinescript checks them but does not infer them from the text: you write them.
  • 3D avatars are a glTF file whose mouth, blink and expressions are morph targets by name, with gestures as the model's clips. examples/avatar3d.json is one, built from examples/make_media.py.
  • Check before rendering. kine avatar check FOLDER validates a folder. kine look FOLDER draws every pose on one sheet; a broken pivot or a missing mouth shape shows there. kine check on the scene reports mouth-approximate (no word times), avatar-missing-shape and gesture overlaps.
  • Cost. Drawn avatars cost little more than still pictures of them. A 3D head costs about as much as any model of its triangle count, moving or not.

Narration

  • Pace. Oskar wants narration at 165–180 wpm, energetic, with varied intonation. Punctuation pauses slow real narration well below the 200–240 wpm a bare sentence gives at rate 1: lines of two or three sentences in the 2:1 fable/lewis blend measured 139 wpm at rate 0.78 and 179 at 0.95. Start near "rate": 0.95, measure words ÷ speaking time from the words files after kine voice, and scale the rate.
  • Mix. Give sounds a role and Kinescript levels them: voice at −18 LUFS, music at −16 and ducked 14 dB under a voice, with no manual mixing.
  • Cue from words, not from the file start. A Kokoro clip opens with 0.3–0.4 s of silence.
  • Kokoro cannot raise its pitch for a question or act on tags such as [laughs]. For a line that needs either, set that sound's backend to elevenlabs or chatterbox (see kine docs, SOUND).
  • Muninn's voice is "speaker": "bm_fable,bm_fable,bm_lewis", a 2:1 blend of fable's rasp and lewis's low register.

Gotchas seen in use

The version in brackets is the one where each was seen. Check it against kine docs and the CHANGELOG before working around it.

  • A colour key takes hsl() only as an expression (2.5.0). "fill": "=hsl(200, 0.6, 0.5)" works; "fill": "hsl(200, 0.6, 0.5)" is a bad-colour error. A scene builder can write hex instead.

  • Group children are placed from the group's anchor, not its top-left corner (1.4.0). A centre-anchored group needs a child at (-w/2, -h/2) to fill it. kine check reports a misplaced child as off-screen.

  • loop: true does not lengthen a video layer (1.4.0). Give the layer a dur.

  • Video and shader layers default to a top-left anchor even when x is given (1.3.1, by design). Set anchor explicitly.

  • Capturing a web page as footage frame by frame with Playwright's fake clock: call page.clock.install(), reload, then pause the clock and step it. Installing it after load stalls most games. WebGPU pages also need the GPU queue drained before each screenshot. The games promo's capture/lib.cjs does both.

  • The Kinescript repo's CLAUDE.md mandates the maintainer's c3_* MCP tools. Where they are not installed, its first line says to use the usual tools.

Contributing upstream

Work on a branch of drknowhow/kinescript and open a PR. Dimitri merges his own work as PRs too, so never commit to its main. Run python -m pytest first. A visual change also needs kine check and kine look --boxes.

Making videos with Remotion

Remotion renders a React component once per frame in headless Chromium and encodes the frames with ffmpeg. The DOM and CSS are the drawing API, so type, layout, screenshots and UI mockups come out sharp and on-brand with little code. Each frame is a pure function of useCurrentFrame().

Before building, name the route and why (Oskar's standing steer, memory 06886186). Remotion fits when the video is mostly type, layout, product UI and 2D motion. It is not the cheapest route for particle fields or generative art.

When NOT to use this skill

SituationUse instead
Hand-drawn character animation, p5.js, painterly framesanimating-with-code
AI-generated footage (Veo, Gemini video)creating-video
Trim, transcode, concatenate or caption existing footageprocessing-video
A comic or storyboard drawn from existing footagedrawing-video
A looping in-page animation the user views in a browseran HTML artifact; no render needed

Abandon Remotion mid-run when a frame takes more than about 5 s to render after the bundle is built. That points to heavy WebGL or canvas work, which a pre-rendered frame sequence or a native ffmpeg/numpy route does better.

Procedure

  1. Scaffold with pinned versions (all @remotion/* packages at one exact version, at least two weeks old):
    bash /mnt/skills/user/making-remotion-videos/scripts/scaffold.sh /path/to/project
    
    It writes package.json, tsconfig.json, src/index.ts, a hello Root.tsx, copies the phone and scroll components into src/, and runs npm install (about 40 s).
  2. Vendor the fonts. Never use @remotion/google-fonts here: see failure modes below.
    cd /path/to/project
    python3 /mnt/skills/user/making-remotion-videos/scripts/vendor_fonts.py "Instrument Serif:ital@0;1" "DM Sans:wght@400;700"
    
    Paste the printed loadFont() calls into a theme.ts that every scene imports.
  3. Prove the toolchain before writing scenes:
    bash /mnt/skills/user/making-remotion-videos/scripts/render.sh still Main out/probe.png 45
    
  4. Storyboard in writing. List the scenes with frame durations and one read each: the single thing the viewer must take away. A caption needs about 1.5 s plus 0.3 s per word on screen. Put durations in a data array and derive the total from it. Never hand-sum frame counts.
  5. Build scenes under the contract below. Sequence them with TransitionSeries from @remotion/transitions. Each transition overlaps its neighbours, so the total is the sum of durations minus the transition lengths. Compute that in code and export it to <Composition>.
  6. Review loop with stills. Render one still per read (the frame where it should be fully legible) and tile them into a contact sheet. Look at it before rendering any video.
    for f in 40 100 260 530; do bash .../render.sh still Main out/s$f.png $f; done
    montage out/s*.png -tile 4x1 -geometry 360x640+6+6 out/sheet.png
    
    Fix layout collisions, illegible type and dead frames here: a still costs about 3 s and a full render minutes.
  7. Render the video in the background if it will outlive the turn (the hub's long-job rule):
    bash .../render.sh video Main out/reel.mp4
    
  8. Verify (next section), then deliver. Keep sources and captures in git and the MP4 out of git unless it is small. Upload large renders to R2 or a release.

The composition contract

  • Every visual value derives from useCurrentFrame() through interpolate, spring or plain arithmetic. No useState timers, no setTimeout, no CSS animation/transition. Remotion seeks frames in any order across parallel tabs, so a CSS animation renders as a random phase or not at all.
  • Randomness comes from random(seed) in remotion, never Math.random().
  • Assets load through staticFile() from public/ with <Img>, <Video> (@remotion/media) or <Audio>. These hold the frame until loaded. A plain <img> does not, and the frame renders blank.
  • Async data (fetch, fonts outside @remotion/fonts) wraps in delayRender() / continueRender(), or goes through calculateMetadata.
  • Pass extrapolateLeft/Right: "clamp" to every interpolate that should hold its end value. The default extrapolates, so a fade continues past 0 and 1.
  • Keep the canvas at the delivery size (1080×1920 for vertical, 1920×1080 for landscape) and design in those pixels. Scaling a smaller composition up softens type.

Product demo reels

For a reel of a real web app inside a phone, follow references/product-reel.md. It covers capturing the live UI at phone size with Playwright, stepping live animations frame by frame with Playwright's fake clock, the phone frame, sticky headers, tap indicators and the timeline the aeyu.io reel used.

Narration

Mix the voiceover outside Remotion into one WAV the length of the composition. Put each line at its slot start, apply linear gain only (peaks at about −2 dBFS; never loudnorm, see below), and play it with <Html5Audio src={staticFile(vo)} /> at the root. Pass the track as a prop (--props='{"vo":"vo/x.wav"}') so voice A/B tests reuse one composition. Write the script as lines with start/end slots taken from the scene timeline, then size each line to its slot: words ÷ slot seconds × 60 should be 140–180 wpm.

  • TTS routes, measured 2026-10-29 on the aeyu reel (details in references/narration.md): Gemini 3.8 Flash TTS (invoking-gemini generate_speech, prebuilt or designed voice, style string) took 2.5 min for 27 takes and had the most varied pitch. OmniVoice (k2-fsa, the engine behind VoiceStudio) runs locally in the jail at about 29× slower than real time on 4 CPU cores. Its tag-based voice design (gender, age, pitch, accent) cannot ask for energy and came out flat. Cloning a lively 6 s reference worked better, and its per-line duration argument sets pace exactly.
  • Take three per line and pick by ASR. Score each take with faster-whisper WER against the script, then pick the longest one that fits its slot. Gemini rendered "Year bests" as "your best" in 6 of 6 takes, with or without CAPITALS stress; "Best of the YEAR" survived 3 of 3.
  • A coined name: test it by ear, and spell it if voices mangle it. ASR cannot judge it, since it mapped all eight mangled readings of "aeyu" to one transcript. See references/narration.md.
  • Record a coined name once and splice the clip into every line that says it. Pick its spelling by generating 3 takes per candidate and keeping the one whose takes all transcribe alike. OmniVoice takes CMU phonemes inline: [EY1 Y UW0].
  • Fill the slot, don't just fit it. Oskar on the aeyu reel: "weirdly hurried, then with a gap between slides". Lines that merely fit ran at 180–280 wpm and left up to 1.7 s of silence before each cut. Size each line to about 160 wpm over its slot, cap pauses inside a take at about 0.2 s, and fit length by picking among takes (generate 5–7 for a tight line), not by stretching. A take may run until 0.1 s before the next line starts.
  • Never stretch speech with librosa's phase vocoder. At 6–12% it made the aeyu voiceover sound "recorded from a laptop microphone in a slightly echoey room" on exactly the stretched lines. If a take needs a nudge, use ffmpeg rubberband=tempo=R:formant=preserved:pitchq=quality:window=short, and only within ±4%.
  • Check pace, not just fit. Unpaced OmniVoice averaged 211 wpm with lines at 283. A take that fits its slot can still be rushed.
  • Verify the mix: ffmpeg -i out/reel.mp4 -af silencedetect=noise=-40dB:d=0.25 -f null - lists the speech onsets, which should match the slot starts. -af ebur128=peak=true gives integrated loudness and peak.
  • Don't loudness-normalize TTS up to −16 LUFS. TTS already peaks near 0 dBFS (the aeyu mix measured −21.1 LUFS with peaks at −1.1 dBFS), so a single-pass loudnorm to −16 adds about 5 dB of limiting. Oskar heard it as "amplified too much". Linear gain to a −2 dBFS peak landed at about −22 LUFS and kept the speech's dynamics.

Verification

ffprobe -v error -show_entries stream=codec_name,width,height,pix_fmt,r_frame_rate:format=duration -of default=nw=1 out/reel.mp4
ffmpeg -v error -i out/reel.mp4 -vf "select='not(mod(n\,90))',scale=270:-1,tile=8x2" -frames:v 1 out/strip.png

Expect h264, the composition size, yuv420p, the composition fps and a duration of total frames divided by fps. Then read the strip. A bad success is a file that probes correctly but has blank screens where an <img> raced its load, fallback serif type where a font failed, or a scene that never appears because a duration constant was wrong. Only looking at frames catches these.

Common failure modes

  • ERR_CERT_AUTHORITY_INVALID on fonts.gstatic.com, then TypeError: Failed to fetch, and the render aborts → the render browser does not trust the egress proxy's CA, though curl does. Vendor fonts with vendor_fonts.py. The same applies to any HTTPS asset fetched during render: download it into public/ first. Diagnosed 2026-10-29 on the aeyu.io reel. Every still failed until fonts went local.
  • "browserExecutable" was specified as ... but the path doesn't exist → the Playwright headless shell lives at /opt/pw-browsers/chromium_headless_shell-<rev>/chrome-headless-shell-linux64/chrome-headless-shell, with -linux64 in the directory name. render.sh globs for it. Without the flag Remotion tries to download Chrome Headless Shell.
  • Output probes as yuvj420p → the default for the probe render. Some players and uploaders mishandle full-range video. render.sh passes --pixel-format=yuv420p --color-space=bt709.
  • Text below the caption collides with the next element → layout was designed at one scale and an element was resized later. Catch it on the contact sheet: the aeyu reel's first sheet had every subtitle under the phone's top edge.
  • The video is shorter than planned and the last scene is cut → total duration was hand-summed while TransitionSeries overlaps. Derive it.

Cost on this box

4 cores, 15 GB. The bundle step takes about 15 s per invocation, including for stills. The 35.5 s aeyu reel (1065 frames, 1080×1920, four full-page screenshots and 285 captured frames) rendered in 79 s at the default concurrency of 4, to a 5.5 MB file at crf 20.

Parsing Video

Claude cannot play video, but it can read images. To interpret a video, sample frames evenly across its duration, tile them into timestamped contact sheets, and Read the sheets. A 4×4 sheet compresses ~16 moments into one image, preserving narrative flow — what changed, in what order, roughly when.

Requires ffmpeg/ffprobe (apt-get update && apt-get install -y ffmpeg if missing).

Workflow

1. Probe first

ffprobe -v quiet -print_format json -show_format -show_streams input.mp4

Note duration, resolution, and whether there's an audio stream. Duration drives how many sheets you need.

2. Generate contact sheet(s)

python3 scripts/contact_sheet.py input.mp4                          # 1 sheet, 4x4, whole video
python3 scripts/contact_sheet.py input.mp4 --sheets 3               # 48 frames across 3 sheets
python3 scripts/contact_sheet.py input.mp4 --start 120 --end 300    # zoom into 2:00–5:00
python3 scripts/contact_sheet.py input.mp4 --grid 3x3 --tile-width 500  # fewer, larger tiles

The script probes duration, samples frames at interval midpoints, stamps each tile with its source timestamp (H:MM:SS, bottom-left), and tiles them into <name>_sheet_NN.png. It prints each sheet's time range.

Sheet budget — more sheets = more Read calls; scale to duration and task:

DurationSheetsSampling interval
< 2 min1 (4×4)~4–7 s
2–10 min2–4~10–40 s
10–60 min4–8, or coarse-then-zoom~1–2 min
> 1 hourcoarse pass, then zoomvaries

3. Read and interpret

Read each sheet image. Tiles run left-to-right, top-to-bottom in time order; use the stamped timestamps to anchor observations ("the scene changes around 1:42"). Cross-sheet continuity: the last tile of sheet N immediately precedes the first tile of sheet N+1.

4. Zoom when needed

Contact sheets trade resolution for coverage. When something needs a closer look:

# Re-sheet a narrower window at higher tile resolution
python3 scripts/contact_sheet.py input.mp4 --start 95 --end 125 --grid 3x3 --tile-width 500

# Or extract a single full-resolution frame at the moment of interest
ffmpeg -ss 00:01:42 -i input.mp4 -frames:v 1 detail.png

Content-aware sampling

Uniform sampling guarantees temporal coverage but ignores structure: it can straddle a cut mid-interval, waste tiles on a static shot, or land mid-pan on a motion-blurred frame. Two refinements, both feeding --at. Choose by footage type: edited content (films, trailers, TV) → shot boundaries; continuously shot footage (handheld, drone, screen recordings, dashcam) → dwell points; unknown → uniform first, refine after.

Shot boundaries (cuts)

Detect cuts and align tiles to them:

# ffmpeg scene score: frames whose difference from the previous frame exceeds 0.3
TS=$(ffmpeg -i input.mp4 -vf "select='gt(scene,0.3)',metadata=print:file=-" -f null - 2>/dev/null \
     | grep -oP 'pts_time:\K[0-9.]+' | paste -sd,)
python3 scripts/contact_sheet.py input.mp4 --at "$TS"

Each selected frame is the first frame of the new shot (the score compares against the previous frame). Threshold 0.3–0.4 suits most content; lower it for subtle cuts, raise it for noisy footage.

Know what scene detection misses. The scene score is a frame-pair difference metric:

  • Gradual transitions (dissolves, fades, wipes) spread the change across many frames, each below threshold — they often go undetected.
  • Within-shot content changes: between two cuts, objects can enter, leave, or change substantially with no boundary ever firing. A long static-camera shot with lots of action yields one tile under pure scene-aligned sampling.
  • Camera motion (pans, handheld shake) can fire false positives.

So treat scene-aligned sampling as a refinement, not a replacement: run uniform sheets first for guaranteed temporal coverage, then a scene-aligned sheet (or a union of both timestamp sets via --at) when shot structure matters. If cut detection quality itself matters, the purpose-built tool is PySceneDetect (uv pip install --system scenedetect[opencv-headless], then scenedetect -i input.mp4 list-scenes) — its content/adaptive detectors are more robust to motion and noise than the raw ffmpeg score, but they are still cut-oriented and share the within-shot blindness above.

Dwell points (where the camera settles)

In continuously shot footage cuts are rare or absent — the structure lives in camera moves. In the frame-difference signal, held compositions are the valleys and pans/zooms are the peaks between them. The valley midpoints are the frames worth sampling: sharp, deliberately framed, one per composition — where uniform sampling would land mid-pan on smeared pixels.

python3 scripts/contact_sheet.py input.mp4 --at "$(python3 scripts/dwell_points.py input.mp4 --max 16)"

dwell_points.py computes the motion signal cheaply (4 fps at 160 px via ffmpeg's scene metric), smooths it over ~1 s, and takes spans below a relative threshold (default p40 of the signal) lasting at least --min-dwell (1 s). It keeps the --max longest holds and prints their midpoints; stderr lists each hold span with its duration so you can see the video's rhythm before reading a single frame.

Caveats:

  • The threshold is relative, so on constant-motion footage (one unbroken pan) it degrades to picking the least-motion moments — harmless but arbitrary; prefer uniform sampling there. It exits non-zero when no hold lasts --min-dwell.
  • Subject motion during a held shot (a person gesturing in a static frame) raises the valley floor but rarely fills it; if a busy-but-held shot is missed, lower --percentile.
  • Dwells say where the camera settled, not what changed — combine with uniform coverage for anything time-critical.

Overseeing a multi-clip edit

When acting as the editing agent over a generated or assembled cut (see the creating-video skill), review at two levels:

  1. The whole assembly. Contact-sheet the final stitched cut, not just the individual clips. Per-clip sheets each look fine in isolation; character drift, prop jumps, and logic breaks only appear when the shots sit in sequence. One 4×4 sheet over a 30 s cut gives ~2–3 tiles per scene — enough to catch them.

  2. The seams. A cut hides continuity errors at the boundary — the outgoing clip's last frame vs the incoming clip's first frame. scripts/seams.py pairs them, one row per cut, for a one-look check:

    python3 scripts/seams.py clip1.mp4 clip2.mp4 ... clipN.mp4 --out seams.png
    

Read every sheet against the continuity checklist for generated video:

  • Character — same face/hair/wardrobe across every shot they appear in.
  • Prop — same identity, size, color, and attachment point shot to shot (the classic failure: a small object that changes size or moves between shots).
  • Physical logic — is the world coherent across the cut (a window open before something enters through it; an action's result matching its setup)?
  • Action completeness — is the key beat actually shown, not cut around?

Report which scene or seam fails and what the fix is (tighten the prompt, or regenerate just that shot) — that verdict is the deliverable the editing agent acts on.

Interpreting honestly

  • Report what is visible in the sampled frames; events between samples are invisible. Say "between 1:30 and 1:40 the scene changes from X to Y", not fabricated specifics about the transition.
  • Fast action (a ball in flight, a single gesture) can fall entirely between samples — tighten the window and re-sheet before concluding something didn't happen.
  • Timestamps are accurate to ~1 s (seek + rounding), fine for general video.

Limits — when NOT to use contact sheets

  • Fine print, dense text, small UI details: tiles are ~380 px wide; text becomes unreadable. Extract full-resolution frames at the relevant timestamps instead.
  • Audio content: sheets are silent. If speech matters, extract the audio (ffmpeg -i in.mp4 -vn audio.mp3) and transcribe it separately; note to the user if no transcription path is available.
  • Frame-exact analysis (sports officiating, VFX QC): sampling misses frames by design; extract every frame in a narrow window (ffmpeg -ss 84 -to 86 -i in.mp4 frames_%03d.png).

For transformation tasks — convert, trim, merge, compress, GIF, subtitles — use the processing-video skill.

Image Processing Tools

This container has a rich image processing toolkit. Before writing any image code, scan this inventory to pick the best tool for the job — don't default to Pillow for everything.

Tool Selection by Task

Format Conversion & Batch Operations

ImageMagick convert is the default choice. Handles 260+ formats, single command, no code needed.

convert input.png output.webp
convert input.png -quality 85 output.jpg
mogrify -format webp *.png          # batch in-place

For animated formats (GIF↔WebP↔APNG, video↔frames), prefer ffmpeg.

Resize, Crop, Thumbnails

ImageMagick for CLI one-liners. Pillow when already in Python pipeline.

convert input.jpg -resize 800x600 output.jpg          # fit within box
convert input.jpg -resize 800x600^ -gravity center -extent 800x600 output.jpg  # fill+crop
convert input.jpg -thumbnail 200x200^ -gravity center -extent 200x200 thumb.jpg

IM supports 30+ resize filters: -filter Lanczos (sharp downscale), -filter Mitchell (balanced), -filter Point (nearest-neighbor/pixel art).

Compositing & Overlays

ImageMagick composite or convert with -composite. Supports all Porter-Duff modes.

composite -gravity southeast watermark.png photo.jpg output.jpg
convert base.png overlay.png -gravity center -composite result.png

For complex multi-layer work or programmatic positioning, use Pillow (Image.paste, Image.alpha_composite).

Montages, Collages, Contact Sheets

ImageMagick montage — purpose-built for grid layouts.

montage *.jpg -geometry 200x200+5+5 -tile 4x3 sheet.jpg
montage *.png -label '%f' -geometry +4+4 catalog.png

Animated GIFs & Frame Sequences

ImageMagick for simple GIF assembly. ffmpeg for anything involving timing control, video sources, or optimization.

convert -delay 10 -loop 0 frame_*.png animation.gif        # IM
ffmpeg -framerate 10 -i frame_%03d.png -vf palettegen palette.png && \
ffmpeg -framerate 10 -i frame_%03d.png -i palette.png -lavfi paletteuse output.gif  # optimized

imageio is convenient for frame-sequence GIFs from Python arrays.

Image Analysis & Measurement

identify for quick metadata and stats. OpenCV for structural analysis. scikit-image for scientific measurement.

identify -verbose image.png          # full metadata dump
identify -format '%wx%h %[colorspace] %[depth]bit' image.png   # targeted
  • OpenCV (cv2): histograms, contour detection, template matching, edge detection, color distribution
  • scikit-image (skimage): region properties, morphology, thresholding (Otsu, adaptive), feature detection (SIFT via OpenCV, ORB), SSIM comparison
  • scipy.ndimage: convolution, interpolation, labeling connected components

Image Comparison & Diffing

ImageMagick compare produces visual diffs directly.

compare image1.png image2.png diff.png
compare -metric RMSE image1.png image2.png null: 2>&1   # numeric similarity

For structural similarity: skimage.metrics.structural_similarity (SSIM).

Color Operations

ImageMagick handles color space conversion, channel manipulation, color quantization.

convert input.jpg -colorspace Gray output.jpg
convert input.png -colors 16 reduced.png           # quantize
convert input.jpg -modulate 110,130,100 output.jpg  # brightness,saturation,hue
convert input.png -channel R -separate red_channel.png

For programmatic color analysis (dominant colors, palettes): OpenCV k-means on pixel arrays, or Pillow getcolors().

Effects & Filters

ImageMagick has extensive built-in effects:

convert in.png -blur 0x3 out.png              # Gaussian blur
convert in.png -sharpen 0x1 out.png
convert in.png -shadow 60x4+2+2 out.png       # drop shadow
convert in.png -vignette 0x40 out.png
convert in.png -sketch 0x10+120 out.png
convert in.png -charcoal 2 out.png
convert in.png -edge 1 out.png
convert in.png -emboss 1 out.png
convert in.png \( +clone -background black -shadow 60x4+0+0 \) +swap -background none -layers merge +repage rounded.png

For advanced/custom convolution kernels: scipy.ndimage or OpenCV.

Metadata & EXIF

identify -verbose reads all metadata. Pillow reads/writes EXIF programmatically.

identify -verbose image.jpg | grep -A20 'Properties:'
from PIL import Image
img = Image.open("photo.jpg")
exif = img.getexif()  # dict-like access to EXIF tags

Note: no exiftool in this container. Use identify or Pillow for metadata tasks.

Icons, Favicons, App Icons

ImageMagick generates multi-size ICO files directly.

convert icon.png -define icon:auto-resize=256,128,64,48,32,16 favicon.ico

For icon sets (iOS/Android), batch-resize with convert or mogrify to each required size.

PDF & EPS (limited)

ImageMagick can read/write PDF and EPS but Ghostscript is not installed — complex PDF rasterization may fail. For PDF-to-image, prefer Pillow (for simple cases), pdfplumber (text/table extraction), or the pdf skill (for full PDF manipulation). IM's built-in PDF delegate handles basic operations.

Seam Carving (Content-Aware Resize)

ImageMagick has Liquid Rescale (LQR) built in:

convert input.jpg -liquid-rescale 80x100%! output.jpg   # shrink width 20%, preserve content

SVG Handling

ImageMagick rasterizes SVG via its built-in delegate. For SVG→PNG at specific sizes:

convert -density 300 input.svg -resize 800x output.png

No Inkscape or rsvg-convert available. For SVG manipulation, work with the XML directly or use Python's lxml.

Complete Tool Inventory

ToolTypeKey Strength
ImageMagick 6.9CLI suite260 formats, effects, compositing, batch ops
ffmpegCLIVideo/animation, frame extraction, optimized GIFs
GraphvizCLIDiagram→image rendering (dot, neato, etc.)
LibreOfficeCLIDocument→image conversion
Pillow 12.1PythonGeneral-purpose, EXIF, drawing, WebP/AVIF, freetype
OpenCV 4.13PythonComputer vision, histograms, contours, morphology
scikit-image 0.26PythonScientific analysis, SSIM, segmentation, features
Wand 0.6PythonFull ImageMagick API from Python
imageio 2.37PythonUnified I/O, GIF frame sequences
scipy.ndimagePythonConvolution, interpolation, labeling
numpyPythonRaw pixel array math
reportlabPythonPDF generation with embedded images

Key Constraints

  • No Ghostscript — PDF/EPS rasterization is limited to IM's built-in delegate
  • No potrace/autotrace — bitmap-to-vector tracing unavailable
  • No standalone optimizers — no optipng, pngquant, jpegoptim, gifsicle (use IM quality settings or ffmpeg instead)
  • No exiftool — use identify or Pillow for metadata
  • No Inkscape — SVG manipulation is XML-level only
  • HEIC/AVIF read-only in ImageMagick; Pillow supports AVIF read/write
  • ImageMagick is v6 — use convert/identify commands, not magick (v7 syntax)

ffmpeg Toolkit

ffmpeg 6.1.1 is pre-installed with a full-featured build. Also available: ffprobe (media analysis) and ffplay (playback, limited use in container).

Before writing custom Python for media tasks, check whether ffmpeg handles it in a single command.

To interpret video content (summarize, describe, find scenes) rather than transform it, use the parsing-video skill — it samples frames into timestamped contact sheets that can be read as images.

Task Reference

Probe & Inspect Media

ffprobe — always start here to understand what you're working with.

ffprobe -v quiet -print_format json -show_format -show_streams input.mp4
ffprobe -v quiet -show_entries format=duration,bit_rate -of csv=p=0 input.mp4
ffprobe -v quiet -select_streams v:0 -show_entries stream=width,height,r_frame_rate,codec_name -of csv=p=0 input.mp4

Video Format Conversion

ffmpeg -i input.avi output.mp4                             # container swap (re-encode)
ffmpeg -i input.avi -c copy output.mp4                     # container swap (no re-encode, fast)
ffmpeg -i input.mp4 -c:v libx265 -crf 28 output.mp4       # H.265
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 out.webm  # VP9 WebM
ffmpeg -i input.mp4 -c:v libsvtav1 -crf 35 output.mp4     # AV1 (SVT, fastest AV1 encoder)
ffmpeg -i input.mp4 -c:v libjxl output.jxl                # JPEG XL (single frame)

Encoder selection guide:

GoalEncoderTypical flags
Compatibilitylibx264-crf 23 -preset medium
Better compressionlibx265-crf 28 -preset medium
Web deliverylibvpx-vp9-crf 30 -b:v 0
Best compressionlibsvtav1-crf 35 -preset 6
Lossless archivallibx264-crf 0 -preset veryslow

Lower CRF = higher quality. x264 default 23, x265 default 28, SVT-AV1 default 35 are visually similar.

Audio Format Conversion

ffmpeg -i input.wav -c:a libmp3lame -q:a 2 output.mp3     # MP3 VBR ~190kbps
ffmpeg -i input.wav -c:a libopus -b:a 128k output.opus    # Opus (best quality/size)
ffmpeg -i input.wav -c:a aac -b:a 192k output.m4a         # AAC
ffmpeg -i input.wav -c:a flac output.flac                  # FLAC lossless
ffmpeg -i input.mp3 -ar 44100 -ac 2 output.wav            # to WAV, set sample rate/channels

Extract Audio from Video

ffmpeg -i video.mp4 -vn -c:a copy audio.aac               # extract without re-encoding
ffmpeg -i video.mp4 -vn -c:a libmp3lame -q:a 2 audio.mp3  # extract as MP3
ffmpeg -i video.mp4 -vn -c:a libopus -b:a 128k audio.opus

Trim & Cut

ffmpeg -i input.mp4 -ss 00:01:30 -to 00:03:00 -c copy clip.mp4        # fast, keyframe-aligned
ffmpeg -ss 00:01:30 -i input.mp4 -to 00:01:30 -c copy clip.mp4        # -ss before -i = faster seek
ffmpeg -i input.mp4 -ss 00:01:30 -to 00:03:00 -c:v libx264 -c:a aac clip.mp4  # frame-accurate (re-encode)

-ss before -i seeks by keyframe (fast, may be imprecise). After -i decodes from start (slow, precise). For frame-accurate cuts, re-encode.

Concatenate / Merge

Demuxer method (same codec, no re-encode):

# Create file list
printf "file '%s'\n" clip1.mp4 clip2.mp4 clip3.mp4 > list.txt
ffmpeg -f concat -safe 0 -i list.txt -c copy merged.mp4

Filter method (different formats, re-encodes):

ffmpeg -i clip1.mp4 -i clip2.mp4 -filter_complex "[0:v][0:a][1:v][1:a]concat=n=2:v=1:a=1[v][a]" -map "[v]" -map "[a]" merged.mp4

Resize & Scale

ffmpeg -i input.mp4 -vf "scale=1280:720" output.mp4                   # exact size
ffmpeg -i input.mp4 -vf "scale=1280:-1" output.mp4                    # width 1280, auto height
ffmpeg -i input.mp4 -vf "scale=-1:720:flags=lanczos" output.mp4       # height 720, Lanczos
ffmpeg -i input.mp4 -vf "scale=iw/2:ih/2" output.mp4                  # half size
ffmpeg -i input.mp4 -vf "pad=1920:1080:(ow-iw)/2:(oh-ih)/2" output.mp4  # letterbox to 1080p

Framerate

ffmpeg -i input.mp4 -r 30 output.mp4                       # simple (drops/dupes frames)
ffmpeg -i input.mp4 -vf "fps=24" output.mp4                # filter-based
ffmpeg -i input.mp4 -vf "minterpolate=fps=60" output.mp4   # motion interpolation (slow)

Animated GIF from Video

Use the two-pass palette method for quality:

ffmpeg -i input.mp4 -vf "fps=10,scale=480:-1:flags=lanczos,palettegen" palette.png
ffmpeg -i input.mp4 -i palette.png -lavfi "fps=10,scale=480:-1:flags=lanczos[x];[x][1:v]paletteuse" output.gif

Single-pass (simpler, lower quality):

ffmpeg -i input.mp4 -vf "fps=10,scale=480:-1" output.gif

Animated WebP from Video

ffmpeg -i input.mp4 -vf "fps=15,scale=480:-1" -c:v libwebp -lossless 0 -q:v 75 -loop 0 output.webp

Extract Frames

ffmpeg -i input.mp4 -vf "fps=1" frame_%04d.png             # 1 frame/second
ffmpeg -i input.mp4 -vf "select='eq(pict_type,I)'" -vsync vfr keyframe_%04d.png  # keyframes only
ffmpeg -i input.mp4 -vf "thumbnail=300" -frames:v 1 thumb.png   # best thumbnail from first 300 frames
ffmpeg -ss 00:00:05 -i input.mp4 -frames:v 1 screenshot.png     # single frame at timestamp
ffmpeg -i input.mp4 -vf "select='gt(scene,0.3)'" -vsync vfr cut_%04d.png  # first frame of each detected cut

The scene score is a frame-pair difference: it catches hard cuts but misses gradual transitions (dissolves/fades) and within-shot content changes; pans can false-positive. For robust shot detection use PySceneDetect (scenedetect -i input.mp4 list-scenes).

Subtitles

ffmpeg -i input.mp4 -vf "subtitles=subs.srt" output.mp4           # burn in SRT
ffmpeg -i input.mp4 -vf "ass=subs.ass" output.mp4                 # burn in ASS (styled)
ffmpeg -i input.mp4 -i subs.srt -c copy -c:s mov_text output.mp4  # soft subs in MP4

Subtitle rendering uses libass (full ASS/SSA styling support).

Text & Watermark Overlays

ffmpeg -i input.mp4 -vf "drawtext=text='Hello':fontsize=48:fontcolor=white:x=10:y=10" output.mp4
ffmpeg -i input.mp4 -i watermark.png -filter_complex "overlay=W-w-10:H-h-10" output.mp4
ffmpeg -i input.mp4 -vf "drawtext=text='%{pts\:hms}':fontsize=24:fontcolor=white:x=10:y=H-30" output.mp4  # timestamp

Audio Processing

ffmpeg -i input.mp4 -af "volume=1.5" output.mp4                    # volume boost
ffmpeg -i input.mp4 -af "loudnorm=I=-16:TP=-1.5:LRA=11" output.mp4  # EBU R128 normalization
ffmpeg -i input.mp4 -af "afade=t=in:d=2,afade=t=out:st=58:d=2" output.mp4  # fade in/out
ffmpeg -i input.mp4 -af "highpass=f=200,lowpass=f=3000" output.mp4  # bandpass
ffmpeg -i input.mp4 -an output_silent.mp4                           # strip audio
ffmpeg -i video.mp4 -i audio.mp3 -c:v copy -map 0:v -map 1:a output.mp4  # replace audio track

Video Stabilization (libvidstab, two-pass)

ffmpeg -i shaky.mp4 -vf "vidstabdetect=shakiness=5:accuracy=15" -f null -
ffmpeg -i shaky.mp4 -vf "vidstabtransform=smoothing=10:input=transforms.trf" stabilized.mp4

Crossfade & Transitions

ffmpeg -i clip1.mp4 -i clip2.mp4 -filter_complex "xfade=transition=fade:duration=1:offset=4" output.mp4

Transition types: fade, wipeleft, wiperight, slideup, slidedown, circlecrop, dissolve, and ~40 more.

Waveform & Spectrum Visualization

ffmpeg -i audio.mp3 -filter_complex "showwavespic=s=1280x240:colors=0x1e90ff" -frames:v 1 waveform.png
ffmpeg -i audio.mp3 -filter_complex "showspectrumpic=s=1280x720" -frames:v 1 spectrum.png

Compress / Reduce File Size

ffmpeg -i input.mp4 -c:v libx264 -crf 28 -preset faster -c:a aac -b:a 128k smaller.mp4
ffmpeg -i input.mp4 -c:v libx265 -crf 32 -preset medium -c:a libopus -b:a 96k smallest.mp4

CRF is the primary quality knob. Preset trades encoding speed for compression efficiency.

Available Codecs & Libraries Summary

Video encoders: libx264, libx265, libvpx (VP8), libvpx-vp9, libsvtav1, librav1e, libaom-av1, libjxl (JPEG XL), gif, png, apng, libwebp Audio encoders: aac, libmp3lame, libopus, libvorbis, flac, ac3, pcm_s16le Subtitle: ASS/SSA (libass), SRT, DVB, DVD, MOV text 426 formats, 562 filters, hardware accel stubs (vaapi, vulkan, opencl — limited use in container)

Key Constraints

  • No GPU acceleration — CUDA device count is 0; vaapi/vulkan/opencl listed but no hardware available. All encoding is CPU-only.
  • Container is ephemeral — long encodes on large files are feasible but the container resets between tasks. Work in /home/claude/, deliver to /mnt/user-data/outputs/.
  • 4 CPU cores, 9 GB RAM — encoding is parallel but constrained. Use -preset faster or -preset veryfast for large files to avoid timeouts.
  • ffplay exists but display is unavailable — use for probing only, not playback.
  • No GPU-accelerated filters — stick to CPU filter variants.

Reading Business Cards

Turn photos or scans that pack many business cards into one image into a clean contact list. The job is two stages, in order: preprocess with the script, then read the tiles it produces. Read the tiles, not the original sheet — the original is too low-resolution per card once the model downscales it.

Why preprocess first

The model downscales any input image to ~1568px on the long edge before it sees it. A phone photo of 50-60 cards is often 4000-6000px; downscaled to one image, each card lands ~150px wide — unreadable, which forces you onto a more expensive model. The script cuts the sheet into overlapping tiles, each near the downscale cap (~1300px), so every card in a tile keeps 500px+ of real resolution. That resolution recovery is what lets a cheaper model (Sonnet) read cards that only the expensive one (Opus) could read before — and it lets Opus read cards from far fewer tiles. The grid is sized to the reader model (see Stage 1): pass --model and the script tiles aggressively for Haiku, moderately for Sonnet, and least for Opus, because a stronger reader has a lower resolution floor. The floor is set by the pixels, not the model's intelligence: at ~220px/card (a whole dense sheet) even Opus reads only company and some names, not phone/email — so even Opus needs a few tiles for fine print on dense sheets.

De-glaring (illumination flattening + local contrast) is applied to each tile. It corrects uneven lighting and the haze off glossy cards and plastic binder sleeves. It cannot recover text where glare has clipped pixels to pure white — that data is gone (see Limits).

Stage 1 — Tile the sheets

The person only provides the images and the goal ("read these cards"). Derive every parameter yourself; do not ask them to choose grid sizes or flags.

Real sheets are messy: cards scattered at angles, packed in binder sleeves, overlapping, piled. Detecting individual card boundaries fails on all of these. Tiling ignores card boundaries — it slices the sheet into a grid of overlapping rectangles. Each tile holds a few cards at high resolution; the overlap means a card split by one tile's edge is whole in its neighbour.

Run it sized to whichever model will read the tiles — pass --model and the script measures each sheet's native card size, then derives the coarsest grid whose cards still clear that model's OCR floor:

python3 scripts/prep_cards.py /mnt/user-data/uploads --out /home/claude/cards_work --model opus

--model opus|sonnet|haiku (or a full model id string) sets the floor: Opus tiles least, Haiku most. Default is sonnet (the safe middle). On a dense ~660px-card sheet this yields roughly 6 tiles/sheet for Opus, 9-12 for Sonnet, 12-20 for Haiku. If the reader is the in-conversation model (the no-key path, below), set --model to match whatever you're running. Overrides: --target-px forces an explicit tile size, --card-px overrides the auto card-size estimate, --floor-px overrides the per-model floor.

It prints the grid it chose per image (e.g. auto 3x2 from 4284x5712 [model=opus floor=350 card~668px -> target 2993]) and writes tiles to cards_work/tiles/ (<sheet>__r{R}_c{C}.png) plus a manifest.json. If a sheet's cards are smaller than the model's floor even at native resolution, it warns — that sheet needs a re-shoot, not a finer grid.

Then verify and self-adjust by inspection — this is your judgment, not the person's:

  • View one representative tile. If the cards in it are crisp and fully legible, proceed to read them all.
  • If cards look small or dense (text fuzzy), rerun with a smaller cap for a finer grid: --target-px 1100.
  • If cards are clipped in half at tile edges, rerun with more overlap: --overlap 0.2.
  • If the cards are matte and text-only and a light haze remains, rerun with --binarize. Skip binarize for color or logo-heavy cards — it flattens them.

De-glaring (illumination flatten + local contrast) is on by default and is non-destructive. Decide on the extra flags by looking at a tile, then commit to the full read.

Stage 2 — Read and extract

There are two ways to read the tiles. In-session reading is the universal path and works for everyone, key or no key: view the tiles with the view tool and transcribe them in this conversation, applying the rules in prompts/haiku_extract.md (one row per fully-visible card, skip edge-clipped cards that are whole in a neighbour, never invent, confidence low when unsure). The coarse model-sized grid is what makes this tractable — a dozen tiles per sheet, not eighty — and the conversation model reading them is exactly the model that reads cards well. Cost here is the subscription's usage allowance, not per-token dollars; the coarser the grid (Opus), the fewer reads.

The API runner is an optional optimization, only when an API key is present (API_KEY in env / /mnt/project/claude.env). It sends each tile to a model in a separate parallel call (temperature 0 on legacy models; current models reject sampling parameters) using the distilled prompt, dedupes, and writes the CSV — reading outside the chat context, so it is cheaper per token and runs many tiles at once. Without a key it cannot run; use the in-session path. The runner bills the API account per token; it buys parallelism and a cheaper meter, never accuracy.

Running the API runner (keyed path)

Validate the model on a sample before the full run. Haiku (default claude-haiku-5-5, $0.10/$0.50 per MTok, ~20x cheaper per token than Sonnet 5.5) has weaker OCR; on phone photos of loose or angled cards it confidently misreads names and marks the errors high confidence. (Gemini's free tier was tested 2026-06-15 and read these poorly — do not reach for it here.) Tested guidance:

  1. Sample run with Haiku:
    python3 scripts/extract_cards.py --work /home/claude/cards_work \
        --out /home/claude/sample.csv --limit 8
    
  2. View 2 of the tiles it read and compare against sample.csv. Check: are names and companies correct? Is confidence honest (clear cards high, blurry/angled low)?
  3. Decide:
    • Clean, high-resolution, upright cards (e.g. a flatbed scan), Haiku reads them correctly → keep Haiku for the full run.
    • Errors, or high confidence on wrong text → switch to Sonnet: --model claude-sonnet-5-5. Sonnet via this same script is far cheaper than reading tiles in-conversation and is accurate on messy phone photos.
  4. Full run with the chosen model and your real output path:
    python3 scripts/extract_cards.py --work /home/claude/cards_work \
        --out /mnt/user-data/outputs/cards.csv [--model claude-sonnet-5-5]
    

The script prints raw vs unique counts and the low-confidence / parse-error tally. The CSV columns are sheet, tile, name, title, company, phone, email, website, address, confidence.

Stage 3 — Triage the failures

From the finished CSV, take every row with confidence = low (and any parse-error). Re-run just those — extract the relevant tiles into a small work dir and run extract_cards.py on them with --model claude-sonnet-5-5 (or Opus in-chat for the worst). Cards still wrong after that are too small, angled, or glare-clipped in the source — flag them for a re-shoot rather than re-running.

Note Haiku's confidence is not reliable enough to drive this triage on its own; if you used Haiku, sanity-check high rows too, or just use Sonnet for the run.

Reading tiles by hand (fallback)

If the API runner is unavailable (no key), you can read tiles yourself with the view tool, applying the rules baked into prompts/haiku_extract.md: one row per fully-visible card, skip edge-clipped cards, never invent, confidence low when unsure. Work in batches and write the CSV incrementally. This burns far more tokens than the runner — prefer the runner.

Other layouts

Two narrower modes exist for clean inputs. Choose them yourself only when a spot-check shows the cards are cleanly separated and unrotated; never the default:

  • --rows R --cols C (grid slice): exact, for cards in a perfect non-overlapping grid. Builds numbered montages instead of tiles.
  • --detect (contour auto-detect): crops individual well-separated cards. Best-effort only — it merges touching cards and misses rotated/glare-covered ones. On real scattered or binder sheets it under-detects badly; stay on the default tiling.

In both, the read units are numbered montages (montage_NNN.png) whose cells carry a red #index matching the manifest; extract one row per #index.

Limits

  • Tiling recovers resolution and corrects lighting; it does not invent pixels that glare clipped to white. Severe specular blowout is unrecoverable — those cards need a re-shoot under diffuse light.
  • Cards photographed at an extreme angle or far smaller than their neighbours may still read low; a finer tile grid helps, a re-shoot helps more.
  • --binarize trades color/logo fidelity for text crispness. Use it only for text-only cards, not as a default.
  • Model accuracy is the real bottleneck, not the pipeline. On tested phone (Haiku 4.5, Sonnet 4.x; not re-measured on the 5.5 models) photos of loose/angled cards, Haiku confidently misread names and companies (e.g. "Clint Emerson/CableQuest" → "Cliff Emerson/Quest") and marked errors high confidence; its inconsistent misreadings also defeated cross-tile dedup. Sonnet on the identical tiles read them correctly with calibrated confidence. Reserve Haiku for clean, high-resolution, upright scans, and validate it on a sample (Stage 2) before trusting a full run.

Seeing Images

Compensatory vision tools based on blindspots measured by vision diagnostic v1-v4 on 2026-03-25. Which model it ran on is not recorded here and it has not been re-run since, so read the thresholds as measured-then: they say a tool exists for each failure, not that the current model fails at exactly that number. Re-run the diagnostic before relying on a specific threshold.

When to Use

Activate this skill when:

  • Describing an uploaded image in detail
  • Reproducing an image as SVG (use BEFORE drawing to establish ground truth)
  • Comparing two images or regions for differences
  • Reading text in degraded/compressed/low-contrast images
  • Identifying subtle features (gradients, faint overlays, reflections)
  • Any image task where accuracy matters more than speed

Known Blindspots (from diagnostics)

Measured, not guessed:

BlindspotThresholdCompensatory Tool
Luminance contrast~15-20 RGB steps invisibleenhance, histogram, sample
Gradients<30-step range invisiblegradient_map, enhance
Context color biasDress effect, simultaneous contrastisolate, sample
Small elements<15px effectively invisiblecrop, grid
Dense countingDegrades >15 items, ~50% error at 30count_elements
Subtle atmosphericsSteam, faint reflections lost in noiseenhance, denoise

Workflow

Setup (one line, every time)

import sys; sys.path.insert(0, '/mnt/skills/user/seeing-images/scripts')
from see import grid, sample, enhance, edges, histogram, isolate, palette, compare, count_elements, gradient_map, denoise, crop

Quick Analysis (2-3 tool calls)

grid(path, rows=2, cols=2)   # → view the output
sample(path, [(x1,y1), ...]) # → verify colors at points of interest

Deep Analysis (for SVG reproduction, spot-the-difference, etc.)

grid(path, rows=3, cols=3)                    # 1. Overview
palette(path, n=10)                           # 2. Dominant colors
edges(path, threshold=30)                     # 3. Shape boundaries
sample(path, [(x1,y1), (x2,y2), ...])        # 4. Exact RGB at points
enhance(path, region=(x,y,w,h), mode='auto')  # 5. Reveal low-contrast areas
isolate(path, region=(x,y,w,h))              # 6. Remove context bias

Tool Reference

All functions in scripts/see.py. Every function that produces an image saves to /home/claude/see_*.png and returns the path. Use view tool on the returned path.

grid(path, rows=3, cols=3, labels=True)

Splits image into labeled cells for systematic inspection. Call it first: it reduces attentional competition.

sample(path, points, radius=3)

Returns exact RGB values at specified pixel coordinates. Use to verify what you think you see. Averages over a small radius to handle noise.

histogram(path, region=None)

Color histogram showing value distribution. Reveals bimodal distributions (hidden gradients), dominant colors, and contrast range. With region=(x,y,w,h), analyzes only that area.

enhance(path, region=None, factor=2.0, mode='contrast')

Boosts contrast in the image or a region. Modes: 'contrast', 'brightness', 'color', 'sharpness'. Use factor=3-5 for near-threshold features.

edges(path, threshold=50)

Sobel edge detection revealing shape boundaries invisible at low contrast. Lower threshold = more edges (noisier). Output is a white-on-black edge map.

gradient_map(path, region=None)

Computes local gradient magnitude across the image. Bright = high gradient, dark = flat. Reveals gradients below the 30-step detection threshold.

isolate(path, region, padding=20, bg=(128,128,128))

Extracts a region and places it on a neutral gray background. Removes surrounding context that causes simultaneous contrast and Dress-type illusions. The bg parameter defaults to mid-gray to minimize context bias.

compare(path, r1, r2)

Side-by-side comparison of two regions with diff overlay. Highlights pixel-level differences with amplification. Use for spot-the-difference tasks.

count_elements(path, region=None, color_range=None, min_size=3)

Programmatic element counting using connected component analysis. Specify approximate color_range as ((r_min,g_min,b_min), (r_max,g_max,b_max)) to count specific colored elements.

denoise(path, region=None, strength=3)

Median filter to reduce photographic noise, revealing subtle features hidden in the noise floor (like steam, faint reflections).

palette(path, n=8)

Extracts the n most dominant colors using k-means clustering. Returns RGB values and their proportions. Essential for SVG reproduction.

Accuracy Notes

Call grid() first on a complex image. Verify colors near context boundaries with sample() or isolate(), counts above 15 with count_elements(), gradients with gradient_map(), and faint features with enhance() before describing them. Each of these is a row of the blindspot table, so the tool call is the evidence — perception alone is not.

SVG Portrait Mode

Selective simplification: one pipeline pass at high K → zone-aware contour simplification → optional per-zone style transforms. Like phone portrait mode, but vectorized — not blur, but stylistic separation of foreground and background.

Quick Start

Agent-annotated (recommended)

The agent looks at the image first, identifies important regions with rough bounding boxes, then calls:

from portrait_mode import portrait_mode

svg, stats = portrait_mode("photo.jpg",
    focus_targets=[
        {'bbox': (215, 125, 295, 195), 'label': 'face'},
    ],
    focus_edges=[
        {'bbox': (214, 170, 310, 290), 'label': 'beard'},
        {'bbox': (210, 415, 300, 505), 'label': 'hands'},
        {'bbox': (195, 95, 330, 140), 'label': 'hat'},
    ])

With style transforms

svg, stats = portrait_mode("photo.jpg",
    focus_targets=[{'bbox': (215, 125, 295, 195), 'label': 'face'}],
    focus_edges=[{'bbox': (214, 170, 310, 290), 'label': 'beard'}],
    style_transforms={
        'background': 'desaturate:0.7',
        'periphery': 'desaturate:0.3',
    })

Backward-compatible (MP-only)

Without annotations, falls back to MediaPipe face detection:

svg, stats = portrait_mode("photo.jpg")

How It Works

image-to-svg pipeline (K=96, unified palette)
    → zone detection (agent bboxes + optional MediaPipe landmarks)
    → assign contours to zones (by centroid)
    → per-zone simplification (epsilon, min_area)
    → optional per-zone style transforms
    → single SVG output (zone-labeled <g> groups, no clipPaths)

Subtractive, not additive. One pipeline pass produces full detail everywhere. Zones that don't need detail get simplified by coarsening contour approximation and raising minimum area thresholds. Subject stays sharp; background gets abstract.

Why This Works

v0.5.0 ran four independent pipelines (one per zone) with different K-means palettes, then composited via clipPaths. This produced tonal discontinuities at zone boundaries and was 4-7x slower. v0.6.0 uses a single palette so colors harmonize naturally, and simplification is a cheap post-process on contours the pipeline already extracted.

Agent Workflow

  1. Look at the image — identify what's compositionally important
  2. Provide rough bounding boxes as (x1, y1, x2, y2) pixel coordinates
    • Precision is NOT required (±30px is fine)
    • Use focus_targets for where the eye goes first (face, eyes)
    • Use focus_edges for compositionally important areas (beard, hands, hat, props)
  3. Call portrait_mode() — skill handles zone detection, extraction, and assembly
  4. Review output — check stats for path distribution across zones

Four Zones

ZonePurposeEpsilonMin AreaExamples
TargetWhere the eye goes first0.5× (tight)15 px²Face, eyes, key subject
EdgeCompositionally important1.0× (default)40 px²Beard, hands, hat, props
PeripheryContext, not focal2.5× (loose)100 px²Torso, clothing, limbs
BackgroundAtmosphere5.0× (very loose)200 px²Sky, walls, landscape

Epsilon multiplies the base simplification factor (0.002 × perimeter). Higher = fewer vertices = more abstract. Min area filters out small shapes entirely.

Zone Assignment

Each contour is assigned to the highest-priority zone covering >30% of its area. This prevents focal shapes that straddle a zone boundary from getting simplified. Small contours (<500 px²) use centroid lookup for speed.

Periphery Generation

When focus targets or edges are specified, periphery is automatically generated as a buffer zone around the foreground (dilated union of target + edge zones). This creates a smooth detail gradient from subject to background.

Per-Zone Style Transforms

With zone-tagged shapes sharing a unified palette, backgrounds can be independently styled without affecting subject colors:

style_transforms={
    'background': 'desaturate:0.7',   # 70% desaturated
    'periphery': 'mute:0.3',          # 30% toward mid-gray
}

Available transforms:

TransformEffect
desaturate:NShift toward gray (0=none, 1=grayscale)
grayscaleFull grayscale
mute:NShift toward mid-gray (0=none, 1=flat gray)
warm:NWarmer color temperature
cool:NCooler color temperature
opacity:NGroup opacity (0=invisible, 1=full)

Parameters

portrait_mode(image_path,
    # Zone annotations
    focus_targets=None,   # [{'bbox': (x1,y1,x2,y2), 'label': str}, ...]
    focus_edges=None,     # [{'bbox': (x1,y1,x2,y2), 'label': str}, ...]

    # Pipeline settings
    K=96,                 # Color clusters (higher = more tonal detail)
    smooth=None,          # ImageMagick preprocessing ("oilpaint", "kuwahara:N")
    svg_width=800,

    # MediaPipe options
    use_landmarks=True,   # Try MP face landmarks for precise face geometry

    # Per-zone simplification overrides
    zone_simplification=None,  # {ZONE_TARGET: {'epsilon_mult': 0.3, 'min_area': 10}}

    # Per-zone style transforms
    style_transforms=None,  # {'background': 'desaturate:0.7'}
)

Performance

Single pipeline pass (~8-12s for a typical photo at K=96) vs v0.5.0's four independent passes (~40-60s). Zone detection and contour assignment add <1s. Style transforms are string operations with zero computational cost.

Requirements

Cross-skill dependencies:

  • image-to-svg pipeline (/mnt/skills/user/image-to-svg/)
  • flowing DAG runner (/mnt/skills/user/flowing/)
  • seeing-images (/mnt/skills/user/seeing-images/) — for agent's visual inspection

Optional MediaPipe models (auto-downloaded on first use):

  • blaze_face_short_range.tflite — face detection fallback
  • face_landmarker.task — precise face oval (478 mesh points)

Note: MediaPipe selfie segmenter is not used. Zone detection comes from agent bboxes, with MP used only for face landmark refinement.

pip install opencv-python-headless scikit-image scipy scikit-learn --break-system-packages -q
apt-get install -y librsvg2-bin -qq

Verification Protocol

After EVERY run, render and visually compare side-by-side. Same as image-to-svg.

import subprocess
from PIL import Image

subprocess.run(['rsvg-convert', '-w', '1400', 'output.svg', '-o', 'output.png'])

orig = Image.open('source.jpg')
rendered = Image.open('output.png')
target_h = 800
orig_r = orig.resize((int(orig.width * target_h / orig.height), target_h))
rend_r = rendered.resize((int(rendered.width * target_h / rendered.height), target_h))
gap = 20
comp = Image.new('RGB', (orig_r.width + rend_r.width + gap, target_h), (255,255,255))
comp.paste(orig_r, (0, 0))
comp.paste(rend_r, (orig_r.width + gap, 0))
comp.save('comparison.png')

Transcribing Images

Read what a slide, page, or image actually shows — text plus charts, diagrams, screenshots, and layout — by rasterizing it and sending the picture to a vision model. This is the fix for the gap the built-in pptx and pdf skills leave: they extract embedded text only, so an image slide, a chart, or a scanned figure reads as empty. Visual transcription reads it the way a person looking at the slide would.

When to reach for this vs. the built-in skills

Use the pptx / pdf skills first for text-native documents — a normal deck or report where the content is real text boxes. They are faster and exact.

Switch to this skill when text extraction comes back thin or empty on a file you can see is visually rich, or whenever the meaningful content is a picture: chart, graph, diagram, screenshot, photo, scanned page, or a slide exported as one flat image. Don't guess which case you're in — if pptx/pdf returned little from a file that clearly has content, that is the signal.

The pipeline

Everything routes through scripts/transcribe_pages.py, which handles all three ingress paths and one bad page never aborts the rest:

  • .pptx / .ppt → LibreOffice headless → PDF → pdftoppm → one PNG per slide
  • .pdf → pdftoppm → one PNG per page
  • image file → used directly as a single page

Each page image is then transcribed by a vision model. Run it directly:

python3 scripts/transcribe_pages.py deck.pptx                  # all slides
python3 scripts/transcribe_pages.py report.pdf --pages 3-7     # subset
python3 scripts/transcribe_pages.py slide.png --model opus     # one image
python3 scripts/transcribe_pages.py deck.pptx --json out.json  # structured

Or import transcribe_file(...) for programmatic use; it returns a list of {page, image, text, error} dicts.

Choosing the model

The transcription core and its empirical cost/recall data are reused from browsing-bluesky/scripts/image_transcribe.py — same registry, kept in sync. Pick with --model:

  • gemini-lite (default) — gemini-3.5-flash-lite. Cheapest and fastest: ~$0.0009 and ~2s per image on 3 synthetic terminal screenshots (2026-11-20), where it read every token. Dense slides cost more. Right for routine deck reading.
  • gemini-flash — gemini-4-hydrogen at thinking level low. About 8x the cost and 3x the latency of lite. Use when a page needs interpretation, not just reading. (--model gemini-3.5-flash also runs gemini-4-hydrogen: the 3.5 Flash model is retired, and the name is kept so existing commands work.)
  • opus — for interactive sessions where you want the reading in your own context anyway.
  • haiku — only if constrained to single-vendor Anthropic; weak at dense transcription (tends to summarize instead of transcribe). That finding is from Haiku 4.5; the alias calls claude-haiku-5-5, which has not been measured.

Default to gemini-lite. Move to gemini-flash when a page needs interpreting, and to opus when the reading belongs in an interactive conversation.

OCR fallback (tesseract)

Tesseract 5.x is installed (eng + osd language packs only) and is exposed as --engine tesseract. It returns glyphs, not a reading — no chart interpretation, no diagram description, no layout meaning. Use it only for pages you already know are plain scanned text, when you want a zero-cost, fully-offline pass. For anything with a chart, diagram, or visual layout, the vision path is the correct tool; tesseract on those pages will quietly lose the content that mattered.

When the deliverable is a file rather than a reading, use ocring-pdfs instead. It installs ocrmypdf at runtime (~18s) and writes the OCR back as an invisible text layer over the original page images, so the output is a PDF that pdftotext and grep can read. The --engine tesseract path here returns loose text with no page anchoring.

Interactive shortcut

In an interactive session you can often skip the model call entirely: rasterize with scripts/transcribe_pages.py … --json to get the page PNGs, or just convert and view each page image yourself — Claude reads images natively. The script's vision-model path exists for batch and autonomous runs where no human-in-loop reader is available, or when a deck has more pages than is practical to view one by one.

DPI

Default raster is 150 DPI — legible for a vision model and safely under the 5 MB/image base64 ceiling. Bump to --dpi 200–300 only for pages with dense small fonts; higher DPI risks exceeding the per-image size limit and costs more tokens for no gain on normal slides.

Sponsors

Details

Marketplace
oaustegard/claude-skills
Source
plugins/media-processing
Category
Media Processing
License
MIT
Keywords
animating-with-code, converting-files, creating-video, detecting-tips-zones, drawing-video, drawing-with-pycairo, image-to-svg, listening-to-music, making-kinescript-videos, making-remotion-videos, parsing-video, processing-images, processing-video, reading-business-cards, seeing-images, svg-portrait-mode, transcribing-images

More from oaustegard/claude-skills

All 14