One loop — work — for code, data, writing and workshop presentations: an approved plan as the run's only authority, delegated implementation, independent verification against computed gates, and human review

version
6.62.0
GitHub stars
27

Install

/plugin install workflows@edwinhu-plugins

Run it inside Claude Code after adding the marketplace with /plugin marketplace add edwinhu/workflows.

Contents

Discovered from source

Skills 91

afk — a standing mandate until the armed ceiling

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

!bash ${CLAUDE_SKILL_DIR}/scripts/arm.sh "${CLAUDE_SESSION_ID}" "$ARGUMENTS"

The line above armed (or declined to arm) the hold when this skill loaded. Its output is the authority on what is armed; if it says "Not armed", say so, do not arm by hand, and still follow steps 1-5: the heartbeat is then the only keep-alive.

**NO QUESTIONS UNTIL THE CEILING** — the time the armed line prints: 18:00 for a daytime /afk, otherwise 09:00. The user is away. Asking is not careful, it is an idle session and lost hours. Take the Recommended branch, state the choice in one line, and keep going.
  1. Heartbeat. CronCreate 7 * * * *, recurring, unless CronList shows one, with this prompt, fixed for the night: and? (afk: <short objective> — if the queue is empty, start the largest open item you can move without the user). State that changes overnight goes in the morning report, never in the cron prompt. Never CronDelete it: CronDelete is in permissions.ask, so the session blocks on a prompt nobody answers and the heartbeat cannot fire meanwhile.
  2. Branches. Take the Recommended one instead of AskUserQuestion.
  3. Commit locally. Explicit paths. Never push, never delete data.
  4. Work the queue. Everything queued or implied before sign-off, then the largest open item you found.
  5. Morning report before the ceiling: what landed, what is blocked, what needs the user.
  6. Release is the user's: work-hold.sh --disarm.
About toWhy wrongDo instead
End a turn announcing the next stepA check-less afk hold allows every stop, so nothing re-enters the session until the next tickTake the step in the same turn
Launch a grind loop on this objectiveA hold beside a grind deadlocks bothOne driver per objective; the heartbeat covers the grind
CronDelete or re-create the heartbeat to change its promptThe permission prompt blocks the session and silences the heartbeatKeep the prompt fixed; write state into the morning report
Answer a tick with "nothing new" / "no change"That is the idle night this skill exists to preventTake step 4: start the largest open item
About to start a Bash command with cd <dir>; or splice quotes with '"'"'Claude Code can show a permission prompt for these even in bypass mode, and nobody answersUse absolute paths and git -C; pass long text through a file (jq --rawfile)

AI Writing Anti-Patterns

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Field guide for detecting and revising AI-generated content indicators based on Wikipedia's "Signs of AI writing" guide.

When to Use

Invoke this skill:

  • Before finalizing ANY AI-assisted writing
  • When reviewing text for AI writing indicators
  • When editing content to sound more natural
  • After completing writing tasks (automatic via hooks)

The Iron Law

Check every piece of AI-assisted writing against these patterns before submission.

This is not optional. AI writing patterns are detectable and undermine credibility.

Quick Screening Order

Start with the most objective indicators:

PrioritySectionWhat to Check
1ChatGPT Artifactsturn0search0, oaicite, contentReference
2Citation ProblemsHallucinated DOIs, dead links, non-existent sources
3Prompt Refusals"As an AI language model...", "I hope this helps"
4Puffery"stands as a testament to", "rich tapestry", "nestled"
5StructureSection summaries, "Despite challenges", rule of three
6Mannered ProseMetaphor where a literal phrase was available (see below)

Critical Patterns to Avoid

CRITICAL Severity (Immediate Revision Required)

These patterns are unambiguous AI artifacts:

ChatGPT-Specific Artifacts:

  • turn0search0, turn1search2 (internal search references)
  • oaicite:X (citation placeholders)
  • contentReference[oaicite:X] (unresolved references)
  • JSON attribution blocks in output

Prompt Refusals:

  • "As an AI language model..."
  • "I cannot provide..."
  • "I hope this helps!"
  • "I hope this email finds you well"

HIGH Severity (Strong Revision Recommended)

THE SHIPPED LIST IS ~/.claude/skills/ai-tic/linter/tics.yaml, NOT THIS TABLE. That file holds both halves — the tics that cleared the ~0-human-rate gate against 14,294,148 sentences, and a rejected: block naming the phrases that did not, so they are not re-proposed. Six phrases were listed here as AI tells while sitting in that rejected: block: serves as (141.35/M), it is important to note that (44.42/M), plays a crucial/vital/key role bare (21.94/M), delve into bare (11.69/M), underscores the importance (5.93/M), and Of course, (386.94/M). They are gone from the tables below and from every executable table in the repo. This is the same correction the User-Voice section thirty lines down already made for a different list — "Keep these as voice preferences if you like them … Do not present them as AI detection." — applied to this one. To check a phrase, run /ai-tic <phrase>; do not add a row here.

Puffery and Exaggeration (those that cleared the gate):

  • "stands as a testament to"
  • "rich tapestry of"
  • "nestled in/among"
  • "the landscape of"

Promotional Language:

  • "groundbreaking", "transformative", "revolutionary"
  • "unparalleled", "unprecedented"
  • "cutting-edge", "state-of-the-art"

MEDIUM Severity (Review and Consider)

Structural Patterns:

  • Section summaries that repeat the heading
  • "Despite [challenge], [positive outcome]" formula
  • Negative parallelisms: "However... Nevertheless..."
  • Rule of three: exactly three examples every time
  • Weasel wording: "some experts say", "it is believed"

Stylistic Quirks:

  • False precision — a summarising figure carried to spurious decimal places ("a rate of 1.3771 percent", "covering 85.63 percent of the universe"). Prefer a high-level fraction in the abstract and introduction, where a figure summarises rather than reports; keep the exact value next to the exhibit that backs it. Enforced as style·false_precision.
  • Elegant variation (synonym cycling to avoid repetition)
  • False ranges ("from X to Y" without real data)
  • Title Case In All Headings
  • Em dash overuse (—)
  • Excessive boldface for emphasis

User-Voice Preferences (NOT AI tells — corpus-checked)

This section is a personal style preference, not a linter. Nothing here is in an executable table, so a writing-verify pass knows none of these phrases. Three of the four were measured against 14,294,148 sentences of human scholarship (8.73M finance/accounting + 5.56M law review) and they are normal human prose:

phrasefinancelawverdict
has bite / have more bite1.9/M5.2/Mhuman — incl. a law review title, "Do the SEC's New Rating Agency Rules Have Any Bite?"
the cut (regulatory reduction)8.7/M4.7/Mhuman — "the cut in the corporate tax rate"
Of course,299.9/M523.7/Mhuman
To be sure, ← not an AI tell; do not replace it11.5/M194.0/Mhuman, and 3× commoner in law reviews than what it replaces
Admittedly,15.5/M63.3/Mhuman
cuts against0.9/M13.1/Mhuman

Keep these as voice preferences if you like them — they are unfalsifiable and that is fine. Do not present them as AI detection. Full record: docs/investigations/2026-08-05_emphasis-enforcement.md.

What DID survive the gate, and is now enforced as a span:

  • ai-tic·sev3·rule-bites — "the reform should bite hardest". The verb is unattested (1/14.29M, and that hit is a cited Financial Times headline in a footnote). The noun above is not. That noun/verb split is the whole rule.
  • ai-tic·sev2·sharpest-version — "the sharpest version of the objection". 0/14.29M, while all four hits of <superlative> version of are "the strongest version of" and the sharpest <noun> at large is 86 hits. The word is ordinary; the collocation is the tell.

Bridge repetitions remain the one genuinely enforced entry from the old list — skills/writing-verify/scripts/bridge_repetition_check.py, invoked from workflows/writing-verify.js. It is real logic over section openings, not a phrase table.

To check a phrase yourself: /ai-tic <phrase> — it runs the FP-hunt against both corpus halves and refuses to add anything over the eligibility gate.

How to Revise

For Puffery

AI PatternHuman Alternative
"stands as a testament to""shows" or "demonstrates"
"rich tapestry of"describe specifically what it contains
"nestled in the heart of""in" or "located in"

For Structure

AI PatternHuman Alternative
Section summary of headingStart with substance, not meta-commentary
"Despite challenges..."State the reality directly without formula
Exactly three examplesUse the number that fits: 2, 4, 5, or just 1

For Mannered Prose

Definition (Anthropic guidance — quoted, do not paraphrase):

Mannered prose substitutes metaphor and flourish for direct statement. Instead of "a parameter worth varying," the mannered writer produces "a dial worth turning." Instead of "this point still matters," they write "this point earns its keep." The phrases exist to display the writer, not to convey the idea, and readers can tell. That is why mannered prose irritates: it makes the reader work harder so the writer can perform. It is also imprecise. Metaphors drag in connotations the writer did not choose and cannot control. The fix is to say what you mean. When a literal phrase is available, use it.

The short form — "Please remove all mannered prose" — works on its own and is often enough.

The mechanism is metaphor-for-literal substitution. That is narrower than "flourish" generally: the test is whether a literal phrase was available and passed over. "A dial worth turning" for "a parameter worth varying" is mannered; a long sentence is not, and an ornate one is purple, a different fault. Puffery inflates ("stands as a testament to"); mannered prose performs.

The imprecision is the load-bearing objection, not the irritation. A metaphor imports connotations the writer did not choose: "the bar routed around the holding" implies coordinated evasion, where "lawyers filed the same claims as mootness-fee requests" makes no claim about motive. Revising for taste alone misses that the mannered version often asserts something the writer cannot support.

The revision, in order:

  1. Replace every mannered phrase with the literal thing it avoided.
  2. Table each swap: the flourish, the plain version, and the connotation it smuggled in.
  3. Name the habit the writer repeats most.

The three constraints — violating any makes the pass worse than not running it:

  • Keep the meaning and the structure. A diction pass, not a rewrite.
  • Leave direct sentences untouched. A sentence with no metaphor has nothing to swap.
  • Don't shorten for its own sake. Literalness is the goal, not brevity. A plain replacement that runs longer than the flourish is still the right replacement.

Why the table is the deliverable, not just the edits. The third column is the point. Listing swaps without naming what each smuggled in hides the substantive change. And naming the repeated habit matters more than any single swap: individual metaphors are cheap to fix and recur immediately, while the habit is the thing the writer can act on.

A related failure, one level up (supporting research, not the definition). Density, not the device, is what makes rhetorical figures read as machine-written: "The problem is not that LLMs use these techniques: it's that they're so robotically consistent in how they use them that it becomes an abuse… What the LLM lacks is not technical ability, but taste" (Colin Gorrie, Why ChatGPT writes like that). The figures to watch are antithesis ("not X, but Y"), tricolon, isocolon, and unearned profundity. A balanced two-part closer at the end of every section is this failure, not metaphor substitution — so it is out of scope for the swap table above, and needs naming as a habit instead.

Model note. Claude Fable 5.1 writes denser prose than Fable 5 — longer sentences, fewer paragraph breaks. Defining the anti-pattern in a user message (preferred) or the system prompt is the documented mitigation.

For False Precision

A figure in the abstract or introduction is SUMMARISING; a figure beside its table is REPORTING. Only the second earns its decimal places. Carrying four of them into a summary implies a precision the estimate does not have and reads as machine output.

AI PatternHuman Alternative
"a rate of 1.3771 percent""about one and a half percent"
"roughly ten times the 2.7361 percent rate""roughly ten times the abstention-wide rate"
"6,612 of 575,553 testable items — 1.15 percent — fall below""about one percent of testable items fall below"
"covering 85.63 percent of the testable universe""covering roughly six in seven of the testable universe"
"the 28.3328 percent figure""that figure"

Exceptions the rule already carries, and which must stay exceptions: exact values inside a table, figure caption, code block or footnote; years, statutory and rule cites, docket, page and version numbers; and money where the cents are the point.

For Promotional Language

AI PatternHuman Alternative
"groundbreaking"describe what it actually does
"revolutionary"compare to what came before
"cutting-edge"specify the technology
"transformative"show the transformation with evidence

Reference Files

For detailed patterns and extensive examples, consult:

THE "ENFORCED BY" COLUMN IS THE POINT OF THIS TABLE. Eleven chapters listed as reference material reads as eleven chapters of coverage; it is not. Four of them have no executable module and are marked reference-only here so nobody has to rediscover the gap by shipping a draft through a clean review pass. (That is exactly how chapter 05's boldface entry went unenforced until v5.134.0 — see docs/investigations/2026-08-05_emphasis-enforcement.md.)

FileContentsEnforced by
references/_index.mdOverview and quick screening guide—
references/01-puffery-and-exaggeration.md"Stands as", superficial analyseswikipedia-puffery
references/02-promotional-language.md"Rich tapestry", disclaimerswikipedia-promotional
references/03-structural-patterns.mdSection summaries, negative parallelismswikipedia-structural
references/04-stylistic-quirks.mdElegant variation, false rangesreference only — regexable, but must clear the ai-tic corpus gate first
references/05-formatting-and-typography.mdBoldface, em dashes, emojisemphasis, formatting, em-dash (boldface/emoji since v5.134.0); inline-header lists partly
references/06-communication-patterns.mdSubject lines, "I hope this helps"wikipedia-communication
references/07-template-artifacts.mdMad Libs patterns, placeholderswikipedia-template-artifacts
references/08-markup-issues.mdMarkdown vs wikitext confusionreference only — wikitext-specific; no in-repo analogue
references/09-chatgpt-specific-artifacts.mdturn0search, oaicitewikipedia-chatgpt-artifacts
references/10-citation-problems.mdHallucinated DOIs, dead linksreference only — covered better by skills/cite-check + skills/source-verify and the cite-fidelity-* constraints, which check citations against sources rather than pattern-matching them
references/11-meta-indicators.mdAbrupt cutoffs, style discrepanciesreference only — judgement; lives in the user-agents/writing-reviewer.md rubric

Every named system is a ${CLAUDE_PLUGIN_ROOT}/scripts/prose-audit.py system, so a finding from it carries a span id. A chapter marked reference only produces no span, which by THE READER'S RULE means anything it surfaces is judgement, not a deterministic finding.

Automatic Detection

This plugin includes PostToolUse hooks that automatically scan Write/Edit output for anti-patterns. When patterns are detected:

  1. Hook emits a warning with specific patterns found
  2. Claude immediately revises the content
  3. Revision removes or replaces flagged patterns

The hook checks for all CRITICAL and HIGH severity patterns automatically.

Review Facts

  • A user's style request ("make it punchy", "professional tone") is a request for an outcome, not for AI-smell — delivering it with puffery, hedges, or bold-emphasis patterns intact ships text that reads as obviously AI-generated and damages the user's credibility.
  • Skimming is not checking. The check is sentence-by-sentence against the pattern list; a skim that lets puffery pass because flagging it felt pedantic presents unreviewed text as reviewed — an unverified claim.

Key Principles

From Wikipedia's guide:

  1. These are signs, not proof - Multiple indicators strengthen the case
  2. Context matters - Some patterns appear in human writing too
  3. Focus on deeper issues - Surface defects point to synthesis and quality problems
  4. Don't rely on detection tools - Human judgment required

Related Skills

  • /writing - Core writing principles from Elements of Style
  • /writing-legal - Legal writing (Phase 2)
  • /writing-econ - Economics writing (Phase 2)

Batch router

Picks the provider for one batch job and hands off. It never submits: the handoff is to claude-batch, openai-batch or gemini-vertex, which own the request format, the pilot and the submission.

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

## IRON LAW: BATCH BY DEFAULT, NEVER SYNC — AND NO FULL PAID RUN WITHOUT EDDY'S YES

Every bulk LLM job goes through a provider's Batch API (half price), never a loop of synchronous calls and never an interactive-agent fan-out; one synchronous verify per job is the only sync call. The full run waits for the pilot's results --input exit 0 and a message from Eddy approving the cost this router printed. A sync loop that "already works" still costs double, and a fan-out of agents bills the shared Claude accounts per document.

Route

uv run ${CLAUDE_SKILL_DIR}/scripts/route_batch.py --requests 5000 --in-tokens 3000 --out-tokens 300 \
  [--tier cheap|mid|top|frontier | --model <id>] [--needs vision|pdf|web-search|long-context] \
  [--balance uva=200 --balance nyu=200] [--json]
uv run ${CLAUDE_SKILL_DIR}/scripts/route_batch.py --jsonl requests.jsonl --out-tokens 300   # measured

It prints a ranked table with a reason per row and the next command. The ranking:

  1. Prepaid credit that expires soonest, covers the estimated cost +20%, and has a usable key.
  2. Other prepaid credit (short balance, or expiry within 24h).
  3. Pay-per-token, cheapest model that fits; Gemini Vertex wins a tie and is the default fallback.
  4. Blocked: no key, or the batch path lacks a needed capability. Listed, with why.

Over a provider's per-batch limit (Claude 100k / 256 MB, OpenAI 50k / 200 MB, Gemini 200k) the row says how many shards. Details, prices and limits: routing rules.

Then invoke the skill the top row names (workflows:claude-batch, workflows:openai-batch or workflows:gemini-vertex) and follow it: validate exit 0 → one verify → 5–10 row pilot whose results --input exits 0 → Eddy's yes to the printed cost → full run → results --input exit 0.

Facts

  • Balances are never stored. There is no balance API for Claude credits; the Console is the only source. Pass --balance <account>=<usd> from what Eddy or the Console reported today; without it the router assumes at most the monthly credit and prints balance UNVERIFIED. Writing a balance into accounts.toml creates a number that is silently wrong a day later.
  • Accounts are config, at $BATCH_ACCOUNTS (default ~/.config/batch-router/accounts.toml; shape in references/accounts.example.toml). This repo is public: org ids, emails and key paths never go in it.
  • OpenAI is pay-per-token until its account says credit_linked = true: the ChatGPT Pro credit page never names Batch. Its batches can queue for hours; not for same-day results.
  • Credits expire at cycle end with no rollover. An org whose credit lapses in four days with $200 unspent is worth more than any per-token discount, which is why expiry outranks price.
  • Web search is priced per search, not per row: Claude and OpenAI $10/1k, Gemini $14/1k after 5k free a month. The router assumes 2 searches per row (--searches-per-row); a grounded Gemini run once cost $390 in searches against $30 in tokens.

Red flags — STOP

About toDo instead
Send bulk work straight to gemini-vertex out of habitRun the router; a prepaid Claude credit may be expiring unused
Loop synchronous API calls over the rowsBatch; the only sync call is the one verify
Fan out agents or farm rows, one per documentOne batch job over pre-cut inputs
Submit on the router's word aloneThe router never submits; follow the provider skill's pilot and get the go-ahead
Quote an UNVERIFIED balance as the real oneSay it is assumed, or ask Eddy for the Console figure
Create an API key for an org that shows no keyAsk Eddy; he adds the key source to accounts.toml

Bib-manage

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Owns the bibliography as an index: every entry pointing at the right PDF, and that PDF being the version of record.

pincite CONSUMES this index — it resolves footnote → source through file = {...} and through nothing else. So the two failures this skill exists to prevent are both silent in pincite: an entry with no file is a pincite that is never attempted, and an entry pointing at a preprint is a pincite that is confidently wrong.

audit    entries with/without file, files missing on disk, entries without a
         DOI, PDFs no entry claims          <- run this first
link     citekey -> PDF, written back as file = {...}
version  is each entry's PDF the published version, or a preprint/proof?
doi      backfill missing DOIs from Crossref, verified against the bib's pages,
         and the PAGE RANGE that same verified record already carries
reconcile
         rewrite the entry's OWN FIELDS from a better source:
         Paperpile > Crossref > the extracted value. Never `file`, never a key
fedreg   fetch the Federal Register PDFs the entries cite

CLI

B="${CLAUDE_SKILL_DIR}/scripts/bibman.py"

python3 "$B" audit   --root .                 # start here
python3 "$B" link    --root . --dry-run       # inspect the mapping first
python3 "$B" link    --root . --backup        # then write
python3 "$B" version --root .
python3 "$B" doi     --root . --dry-run --only aggarwal2015
python3 "$B" reconcile --root . --dry-run            # read the full set FIRST
python3 "$B" reconcile --root . --backup --fields doi,pages
python3 "$B" fedreg  --root . --dry-run

Every path is a flag — --bib, --pdf-dir, --fedreg-dir — defaulting to the layout pincite uses, so the tool works on any manuscript. --only scopes version and doi to named citekeys; --dry-run reports without writing; --backup leaves a .bak; --strict makes version exit 1 on any entry that is not the version of record.

--fields scopes reconcile to a subset of author,title,journal,volume,pages, year,doi,publisher; --pp-index points at the Paperpile CLI's index cache.

audit, link and version never touch the network. doi and fedreg do. reconcile reads Paperpile locally and reaches Crossref only for what Paperpile does not cover.

Facts

  • The link threshold depends on the YEAR, and that is the whole of its accuracy. Two matchers run in order: PDF stem == citekey (48 of 79 on the test bibliography), then Paperpile-style Author Year - Title.pdf on first-author surname plus title similarity (the other 31). Same year, the year corroborates and 0.55 is enough. DIFFERENT year — a working paper against its published version, Lund 2017 → lund2018 — demand ≥ 0.95, because only the title separates two papers by one author. At 0.55 with the year ignored, alexander2010 ("Interim News and the Role of Proxy Voting Advice") claims the PDF "The role of advisory services in proxy voting" at 0.52. That is a wrong file in the bib, which is the one input pincite trusts absolutely — a mis-link there sends a footnote to another paper and can verify CONFIRMED while doing it.

  • Never force a match; REPORT what stays unmapped. On the test bibliography four PDFs stay unclaimed and every one is correct: Alexander et al. 2009 (0.52), Cremers and Romano 2011 whose filename is truncated mid-title by the filesystem (...proxy voting on co ... ion plans..., best match 0.89 against a 2007 entry), and two whose filenames are not Author Year - Title at all. An unmapped PDF costs a minute of human attention. A forced one costs a wrong citation nobody looks at again.

  • Paperpile writes SURNAMES ONLY, so the first-author surname is the FIRST token. Barzuza et al., Bebchuk and Hirst, Rock. Taking the last token — the correct habit for a bibtex author field — reads "Barzuza et al." as an author named "al." and unmapped 14 of 83 PDFs here before it was caught.

  • A 6-digit pages is an Elsevier ARTICLE NUMBER, not a page. J. Fin. Econ. 154 (2024) 103810 is paginated 1–N in its own PDF and IS the version of record. Range-testing it reports every such entry as a preprint — shu2024 and cabezon2025 are both genuinely published and both false alarms without this exception.

  • A publisher's own PDF can be an ADVANCE-ACCESS PROOF. OUP served iliev2021 as pages 1–49 instead of 5581+, v 00 n 0, internal name RFS-OP-….tex — authenticated, current, and useless for a pincite. Flag when the range test fails AND page 1 carries no nonzero volume designation. A successful fetch never means the pagination problem is solved.

  • Do NOT identify the version of record by matching the journal name against page 1. Bib journal names are Bluebook abbreviations ("Rev. Fin. Stud.") that never appear in a masthead ("The Review of Financial Studies"). That test flagged 46 of 61 entries and is worthless. The derived printed-page range against the bib's start page is the test that works: 5 of 79 flagged, all five real preprints.

  • The page derivation is pincite's, imported not copied. pages_of, page_numbers and page_offset are pure functions and pincite does nothing at import time. Copying them lets the two skills' notion of "what page is this" drift, and a version verdict that disagreed with pincite's own would be worse than no verdict.

  • A bibliographic title query returns the SSRN PREPRINT DOI. Measured on aggarwal2015: title alone returns 10.2139/ssrn.2120797, 10.2139/ssrn.1688993, 10.2139/ssrn.2023480 — the top three hits are all SSRN, none carries a page. Query with title + journal + volume + pages, then VERIFY the returned record's start page equals the bib's pages before accepting: that yields 10.1111/jofi.12284, page 2309–2346 against the bib's 2309. A DOI that fails the page check is reported, never written — it is usually the preprint of the very work the entry means.

  • The verified Crossref record already carries the END page; doi writes it. The match test was always record start page == bib start page, so the range that passes it is the same work's own span — 10.1111/jofi.12284 returns 2309-2346 against a bib holding 2309. Written as pages = {2309--2346}. Four guards, and each blocks a different way of destroying a value that was already right: @article only (a @misc press release's pages = {1} is not an article span); a bare start page only, so a Bluebook pincite someone recorded there survives — on OPV easterbrook1991 (66-67) and jackson2019 (849, 851--52) are the only range-like values in 197 entries and BOTH are pincites, not spans; never an ARTICLE_NUMBER; and the Crossref range must START at the bib's page, re-checked at write time so the DOI test and the range test can never disagree. Measured on OPV: 55 of the 64 bare start pages were eligible, 30 resolved, and the 25 that did not are law reviews, which Crossref indexes thinly.

  • An end page turns version from one test into three. With only a start page the derived span can be asked one question, lo <= start <= hi. With the end page it answers whether the span's BOTH endpoints fall inside, and whether the PDF's page count can hold the printed span at all — n < span is impossible for a version of record, and n > span + 2 (beyond a cover sheet) is a different artifact. aggarwal2015 is a 57-page PDF claiming a 38-page span, ertimur2018 64 pages for 21, hayne2019 59 for 43; each now fails on three counts instead of one, with no header parsing. Entries with no end page keep the start-only path unchanged — on OPV that is still most of them, and all 77 verdicts were byte-identical before and after the backfill.

  • reconcile's precedence is Paperpile > Crossref > the extracted value, and it is implemented as source ORDER, not a merge. An entry confidently matched in Paperpile is never queried against Crossref, so the stronger source cannot be contradicted by the weaker one. On OPV — a bib whose header reads Auto-extracted from sources.md via Gemini Vertex Batch, where every field was written by a model reading a markdown list and no entry carried a DOI — Paperpile covered 33 of 197, Crossref 24, and 140 had neither. Every change prints as citekey.field: old -> new [source]; --dry-run writes nothing and is byte-identical run to run.

  • A CONFIDENT Paperpile match needs the surname, the year AND the title, and an entry with no author needs the year EXACTLY. Surname equal, |Δyear| ≤ 1, title ≥ 0.90 same-year / ≥ 0.95 across a year — the same asymmetry link uses. With NO author there is nothing to corroborate with, and 1.00-vs-0.97 is not enough: secproxyadvice2019 (the SEC's own 2019 release, 84 Fed. Reg. 66,518) matched a 2020 COMMENT LETTER about it at 0.98, whose title is the release's title with RE: in front. Requiring Δyear == 0 there kills it; a document with no author has a fixed date and no working-paper drift to forgive. Two candidates within 0.02 of each other are AMBIGUOUS — reported, never picked, because a wrong library match rewrites author, title, journal and DOI at once.

  • Crossref's start-page verification is @article ONLY, because a pincite is shaped exactly like a page range. Nothing in the VALUE 66-67 says whether it is pages 66 to 67 of an article or a Bluebook pincite into a book. Reading easterbrook1991's 66 as a start page made Crossref accept 10.1111/j.1468-0319.1991.tb00155.x, titled "Discussion Papers", because it happens to begin on page 66 — a proposal to overwrite the title of The Economic Structure of Corporate Law. The entry TYPE is what separates them, and range_blocked already said so. This guard lives in crossref_pick, so doi gets it too.

  • A Paperpile record can be the SSRN PREPRINT of a published article, and taking its fields REGRESSES the citation. choi2009 (S. Cal. L. Rev. 82:649) and bebchuk2019a (B.U. L. Rev. 99:721) are both published; both library records carry journal = SSRN Electron. J. and a 10.2139/ssrn.* DOI. Precedence orders sources by reliability — it does not authorise replacing a real journal name with a preprint server's placeholder. When the source record is SSRN-shaped and the entry ALREADY names a journal, journal, volume, pages, year, publisher and doi are all refused and reported. A genuine working paper has no journal to protect, so it is unaffected.

  • Precedence does not authorise a THINNER rendering of a name. Paperpile answered J Fisch and A Hamdani and S D Solomon for fisch2019's Jill E. Fisch and Assaf Hamdani and Steven Davidoff Solomon, and Crossref answered VICENTE CUÑAT and MIREIA GINE and MARIA GUADALUPE for Vicente Cuñat and Mireia Giné and Maria Guadalupe — initials for given names, and full caps with the accents gone. Both are refused and reported. Note that str.isupper() on the whole author string is ALWAYS False: the and separators are lowercase, so the test must run on the names alone.

  • reconcile never touches file and never touches a citekey. file is not in RECONCILE_FIELDS and is refused by name in field_blocked — the PDF mapping belongs to link, which owns the matcher and its thresholds. The keys are load-bearing for something outside this repo: paper/typst/opv-body.typ references entries through #ref(<key>) and that project's scripts/resolve_refs.py resolves supra numbers through them, so a renamed key silently breaks the manuscript. After writing, the file is re-parsed and the key LIST compared to what it was; a mismatch exits non-zero pointing at the .bak.

  • federalregister.gov bot-blocks every request and lands on unblock.federalregister.gov. Fetch from govinfo's link service, https://www.govinfo.gov/link/fr/<vol>/<page>?link-type=pdf, into --fedreg-dir as fedreg-<vol>-<page>.pdf. Pre-2000 volumes return the WHOLE ISSUE, 200+ MB — trim with qpdf --empty --pages IN a-b -- OUT. The SEC's own release PDF of the same document has different pagination, so it is the wrong source for a Fed. Reg. pincite.

Red flags — STOP

About toWhy wrongDo instead
Run pincite on a bib you have not auditedEntries with no file are skipped silently; here that was 118 of 197audit first
Lower the 0.95 cross-year threshold to map "just one more" PDFAt 0.55 alexander2010 claims the wrong paper at 0.52Leave it unmapped and link it by hand
Hand-add a file to make an unmapped PDF go awayThe unmapped list is the output, not a failureRead the reason it prints first
Report a 6-digit pages entry as a preprintIt is an Elsevier article number; the PDF is the version of recordTrust the article-number skip
Check the version of record by finding the journal name on page 1Bluebook abbreviations never match mastheads — 46 of 61 flaggedRange-test the derived pages against the bib's start page
Accept a Crossref DOI because the title matchedTitle-only returns the SSRN preprint; 6 of 7 test entriesRequire the start page to equal the bib's pages
Write a page range over a pages that already holds oneIt is a Bluebook pincite, not a span — both such values on OPV wereLeave it; doi reports it as SKIPPED
Give a @misc or @book entry an article spanpages = {1} on a press release is not page 1 of an articleThe @article guard; read the SKIPPED list
Trust a freshly fetched publisher PDFIt can be an advance-access proof paginated 1–NRe-run version on anything fetched
Use the SEC release PDF for a Fed. Reg. citeDifferent pagination from the one the cite usesfedreg via govinfo link/fr/<vol>/<page>
Run link without --dry-run on a bib you have not inspectedIt writes into 197 entries--dry-run first, then --backup
Run reconcile without reading the full --dry-run setIt rewrote 163 fields across 47 entries on OPVRead every line, then --backup; --fields to scope
Take a Paperpile record's journal when the entry already names one and the record is SSRNSSRN Electron. J. over B.U. L. Rev. is a regression, not a correctionThe SSRN guard; read the CONFLICTS list
Accept a Paperpile match on a no-author entry a year apartThe SEC's 2019 release matched a 2020 comment letter at 0.98Δyear == 0 when there is no surname to corroborate
Let Crossref verify a non-@article by its start page66-67 is a pincite, and "Discussion Papers" also starts on page 66@article only — the type is what separates them
Add file to RECONCILE_FIELDS "since it is a field too"The PDF mapping is link's, matcher and thresholds includedLeave it; field_blocked refuses it by name

Bloomberg Law EDGAR search

!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

A Bloomberg-specific profile of ui-json-capture: drive the logged-in browser at human pace and keep the JSON the page already receives. Load browser-automation (port 9222, mcp__chrome-devtools__*) and read ui-json-capture for the procedure and its Red Flags.

**Never call the JSON API yourself** (`fetch`, `curl`, replayed POSTs) — bulk use of it is plainly against Bloomberg's terms. Click the UI; capture what it returns. Before paging at all: read the facet counts on page 1, and prefer the results list's own CSV export (`allow_csv: true` in the response) when a filtered slice fits its cap.

Why this route

Measured against SEC EFTS, WRDS web and wrds_sec_search: Bloomberg matches per exhibit (option "Exhibits Only"), runs unstemmed (stemmed_search: false), tolerates extraction glitches ("Non -Competition" still matched), and has Terms & Connectors: "phrase", ^"case-sensitive", ! and * wildcards, N/x, NP/x, S/, P/, ATLx(), ATMx(). Coverage from 1994; ~19.6M EDGAR documents.

Login

The browser is often signed out. Credentials are the 1Password item Bloomberg Law (UVA) (vault "Shared with Agents"); handle them only inside a dispatched agent, never in the main thread.

  • Fields: input#username, input#password. A bare #username matches the <m-text-field> wrapper first. Enter Input.insertText after focusing; React ignores .value.
  • Enter does not submit. Click the button whose text is Sign In; it lands on bloomberglaw.com/start.

Search

  • Form: https://www.bloomberglaw.com/start#advanced-search/features_edgar_all_search/features_edgar_all_search (Keywords; Apply To = Forms & Exhibits / Forms Only / Exhibits Only; Forms; Exhibits; Exhibit title; filer, CIK, company, location, SIC, file number). The favorite link may need one reload.
  • Install scripts/01-install-hook.js before clicking Search, so page 1 is captured.

Wire facts (observed, not for calling)

  • POST /product/blaw/api/v1/search/criteria, body {"criteria":{"model":"features_edgar_all_search","term":…,"keywords_apply_to":"edgar_exhibit","page_size":50,"page":N}}. Page 1 is the form submit; page N comes from the pager. Filter clicks hit the same endpoint with new criteria; the hook keys each page by criteria-minus-page, so check __cap.queries holds exactly one query before assembling.
  • Response: results_page.components.documents.items[] (50 per page; id, title like "WOLFSPEED, INC.; Form 10-K; EX-10.21", date, metadata filer, subtitles file number / filed / period / form, extracts with <mark> hits); .remote_count total (the UI shows "1000+"); components.facets.items[] (form family, form type, exhibit type, company, SIC) with counts.
  • Paging is an in-app route change (URL /product/blaw/search/results/<criteria_id>, same document), so the hook persists. Next control: svg[data-testid="search-results-next-page"], disabled="true" on the last page; click it with a dispatched MouseEvent.

Capture

scripts/01-install-hook.js and scripts/02-run-pager.js here; 03-status.js, 04-dump.js, 05-stop.js and assemble.py from ui-json-capture/scripts/ unchanged:

python3 ~/.claude/skills/workflows/skills/ui-json-capture/scripts/assemble.py \
  --batches scratch/blaw --glob 'batch_*.json' --out data/raw/blaw_hits.jsonl \
  --expect-total <remote_count> --page-size 50 --id-field id

From hits to EDGAR (accession, full text)

A hit carries no accession or CIK. scripts/resolve_exhibits.py resolves an assemble.py file:

uv run --script ${CLAUDE_SKILL_DIR}/scripts/resolve_exhibits.py --hits data/raw/blaw_hits.jsonl \
  --out-dir data/processed/blaw --cache-dir data/raw/edgar_sgml \
  --phrase "consulting agreement" --phrase "non-competition"    # phrases optional
  • Join on WRDS: keys go up as parameter arrays (pgdata is a hot standby: no TEMP tables) and join server-side to registrant (file no.), filing_view (date, form prefix) and wrds_forms (wrdsfname), so only ids and paths come back, never filing text.
  • Files via rclone: raw submissions from wrds:/wrds/sec/archives by wrdsfname, cached; over 10,000 files it uses edgar.md's server-side tar. --ignore-checksum is deliberate: the default md5 check opens an ssh session per file on the login node, and exhausted it at 16 transfers.
  • Parse locally: one pass over the SGML <DOCUMENT> blocks gives TYPE, DESCRIPTION, FILENAME and text for every hit, with no per-hit query.

hits_resolved.jsonl / .parquet: Bloomberg fields, match_status (unique / resolved_by_exhibit / ambiguous / unmatched), resolved_by (type → exact title form → DESCRIPTION → extract text), doc_match, accession, cik, filename, sec_url, text_path, chars, extract_found, phrase_verification. Title quirks it handles: normalized form drops /A (the title keeps it); EX-99: EX-99.1 is TYPE: DESCRIPTION; EX-10.6 2 is the DESCRIPTION of the second EX-10.6; Rule 425 Communications = 425.

Measured on 295 hits: WRDS join 1–27 s; 277 submissions = 2.62 GB by rclone in 27 s; parse + extract ~20 s. 294 resolved, 1 unmatched (CB/A: absent from the WRDS tables). Exhibit text equals sec.gov's document minus its SGML header line in 275/275 checked.

Run a first pass on extracts alone; send only positives and unsure rows to a full-text pass.

Filters

Results-page facets are <label> elements (click the label, not the a11y checkbox node); each click is a new query. Form type lands in criteria as "facets":{"normalized_form_type":["Form 8-K"], "form_family":[], "display_exhibit_type":[], "cik":[], "sic":[]}. A facet slice under 1,000 needs no date filter: capture it whole and filter locally. Date presets sit under [data-testid="date-selector-preset-dropdown-button"] (Any, Single date, Date range, Last 7/30 days, Last 12 months, Last 5 years) and land as e.g. "date_type":"last_5_years".

Verified: three-phrase query, Exhibits Only, Form 8-K → 356 hits, 8 pages captured by the pager in ~4 min, assemble.py exit 0 (no gaps, 356 unique ids).

Not yet verified

Whether paging continues past result 1,000; the CSV export's row cap; the field names for a custom date range.

Bluebook Footnote Audit Workflow

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Systematic Bluebook 21st edition compliance audit for law review manuscripts in DOCX format.

Announce: "Using bluebook-audit to run a systematic Bluebook compliance check."

Overview

Seven-phase linear workflow: Extract -> Check -> Report -> Correct -> Verify -> Archive -> Cross-Refs

/bluebook-audit  -> extract -> check -> report -> [USER REVIEWS] -> correct -> verify -> archive -> crossrefs
/bluebook-audit-fix -> diagnose -> route to {re-check, re-correct, re-verify}

Phase Summary

PhaseResponsibilityGate
ExtractParse DOCX -> structured JSON with formattingfootnotes_data.json exists, all FNs extracted
CheckMechanical checks → Gemini Batch per-footnote → Claude cross-footnote reviewaudit_findings.json exists, ALL FNs covered, three-layer merge complete
ReportPresent findings to user for reviewAUDIT_REPORT.md exists, user acknowledges
CorrectApply fixes to DOCX via lxmlCorrected DOCX exists, fix counts match
VerifyRe-scan to confirm fixes appliedZero remaining issues in re-scan
Archiveperma.cc URL archivingAll URLs archived, links written to DOCX
Cross-RefsConvert supra/infra notes to NOTEREF fieldsAll cross-refs are auto-updating fields

How to Start

  1. User provides a DOCX file path
  2. Workflow creates scratch/ directory for intermediate artifacts
  3. Proceeds through phases sequentially

Next Step

Read the entry command:

Read("commands/bluebook-audit.md")  # relative to this skill's base directory
## Iron Law: ALL Footnotes Must Be Checked

Every footnote in the document must be audited. No subsets. No sampling.

Auditing only "major-severity" footnotes or a random sample guarantees missed errors. The formatted Gemini audit must cover ALL footnotes, not just previously flagged ones.

Skipping footnotes is NOT HELPFUL — missed errors go to publication and embarrass the user.

## Iron Law: Formatted Text for Gemini Audit

NEVER send plain text to Gemini for typeface auditing. Always include formatting markup.

Plain text produces 10-20x false positives because Gemini cannot see what is already italic/small caps/roman. Inline markup (*italic*, [SC]small caps[/SC]) reduces false positives from ~400 to ~20 for a 239-footnote document — measured on gemini-2.5-flash, so on the current judgment role it is an estimate, not a measurement.

## Iron Law: Verify After Corrections

After applying corrections, ALWAYS re-run the scanner to verify fixes were applied.

NBSP characters, run boundaries, and cross-run text cause silent failures. A fix that "applied" in code may not have actually changed the DOCX. Re-scanning is mandatory.

## Iron Law: Mechanical Findings Override Gemini

Never drop a mechanical finding because Gemini didn't flag it.

Deterministic checks (signal italic, terminal periods, Id. chains) are 100% reliable. Gemini misses ~30% of signal formatting issues because it focuses on citation-level analysis and lacks a dedicated signal-checking output field. During merge/dedup, mechanical findings are authoritative for their rule categories. Gemini adds value only for judgment calls (source type classification, abbreviation tables).

Previous failure: Gemini reported FN103 as having only typeface issues on article titles, completely missing that "See also" was not italicized — which the mechanical checker caught.

## Iron Law: Source Type Classification Requires Human Review

Gemini consistently misclassifies non-standard source types. Never auto-fix Gemini's typeface suggestions for SEC releases, executive orders, working papers, or regulatory materials.

The hardest part of a Bluebook audit is determining the correct typeface for non-standard sources. Gemini defaults to "everything should be italic or small caps" but many source types are correctly roman:

  • SEC releases/rules/concept releases → roman (regulatory material, Rule 14.6)
  • Executive order titles → roman (Rule 14.7)
  • Working paper series designations → roman (parenthetical)
  • Company names in no-action letters → roman

The audit report MUST separate "verified fixes" (clear violations) from "judgment calls" (source type dependent) and include a "correct as-is" section documenting why Gemini's suggestions were rejected. See references/audit-patterns.md for the full source type reference table.

Previous failure: Gemini flagged 10+ items as needing italic/small caps that were actually correct as roman (SEC releases, exec orders, working paper designations). Without the source type reference table, these would have been incorrectly "fixed."

## Audit Facts
  • Gemini hallucinates citation formats and fabricates citation details. Its output is a lead sheet, not a verdict — run the mechanical checker independently, merge results, and spot-check before applying anything. A Gemini suggestion applied unverified is an unverified claim applied as a fix.
  • Fixes cascade: a correction in one phase creates new errors in dependent phases, and every phase catches a different error type. Re-run the affected scanner after every correction pass — "earlier phases were clean" says nothing about errors the fixes just introduced.
  • A hardcoded journal list and a previously-extracted sources.bib are both LAGGING records, and each one hid real errors in the same document: J. Corp. L. was not in JOURNAL_NAMES, and 19 of 53 article cites were missing from a bib extracted four months earlier. Validating a manuscript against either has the dependency backwards — it hides precisely the citations most recently reworked, which are the ones most likely to be wrong. Run bib_gaps.py BEFORE the small-caps check so the bib is current when the check reads it.
  • The bib's journal field only exists on @article entries, so it covers article periodicals and nothing else — 23 of one document's 54 small-capped names were newspapers, magazines, blogs and BOOK titles, which live in @misc/@book. Treating "scan the bib" as complete coverage silently exempts every book and newspaper in the manuscript.
  • One wrong supra reference invalidates the reader's trust in ALL footnotes. Sampling, eyeballing, or marking a footnote correct without checking the reporter/volume audits a subset and asserts compliance for the whole — a claim the audit never verified. Every footnote, every phase, regardless of document length.
## Delete & Restart

If you sent plain text to Gemini instead of formatted text with footnote markers, DELETE the results and START OVER with properly formatted input. Gemini cannot audit what it cannot parse.

Phase 6: Archive

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Archive all non-permanent URLs in footnotes via perma.cc API.

What This Phase Does

  1. Extract URL inventory from footnotes data
  2. Deduplicate URLs (same URL across multiple footnotes)
  3. Archive each URL via perma.cc API
  4. Write perma.cc links back to DOCX footnotes
  5. Save progress after each successful archive

Prerequisites

  • Perma.cc API key in .env file (project-level or user home directory)
  • For institutional accounts: organization folder ID (unlimited archives)
  • Free accounts: 10 links/month limit

Script

uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/permacc_archive.py" --docx <path> --data scratch/footnotes_data.json

Institutional Account Setup

# Find your organization and folder ID
curl -H "Authorization: ApiKey YOUR_KEY" https://api.perma.cc/v1/organizations/

Use the folder parameter when creating archives:

requests.post("https://api.perma.cc/v1/archives/", json={
    "url": url,
    "folder": FOLDER_ID,  # enables institutional limits
})

Gate: Exit Archive

  • All non-perma.cc URLs archived
  • perma.cc links written to DOCX
  • scratch/permacc_archives.json contains all mappings

Next Phase

Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-crossrefs/SKILL.md and follow its instructions.

Phase 2: Check (Mechanical + AI Audit)

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Two-stage checking: Python mechanical checks catch definite errors; Gemini batch audit catches judgment-call issues.

Stage 2a: Mechanical Checks (Python)

uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/scan_formatting.py" --docx path/to/file.docx

Checks performed on ALL footnotes:

  1. Journal name small caps - Comprehensive pattern list (law reviews, finance journals, newspapers, periodicals, forums)
  2. Book title small caps - Detect italic book titles that should be small caps
  3. Id. chain validation - Rule 4.1 mechanical check (single-source predecessor)
  4. Signal italic consistency - All signals (see, cf., e.g.) must be italic
  5. Terminal period - Every footnote must end with a period
  6. Hereinafter consistency - Defined at first citation, used consistently after
  7. Author name supra format - Text before *supra* should be roman, not italic (unless it's a case name short form). Catches *Manne, supra* → should be Manne, *supra*
  8. Italic spillover - Trailing/leading spaces inside italic or small caps runs (e.g., *supra * should be *supra* ). These don't affect Word display but cause Gemini misparses

Stage 2a-ii: Bibliography drift

uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/bib_gaps.py" \
  --docx path/to/file.docx --bib paper/references/sources.bib --out scratch/bib_additions.bib

make_bib_from_docx.py bootstraps a bib for a manuscript that has none; this is the counterpart for every round after the first. It reports article cites present in the footnotes and absent from the bib, and emits BibTeX for review.

Parsing is deterministic, not model-driven — a fabricated bib entry is worse than a missing one because it looks answered. A title containing a comma ("The Millennial Corporation: Strong Stakeholders, Weak Managers") cannot be separated from an author list positionally, so review the emitted entries before merging.

NBSP Handling

DOCX uses non-breaking spaces (\xa0) in abbreviations. ALL search functions must handle both \x20 and \xa0:

  • No.\xa02106, Feb.\xa07, Oct.\xa021
  • Wall St.\xa0J., Corp.\xa0Governance

Stage 2b: Gemini Batch Formatted Audit

Default: Use Gemini Batch API (50% cheaper, handles all footnotes in one job). Fallback to sync calls if batch is unavailable.

Step 1: Extract formatted footnotes

uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/gemini_audit.py" --docx path/to/file.docx --extract-only

This outputs a JSON file mapping footnote numbers to formatted text with inline markup:

  • *text* = italic
  • [SC]text[/SC] = small caps
  • Plain text = roman

Step 2: Submit Gemini Batch Job

Build a JSONL file with one request per footnote, then submit via Batch API:

# Build JSONL (one line per footnote)
for fn_num, formatted_text in footnotes.items():
    request = {
        "custom_id": f"fn-{fn_num}",
        "body": {
            "contents": [{"parts": [{"text": PROMPT.format(fn_num=fn_num, formatted_text=formatted_text)}]}],
            "generationConfig": {"responseMimeType": "application/json", "temperature": 0.1}
        }
    }

# Submit batch job (see /gemini-vertex skill for full pattern)
# Use examples/batch_processor.py pattern — DO NOT guess API parameters

IMPORTANT: Follow the /gemini-vertex skill's Iron Law — read examples/batch_processor.py before writing batch code.

Fallback (sync): If batch is unavailable, use:

uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/gemini_audit.py" --docx path/to/file.docx

Gemini Prompt Focuses On:

  • Source type classification (case, statute, article, book, newspaper, working paper, hearing, letter, regulation)
  • Typeface correctness per Rule 2 (italic vs small caps vs roman)
  • Abbreviation correctness per T6/T13
  • Short form validity (cases must not use supra)
## Iron Law: Audit ALL Footnotes

The Gemini audit MUST cover every footnote, not a subset. Auditing only "major" or "flagged" footnotes guarantees missed errors.

Previous failure: Auditing 45 of 239 footnotes missed 41 journal names needing small caps.

Stage 2c: Claude Cross-Footnote Review

Never trust a single agent. After Gemini's per-footnote audit, Claude reviews the full set for cross-footnote patterns that per-footnote analysis misses.

Claude receives:

  1. ALL formatted footnotes (fits in 1M context at ~20-40K tokens)
  2. Gemini's per-footnote findings
  3. Mechanical check results

Claude reviews for:

  • Supra chain validity — does supra note 42 actually point to the right source?
  • Hereinafter consistency — defined at first citation? Used consistently after?
  • Repeated source type errors — if Gemini misclassified one SEC release, it likely missed them all
  • Id. chain context — predecessor footnote analysis that per-footnote Gemini can't see
  • Gemini false positive filtering — flag Gemini suggestions that are likely wrong (SEC releases, exec orders, working papers)

Output: Annotated version of Gemini findings with cross-footnote issues added and false positives flagged.

Red Flags

  • Sending plain text to Gemini → always include inline markup; plain text produces 10-20x false positives without formatting info.
  • Auditing a subset of footnotes → audit ALL footnotes; subsets guarantee missed errors.
  • Skipping NBSP variants in mechanical checks → always try both space types; NBSPs cause silent search failures.
  • Trusting Gemini results without the Claude cross-check → always run Stage 2c; per-footnote review misses cross-footnote patterns.
  • Trusting Claude review without mechanical checks → mechanical checks are authoritative for their categories; Claude misses deterministic patterns.
  • Skipping any stage → run all three (mechanical → Gemini → Claude); each catches a different error class.

Merging Three-Layer Findings

Priority order: Mechanical > Claude cross-review > Gemini per-footnote

Mechanical checks are authoritative for deterministic rules:

  • Signal italic formatting → trust mechanical checker (regex on run-level XML)
  • Terminal periods → trust mechanical checker
  • Id. chain validation → trust mechanical checker
  • Journal/book small caps patterns → trust mechanical checker

Claude cross-review is authoritative for:

  • Cross-footnote consistency (supra chains, hereinafter definitions)
  • Gemini false positive filtering (source type misclassifications)
  • Patterns across footnotes that per-footnote analysis misses

Gemini per-footnote is authoritative for:

  • Individual source type classification (when not overridden by Claude)
  • Abbreviation correctness (T6/T13 tables)
  • Short form validity

Never drop a mechanical finding because Gemini or Claude didn't flag it. The mechanical checker catches 100% of signal issues by design.

Gate: Exit Check

Before proceeding to Report phase:

  • scratch/audit_findings.json exists
  • Mechanical check results cover ALL footnotes
  • Gemini audit results cover ALL footnotes (verify count matches extract)
  • Claude cross-footnote review complete
  • Findings merged (mechanical > Claude > Gemini priority)
  • bib_gaps.py run FIRST; any emitted entries reviewed and merged
  • scan_formatting.py's journal list reconciled against the bib's journal values (see below)

Order matters. bib_gaps.py is what keeps sources.bib current, and the small-caps check is only as good as the list behind it. Run the gap check first and the bib supplies the article periodicals; run it last and the small-caps check is validating against a record that has already fallen behind — which is how J. Corp. L. survived a full audit.

Note what the bib does NOT cover: its journal field exists only on @article entries, so newspapers, magazines, blogs and BOOK titles are absent — 23 of this document's 54 small-capped names. Those stay the name list's job.

Next Phase

Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-report/SKILL.md and follow its instructions.

Phase 4: Correct

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Apply approved corrections to the DOCX file via lxml XML manipulation.

What This Phase Does

  1. Back up the original DOCX
  2. Apply corrections in order: a. Cross-reference resolution (fill [_] placeholders) b. Small caps for journal/periodical names (run-splitting) c. Small caps for book titles (italic -> small caps) d. Signal italic fixes e. Id. chain corrections f. Terminal period additions g. Other typeface fixes
  3. Write corrected DOCX

Run-Splitting Approach

When formatting a substring within a larger run:

  1. Find the run containing the target text
  2. Split into 3 runs: prefix (original) + target (new format) + suffix (original)
  3. Clone rPr from original run via deepcopy
  4. Add new formatting only to target run
  5. Set xml:space="preserve" on all <w:t> elements

Critical Gotchas

NBSP Variants

All search operations MUST handle \xa0 (non-breaking space):

def find_in_run(text, target):
    if target in text:
        return text.find(target)
    nbsp_target = target.replace(' ', '\xa0')
    if nbsp_target in text:
        return text.find(nbsp_target)
    # Try regex with [\s\xa0] for mixed
    import re
    pattern = re.escape(target).replace(r'\ ', r'[\s\xa0]')
    m = re.search(pattern, text)
    return m.start() if m else -1

footnoteRef Space Run Bug

The run after <w:footnoteRef/> often contains the full footnote text, not just a space. When replacing entire footnote content, keep ONLY the footnoteRef run and add an explicit space run.

Cross-Run Text

supra note 10 spans italic + roman runs. Target the specific run containing the text you need to change (e.g., just "note 10" in the roman run).

Multi-Split Footnotes

Some footnotes need multiple formatting changes in the same run (e.g., FN91: italic the letter title AND small-caps the annual report title, both in one roman run). Process splits sequentially left-to-right:

  1. First split creates 3 new runs from the original
  2. Second split finds target text in one of the NEW runs and splits again
  3. The find_run() search re-scans the footnote element each time, so it finds the new runs

Example: Jamie Dimon, Chairman & CEO Letter to Shareholders, in JPMorgan Chase & Co., 2023 Annual Report 1 (2024)

  • Split 1: "Chairman & CEO Letter to Shareholders" → italic (creates 3 runs)
  • Split 2: "JPMorgan Chase & Co., 2023 Annual Report" → small caps (splits the third run from Split 1)

Italic Spillover Cleanup

After all substantive fixes, clean up trailing/leading spaces in italic runs. Word displays these fine, but they cause Gemini annotation issues on re-audit:

# Find italic runs with trailing spaces
if text.endswith(' ') and is_italic:
    t.text = text.rstrip(' ')
    # Insert a new roman space run after
## Iron Law: Verify Every Fix

After each category of corrections, verify the fix was applied by reading back the modified XML. Silent failures from NBSP, run boundaries, or wrong-run targeting are common.

Skipping read-back verification is NOT HELPFUL — silent failures from NBSP or run boundaries mean the user's document still has errors.

Correction Facts

  • NBSP characters and run boundaries cause silent failures and wrong-run targeting — code that "applied" is not code that worked. A fix reported without XML read-back is an unverified claim presented as a correction.
  • Verification batched to the end can't attribute a failure to the fix that caused it; verify after each category.
  • Every footnote has a unique run structure — a pattern that worked on the last footnote proves nothing about this one.
  • One XML corruption destroys the whole document; skipping the backup gambles the user's manuscript to save seconds.

Gate: Exit Correct

Before proceeding to Verify phase:

  • Backup DOCX exists (original preserved)
  • Corrected DOCX exists
  • Per-category fix counts logged
  • Spot-check verification passed (at least 5 random fixes checked)

Next Phase

Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-verify/SKILL.md and follow its instructions.

Phase 7: Cross-References

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Convert hardcoded supra/infra note numbers to NOTEREF field codes that auto-update when footnotes are renumbered, then tie each cross-reference to a bibkey from the document project's references/sources.bib so the bibliography becomes the semantic identity layer.

Canonical 3-script pipeline

sources.bib ← make_bib_from_docx.py       # BOOTSTRAP (once per paper, Gemini)
                       ↓
docx ← create_crossrefs.py                # ONE-TIME conversion of hand-typed text
                       ↓
docx ← audit_crossref_targets.py --grep --apply        # DRIFT CORRECTION (deterministic)
                       ↓
docx ← bib_integrate.py --bib references/sources.bib   # BIBKEY-TAG SYNC (deterministic)
       (also pre-computes cached display values)

All four live in ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/.

Maintenance one-liner

After a Word session that added/edited footnotes (including new hand-typed <X>, supra note N text), run:

"${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/sync_crossrefs.sh" \
  draft.docx references/sources.bib

The shell wrapper runs create_crossrefs → audit_crossref_targets → bib_integrate in sequence. All three are idempotent and fully deterministic — re-running on a clean doc does nothing; ~5 seconds total runtime.

StageScriptGemini?When
Bootstrapmake_bib_from_docx.pyYes (one-off batch)Paper has hand-typed footnotes and no sources.bib yet
Initial conversioncreate_crossrefs.pyNoConvert hardcoded supra note N text → NOTEREF fields
Drift correctionaudit_crossref_targets.py --grepNoAfter footnote edits — fix any supras whose target drifted
Drift correction (ambiguous)audit_crossref_targets.py --batchYesOnly for refs grep can't resolve uniquely
Bibkey-tag syncbib_integrate.pyNoAfter cross-refs settle — rename bookmarks to _RefBib_<bibkey> and pre-compute cached display

Initial conversion (one-time)

Convert hardcoded <X>, supra note N text to NOTEREF cross-reference fields.

# Preview
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/create_crossrefs.py" --docx <path> --dry-run

# Apply
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/create_crossrefs.py" --docx <path>

Workflow

  1. Bootstrap the bib (only if no references/sources.bib yet):
    uv run --with lxml --with google-genai --with google-cloud-storage python3 \
      "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/make_bib_from_docx.py" \
      --docx <path> --out references/sources.bib
    
    Walks the docx footnotes, skips supra-only / bio fns, sends each first-cite candidate to Gemini Vertex Batch, emits BibTeX with note = {fnN} linking each entry to its source footnote.
  2. Dry run create_crossrefs — review the cross-reference map and bookmark plan
  3. Apply create_crossrefs — the script backs up the original before writing
  4. Audit + retarget — run audit_crossref_targets.py --grep --apply (see below). Hand-typed supra note N numbers go stale as footnotes are added or reordered; the create_crossrefs script faithfully bookmarks the literal N, which can be silently wrong for every reference. Grep resolves the easy cases deterministically; only ambiguous refs need --batch.
  5. Bibkey-tag — run bib_integrate.py --bib references/sources.bib --apply. Renames bookmarks to _RefBib_<bibkey>, retargets supras by bibkey, and pre-computes cached display values so the doc renders correctly on first read (no F9 needed).
  6. Verify in Word — open the DOCX, spot-check 5-10 supras. Cmd+A → fn+F9 (Mac) or Ctrl+A → F9 (Windows) to force-refresh fields if needed.
  7. Renumber test (optional) — add a footnote before a referenced target and confirm the supra numbers update.

audit_crossref_targets.py (drift correction, also runnable standalone)

# Mechanical audit only — flags refs whose surnames don't appear in the target footnote
uv run --with lxml python3 \
  "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/audit_crossref_targets.py" \
  --docx <path>

# Deterministic first pass — grep each reference for its first cite. No LLM cost.
#   On a 138-ref test set: resolved 62 refs unique with 98% accuracy, deferred
#   76 truly ambiguous cases. Always run this BEFORE invoking Gemini/Batch — it
#   strips the easy cases and saves LLM calls.
uv run --with lxml python3 \
  "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/audit_crossref_targets.py" \
  --docx <path> --grep --apply

# Grep + production LLM batch for the residue (recommended for full pipeline)
uv run --with lxml --with google-genai --with google-cloud-storage python3 \
  "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/audit_crossref_targets.py" \
  --docx <path> --grep --batch --apply --location global

# LLM-only paths (skip grep — coverage-risky for --gemini):
uv run … --docx <path> --batch --apply   # --model overrides the resolved role
uv run … --docx <path> --gemini --apply

bib_integrate.py (bibkey-tag sync, fully deterministic)

uv run --with lxml python3 \
  "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/bib_integrate.py" \
  --docx <path> --bib references/sources.bib --apply

What it does:

  1. Parses sources.bib.
  2. For each bib entry, derives signature tokens (author surnames + distinctive title words + institutional bibkey prefix) and finds the docx footnote whose body text contains all of them. This re-derives the fn_id → bibkey map from current docx content — sidesteps any drift between bib's note={fnN} tag and current numbering.
  3. For each supra/infra NOTEREF, picks the best matching bibkey (target-fn bonus + global token scoring).
  4. Creates _RefBib_<bibkey> bookmarks wrapping body footnoteReferences. Multi-cite footnotes get multiple bookmarks (one per source) — each pointing at the same body footnoteReference, so the displayed number stays the same; only the bookmark name differs by source.
  5. Rewrites each NOTEREF instr text to point at its bibkey-named bookmark.
  6. Pre-computes cached NOTEREF display values (using --bio-count, default 3 for *, †, ‡ author bios that don't consume display numbers when numRestart=eachSect is set).
  7. Emits SUPRA_BIB_AUDIT.md listing any supras whose surnames don't match the picked bibkey's haystack (author + title + howpublished + bibkey).

Fully deterministic — no LLM calls. Idempotent on re-runs.

make_bib_from_docx.py (bootstrap, Gemini)

uv run --with lxml --with google-genai --with google-cloud-storage python3 \
  "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/make_bib_from_docx.py" \
  --docx <path> --out references/sources.bib

Walks the docx footnotes, splits multi-cite footnotes on ;, sends each first-cite candidate to a Vertex AI Batch job (one independent request per citation; the bulk role by default). Emits BibTeX entries with bibkey conventions firstauthorlastYEAR for academic works and short slugs (gao2017, crs2024, secReg2020) for institutional sources.

Each entry includes note = {fnN} linking back to the source footnote.

Once sources.bib exists, maintain it directly — do not regenerate. Edits to the bib (typos, missing fields, new sources) should be in-place.

Vertex AI batch requires:

  • gcloud auth application-default login for ADC
  • --project (default $GOOGLE_CLOUD_PROJECT or activist-defense-nal)
  • --gcs-bucket (default $GEMINI_BATCH_BUCKET or nal-batch-extraction)
  • Location: us-central1 (default)

For gemini-3.x models, thinkingLevel: MINIMAL is set automatically — without it, batch responses silently return empty content (see /gemini-vertex SKILL gotcha 12).

Output written to <docx-dir>/scratch/:

  • crossref_audit.json — every cross-reference with current target + match status
  • crossref_remap.json — Gemini's proposed corrections (with confidence)
  • CROSSREF_AUDIT.md — human-readable diff report

The audit is safe to run standalone without re-running the rest of the bluebook-audit workflow. Use it any time create_crossrefs has been run and you want to validate that hand-typed supra/infra numbers were correct.

Recommended pipeline order

  1. --grep first. Cheapest, deterministic, ~98% precision on unique matches. Resolves ~half the refs at $0 cost.
  2. --batch (Vertex AI, per-ref independence) on the deferred set.
  3. Manual first-cite verification on remaining mechanical flags (Step 4 below).

How --grep works

A cross-reference reads <short-form identifier>, supra/infra note N. The identifier can be any Bluebook short form, not just author surnames:

  • author group: Hu, Malenko & Zytnick
  • hereinafter tag: GAO Report, CRS Report, Rosenberg, 2020 SEC Regulation
  • institutional doc: Best Practice Principles, Policies and Procedures, Senate Banking Committee Letter
  • case name: ISS v. SEC
  • title fragment: Choi, Fisch & Kahan, Power of Proxy Advisors

For each reference the script:

  1. Tokenizes the identifier (filters Bluebook signals like See, But see, E.g.).
  2. Strips every <X>, supra/infra note <N> phrase from each candidate footnote, so a footnote that merely cites the same identifier via supra doesn't count as the first cite.
  3. Searches every footnote — except the source — for one whose post-supra-strip text contains all the identifier tokens with word-boundary matches.
  4. If exactly one footnote matches → unique, apply deterministically.
  5. If multiple match → ambiguous, defer to LLM (picking the earliest is wrong ~17% of the time — typically when the identifier appears in passing in one footnote and as a first cite in another).
  6. If none match → no_match, defer to LLM (which has the catalog and can guess, or correctly say "not in document").

Step 4 (mandatory): Manual first-cite verification on residuals

Iron Law: Neither Gemini pass is fully trustworthy on its own. A two-batch consensus is high-confidence, but the residual disagreements ALWAYS require human first-cite verification. Do not stop after --apply.

The failure modes we observed on a real 248-footnote draft:

  • Single-batched call: model returns uniform "medium" confidence — a smoking-gun signal it didn't reason per-reference.
  • Per-reference Vertex Batch: model returns uniform "high" with detailed reasoning, but still confidently picks wrong targets when:
    • the cited work isn't in the doc (model hallucinates a plausible-sounding nearby match)
    • the first cite is buried mid-footnote in a See also or multi-cite list (model misses it and picks a different paper by the same author)
    • the reference is to a self-referencing pattern (model returns the source footnote itself)
  • Mechanical surname check: only inspects the leading 250 chars of the target footnote, so first cites buried after See/hereinafter/inline-discussion lead-ins are flagged as mismatches even when correct.
Recipe
  1. Compare both Gemini passes, taking the intersection where they agree as the high-confidence consensus. Apply that subset first.
  2. For each disagreement, search the doc for the first non-supra cite of the disputed surname / title fragment. The script's regex below catches mid-footnote occurrences:
    for fn in fns.findall('.//w:footnote'):
        txt = ''.join(t.text or '' for t in fn.iter('w:t'))
        if re.search(r'<surname-pattern>', txt) and 'supra' not in txt[:300].lower():
            print(fn.get('w:id'), txt[:200])
    
  3. Build a manual remap of (source_fn, surnames, current_bookmark) → correct_fn_id, save as crossref_remap_manual.json, and apply with --remap PATH --apply.
  4. For the "no first cite in document" residue (works the author cites without ever having introduced):
    • Search broadly first — the first cite is often mid-footnote in a See also/multi-cite list.
    • Only if truly absent: flag for an author edit. Write SUPRA_UNRESOLVED.md listing the source FN, surnames, and the search patterns tried.
  5. Final re-audit will still show "mechanical mismatches" for the false-positive cases — verify each by spot-checking the target footnote content. Do not retarget these.
Gate
  • Consensus retargets applied (intersection of --gemini and --batch)
  • Manual remap built and applied for disagreements
  • Each surviving "mismatch" flag spot-checked against actual target content
  • No supra/infra reference points at an unrelated footnote
  • SUPRA_UNRESOLVED.md written (or empty if all resolved)

What Gets Converted

PatternExampleResult
Single suprasupra note 42NOTEREF to FN42 bookmark
Single infrainfra note 188NOTEREF to FN188 bookmark
Rangeinfra notes 209-210Two NOTEREFs with separator
With pincitesupra note 42, at 15NOTEREF + roman , at 15
Existing NOTEREF(already converted)Skipped

What Is NOT Converted (Phase 2 — Future)

  • Part/Section references (supra Section I.A., infra Part III)
  • These require a heading-to-bookmark mapping strategy since heading numbering is partially auto-generated

Gate: Exit Cross-References

  • Dry run reviewed — cross-reference map is correct
  • NOTEREF fields created for all supra/infra note references
  • Target audit passed — audit_crossref_targets.py run + Step 4 manual verification complete; no NOTEREF points at an unrelated footnote
  • Backup DOCX exists
  • Word field update (Ctrl+A, F9) confirms correct numbers

Workflow Complete

Present final summary to user:

  • Total formatting corrections applied (from Phase 4)
  • Total URLs archived (from Phase 6)
  • Total NOTEREF fields created (from Phase 7)
  • Final DOCX file path

Phase 1: Extract Footnotes

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Parse the DOCX file and build structured data for all subsequent phases.

What This Phase Does

  1. Parse word/footnotes.xml via lxml
  2. Extract each footnote's runs with formatting flags (italic, small caps, bold)
  3. Parse word/_rels/footnotes.xml.rels for hyperlink URLs
  4. Build citation registry (hereinafter definitions, author-to-first-cite mapping)
  5. Resolve cross-references (supra note [_] placeholders)
  6. Extract all URLs for archiving inventory

Script

uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/extract_footnotes.py" --docx <path>

Output: scratch/footnotes_data.json

Gate: Exit Extract

Before proceeding to Check phase:

  • scratch/footnotes_data.json exists
  • Contains entries for ALL footnotes in the document (verify count)
  • Each entry has formatted_text field with inline markup
  • Citation registry has hereinafter definitions
  • URL inventory extracted

If footnote count doesn't match document: STOP. Investigate missing footnotes before proceeding.

Next Phase

Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-check/SKILL.md and follow its instructions.

Phase 3: Report

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Generate a human-readable audit report and present to the user for review before applying corrections.

What This Phase Does

  1. Merge three-layer findings: mechanical + Gemini per-footnote + Claude cross-footnote review (mechanical > Claude > Gemini priority — see audit-check merge rules)
  2. Categorize by issue type and severity
  3. Flag items needing manual review (low-confidence cross-refs, ambiguous citations)
  4. Generate scratch/AUDIT_REPORT.md
  5. Present summary to user

Three-Layer Merge Priority

Mechanical > Claude cross-review > Gemini per-footnote

  • Mechanical findings (signal italic, terminal periods, Id. chains, small caps patterns) are deterministic and must never be dropped
  • Claude cross-review adds cross-footnote patterns (supra chains, hereinafter consistency) and filters Gemini false positives
  • Gemini per-footnote adds individual source type classification and abbreviation checks

Report Structure

# Bluebook Audit Report

## Summary
- Total footnotes: N
- Clean: N (XX%)
- Issues found: N across M footnotes

## Fix Counts by Category
| Category | Count | Auto-fixable |
|----------|-------|-------------|
| Journal name small caps | N | Yes |
| Book title small caps | N | Yes |
| Cross-reference resolution | N | Yes (high confidence) |
| Id. chain errors | N | Partial |
| Signal formatting | N | Yes |
| Terminal periods | N | Yes |
| Typeface errors (other) | N | Manual |

## Issues by Footnote
[sorted by footnote number]

## Items Needing Manual Review
[low-confidence cross-refs, ambiguous citations, judgment calls]

## Correct As-Is (Gemini False Positives)
[Items Gemini flagged but are actually correct, with reasoning]
[Group by source type: SEC releases (roman), exec orders (roman), etc.]
[Reference: audit-patterns.md Source Type Typeface Reference table]

Why "Correct As-Is" Matters

Many Gemini suggestions are wrong — especially for non-standard source types (SEC releases, exec orders, working paper designations). Documenting WHY these are correct:

  1. Prevents re-flagging if someone re-runs the audit
  2. Forces the reviewer to consciously evaluate each judgment call
  3. Creates a record of the source type classification decisions

Gate: Exit Report

This is a user gate. The workflow pauses here.

  • scratch/AUDIT_REPORT.md exists
  • User has reviewed the report
  • User approves proceeding to corrections

Do NOT proceed to corrections without user acknowledgment.

Next Phase

After user approval: Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-correct/SKILL.md and follow its instructions.

Phase 5: Verify

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Re-scan the corrected DOCX to confirm all fixes were applied and no new issues were introduced.

What This Phase Does

  1. Re-extract all footnotes from the corrected DOCX
  2. Re-run mechanical checks (small caps scanner, signal checker, etc.)
  3. Compare findings against the original audit - all flagged issues should now be clean
  4. Report any remaining issues

Verification Checks

  1. Formatting scanner: Re-run scan_formatting.py --docx corrected.docx - expect 0 findings (or only known false positives like "Institutional Investors" in titles)
  2. Cross-reference check: All [_] placeholders should be resolved
  3. Signal formatting: All signals should be italic
  4. Terminal periods: All footnotes should end with periods
  5. Id. chains: All id. references should have single-source predecessors
  6. Gemini re-audit on fixed footnotes: Re-run gemini_audit.py --subset [all previously flagged FNs] to catch issues introduced by fixes or missed in the initial pass

Re-Scan Catches Real Issues

In the "Other People's Votes" audit, the Gemini re-scan on fixed footnotes caught a Wells Fargo press release title (FN206) that wasn't in the original "judgment call" list — it was in the same footnote as other fixes but wasn't flagged initially. The verify phase is not ceremonial; it finds real issues.

## Iron Law: Re-Scan Is Not Optional

The verify phase MUST re-run the scanner on the corrected DOCX. Skipping verification misses small-caps fixes.

If the re-scan finds issues, go back to Correct phase. Do NOT proceed to Archive with unresolved issues.

Red Flags

  • Skipping the re-scan because "all fixes applied" → run the scanner; silent fix failures are common.
  • Dismissing remaining findings as false positives → investigate each one; some "false positives" are real.
  • Proceeding to Archive with >0 real issues → fix them first; uncorrected issues persist forever.

Gate: Exit Verify

Before proceeding to Archive phase:

  • Re-scan completed on corrected DOCX
  • Zero remaining issues (or all remaining are confirmed false positives)
  • If issues found: returned to Correct phase and re-verified

Next Phase

Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-archive/SKILL.md and follow its instructions.

Bluebook Citation (22nd edition; 21st still reachable)

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Citation formatting for law reviews and legal scholarship per The Bluebook: A Uniform System of Citation (21st ed. 2020).

Announce: "I’m using the bluebook skill for citation formatting."

When to Use

Invoke this skill for:

  • Formatting case citations (federal, state, foreign)
  • Statutory and regulatory citations
  • Secondary sources (books, articles, treatises)
  • Short form citations (id., supra, hereinafter)
  • Introductory signals and parentheticals
  • Citation sentences vs. citation clauses

For legal writing style: Use /writing-legal skill (Volokh) For general writing: Use /writing skill (Strunk & White)

To RENDER these rules in a Typst manuscript: this skill states the rules; it does not implement them. docx-typst carries the implementation — assets/bluebook.typ is a #show cite: rule supplying the three things typst's built-in bibliography cannot (supra note N, small-caps reporters, shortjournal → container-title-short), because hayagriva is statically linked into the typst binary and renders none of them. Reach for it whenever a Typst document needs the short forms in references/short-forms.md to renumber themselves rather than be typed by hand.

## IRON LAW #1: NO CITATION WITHOUT VERIFICATION

If you haven’t verified EVERY element of a citation, DO NOT write it.

Before writing ANY citation:

  1. Verify case name spelling and procedural posture
  2. Verify reporter volume and page numbers
  3. Verify court and year
  4. Verify pinpoint page exists

Guessing reporter volumes or page numbers is NOT HELPFUL — the user publishes with wrong citations that fail verification. Period.

## IRON LAW #2: NO SHORT FORMS WITHOUT FULL CITATION FIRST

Id., supra, and hereinafter REQUIRE a preceding full citation.

Before using ANY short form:

  1. Locate the full citation in the document
  2. Verify no intervening citations (for id.)
  3. Verify the supra reference is unambiguous

Using id. after intervening citations creates ambiguity. Delete and cite in full.

## IRON LAW #3: FOOTNOTE VS. TEXT CITATION FORMAT

Law review citations use footnote format (Rule 1). Court documents use text format (Bluepages).

FOOTNOTE (law reviews):    Smith v. Jones, 500 U.S. 1, 5 (1991).
TEXT (court documents):    Smith v. Jones, 500 U.S. 1, 5 (1991)

FOOTNOTE (statutes):       18 U.S.C. § 1001 (2018).
TEXT (statutes):           18 U.S.C. § 1001 (2018)

If writing for a law review and using text format conventions, DELETE and reformat.

The Gate Function

Before writing ANY citation:

1. IDENTIFY → What type of source? (case, statute, article, book)
2. LOCATE   → Find the correct rule in Bluebook
3. VERIFY   → Confirm ALL elements (volume, page, court, year)
4. FORMAT   → Apply correct typeface and punctuation
5. CHECK    → Does this match examples in the rule?
6. WRITE    → Only after steps 1-5

Skipping any step produces unreliable citations.

Citation Facts

  • An intervening citation breaks id. — id. after an intervening cite is ambiguous and must become a full short form. Supra only works when the full citation it points to actually exists earlier in the document.
  • Signals are checked against Rule 1.2 examples, not intuition — a wrong signal misleads the reader about how the source supports the proposition.
  • Parentheticals explain the source's relevance; pinpoints prove the specific claim. A cite deferred ("I'll add the pinpoint later") ships without one.
  • Typeface (Rule 2) is mandatory, not stylistic. Abbreviations come from tables T6, T10, T12 — "common" or "obvious" abbreviations that don't match the tables fail cite-check.
  • "Pretty sure" about a reporter volume or page number means unverified — a guessed element presented as a citation is an unverified claim, and exact pinpoints are required.

Quick Reference: Common Citation Forms

Cases (Rule 10)

Full citation:
Brown v. Board of Education, 347 U.S. 483, 495 (1954).

Short form (same footnote or five footnotes with no intervening):
Id. at 496.

Short form (different footnote, no intervening):
Brown, 347 U.S. at 497.

Short form (intervening citations):
Brown v. Board of Education, 347 U.S. at 498.

Statutes (Rule 12)

Full citation:
42 U.S.C. § 1983 (2018).

Multiple sections:
42 U.S.C. §§ 1983-1985 (2018).

Short form:
§ 1983 or id. § 1984

Law Review Articles (Rule 16)

Full citation:
Cass R. Sunstein, *On the Expressive Function of Law*, 144 U. Pa. L. Rev. 2021, 2030 (1996).

Short form:
Sunstein, supra note 12, at 2035.

Books (Rule 15)

Full citation:
Richard A. Posner, Economic Analysis of Law 45 (9th ed. 2014).

Short form:
Posner, supra note 5, at 52.

Typeface Rules (Rule 2)

Source TypeLaw Review Format
Case namesItalics: Brown v. Board
Book titlesSmall caps: ECONOMIC ANALYSIS OF LAW
Article titlesItalics: On the Expressive Function
Journal namesSmall caps: U. PA. L. REV.
Periodical names (non-consecutively paginated)Italics: N.Y. Times
StatutesRoman: 42 U.S.C. § 1983

Introductory Signals (Rule 1.2)

SignalMeaningUse When
[no signal]Direct supportSource directly states proposition
SeeImplicit supportSource supports but doesn’t directly state
See, e.g.,One of severalMultiple sources support; citing representative
Cf.Analogous supportSource supports by analogy
Compare ... withComparisonSources illustrate through contrast
See generallyBackgroundSource provides helpful background
But seeContradictionSource contradicts proposition
ContraDirect contradictionSource directly contradicts

Signal Order (Rule 1.3)

Within a single citation sentence, signals appear in this order:

  1. [no signal]
  2. E.g.,
  3. Accord
  4. See
  5. See also
  6. Cf.
  7. Compare
  8. Contra
  9. But see
  10. But cf.
  11. See generally

Common Errors Checklist

Case Citations

  • Party names shortened properly (omit "Inc.", "Ltd." unless only identifier)
  • "United States" abbreviated to "U.S." (as party, not "United States of America")
  • Reporter abbreviation matches T1
  • Court identifier included unless obvious from reporter
  • Year is decision year, not argument year
  • Pinpoint included for specific propositions

Statutory Citations

  • Current official code used (not session laws for current statutes)
  • Section symbol (§) used, not "Section"
  • Space between § and number
  • Year is code edition year, not enactment year
  • Supplements cited when applicable

Short Forms

  • Full citation appears earlier in same document
  • Id. used only when no intervening citation
  • Supra refers to footnote number where full cite appears
  • Hereinafter defined in first full citation

Progressive Disclosure

For detailed rules, consult:

Reference Files

  • references/cases.md - Complete case citation rules (R. 10)
  • references/statutes.md - Statutory and regulatory citations (R. 12-14)
  • references/secondary-sources.md - Books, articles, treatises (R. 15-17)
  • references/short-forms.md - Id., supra, hereinafter rules (R. 4)
  • references/quotations.md - Block quotes, alterations, ellipses (R. 5)
  • references/signals-parentheticals.md - Signals, parentheticals, order (R. 1)
  • references/audit-patterns.md - Citation audit patterns and validation
  • references/abbreviations.md - Bluebook abbreviation tables
  • references/editions-21-to-22.md - What changed in the 22nd edition, from the publisher's preface plus rule-by-rule checks

Every reference file below is checked against the rule text. editions-21-to-22.md is not in the table because it is the SOURCE that verification was done against, not a subject of it.

filerulesstatus
quotations.md5verified vs 21e scan AND 22e; 6 errors corrected
signals-parentheticals.md1rebuilt from 22e; carries the new contrast signal
short-forms.md4rebuilt from 22e
cases.md10rebuilt from 22e
statutes.md12 (13-14 NOT in corpus)rebuilt from 22e; 13-14 flagged in place
secondary-sources.md15, 16, 17rebuilt from 22e
abbreviations.md10.2.2, 15.1(e), 16.1, T6rebuilt from 22e; T6 reproduced from the capture, table gaps flagged in place
audit-patterns.mdcross-cutting (1, 4, 10, 12, 15-18, T6)rebuilt from 22e; every check names its rule, untraceable checks marked in place

Five were then adversarially re-verified by separate agents against the same corpus, with instructions to find errors rather than agree: signals-parentheticals.md and short-forms.md came back with zero contradictions (every imperative traced to a governing sentence); cases.md, statutes.md and secondary-sources.md each had defects, all now corrected. quotations.md, abbreviations.md and audit-patterns.md had no second pass. Reports: scratch/bb22/verify/.

All eight were checked against subsection pages extracted verbatim from the official Bluebook Online 22nd edition; page cites in them are 22e pages. Claims the corpus did not cover are marked UNVERIFIED in place rather than left looking checked. Per-file reports: scratch/bb22/reports/. Rule 5 verdicts: scratch/bluebook-verify/REPORT.md.

What is still unchecked, and why. "Rebuilt from 22e" means every claim traces to the extracted corpus or is marked in place — it does not mean the whole rule surface was available. These gaps are real and are flagged inside the files rather than papered over:

gapstate of the captureconsequence
Table T10 (geographical terms)captured in full — T10.1 (22e pp. 340-42), T10.2 and T10.3 as separate subtable pagesits values may be stated
Table T13 (institutional names in periodical titles, 22e pp. 346-48)captured in full — the earlier empty capture used a wrong slug (t13-institutional-names-... rather than t13-periodicals)its values may be stated
Tables T1, T7, T11, T12 and the restnever fetchedcross-references to them are reported; their contents are not
Rules 6, 7, 8, 9, 11, 19never fetchedchecks resting on them are marked, not asserted. Rules 3, 5, 20, 21 and 23 have since been extracted

Nothing from these gaps has been filled from training knowledge. A claim depending on one is either absent or carries a marker within a line of itself.

Looking a rule up in the actual book

Best source: the official Bluebook Online, 22nd edition. UVA Law provides institutional access and the browser on CDP 9222 is already signed in. This is live publisher text — no OCR, no auth expiry, no retrieval tricks — and it is the source of record.

https://www.legalbluebook.com/bluebook/v22/rules/<n>-<slug>
https://www.legalbluebook.com/bluebook/v22/rules/<n>-<slug>/<n>-<m>-<slug>

e.g. /bluebook/v22/rules/5-quotations/5-2-alterations-and-quotations-within-quotations

Swap v22 for v21 to read the 21st edition; the site's own dropdown does the same. Drive it with the browser-automation skill (Linux → mcp__chrome-devtools__* on 9222). If that MCP server is not connected this session, raw CDP over http://127.0.0.1:9222/json/list works — and open a new tab rather than navigating the user's.

EDITION MATTERS, IN TWO WAYS. The 22nd is now live and is the default.

references/editions-21-to-22.md is the guide — the publisher's preface (which names changes in rules 1.2, 10.8.3, 12.4(f), 14.4, 15.1(d), 15.8, 18, 20.2.4, new 22 and 23, and tables T1.3, T1.5, T2, T6, T10) plus the rule-by-rule differences verified here. Read it before trusting anything in this skill against a 22e manuscript. Three things that bite immediately:

  • Rules 22 and 23 are NEW; the 21st stops at 21. Rules 1-21 keep their numbers.
  • Pagination moved — Rule 5 is 22e pp. 87-91 against 21e-scan pp. 103-108. A bare page cite is ambiguous; name the edition.
  • The preface's list is partial. Rule 5.1(a)(i) gained "single spaced" in the 22nd and the preface never mentions it. Absence from the preface is not evidence a rule is unchanged.

Pinpoint — the 21st edition, full scan

Pinpoint holds the full 21st edition — use it. A complete 394-page scan is in the Bluebook collection (b7425c3f3368f9c9), OCR'd by Google, searchable to the page. This is the authoritative lookup for any rule in this skill, and it is what references/quotations.md was finally verified against.

pinpoint search Bluebook "<terms>" --pages --order density --limit 60
pinpoint generate ask Bluebook "<question>"       # locates fast; do NOT quote its paraphrase

Four things learned the hard way, all of which cost a pass:

  • --order density, not the default. Document order means a bounded run stops early — Rule 5 sits at scan pp. 103-108 and a default-ordered run never reached past p. 71.
  • It OR-matches; a quoted phrase returns zero. Cast wide and filter locally with awk.
  • --no-dedupe when a page you know exists will not come back. Rule 1.5(b) at p. 86 returned nothing across four term sets at limits up to 120; --no-dedupe --limit 200 returned it in full.
  • Quote from --pages, never from generate ask. It locates well and paraphrases confidently — it produced a fluent synthesis of a capitalization test that no line in the book supports.

Auth expires within the hour (API error 7 (PermissionDenied)); pinpoint auth re-lifts the session from the CDP browser. Re-auth and retry — do not read an auth failure as "the rule is not in the book."

NotebookLM — narrow, and mostly superseded

The notebook (f70a9976-b443-43d5-b5fd-43ff86b2b700) holds a 53-page excerpt, not the book. Verified: the only full rule text is Rule 1 and 1.1-1.4 (pp. 51, 53) plus the Quick Reference tables. Everything else is cover scans and cropped strips of the printed book's thumb tabs — so "rule 10", "rule 12" appear as tab labels with nothing behind them, and a model asked what the PDF contains will read those tabs and name rules it cannot quote. It did exactly that before retracting under stricter questioning. It cannot answer on Rule 5 at all.

Refuse its web-research offer. On a rule it does not hold it replies "Would you like me to perform some web research?" — observed three times. Accepting turns a web search into something formatted as a source-grounded answer, which is how an unverified reference file gets written by an author who believes they consulted the book.

Prefer Pinpoint above. Reach for the notebook only for Rule 1 or the Quick Reference tables:

When to query the notebook:

  • Rule wording is ambiguous in reference files
  • Formatting international or specialized materials
  • Checking obscure abbreviations not in quick reference
  • Resolving conflicts between rules
  • Understanding historical changes from previous editions

When to Load References

Load the specific reference when:

  • Formatting an unfamiliar source type
  • Encountering edge cases (unpublished cases, foreign sources)
  • Checking state-specific reporter requirements
  • Working with complex statutory schemes
  • Formatting international materials

Integration

Use with /writing-legal for complete legal scholarship workflow:

  1. /bluebook formats citations correctly
  2. /writing-legal ensures argument structure and evidence handling
  3. /ai-anti-patterns catches AI writing indicators before submission

Delete & Restart Pattern

When to delete and restart:

  1. Citation uses guessed page numbers → Delete, verify source, cite with real numbers
  2. Id. follows intervening citation → Delete id., use full short form
  3. Wrong signal used → Delete, reread Rule 1.2, apply correct signal
  4. Typeface incorrect → Delete, apply Rule 2 typeface
  5. Abbreviation doesn’t match Bluebook tables → Delete, use table abbreviation

How to restart:

Old: See Smith v. Jones, 500 U.S. at 15. Id. at 20. [intervening cite] Id. at 25.
New: See Smith v. Jones, 500 U.S. at 15. Id. at 20. [intervening cite] Smith, 500 U.S. at 25.

The third cite cannot use id. after an intervening citation.

Citation Verification with Gemini File Search

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Scan pandoc-flavored markdown drafts for citations, upload source PDFs to a Gemini File Search store, and verify each citation is grounded in its source. Produces a structured REVIEW-CITES.md report.

Prerequisites

  • GOOGLE_API_KEY env var set (Google AI Studio; on this machine: export GOOGLE_API_KEY="$(cat $GEMINI_API_KEY_FILE)")
  • Bun runtime
  • rclone with a google-drive: remote configured (used to bypass Google Drive FUSE deadlocks)
  • python3 with pymupdf4llm installed (used for PDF text extraction in passage grounding)
  • readwise CLI installed and authenticated (for Readwise article export in source materialization)
  • One or more .bib files with file fields mapping bibkeys to PDF paths (e.g., Paperpile's paperpile.bib)

Source Materialization

Before running cite-check, materialize all sources locally:

cd ${CLAUDE_SKILL_DIR}
bun materialize-sources.ts \
  --bib ~/Google\ Drive/My\ Drive/resources/Paperpile/paperpile.bib \
  --bib ./references/sources.bib \
  --refs ./references \
  --drafts ./drafts \
  --debug

This populates references/ with local copies of all cited sources:

  • Paperpile PDFs → batch rclone copy from Google Drive → references/<bibkey>.pdf
  • Readwise articles (reports, news, speeches without PDFs) → search by title, export markdown → references/<bibkey>.md
  • Gaps → printed at the end for manual action (Obsidian web clipper or manual sourcing)

After materialization, cite-check operates purely locally.

Usage

cd ${CLAUDE_SKILL_DIR}
bun install  # first time only

# Single bib file
bun cite-check.ts --bib ~/Google\ Drive/My\ Drive/resources/Paperpile/paperpile.bib --drafts <path-to-drafts>

# Multiple bib files (Paperpile + project-local; first bib wins on duplicate keys)
bun cite-check.ts \
  --bib ~/Google\ Drive/My\ Drive/resources/Paperpile/paperpile.bib \
  --bib ./references/sources.bib \
  --drafts <path-to-drafts>

CLI Flags

FlagRequiredDefaultDescription
--bib <path>Yes*--Path to .bib file (repeatable; first wins on duplicate keys)
--store <id>Noauto-createUse existing File Search store ID
--drafts <dir>No./draftsDirectory with markdown draft files
--out <path>No<drafts>/REVIEW-CITES.mdOutput report path
--limit <n>NoallCheck only first N citations (smoke test)
--dry-runNofalsePrint prompts without querying
--sequentialNofalseRun queries one-at-a-time instead of Batch API (default: batch)
--retry-model <model>Nothe pro roleRetry UNSUPPORTED results with a stronger model
--auditNofalseAudit source availability without querying (checks Paperpile PDFs)
--debugNofalseVerbose logging

*Either --bib or --store is required.

Ask Mode: Targeted Source Queries

Ask a specific question about a single source:

# Does Bebchuk2019 support a specific claim?
bun cite-check.ts ask @Bebchuk2019-uq "do expense ratios fall since 2010?" --bib paperpile.bib

# What does a source say about a topic?
bun cite-check.ts ask @Brav2022-ht "what are retail turnout rates?" --bib paperpile.bib --bib sources.bib

The ask mode uploads the single source PDF via the legacy Files API (with manifest caching, 48h TTL), queries Gemini with inline file references, and prints the answer with supporting passages to stdout. No File Search store is created and no report is generated.

Cross-Directory File Resolution

When multiple --bib files are provided, file paths are resolved across all bib directories. This handles the common case where a project-local sources.bib has file = {All Papers/...} paths that are relative to the Paperpile folder rather than the project's references/ directory. The tool tries each bib directory as a fallback when the primary path doesn't exist on disk.

How It Works

  1. Extract citations from markdown using pandoc [@bibkey] syntax

  2. Parse bib file to map bibkeys to PDF file paths via file fields

  3. Sync the File Search Store, per file — PDFs for cited bibkeys are imported into a persistent Gemini File Search store with bibkey metadata. Google Drive FUSE paths are copied locally via rclone to avoid EDEADLK deadlocks. Stores persist across runs (no 48h TTL). "Changed" means the SHA-256 of the bytes of a cited source differs — not just its bibkey or path — so swapping a PDF at the same path under the same bibkey (a working paper replaced by the published version) invalidates it. A missing or unreadable source hashes to a <missing> sentinel rather than erroring; coverage is reported separately.

    State (.cite-check-store.json, in the drafts directory) records a per-bibkey hash, so invalidation is surgical rather than whole-store: one swapped PDF among 46 re-imports one document, not 46. The run takes one of three paths and always logs which and why on stderr:

    • [store] REUSE — no cited source changed; zero API calls.
    • [store] SURGICAL — changed and removed keys have their document deleted, then changed and added keys are imported. Delete comes first, so a failed import leaves the key absent from the store and it reports NOT_IN_STORE rather than being verified against stale bytes. Documents are located by paginating documents.list and matching customMetadata.bibkey — never displayName, which the store replaces with a random id on import.
    • [store] FULL REBUILD — the store is deleted and everything re-imported. This is the fallback for anything that leaves the store's contents unknowable: no prior state, state written before per-key hashes existed (a one-time migration), a documents.list failure, a key the state claims was imported but no document carries, or a failed delete. A silent partial update is the failure mode this guards against, so the reason is always printed.
  4. Query Gemini with structured prompts for each citation, using the fileSearch tool with metadata filtering to scope each query to the relevant source documents

  5. Classify each citation as SUPPORTED / PARTIAL / UNSUPPORTED / NOT_IN_STORE / ERROR

  6. Verify grounding — for SUPPORTED/PARTIAL results, extract source PDF text via pymupdf4llm and run token-level LCS alignment to confirm the passage Gemini quoted actually exists in the source. Ungrounded passages are flagged [UNGROUNDED] in the report.

  7. Write report to REVIEW-CITES.md

Bib File Format

The --bib flag expects a .bib file where entries have a file field with a path relative to the bib file's directory. Paperpile's exported paperpile.bib follows this convention:

@article{Hu2024-bm,
  author = {Edwin Hu and ...},
  title = {{Custom proxy voting advice}},
  file = {All Papers/H/Hu et al. 2024 - Custom proxy voting advice.pdf},
  year = {2024}
}

All bib entries are parsed. Entries with a file field (~95% of Paperpile entries) are imported into the File Search store. Only sources for bibkeys that are actually cited in the drafts are imported.

Citation Features

  • Bracketed [@key] and in-text @key citations
  • Locators: [@key, p. 42]
  • Compound cites: [@a; @b] (queried together)
  • Footnote indirection: citations in [^id]: footnote bodies
  • Bluebook signals: see, cf., see also, etc. (softens verification)
  • Parenthetical extraction: [@key] (holding that X)

Output

REVIEW-CITES.md with:

  • Summary counts (supported/partial/unsupported/not in store/error/ungrounded)
  • Details table: status, file:line, bibkey, claim, response
  • [UNGROUNDED] flag on any SUPPORTED/PARTIAL result whose passage failed grounding verification

Batch Mode (Default)

By default, all citation queries are submitted as a single Gemini Batch API job using the File Search tool with metadata filtering. Each query is scoped to the relevant source documents via bibkey metadata, so there is no cross-contamination between queries.

# Default (batch)
bun cite-check.ts --bib paperpile.bib --drafts ./drafts

# Sequential (one query at a time, useful for debugging)
bun cite-check.ts --bib paperpile.bib --drafts ./drafts --sequential

The --sequential flag runs each query as an individual generateContent call instead of a batch job. This is useful for debugging or when batch jobs hit rate limits.

Audit Mode

Run --audit before checking citations to see which sources are available and which need to be added:

bun cite-check.ts --bib paperpile.bib --bib sources.bib --drafts ./drafts --audit

The audit checks each cited bibkey for PDF availability on disk (via bib file field with cross-directory resolution). Missing sources should be added to Paperpile.

Exit code is 1 if any sources are missing, 0 if all sources are available. No Gemini store is created and no queries are sent.

Passage Grounding

After Gemini returns a SUPPORTED/PARTIAL result with a supporting_passage, the tool verifies the passage actually exists in the source PDF text using token-level LCS alignment (ported from langextract's WordAligner). Two gates reject bad matches:

  • Coverage gate (default 0.75): at least 75% of passage tokens must appear in the matched source span
  • Density gate (default 0.33): matched tokens must be at least 33% of the source span length (rejects scattered matches)

Signal cites (see, cf., etc.) use relaxed thresholds (0.5 coverage / 0.2 density) since they only need conceptual alignment.

Grounding requires extracting text from the source PDF. This uses pymupdf4llm (via extract-pdf-text.py) which preserves document structure, footnotes, and tables as clean markdown. Extracted text is cached in <drafts>/.cite-check-text/.

The Gemini File Search API behind the primary grounding signal — store creation, metadata filtering, and the groundingMetadata shape grounding.ts parses — is documented in skills/gemini-vertex/references/file-search.md.

Google Drive FUSE Bypass

PDF files stored on Google Drive Desktop's FUSE mount (~/Google Drive/My Drive/) are subject to EDEADLK deadlocks when accessed concurrently or when not locally cached. The tool detects Google Drive paths — including through symlinks (e.g., references/All Papers → ~/Google Drive/.../Paperpile/All Papers) — and uses rclone copyto to fetch them to a local cache (~/.cache/cite-check-pdfs/) before upload or text extraction. Requires rclone with a google-drive: remote configured.

Architecture

cite-extract.ts          -- Pure citation extraction (no I/O)
gemini.ts                -- Gemini API wrapper (File Search store CRUD, query, legacy upload for ask mode, rclone FUSE bypass)
grounding.ts             -- Post-hoc passage grounding (tokenizer, LCS aligner)
extract-pdf-text.py      -- PDF text extraction via pymupdf4llm
materialize-sources.ts   -- Copy Paperpile PDFs + Readwise articles to references/
cite-check.ts            -- CLI orchestrator (extract -> import -> query -> ground -> report)

Claude Message Batches

Paid from the prepaid API credit an Ultra plan grants one linked Console org. Every action goes through one script; the accounts it can bill are in the batch-router accounts config ($BATCH_ACCOUNTS, default ~/.config/batch-router/accounts.toml, shape in ../batch-router/references/accounts.example.toml).

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

## IRON LAW: NO FULL `submit --run` UNTIL FOUR EXITS ARE 0 AND EDDY SAID YES

The full run waits for: validate exit 0 on the full file, verify exit 0, the 5–10 row pilot's results --input exit 0, and a message from Eddy approving the cost route_batch.py printed. A batch that is wrong is wrong 100,000 times, and the credit it burns expires at cycle end with no rollover: skipping the pilot spends the month's credit on a bad dataset.

The sequence

S=${CLAUDE_SKILL_DIR}/scripts/claude_batch.py
uv run $S accounts                                    # which orgs have a key; free
uv run $S validate requests.jsonl                     # schema, custom_id, limits; free
uv run $S verify requests.jsonl --account gmail       # ONE sync request at full price; prints its cost
uv run $S submit pilot.jsonl --account gmail --run    # 5-10 rows first; refuses without --run
uv run $S status <batch_id> --account gmail --wait    # run in the background, not a narrated poll
uv run $S results <batch_id> --account gmail --input pilot.jsonl --out pilot.out.jsonl

Input is one {"custom_id": "...", "params": {<Messages API body>}} per line. results writes the succeeded rows and prints every errored, canceled, expired, missing or unexpected custom_id; it exits 1 unless all of them reconciled. Pick the model from the current models overview at use time, never from memory; the price table the script uses is in limits and prices.

Facts

  • Credit: $200/month per Ultra plan to ONE linked org; covers the Claude API including Message Batches, not Claude Code. Expires at the end of each billing cycle, no rollover; included credit spends before purchased credit. Spending the soonest-expiring org first is free money; spending a purchased card balance while an idle credit expires is the waste this skill exists to stop.
  • No balance API: balance and expiry show only in Console Billing > Promotional credits. Never write a balance into config; report it as UNVERIFIED unless Eddy or the Console gave a number today.
  • Exhausted credit fails the request with Your credit balance is too low to access the Anthropic API. Record that text verbatim and stop; switching to another org's key without saying so bills an account Eddy did not pick.
  • Keys are per org, and batches are per workspace: status, results and cancel must use the account that submitted. A UVA or NYU account with key = "none" exits with "no key"; never create a key.
  • Limits: 100,000 requests or 256 MB per batch; most finish within 1h; unfinished requests expire at 24h; results kept 29 days; only succeeded rows are billed. stream, speed, fallbacks and max_tokens: 0 are rejected in batch. PDFs past the cap: upload under the SAME --account as submit and reference the file_id (proven in a batch 2026-11-05; another org's key gets 404) — Files API.
  • Shared prefix: use 1-hour prompt caching (cache_control with ttl: "1h"); a 5-minute cache entry expires before most batch rows run.
  • Spend counts toward the org's monthly spend cap (resets on the 1st), and a batch can overshoot the workspace spend limit slightly.

Red flags — STOP

About toDo instead
submit --run without a verify and a 5–10 row pilot on the same model and paramsRun them; a paid full batch needs Eddy's go-ahead
Create an API key for the UVA or NYU orgAsk Eddy; record the key source in accounts.toml once he has made one
Loop messages.create over many rowsBuild JSONL and submit a batch: 50% off and no rate-limit fight
Accept processing_status: ended as successresults --input; done only on exit 0, else resubmit the custom_ids it lists
Poll status from the model turn by turnstatus --wait in a background Bash, notified on exit

Consensus CLI

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Search Consensus.app for academic papers via the consensus CLI tool.

Binary: ~/projects/consensus-cli/consensus

Requires: a Chrome/Chromium signed in to consensus.app with CDP on port 9250 (override with CONSENSUS_CDP_PORT, or CDP_PORT for the whole CLI family).

Check: ls ~/projects/consensus-cli/consensus || echo "MISSING: consensus binary not built"

Core Command

consensus search "<query>" [options]
consensus journals [<query>]      # exact journal names accepted by --journal
consensus publishers              # values accepted by --publisher

Flags

FlagDescription
--n <int>Result count (default 20, max 100)
--type <csv>Study types: rct,systematic,meta,non_rct,observational,lit_review,case,animal,in_vitro
--years <range>Year range: 2018-2024 or past N years (e.g. 5)
--min-citations <int>Minimum citation count
--rank <q1|q2|q3|q4>Journal quartile filter (SJR)
--humanHuman studies only
--rctShorthand for --type rct
--open-accessOpen access papers only
--domain <csv>Fields of study (e.g. Medicine,Chemistry)
--country <csv>Country filter (e.g. USA,UK)
--journal <name>Restrict to one journal; repeat the flag for more. Names must match the index exactly
--journals-file <path>Read journal names from a file, one per line (# comments and blanks skipped)
--publisher <name>Restrict to one publisher; repeat for more
--page <int>Page number (default 0)
--sort <field>Client-side sort: citations (descending)

Output Fields (per paper)

{
  "title": "...",
  "authors": ["..."],
  "year": 2023,
  "journal": "...",
  "doi": "...",
  "citations": 150,
  "study_type": "rct",
  "takeaway": "One-sentence finding...",
  "open_access_pdf_url": "https://... or null",
  "url": "https://consensus.app/papers/..."
}

Journal Filtering — the primary quality gate

File: ${CLAUDE_PLUGIN_ROOT}/references/trusted-journals.local.md

A shared resource — the google-scholar and research skills and the librarian agent read the same file. It is the user's curated list of trusted journals, one exact name per line, with # comments. It is the argument to --journals-file — pass the path directly, do not re-type the names:

consensus search "<topic>" --n 50 --sort citations \
  --journals-file ~/projects/workflows/references/trusted-journals.local.md

When the user asks for "journals I like", "relevant journals only", "top journals", or a field they clearly work in, filter SERVER-SIDE with --journals-file (or --journal for a narrower subset). Server-side filtering means all N results are from trusted venues, instead of filtering a mixed result set down to two or three afterwards.

For a narrower cut, pass the subset explicitly — one flag per journal, since journal names contain commas:

consensus search "insider trading enforcement" --n 20 \
  --journal "Journal of Finance" \
  --journal "Journal of Financial Economics" \
  --journal "Review of Financial Studies"

Adding a journal to the list

A journal name that is not in Consensus's index silently matches nothing — it does not error. Verify before adding:

consensus journals "review of financial"

Then append the exact name string to trusted-journals.local.md, under the right # section.

Publishers

--publisher is a coarser cut over a fixed vocabulary (consensus publishers lists it: Elsevier, Wiley, Springer Nature, OUP, CUP, JAMA, NEJM, ...). Use it when the user wants a house rather than a venue; journal filtering is otherwise strictly better.

Still mark, still resolve

--journals-file also makes the ★ pass trivial: every returned paper is from a trusted venue, so mark them all ★ and note the filter in the preamble. Without a journal filter, mark ★ per-paper against the same file, and run the SSRN DOI resolution below.

SSRN Label Detection & DOI Resolution

SSRN label patterns (journal field is NOT the real venue):

  • Contains "eJournal", "Topic)", "SSRN Electronic Journal"
  • Starts with a subject code: PSN:, ERN:, ERPN:, SRPN:, POL:, LSN:

Note that SSRN labels cannot appear when --journal/--journals-file is in play — the filter matches on the indexed journal name, so working-paper labels are excluded by construction. This section applies to unfiltered searches.

When a paper has an SSRN-label journal AND a non-null doi:

curl -s "https://api.crossref.org/works/<doi>" | uv run python3 -c "
import json, sys
d = json.load(sys.stdin)
msg = d.get('message', {})
ct = msg.get('container-title', [])
print(ct[0] if ct else 'NOT FOUND')
"

Use the resolved journal name to re-check against the trusted list. If it matches, mark ★ with a note: ★ (resolved via DOI from SSRN label).

If doi is null or CrossRef returns no container-title: leave as unresolved SSRN label.

Presentation Format

★ [Title](url) — Authors (Year), *Journal*, N citations
  > Takeaway: ...

★ [Title](url) — Authors (Year), *Resolved Journal* (resolved via DOI), N citations
  > Takeaway: ...

[Title](url) — Authors (Year), *Journal* [SSRN label, unresolved], N citations
  > Takeaway: ...

Trusted papers first (confirmed then resolved), then unresolved, then non-trusted.

IRON LAW: Always Use the CLI Binary

NEVER use mcp__consensus__search. ALWAYS use the ~/projects/consensus-cli/consensus binary. This is not negotiable.

The MCP tool is rate-limited to 3 results per search and requires a free account. The CLI binary drives the signed-in enterprise session in the CDP browser and returns up to 100 results.

Red Flags

ActionWhy WrongDo Instead
Using mcp__consensus__search instead of the CLIMCP is rate-limited to 3 results; CLI has no limitAlways use ~/projects/consensus-cli/consensus
Presenting results without reading trusted-journals.local.mdUser expects journal quality signals on every searchRead the shared trusted-journal list first, always
Filtering trusted journals client-side after an unfiltered searchWastes most of the result set — 50 results collapse to 3Pass --journals-file and get 50 trusted results
Passing a journal name you did not verifyAn unindexed name matches nothing, silently — you get zero papers and blame the queryconsensus journals "<partial>" first
Comma-separating journals in one --journalJournal names contain commas; the whole string is treated as one nameRepeat the flag, or use --journals-file
Treating SSRN topic labels as real journals without checking DOIThe paper may be in JF or JAE — you'd miss a trusted hitRun CrossRef DOI lookup first
Skipping DOI resolution because there are many SSRN-labeled papersHigh-citation SSRN-labeled papers are often published in top venuesResolve all of them — it's one curl per paper
Using --rank q1 as a journal quality filterThe API maps SSRN working papers under Q1 labels — it is not reliableUse --journals-file — it is now an explicit server-side filter
Passing --n > 100CLI validates and rejects — exits non-zeroMax is 100

Decision Tree

User wants papers on a topic
    ↓
Read trusted-journals.local.md
    ↓
Does the user want only their journals / "relevant" / "top" journals,
or is the topic squarely in a field the file covers?
    ↓
YES → consensus search "<topic>" --n 50 --sort citations \
        --journals-file <path to trusted-journals.local.md> [other filters]
      → every result is trusted: mark all ★, say which filter was applied
      → zero results? The filter may be too narrow for the topic —
        rerun unfiltered and mark ★ per-paper instead of widening silently
    ↓
NO  → consensus search "<topic>" --n 50 --sort citations [other filters]
      → per paper:
          journal matches the file? → ★
          SSRN label + doi present? → curl CrossRef → re-check → ★ if match
          else → unresolved / non-trusted
      → present ★ confirmed, ★ resolved, then rest

Common Patterns

# Basic search — sort by citations to surface highest-impact papers first
consensus search "mandatory disclosure effects" --n 50 --sort citations

# Restrict to RCTs
consensus search "aspirin cardiovascular" --rct --n 10

# Recent papers, high-citation
consensus search "ESG disclosure" --years 5 --min-citations 50

# Systematic reviews only
consensus search "minimum wage employment" --type systematic

# Only the user's journals — the default for their own fields
consensus search "corporate governance" --n 30 --sort citations \
  --journals-file ~/projects/workflows/references/trusted-journals.local.md

# A narrower cut: the finance top three
consensus search "payout policy" --n 20 \
  --journal "Journal of Finance" \
  --journal "Journal of Financial Economics" \
  --journal "Review of Financial Studies"

# Check a name before adding it to the trusted list
consensus journals "review of financial"

# By publisher house rather than venue
consensus search "machine learning in radiology" --n 20 --publisher Elsevier --publisher Wiley

Operational Notes

  1. Chrome must be running with CDP and signed in. Exit codes follow sysexits: 69 = browser unreachable, 77 = not signed in, 75 = CAPTCHA/rate limit — branch on the code, do not parse stderr
  2. consensus journals is rate-limited hard (429 → exit 75) after ~30 rapid calls. Verifying a batch of names needs ~1.5s between calls and a cool-off after a 429
  3. --rank q1 is imprecise (SSRN papers slip through) — --journals-file is the reliable quality gate
  4. Journal filters are exact-match on the indexed name and fail silently, not loudly — an empty result set usually means a wrong name, not a dry topic
  5. Law reviews ARE indexed — all fourteen T14 flagships resolve. The student-edited business specialties (Journal of Corporation Law, Delaware JCL, Harvard/Columbia BLR, Penn JBL, NYU JLB, Virginia L&B, Berkeley BLJ) are NOT; reach those with scholar lookup --journal "<name>"
  6. study_type comes from Consensus badges and may be null for many papers
  7. open_access_pdf_url is null when no PDF is available (not undefined)

Continuous Learning Skill

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Extract reusable patterns from sessions and save them as learned skills for future use.

How It Works

This skill analyzes the session transcript to identify extractable patterns:

  1. Session Evaluation: Checks if session has enough messages (default: 10+)
  2. Pattern Detection: Identifies error resolutions, workarounds, debugging techniques
  3. Skill Extraction: Saves useful patterns to ~/.claude/skills/

When to Use

  • At end of long sessions with multiple problem-solving cycles
  • After resolving complex errors that might recur
  • When you develop project-specific conventions

Invocation

Use /learn command or invoke directly:

/continuous-learning

Configuration

Edit config.json to customize:

{
  “min_session_length”: 10,
  “extraction_threshold”: “medium”,
  “auto_approve”: false,
  "learned_skills_path": "~/.claude/skills/learned/",
  “patterns_to_detect”: [
    “error_resolution”,
    “user_corrections”,
    “workarounds”,
    “debugging_techniques”,
    “project_specific”
  ]
}

Pattern Types

PatternDescription
error_resolutionHow specific errors were resolved
user_correctionsPatterns from user corrections
workaroundsSolutions to framework/library quirks
debugging_techniquesEffective debugging approaches
project_specificProject-specific conventions

Learned Skills Format

Extracted skills are saved following the standard skill directory structure:

~/.claude/skills/learned/
├── fix-marimo-import-error/
│   └── SKILL.md
├── debug-pixi-environment/
│   └── SKILL.md
└── wrds-connection-pattern/
    └── SKILL.md

Each learned skill follows standard SKILL.md format with:

  • Frontmatter (name and description)
  • Problem context
  • Solution pattern
  • Example usage

Integration

The skill reads from CLAUDE_TRANSCRIPT_PATH (JSON conversation transcript) which is automatically set by Claude Code during sessions.

Related

  • /learn command - Manual pattern extraction mid-session
  • /checkpoint command - Save session state
  • Session-end hook - Auto-evaluates sessions for patterns

Court dockets: retrieving filings from Lex Machina, Docket Alarm and Bloomberg Law

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Three commercial platforms, one job: turn a list of cases into a directory of PDFs plus a manifest you can defend. Verified end to end on 1,597 Delaware Chancery cases — ~4,000 documents retrieved across all three.

Auth is the user's live browser session on CDP 9222. None of these needs a password. Read the cookie with scripts/lm_cookie.py; never log in, and never touch stored credentials unless the user explicitly says to.

Full endpoint chains, request/response shapes and per-vendor quirks: ${CLAUDE_SKILL_DIR}/references/endpoints.md — read it before writing any fetch code.

The scripts are the interface

Do not hand-roll a harvester. These are working, resumable and manifest-writing:

scriptdoes
scripts/lm_cookie.pyreads the live session cookie for any host over CDP
scripts/harvest_complaints.pyLex Machina: docket-search → tag-257 complaint → signed PDF
scripts/da_grab.pyDocket Alarm: serial, boring, survives hanging documents
scripts/fetch_opinion_pdfs.pyopinion/decision PDFs from a plan parquet
scripts/bl_docket_list.pyBloomberg Law: every docket matching facets + term, as a list (no PDFs)

Copy them into the project and adapt. Rewriting from the reference doc reproduces bugs that took hours to find.

## The Iron Law of Identification vs Retrieval

A document that is IDENTIFIED is not a document that is RETRIEVED. Never mark a substitute ok.

Every vendor names the complaint far more often than it serves the file. Measured on the same 1,597 cases: Lex identified 99.6% and served 81.0%; Docket Alarm identified 96.4% and served 81.7%. A fallback that takes "the first document that has a file" silently returns a Rule 5.1 certification, a cover letter or a verification page — and reports success.

Conflating the two overstates coverage by 15-20 points. That is not a rounding error: it is a fabricated denominator under every downstream rate, and shipping it is more harmful than returning nothing, because nobody downstream can see it happened.

## The Iron Law of Absence

Never record a document as unavailable because YOUR SELECTION RULE failed. Enumerate the docket entries and look for a later substantive version before concluding it is not public.

This is the mirror of the law above. That one catches substitutes passed off as complaints; this one catches real complaints recorded as absent — and it is the more expensive failure, because a substitute is visible in the data while an absence looks like a fact about the world.

Delaware Chancery files the complaint confidentially, then files a redacted PUBLIC VERSION days later as a SEPARATE docket entry (Ct. Ch. R. 5.1). Every vendor rule that stops at the first complaint-tagged entry sees no document and reports failure. The public version sits typically around entry 10-15, measured as deep as entry 91, on dockets running to 277 entries.

Measured on two independent Delaware populations:

population"unavailable" caseshad a public versionrate
DGCL 220 complaints25322990.5%
fiduciary-duty follow-on suits807391.2%

Recovering them moved coverage 82.2% -> 95.3% and added 195 confirmed cases to a study population. The pre-recovery write-up asserted "~18% of complaints are not publicly retrievable — a finding about practice, not a collection defect." That sentence was wrong in both halves.

The correct statement is "~18% are filed confidentially, and ~90% of those are publicly available as a redacted version elsewhere on the same docket." Genuine unavailability is ~4%.

Two defects that bite while recovering them

  • Match the public version to a COMPLAINT, not to any document mentioning one. Public Version of Answer to Verified Complaint and Motion to Extend Deadline to File Public Version of the Verified Complaint both contain "public version" and "complaint". Requiring only those two terms returned the wrong document on 10.6% (23/218) and 10.0% (7/73) of cases across two populations — a stable rate, so expect it. Exclude answer|affirmative defense|motion|letter| certificat|opposition|brief|reply|response|stipulation|notice|transcript|order.
  • A resume guard keyed on filename alone silently preserves a known-bad file. After fixing the regex, the re-fetch skipped every corrected case as already_present, because the wrong document from the first pass sat on disk under the same name. Quarantine before re-fetching.

Redaction does not degrade extraction. Measured on the recovered set: misconduct_category, plaintiff_law_firm and requester_type all came back at 0% missing, identical to the unredacted baseline. Captions, parties, statutory citations, prayer for relief and signature blocks survive; body allegations may be blacked out. Carry a source flag and check the not_stated rate before using them for content coding.

Listing Bloomberg dockets by court, county or search term

scripts/bl_docket_list.py collects the docket LIST (ids, parties, judge, case type) for any facet + term search, sliced past both silent caps and asserted equal to the API's remote_count. Run --discover first to learn the facet codes (Nevada=29, Clark=84, Washoe=85), then --count-only, then collect; --csv adds the 41-column export and asserts its id set matches. Flags and examples are in the script header.

--term is sent verbatim, and quoting changes the result set: "Business Court" returns 3,690 dockets, Business Court 17,180. Write the quoted form into the project's methods note.

Vendor facts

  • Lex Machina lists the complaint with an empty files[] on ~18.7% of cases (298/1,597) — which means it was filed confidentially, NOT that it is unavailable; see the Iron Law of Absence above. The document id and title are present; only the file is withheld. That is file_withheld, a recoverable fact worth recording — not "not found", and not a reason to substitute another document.
  • Docket Alarm returns HTTP 500 with a ~40 KB HTML body when it does not hold the file. It is reproducible on a fresh cookie and never recovers, so retrying it is pure waste; at a ~15% failure rate the retry backoff was the dominant cost of a run until it was removed.
  • Some Docket Alarm documents never respond at all. One hanging request plus a 300 s timeout plus an ordered ThreadPoolExecutor.map is indistinguishable from a deadlocked process. Use a 45-90 s timeout.
  • Bloomberg's accessToken is HttpOnly with an ~8 minute TTL, refreshed by the page's own SPA on a timer — not by your requests. The tab must stay open; a cookie snapshot into any external store authenticates for minutes, not hours.
  • Bloomberg's CSV export silently truncates at 1,000 rows — no warning in the file or the UI. Always cross-check an export against the API's own remote_count and date-slice below the cap.
  • Docket Alarm's search endpoint is bot-protected: HTTP 400 outside the browser, 200 via same-origin fetch() in a logged-in tab. Per-docket pages are fine outside. Do not fight it.
  • ~18% of Delaware Chancery §220 complaints are unavailable from all three vendors, and they fail on the same cases — adding two vendors to the best single one bought 17 cases out of 1,597. If coverage stalls near 82%, that is the record, not your pipeline.

Finding decisions in a docket

  • The docket line names the DOCUMENT, not the holding. Entries read Memorandum Opinion; the grant/deny is inside the PDF. Only ~15/152 opinion entries mention an outcome word at all.
  • Chancery disposes of cases through a Magistrate's Final Report, then an order implementing it: Granted ([Proposed] Order Implementing the Court's Post-Trial Bench Ruling). Searching only for Memorandum Opinion / Letter Decision missed 34% of cases with a decision (169 found vs 257 that exist). Always include: final report, master's report, post-trial ruling, bench ruling, opinion and order, final order and judgment.
  • Never search for opinions only within cases you already classified as having one. That is circular and it hides exactly the misclassifications you are trying to find. Scan the whole population.

Red Flags — STOP If You Catch Yourself:

About toWhy wrongDo instead
About to record a complaint as unavailable because the first entry has no fileThe Rule 5.1 public version sits further down the docket ~90% of the timeEnumerate the entries and search for a later substantive version
Matching a public version on "public version" + "complaint"Also matches answers, motions and briefs ABOUT the complaint — 10% wrong-document rate on two populationsExclude answer/motion/brief/letter/certificate terms
Re-fetching after a selection fix without clearing the old filesA filename-keyed resume guard reports already_present and keeps the known-bad documentQuarantine, then re-fetch
Report "session expired" because the cookie read came back emptyAn empty cookie name (['']) is a parsing failure, not an absent session. Picking the first CDP page target returns nothing when that tab is blob:/chrome:// — this cost hours and triggered a needless credential huntUse scripts/lm_cookie.py, which selects a page on the requested host; verify by fetching one document
Pin the session cookie into a Cookie: headerIt goes stale mid-run when the vendor rotates it, and the resulting 401s look exactly like rate limitingPut cookies in the requests cookie jar; re-read from the browser on a 401
Retry a Docket Alarm HTTP 500Reproducible "document not held"; the backoff dominates the runRecord it and move on
Take documents[0] when no title matches the entry textThat is how a sealing motion becomes a "complaint"Mark is_substitute, never ok
Run pdftotext/pdfinfo inside the download loopA long timeout on one malformed PDF stalls a worker and looks like a hangCompute page/text metrics in a post-pass
kill $! after setsid, or pkill -f a pattern matching your own shellKills the wrong PID and leaves crawlers running — three concurrent copies hit one vendor this wayps -eo pid,cmd | awk '/[d]a_grab/' then kill the real pid
Trust a vendor's case-type tag as ground truthTags put a judicial letter, a §225 petition and a notary page in a books-and-records populationScreen from the document text
Conclude coverage is complete without checking the whole populationEvery widening of the net raised the count: 6.5% → 10.7% → 16.1% written decisionsScan all cases, all document-type spellings

CourtListener for empirical legal research

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Free Law Project's corpus. Three access routes with very different properties, and one distinction that people get wrong before they start.

The two corpora are not the same thing

**"CourtListener is federal-only" is true of RECAP and FALSE of opinions.** Getting this wrong sends you to a paid vendor for state-court data that is sitting here free.
corpussourcecoveragecontains
Opinions (case law)scraped from court websitesfederal and statewritten decisions, full text
RECAPPACERfederal onlydocket entries, motions, filings

The tell is on the court record itself: pacer_court_id: null + has_opinion_scraper: true means a scraped state court, not a PACER one. Verified on Delaware Chancery:

GET /api/rest/v4/courts/delch/
  "full_name": "Court of Chancery of Delaware"
  "jurisdiction": "SA"            <- state
  "has_opinion_scraper": true
  "pacer_court_id": null          <- NOT RECAP

Pick the route before writing any code

routeauthbest forcost
Bulk data filesnoneany corpus-scale analysisquarterly snapshot, big files
REST API v4search/courts unauth; dockets needs a tokentargeted queries, current datarate-limited, polite pacing on you
DB replication / MCPmembership or commercialcontinuous synccontact FLP

Default to bulk for research. It is public domain, needs no account, and does not rate-limit you. Reach for the API only when you need data newer than the last quarterly snapshot, or when you want a few hundred records rather than a corpus.

Bulk data

No authentication. Public Domain Mark. Regenerated quarterly on the last day of March, June, September and December. Snapshots, not deltas — each file is the whole table.

https://com-courtlistener-storage.s3-us-west-2.amazonaws.com/?list-type=2&prefix=bulk-data/
aws s3 ls s3://com-courtlistener-storage/bulk-data/ --no-sign-request

Measured sizes, 2026-06-30 snapshot:

filesizecarries
opinions-YYYY-MM-DD.csv.bz254.6 GBopinion full text
opinion-clusters-…2.46 GBcourt, date_filed, case name, docket_number
dockets-…5.01 GBdocket metadata
courts-…<10 MBthe court table
citation-map-…0.52 GBcitation graph

Also published: financial disclosures, judges, oral arguments, and ~2 TB of ModernBERT opinion embeddings at s3://com-courtlistener-storage/embeddings/opinions/.

Filter clusters first, then stream opinions. opinion-clusters is 22× smaller and carries the court and date. Select the cluster ids you want, then stream-decompress the 54 GB opinions file and keep only matching rows. Never materialise the opinions CSV.

Files are PostgreSQL COPY TO output: CSV, UTF-8, header row. Schema dumps are published alongside; field definitions are in the CourtListener GitHub models.py, or via an OPTIONS request to the matching API endpoint.

REST API v4

https://www.courtlistener.com/api/rest/v4/
  • search and courts work unauthenticated — 3,358 delch opinions were pulled in 206 unauthenticated calls.
  • dockets returns {"detail":"Authentication credentials were not provided."} without a token.
  • Token lives at /profile/api/ — which sits behind a "Let's confirm you are human" interstitial. That gate is the site's bot check and belongs to Free Law Project; the user clears it, not you.
  • Search type params: type=o opinions, type=r RECAP dockets.

Pace politely and back off on 429/5xx. This is a nonprofit's free service; without a token there is no quota to raise, so courtesy is the entire budget.

API Facts

  • suitNature is not normalised. The same NOS 160 appears as Stockholders Suits, 160 Contract: Stockholders Suits and 160 Stockholders Suits. Query the text (suitNature:"Stockholders Suits"), not the numeric code, or you silently lose most of the set.
  • Derivative suits are docketed under more than one NOS. "shareholder derivative" returns 1,670 under NOS 160 and 2,020 under 850 (Securities/Commodities). Filtering to 160 alone drops more than half.
  • docketNumber carries the C.A. number, which joins directly to a Delaware Chancery docket from Lex Machina or CourtConnect — measured at 100% match on 2,350 rows. That join is what lets a filing universe from one source meet a ruling from this one.
  • Opinions are written decisions only, and that is a selection, not a gap. Measured against a verified Chancery filing universe: 82% of caption-derivative filings never produce a written opinion, because most dispositions are bench rulings from the transcript. An outcome variable built from opinions describes the selected fifth. Reporting a dismissal rate without that denominator is an unverified claim dressed as a finding.
  • The bulk snapshot lags up to a quarter. The 2026-06-30 file lands two weeks after a 2026-06-15 event, so a study of anything recent gets a truncated post-period until the next drop. Check the snapshot date against your event date before designing the test.

Coverage and what is coming

Delaware Chancery: 3,358 opinions since 2012 (delch). Texas currently covers the Supreme Court, Court of Criminal Appeals and the 15 civil courts of appeal.

From the FLP wiki, verbatim:

"In 2026, we are collecting content from high and appellate courts at the five most populous states: California, New York, Texas, Florida, and Pennsylvania."

"In 2027 we aim to continue this with the Delaware Chancery Court and the next most populous states."

The 2027 item is filings, not opinions — Chancery opinions already exist. Docket-level state coverage would reach the bench rulings the opinion corpus structurally cannot see.

Red Flags — STOP if you catch yourself

About toWhy wrongDo instead
Tell the user CourtListener is federal-onlyTrue of RECAP, false of opinions — state case law is thereCheck pacer_court_id / has_opinion_scraper on the court
Page the REST API for a corpus-scale pullRate-limited, slow, and rude to a nonprofit when the same data is a free downloadBulk files
Clear the human-verification interstitial to get a tokenIt is the site's bot check, and you are a botAsk the user; or use the unauthenticated endpoints, which cover most needs
Decompress the 54 GB opinions CSV to diskYou need a few thousand rows of itFilter opinion-clusters first, then stream and match
Filter derivative suits on NOS 160 alone2,020 of them are docketed 850160 OR 850 plus a text filter, then hand-check
Report a dismissal rate from opinions without the selection denominator82% of filings never get a written opinion; the rate describes the selected fifthQuantify selection against a filing universe and lead with it
Assume the bulk snapshot is currentQuarterly, and the boundary may fall days after your eventRead the snapshot date; use the API for the tail

Related

  • ui-json-capture — when the source is a paywalled UI rather than an open corpus
  • wrds — the SEC-filing side of the same research questions

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Contents

The Problem

CRSP ships annually. The CIZ daily file stops at the last December close and does not error when you query past it — it returns a short panel. LSEG is current to T-1 but is keyed on RIC, not PERMNO, so the join runs through CUSIP.

Verified on WRDS PostgreSQL:

SourceLast dateGap
crsp.stkdlysecurityprimarydata (CIZ)2025-12-31—
crsp.stkmthsecuritydata (CIZ)2025-12-31—
LSEG TRDPRC_1 / TR.TotalReturn1DT-1 (2026-07-27)141 trading days

This skill assumes crsp-v2 (CIZ table and column names) and lseg-data (session, quota, entitlements). It covers only what is specific to joining them.

Splice Enforcement

IRON LAW: NO LSEG PRICE LEVEL SPLICED ONTO A CRSP PRICE LEVEL

**LSEG's `get_history` price series is back-adjusted to the CURRENT share basis. CRSP's `DlyPrc` is the price as traded on the day.** For any security that split after CRSP's cutoff, the two series are on different bases and the seam between them is a fabricated return.

Measured on the Dec-2025 overlap, where both sources have the same 22 trading days: 8 of 300 sampled securities disagree on price by >5%, at a ratio that is constant to std = 0.0000 across all 22 days — the signature of an adjustment factor, not a data error. STRO.O ratio 0.1000 (1:10 reverse split), VISN.O ratio 2.0493.

  • pd.concat([crsp_prc, lseg_prc]) → WRONG. A 1:10 reverse split after the cutoff prints a +900% one-day return on the seam.
  • Chain the gap off CRSP's last DlyPrc with LSEG's daily total returns → CORRECT. p_t = p_cutoff * cumprod(1 + ret).

A fabricated ±90% return on a known date, in a panel that otherwise validates, is worse than a missing tail — it survives every summary statistic and lands in the event window. scripts/crsp_lseg_splice.py::rebuild_price() does the chaining.

The chained series is a total-return index anchored at CRSP's last price, not a quoted price: total-return chaining reinvests dividends into the level. The raw LSEG quote is carried alongside as lseg_prc for anyone who needs the level.

Return Facts

  • TR.TotalReturn1D is the DlyRet analogue and it validates. On the Dec-2025 overlap, 99.12% of permno-days agree to within 1e-5, median absolute difference 2.45e-07, correlation 0.991. This is the field to splice on.
  • It is returned in PERCENT. -6.369427 means −6.37%. CRSP DlyRet is a decimal. Divide by 100 at the boundary or every return in the gap is off by 100×, which is loud in a mean but silent inside a signal that gets standardized.
  • It is CALENDAR-PADDED, and that breaks any coverage count taken from it. Every instrument comes back with the identical number of return days — measured std = 0.0000 across 3,384 RICs — including the 19 that delisted mid-gap. 1.14% of return rows have no trade price that day and 38% of those are ret == 0. Build the panel by inner-joining returns against the price series: otherwise a delisted stock keeps emitting flat rows to the end of the window, and the fill rate reads 99.8% because every security looks complete.
  • It is dividend-inclusive, like CIZ DlyRet, so no distribution merge is needed — and, per crsp-v2, no delisting-return merge either.
  • DlyCap must scale on the PRICE relative, not the return chain. Chaining market cap on TR.TotalReturn1D reinvests every dividend into shares outstanding and inflates cap by the cumulative dividend yield over the gap. lseg_prc_t / lseg_prc_cutoff is the right multiplier: both legs sit on LSEG's single adjusted basis, so splits cancel (as they must for cap, which is split-invariant) and dividends are excluded. TR.CompanyMarketCap exists but is not on CRSP's DlyCap definition or units (CIZ DlyCap is $ thousands).
  • Start the LSEG pull ON the CRSP cutoff date, not the day after. That one overlapping day is the anchor. crsp_prc / lseg_prc on the same day measures each security's adjustment-basis ratio directly: ~1.0 means the two agree, anything else is a post-cutoff corporate action LSEG has back-adjusted for (or a bad link that survived the venue screen). The pipeline carries it as adj_ratio and coverage reports it — it is what makes the splice auditable per security rather than trusted in bulk.
  • LSEG serves data for still-listed instruments. A security that delists inside the gap stops on its delist date, and its RIC gets a ^-stamp (IROQ.O^C26 = delisted March 2026) — which is how you learn the delist happened, since CRSP has not published it yet. There is no delisting return from this path.

Symbology Enforcement

IRON LAW: NO RIC ACCEPTED WITHOUT A US-VENUE CHECK

**LSEG resolves a US CUSIP to a foreign cross-listing's RIC for ~3% of the universe, and that RIC returns prices in a FOREIGN CURRENCY with no error.**

Measured: 101 of 3,485 resolved RICs (2.76%) carry a non-US venue suffix — .TRE (Tradegate), .MU (Munich), .SG (Stuttgart), .TBEA, .MX, .BCU.

CRSPLSEG RICeffect
TPH TRI POINTE HOMEST86f.TREprices in EUR; CRSP/LSEG ratio 1.3294 (≈ USD/EUR)
VERO VENUS CONCEPT0RR0.MUMunich listing; 18 of 22 days differ >1pp
CIVI CIVITAS RESOURCESUS17888H1032.TREISIN-form RIC, Tradegate
  • Taking RIC straight from symbol_conversion → WRONG. An FX series enters the panel as a price series.
  • Filter on the suffix before pulling → CORRECT. US venues are the bare RIC (composite) and .O .N .A .P .K .PK .OQ.

A EUR price series is plausible — right order of magnitude, right shape, moves with the stock. It fails no null check and no range check. It is only caught at the venue, which is why the check belongs before the pull, not after.

Symbology Facts

  • Use CRSP's date-effective cusip9 from crsp.stksecurityinfohist, never stksecurityinfohdr.hdrcusip. CIZ inverted the SIZ naming — CIZ cusip is the historical value and hdrcusip is the header one — so header-CUSIP code attaches a security's most-recent CUSIP to its entire history. For a fill-forward anchored at the cutoff date the two usually agree, but the same script pointed at an earlier as-of date silently mis-links.
  • Pass the 9-character CUSIP to SymbolTypes.CUSIP. CRSP CIZ carries both cusip9 and the 8-character cusip; LSEG returns 9-char. Trimming to CUSIP8 to join is fine as a post-hoc key, but resolve on the 9.
  • Unresolved CUSIPs come back absent, not wrong. Misses land as explicit nulls. The precision risk in this pipeline is the foreign venue and the entity mismatch, not the miss.
  • Entity agreement is the cheap guard: CRSP ticker vs LSEG TickerSymbol agrees for 97.31%, RIC root vs CRSP ticker for 95.06%, either for 97.56%. Most residual disagreement is a rename CRSP has recorded and LSEG reports under the current name (COMM → VISN.O), not a mis-link — flag it, do not drop it.
  • symbol_conversion chunks fine at 200 symbols; the session cap is 500 requests/minute and the binding limit is get_data at 10,000 data points and get_history at 3,000 rows per request.

Measured Coverage

Full numbers, denominators, and the validation method: references/coverage.md.

Universe = the 3,657 CRSP CIZ common stocks (5-column SHRCD 10/11 equivalent, primaryexch IN ('N','A','Q')) trading on 2025-12-31.

Stagecountshare of universe
CRSP common stocks at cutoff3,657100%
CUSIP9 → RIC resolved3,48595.30%
— of those, US venue3,38492.53%
— of those, entity-agreeing—97.56% of resolved

Agreement on the Dec-2025 overlap (300-security sample, 6,244 permno-days):

checkresult
price exact (reldiff ≤ 1e-4)96.28%
price within 5%98.13%
daily return within 1e-599.12%
return available where CRSP has a day98.34%

The 1.87% of price rows outside 5% are the split-adjustment and foreign-venue cases above — both handled by the two Iron Laws, neither by a tolerance.

The Pipeline

scripts/crsp_lseg_splice.py — five cached steps, all runs them in order:

set -a; . $XDG_RUNTIME_DIR/agenix/lseg-credentials; set +a
export RDP_APP_KEY=$LSEG_APP_KEY RDP_USERNAME=$LSEG_USERNAME RDP_PASSWORD=$LSEG_PASSWORD
python scripts/crsp_lseg_splice.py all --out data/ --start 2020-01-01
stepdoeswrites
universeCRSP CIZ common stocks alive at max(dlycaldt) + their daily paneluniverse.parquet, crsp_panel.parquet, asof.txt
mapCUSIP9 → RIC, venue + entity screen, link_status per rowlink.parquet
pullgap-period TRDPRC_1/ACVOL_UNS + TR.TotalReturn1Dlseg_hist.parquet, lseg_ret.parquet
splicereturn-chained continuous panel, source column marks provenancepanel_spliced.parquet
coveragecoverage table + seam sanity checkstdout

Output panel is permno × date with dlyprc, dlyret, dlycap, dlyvol, source, plus RIC, lseg_prc (the raw quote) and adj_ratio on the LSEG rows. source ∈ {CRSP, LSEG}. Keep source in anything downstream — the two halves have different provenance and the LSEG half is unaudited by CRSP.

Red Flags — STOP If You're About To:

  • pd.concat a CRSP price and an LSEG price → STOP. Different adjustment bases; chain on returns (Iron Law 1).
  • Use a RIC without checking its suffix → STOP. 4.3% are foreign-currency venues (Iron Law 2).
  • Use TR.TotalReturn1D without /100 → STOP. It is percent; CRSP is decimal.
  • Join on hdrcusip → STOP. Header CUSIP, not date-effective.
  • Query crsp.dsf / crsp.msf for the recent panel → STOP. Legacy SIZ, frozen at 2024-12-31 — a year before the CIZ cutoff. See crsp-v2.
  • Open a second LSEG session while a pull runs → STOP. One concurrent platform session; the second fails on quota rather than queueing.
  • Count coverage from the return series → STOP. TR.TotalReturn1D is calendar-padded; every security looks complete. Count from the price series.
  • Report the panel as "CRSP data through today" → STOP. It is CRSP through the cutoff and LSEG after, at 91% of the universe. Say so.

Additional Resources

  • references/coverage.md — full measured coverage, per-failure-mode breakdown, validation method, and the reproduction commands
  • scripts/crsp_lseg_splice.py — the pipeline
  • ${CLAUDE_SKILL_DIR}/../crsp-v2/SKILL.md — CIZ tables, the 5-column universe filter, delisting returns, CUSIP inversion
  • ${CLAUDE_SKILL_DIR}/../lseg-data/SKILL.md — session setup, quota, entitlements, rate limits
  • ${CLAUDE_SKILL_DIR}/../lseg-data/references/symbology.md — SymbolTypes enum names and the conversion API
  • ${CLAUDE_SKILL_DIR}/../wrds/SKILL.md — WRDS Postgres connection and .pgpass

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Contents

Format Enforcement

IRON LAW: NO LEGACY SIZ TABLE FOR ANY DATA AFTER 2024-12-31

CRSP's legacy Stock & Indexes Flat File Format 1.0 (SIZ) was **discontinued after the December 2024 data release**. The legacy tables still exist on WRDS and still answer queries — they just stop.

Verified on WRDS PostgreSQL:

TableFormatmax(date)
crsp.dsflegacy SIZ2024-12-31
crsp.stkdlysecurityprimarydataCIZ2025-12-31
crsp.stkmthsecuritydataCIZ2025-12-31
  • SELECT ... FROM crsp.dsf WHERE date >= '2025-01-01' → WRONG. Returns zero rows, silently.
  • SELECT ... FROM crsp.msf WHERE date >= '2025-01-01' → WRONG. Same silent truncation.
  • SELECT ... FROM crsp.stkdlysecurityprimarydata → CORRECT.

A legacy query does not error when it runs off the end of the data. It returns a short panel, the regression runs, and the sample period is quietly wrong.

Handing back a silently truncated panel is not helpful — it is worse than an error, because the user ships it. You reach for crsp.dsf because the legacy names are in your weights and in every paper you have read. The weights are stale. Confirm the format before writing the first SELECT.

Red Flags — STOP Immediately If You're About To:

  • Write crsp.dsf, crsp.msf, crsp.dse, crsp.dsedelist, or crsp.stocknames → STOP. Legacy SIZ, frozen at 2024-12-31. Use the CIZ table (references/tables.md).
  • Write WHERE shrcd IN (10,11) → STOP. shrcd does not exist in CIZ. Five columns replace it (below).
  • Write WHERE exchcd IN (1,2,3) → STOP. exchcd does not exist in CIZ. Use primaryexch.
  • Write abs(prc) or WHERE prc > 0 → STOP. CIZ prices are always positive. Use dlyprcflg.
  • Add a delisting return to dlyret/mthret → STOP. CIZ already embeds it. You are double-counting.
  • Assume mthret reproduces legacy ret → STOP. Different methodology (compounded daily). See references/known-differences.md.
  • Guess what a flag value means → STOP. references/flags.md has all 774 values; crsp.metaflaginfo is live.
  • Substitute comp.secd (or any Compustat price table) because CIZ ran out at 2025-12-31 → STOP. Use the crsp-lseg-splice skill. secd carries no delisting returns, null trfd on ~48% of firm-days, and ajexdi that goes stale through corporate actions (one gvkey prints +17,700% on an unadjusted reverse split). It also applies none of CIZ's universe screens, so its cross-section is not comparable: measured on one event-study design, swapping CRSP for secd left Delaware flat at ~2,075 firms while doubling non-Delaware from 968 to 1,923.

Format Facts

  • The CIZ format shipped in July 2022 and became the only updated format in February 2025 (December 2024 data was the last SIZ release). Both formats sit in the same crsp schema in Postgres and the same crsp library in SAS — the schema name does not tell you which format you are in. The table name does. Reporting "I queried the crsp schema, so this is v2 data" is an unverified claim presented as fact.
  • WRDS-built convenience tables keep their legacy names with a _v2 suffix: crsp.dsf_v2, crsp.msf_v2, crsp.stocknames_v2. CRSP-built tables use the new CIZ names (crsp.stkdlysecuritydata). Both are CIZ; the _v2 ones are pre-joined and wider. Treating the absence of a _v2 suffix as evidence a table is legacy sends you back to the frozen data.
  • On the WRDS Cloud filesystem, SIZ stays at /wrds/crsp/sasdata/a_stock and CIZ is at /wrds/crsp/sasdata/a_stock_v2.
  • Column names carry their frequency as a prefix: Dly, Mth, Qtr, Ann. There is no frequency-agnostic ret or prc in CIZ — DlyRet and MthRet are different columns in different tables, not the same column read from two files.
  • The 62 suffix (crsp.stkdlysecuritydata62) is the 1962-start subset product; the _ind suffix (crsp.stkindmembership_ind) is the full Index database. Stock-only products carry just 4 index series; crsp.indseriesinfohdr_ind carries 274. Querying the wrong one returns zero rows rather than an error, so "that index isn't in CRSP" is usually the wrong suffix, not a missing index.

Universe Enforcement

IRON LAW: NO CIZ COMMON-STOCK SAMPLE WITHOUT ALL FIVE COLUMNS

Legacy `SHRCD IN (10, 11)` was unpacked into **five** CIZ columns across two tables. No single column reproduces it.
sharetype       = 'NS'                 -- No Special Share Type
AND securitytype    = 'EQTY'
AND securitysubtype = 'COM'
AND usincflg        = 'Y'
AND issuertype      IN ('ACOR', 'CORP')
  • WHERE sharetype = 'COM' → WRONG. Returns 0 rows. COM lives at the SecuritySubType level; ShareType is never 'COM' in CIZ. CRSP support confirmed this in writing.
  • WHERE securitysubtype = 'COM' alone → WRONG. Picks up non-US-incorporated firms, ADRs (sharetype='AD'), and REITs (issuertype='REIT') that legacy SHRCD excluded.
  • All five, ANDed → CORRECT.

A universe that silently differs from SHRCD IN (10,11) breaks comparability with every prior paper in the literature — that is an anti-helpful result dressed as a working query. The one-column version is faster to type and returns plausible row counts, which is exactly why it survives review.

Verified distribution in crsp.stksecurityinfohist: EQTY/COM/NS/CORP/Y = 90,515 rows and EQTY/COM/NS/ACOR/Y = 21,682 rows are the two SHRCD 10/11 cells; EQTY/COM/AD/CORP/N (6,279 ADR rows) and EQTY/COM/NS/REIT/Y (2,347 REIT rows) are what the sloppy filter lets in.

Exchange Facts

  • EXCHCD is gone. primaryexch is a single letter: N (NYSE), A (NYSE American), Q (NASDAQ), R (NYSE ARCA), B (BATS), I (IEX), C (Consolidated), X (Unknown), N/A.
  • EXCHCD IN (1,2,3) maps to primaryexch IN ('N','A','Q') — but primaryexch alone also carries the halted and suspended records that legacy EXCHCD split into -2 and -1. To reproduce legacy EXCHCD IN (1,2,3) exactly, add conditionaltype = 'RW' AND tradingstatusflg = 'A'.
  • Legacy EXCHCD = -2 (halted) → tradingstatusflg = 'H'. Legacy EXCHCD = -1 (suspended) → tradingstatusflg = 'S'.
  • The universe columns live on the history table (stksecurityinfohist, one row per attribute-change interval), not only the header. Filtering on stksecurityinfohdr applies today's classification to the whole 1925–2025 panel and back-fills survivorship into the sample.

CUSIP Facts

  • CIZ inverted the CUSIP naming. CUSIP is now the historical CUSIP (legacy NCUSIP); HdrCUSIP is the header/most-recent CUSIP (legacy CUSIP).
  • Code ported from SIZ that joins on cusip therefore changes meaning silently — it starts joining on the historical value. For a point-in-time match to Compustat/IBES this is usually what you wanted; for a header match it is a bug.

Return Enforcement

IRON LAW: NEVER ADD A DELISTING RETURN TO A CIZ RETURN

In SIZ, delisting returns lived only in `crsp.dsedelist`/`msedelist` and every researcher hand-merged them in. **CIZ embeds the delisting return directly in the daily and monthly return series.**

Verified for PERMNO 10002 (delisted 2013-02-15):

dlycaldtdlyprcdlyprcflgdlyretdlydelflg
2013-02-142.92TR-0.010170N
2013-02-152.98TR0.020548N
2013-02-190.00DA0.010906Y

The 2013-02-19 row is the delisting return. crsp.stkdelists still exists, but it is for the delisting reason and event detail — not for patching the return series.

  • coalesce(dlyret,0) + coalesce(dlret,0) → WRONG. Double-counts.
  • (1+dlyret)*(1+dlret)-1 → WRONG. Same double-count, compounded.
  • Use dlyret as-is → CORRECT.

Silently inflating delisting-month returns reintroduces exactly the survivorship artifact the merge was supposed to fix. The old merge is muscle memory and the result looks normal — the bias only shows up in the delisting tail, which is where the identification usually lives.

Return Facts

  • MthRet is a compound of daily returns within the month, with dividends reinvested on the ex-date. Legacy RET was a month-end-to-month-end holding period return with dividends reinvested at month-end. These are different estimators, not a renaming. WRDS found 90 stock-months differing by >100% and 3,479 differing by >5%.
  • The same change applies to delisting returns (DelRet), where the divergence can be larger.
  • DLRETX (delisting return without dividends) does not exist in CIZ. There is no substitute.
  • DlyRetMissFlg and DlyRetDurFlg explain missing and multi-period returns. DlyRetDurFlg = 'D1' is the ordinary adjacent-trading-day case; P1–P9 mean the return spans 2–10 trading periods; MR means missing. Filtering on this flag replaces the old ad-hoc "drop returns after a gap" heuristics.
  • DlyPrc is always positive. The bid-ask-average case that legacy encoded as a negative price is now DlyPrcFlg = 'BA'; a real closing trade is 'TR'; a delisting amount is 'DA'. Never call abs().

What Changed at a Glance

ConceptLegacy SIZCIZ (v2)
Daily priceprc (negative = bid/ask avg)dlyprc (always positive) + dlyprcflg
Daily returnretdlyret (delisting return included)
Monthly returnret from msfmthret (compounded daily)
Common stockshrcd IN (10,11)5 columns (see above)
Exchangeexchcd IN (1,2,3)primaryexch IN ('N','A','Q')
Historical CUSIPncusipcusip
Header CUSIPcusiphdrcusip
Delisting codedlstcd (3-digit)delactiontype, delstatustype, delreasontype, delpaymenttype
Distribution codedistcd (4-digit)distype, disfreqtype, dispaymenttype, disdetailtype, distaxtype, disorigcurtype, disordinaryflg
Adjustment factorscfacpr, cfacshr in dsfcrsp.stkdlycumulativeadjfactor (separate table)
Market indexvwretd in dsfindno=1000200 in crsp.inddlyseriesdata
Issuer attributesmixed into stocknamescrsp.stkissuerinfohdr / stkissuerinfohist (PERMCO-keyed)

Table Map

Most-used CIZ tables. Full catalog with verified columns: references/tables.md.

TableGrainUse for
crsp.stkdlysecurityprimarydatapermno × day, 12 colsDaily returns/prices/cap — default daily table, ~7 GB
crsp.stkdlysecuritydatapermno × day, 32 colsAdds bid/ask, high/low, open, prev-price, dividend amounts — ~20 GB
crsp.stkmthsecuritydatapermno × month, 37 colsMonthly aggregates + identifiers
crsp.stkqtrsecuritydata / stkannsecuritydatapermno × qtr / yearPre-aggregated panels; check qtrcompflg/anncompflg
crsp.stksecurityinfohistpermno × intervalUniverse filters, historical CUSIP/ticker
crsp.stksecurityinfohdrpermnoCurrent/header attributes only
crsp.stkissuerinfohdr / stkissuerinfohistpermcoIssuer-level SIC/NAICS/ICB, non-duplicated issuer counts
crsp.stksharespermno × intervalShares outstanding history
crsp.stkdistributionspermno × exdt × seqDividends, splits, factors
crsp.stkdelistspermnoDelisting reason/status detail (not for returns)
crsp.stkdlycumulativeadjfactorpermno × daydlycumfacpr, dlycumfacshr, dlyshrout
crsp.inddlyseriesdata / indmthseriesdataindno × periodIndex returns and levels
crsp.stkindmembership_indpermno × indnoIndex constituents (S&P 500 = indno 1000500)
crsp.dsf_v2 / msf_v2 / stocknames_v2WRDS-builtPre-joined convenience tables (identifiers + data + shrout + factors)
crsp.metafileinfo, metaiteminfo, metaflaginfo, metasiztocizmetadataSelf-documenting schema — query these instead of guessing

Canonical Queries

All queries below were executed against WRDS PostgreSQL. Full set with output: references/queries.md.

Common-stock daily panel (the SHRCD 10/11 + EXCHCD 1/2/3 equivalent):

SELECT d.permno, d.dlycaldt, d.dlyprc, d.dlyret, d.dlycap, d.dlyvol
FROM crsp.stkdlysecurityprimarydata d
JOIN crsp.stksecurityinfohist h
  ON h.permno = d.permno
 AND d.dlycaldt BETWEEN h.secinfostartdt AND h.secinfoenddt
WHERE d.dlycaldt BETWEEN %(start)s AND %(end)s
  AND h.sharetype = 'NS' AND h.securitytype = 'EQTY' AND h.securitysubtype = 'COM'
  AND h.usincflg = 'Y' AND h.issuertype IN ('ACOR','CORP')
  AND h.primaryexch IN ('N','A','Q')
  AND h.conditionaltype = 'RW' AND h.tradingstatusflg = 'A';

The BETWEEN secinfostartdt AND secinfoenddt join is mandatory — it is what makes the classification point-in-time. Dropping it applies the security's final classification to its entire history.

Market capitalization — do not compute it. dlycap (and mthcap) is CRSP's own capitalization in $ thousands, already on the row. dlyprc * shrout reintroduces the precision and rounding differences CRSP documented.

Market index return — join on indno, do not look for vwretd:

SELECT m.permno, m.mthcaldt, m.mthret, i.mthtotret AS vwretd, i.mthprcret AS vwretx
FROM crsp.stkmthsecuritydata m
JOIN crsp.indmthseriesdata i ON i.mthcaldt = m.mthcaldt AND i.indno = 1000200;

1000200 = CRSP NYSE/NYSEMKT/Nasdaq/Arca Value-Weighted (vwretd/vwretx), 1000201 = Equal-Weighted (ewretd/ewretx), 1000502 = S&P 500 Composite (sprtrn = dlyprcret/mthprcret).

Compustat merge — the CCM link is unchanged by CIZ. crsp.ccmxpf_lnkhist is still keyed on lpermno, so existing CCM code ports as-is once the CRSP side is CIZ.

Additional Resources

  • references/tables.md — full CIZ table catalog, verified column lists, grain and key columns
  • references/siz-to-ciz.md — column-by-column crosswalk from every legacy SIZ file
  • references/flags.md — all 774 flag values across 62 flag types, dumped from crsp.metaflaginfo
  • references/known-differences.md — WRDS/CRSP-documented value discrepancies and the monthly-return methodology change
  • references/queries.md — verified query recipes (universe, delisting, factors, indexes, CCM, MSE-style rebuilds)
  • references/indexes.md — INDNO/INDFAM conventions, the +400 monthly rule, S&P 500 series, decile statistics
  • examples/ciz_panel.py — end-to-end Python pull of a CIZ common-stock panel
  • ${CLAUDE_SKILL_DIR}/../crsp-lseg-splice/SKILL.md — CRSP updates annually, so the CIZ daily file stops at the last December. That skill carries the panel forward to T-1 with LSEG via CUSIP→RIC, and has the measured coverage (~91% of the common-stock universe) and the two splice hazards (adjustment basis, foreign-venue RICs).
  • ${CLAUDE_SKILL_DIR}/../../skills/wrds/SKILL.md — connection, .pgpass, WRDS Cloud/SGE rules (this skill assumes them)
  • CRSP source PDFs (WRDS login required): User Guide, Cross-Reference Guide, Metadata Guide, Executive Summary
  • WRDS transition pages: Announcement, Transition FAQ, Index Overview, Recreate MSE Tables

Data Context Extractor

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Extract tribal knowledge about a dataset or database and generate a reusable data context skill.

## The Iron Law of Data Context

YOU MUST interview the user before generating ANY skill content. This is not negotiable.

You MUST NOT:

  • Generate a data context skill from schema alone
  • Skip entity disambiguation
  • Assume metric definitions without confirmation
  • Output a skill without user review of each section

If you're about to write a skill based on your assumptions, STOP. Interview first.

Two Modes

1. Bootstrap Mode

Trigger: No existing data context skill for this project/dataset.

Create a new data context skill from scratch by interviewing the user about their data.

2. Iteration Mode

Trigger: Existing data context skill exists, user wants to add a new domain or update.

Read existing skill, identify gaps, interview for the new domain, merge into existing skill.

Integration with Existing Data Access Skills

Before interviewing, check for existing data access skills that already encode tribal knowledge:

1. READ existing skills: /wrds, /lseg-data, or any project-local data skills
2. IDENTIFY what they already cover (table names, filters, field mappings, gotchas)
3. DO NOT re-document what existing skills handle
4. FOCUS the interview on the project-specific layer:
   - Which specific tables/fields from WRDS or LSEG does THIS study use?
   - How are identifiers linked across sources? (permno ↔ gvkey, RIC ↔ ISIN ↔ cusip)
   - What sample filters define the study universe? (date range, exchange, firm type)
   - What derived variables or transformations are project-specific?

The generated data context skill should reference existing skills rather than duplicate them:

## Data Sources

| Source | Skill | Tables/Fields Used |
|--------|-------|--------------------|
| WRDS | `/wrds` | comp.funda (at, lt, ceq), crsp.msf (ret, prc) |
| LSEG | `/lseg-data` | TR.F.TotRevenue, TR.GICSSector |
| Local | DuckDB | data/processed/merged_panel.parquet |

For connection details and critical filters, see the referenced skills.

The Interview Process

1. DISCOVER data sources
   → What databases/files/APIs? Connection details?
   → For each: what dialect? (PostgreSQL, DuckDB, SQLite, Snowflake, etc.)
   → IMPORTANT: If user mentions WRDS or LSEG, read the corresponding skill first
     and ask only about project-specific usage, not general access patterns

2. MAP entities
   → What are the core entities? (users, transactions, products, etc.)
   → How do they relate? (foreign keys, join paths)
   → CRITICAL: Disambiguate entity names
     - "user" vs "account" vs "customer" — are these the same?
     - "order" vs "transaction" vs "purchase" — clarify overlaps
     - Document the canonical name and any aliases

3. DEFINE key metrics
   → What are the business-critical metrics?
   → For each metric:
     - Exact definition (SQL or formula)
     - Known edge cases
     - Common misinterpretations
     - Time grain (daily, monthly, etc.)

4. DOCUMENT data hygiene
   → Known data quality issues
   → Fields that lie (e.g., "created_at" that's actually "imported_at")
   → Nulls that mean something specific
   → Enums/codes that need translation
   → Date ranges with reliable data vs backfill periods

5. CAPTURE common gotchas
   → Joins that explode (many-to-many lurking as one-to-many)
   → Filters that are always needed (e.g., "WHERE is_deleted = false")
   → Time zones and their traps
   → Slowly changing dimensions
   → Tables that look useful but aren't (deprecated, partial, test data)

6. COLLECT common query patterns
   → Frequently needed aggregations
   → Standard date filters or cohort definitions
   → Boilerplate CTEs that everyone copies

Interview Style

Ask questions in batches of 3-5. Don't overwhelm with everything at once.

Round 1: Data Sources

  • What data sources does this project use? (WRDS, LSEG, local files, other databases?)
  • For known sources (WRDS/LSEG): which specific tables and fields do you pull?
  • For local data: what formats (parquet, CSV, SQLite, DuckDB)? Where stored?
  • How do you link identifiers across sources? (permno, gvkey, cusip, ISIN, RIC)

Round 2: Core Entities (after Round 1 answers)

  • What are the main entities in this data?
  • Walk me through how [Entity A] relates to [Entity B]
  • Are there any entity names that are ambiguous or have aliases?

Round 3: Metrics & Definitions (after Round 2 answers)

  • What metrics does your team track most?
  • For [Metric]: what's the exact calculation? Any edge cases?
  • Are there metrics people commonly get wrong?

Round 4: Data Quality & Gotchas (after Round 3 answers)

  • What are the known data quality issues?
  • Any fields that are misleading or have surprising behavior?
  • What's the most common mistake new analysts make with this data?

Round 5: Common Patterns (after Round 4 answers)

  • What queries do you find yourself writing repeatedly?
  • Any standard filters or cohort definitions everyone uses?
  • What boilerplate would save the most time?

Output: Generated Skill

After the interview, generate a skill with this structure:

project-name/
├── .claude/
│   └── skills/
│       └── data-context/
│           ├── SKILL.md          # Main skill file
│           └── references/
│               ├── entities.md   # Entity definitions and relationships
│               ├── metrics.md    # Metric definitions with SQL/formulas
│               └── gotchas.md    # Data quality issues and common pitfalls

SKILL.md Template

---
name: [project]-data-context
description: "Data context for [project]. Entity definitions, metric calculations, data quality notes, and common patterns for [data domain]."
---

# [Project] Data Context

## Data Sources

| Source | Skill/Dialect | Tables/Fields Used |
|--------|---------------|--------------------|
| [WRDS] | `/wrds` | [specific tables and fields for this project] |
| [LSEG] | `/lseg-data` | [specific fields for this project] |
| [Local] | [DuckDB/CSV/Parquet] | [file paths or database] |

For connection details and critical filters, see the referenced skill. This context covers only project-specific usage.

## Entity Map

[Entity relationship summary — which entities exist, how they connect]

See `references/entities.md` for full definitions.

## Key Metrics

[Top 3-5 metrics with brief definitions]

See `references/metrics.md` for exact calculations and edge cases.

## Critical Gotchas

[Top 3-5 gotchas that catch analysts]

See `references/gotchas.md` for full list.

## Common Patterns

[Frequently used query snippets or data access patterns]

Generating the References

For each reference file, include:

  • entities.md: Entity name, aliases, primary key, important fields, relationships, common joins
  • metrics.md: Metric name, definition (SQL/formula), edge cases, common misinterpretations, time grain
  • gotchas.md: Issue description, why it happens, how to detect, how to handle

Exit Gate

Before writing skill files, execute this gate:

  1. IDENTIFY: All 5 interview rounds completed
  2. RUN: Read back each generated section to the user
  3. READ: Verify entities, metrics, gotchas sections all have substantive content (not placeholders)
  4. VERIFY: User approved each section via explicit confirmation
  5. CLAIM: Only generate skill files after ALL checks pass

Skipping this gate produces a skill based on your assumptions, not the user's knowledge. That skill will mislead every future analysis.

Verification

Before finalizing the skill:

  1. Read back each section to the user for confirmation
  2. Test at least one metric definition against actual data if possible
  3. Check entity relationships match what the user described
  4. Confirm the gotchas list captures what the user considers most important

Iteration Mode Details

When adding to an existing data context skill:

  1. Read the existing skill files
  2. Identify what's already documented
  3. Interview for the NEW domain only (don't re-interview existing content)
  4. Merge new content into existing files, maintaining consistent formatting
  5. Flag any conflicts with existing definitions (e.g., metric redefinition)

Where to Place the Output

  • Project-specific: .claude/skills/data-context/ in the project root
  • The generated skill becomes available to all future Claude sessions in that project
  • Subagents spawned by ds-delegate will have access to it automatically

Data Context Facts

  • Schema and column names carry structure, not semantics: revenue may be gross, net, or recognized; a user_id foreign key does not say whether users can hold multiple active accounts. A metric definition or relationship cardinality inferred from names is an unverified claim presented as fact — every downstream analysis that loads the generated skill inherits the error. The interview takes 10 minutes; weeks of wrong analysis is what it prevents.
  • Gotchas are the most valuable section of the generated skill and the easiest to defer. "Fill in later" means never — capture them during the interview while context is fresh, or the skill ships as a hazard that misleads every future analysis.
  • There is no "standard SQL" in practice. When the user says "standard SQL", get the exact dialect — definitions in metrics.md must be runnable as written.
  • WRDS connection mechanics are covered by /wrds, and LSEG field prefixes by /lseg-data. Re-documenting them here duplicates and drifts; document only the project-specific tables, fields, and filters THIS project uses.

de-ai-revise — make prose read less AI-generated

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

A writing-improvement tool. It audits a draft with three corpus-validated scorers, then rewrites only the flagged spans so the prose reads less like an LLM wrote it — plainer diction, burstier rhythm, fewer machine tics — while leaving already-human passages untouched.

This is the GENERATION side of the AI-writing apparatus, not detection. Detecting polished AI was proven near-impossible (60%+ false-positive rates on real human writing); this skill never renders a verdict on authorship. It improves readability for a human reader. The scorers GUIDE which spans to revise; they are not a target to maximize.

This is a BACKSTOP, not the main event. The primary lever for human-reading prose is the GENERATION contract upstream — writing-draft now drafts topic-sentence-led and proportional (varied paragraph/sentence length), which is what produces human burstiness in the first place. A draft generated well needs little here. If de-ai-revise is finding a lot, the fix usually belongs upstream (the outline's POINTs aren't real topic sentences, or the draft padded uniformly), not in a heavy span-by-span rewrite here. Use this to catch residue, not to manufacture rhythm a flat draft never had.

## The Iron Law of Goodhart

THE SCORERS GUIDE; THEY DO NOT GRADE. NO EDIT THAT IMPROVES A NUMBER BUT NOT THE READING. This is not negotiable.

A human reads the output. Mechanically maxing burstiness (chop every sentence), nuking every em-dash, or swapping every flagged word degrades prose to win a composite — that is the failure this skill exists to prevent. Revise a span only when the rewrite reads better to a person. Leave a flagged span alone when the author's choice is the right one (see Preserve-Human below).

The three scorers (all corpus-gated — do NOT re-derive)

scripts/de_ai_audit.py folds them into one line-anchored span list. Every signal was gated against a 14.3M-sentence law+finance corpus, so flags are AI defaults real scholars don't write — not generic "fancy word" lint.

The scorers themselves now live in ${CLAUDE_PLUGIN_ROOT}/scripts/prose-audit.py, the plugin's single deterministic prose audit, and de_ai_audit.py is a thin wrapper over its --profile de-ai view. The output shape below is unchanged and will stay that way — this skill needs the REWRITE view (a worklist of spans with plain replacements), which is a different shape from the audit's severity-ranked, id-bearing span list. Use prose-audit.py directly for anything that is not a de-AI rewrite: it also carries the wikipedia AI-tell tables, the domain style guides, and the provenance-leak class this profile is blind to.

ScorerCatchesRemedy
Scored AI-tics (ai-anti-patterns/constraints/scored-tics-patterns.py)phrase/structure tics that passed the ~0-human-rate gate (sev1-5)rewrite the construction; these have no honest use
Tiered diction (references/diction.yaml)fancy→plain words, tiered by corpus ratealways_flag → swap on sight; cluster → fix when 2+/para; density → vary at saturation; dropped → never touch (legal-normal)
British spelling (BRITISH in de_ai_audit.py)locale mismatch in US-register prose (recognise, behaviour, whilst, labelled) — LLMs emit these into US documents from mixed training corporaswap for the US form; drop the check for a UK-register document
Stylometrics (ai-anti-patterns/scripts/style_metrics.py)rhythm/structure: composite_human_likeness 0-100, em-dash, metronomic runs, opener transitions, nominalization, false precision, burstiness/passive advisoriesvary sentence length toward bursty; em-dash → semicolon/period; plainer Latinate→Anglo-Saxon; round a summarising figure to a fraction

Modes

ModeTriggerBehavior
rewrite (default)"de-AI this", "make it less AI"audit → rewrite flagged spans → one corrective 2nd pass → return an edits-made + verification report (NOT the whole file)
detect-only"just flag", "scan", "what AI tells are in this", "audit only"audit only; report flagged spans + composite/tic-density; no edits
edit-in-place"fix draft.md directly", "clean the file in place"minimal targeted Edits to the file; preserve already-human paragraphs; re-audit after

Default to rewrite when unspecified.

Process (the spec)

START
  │
  ├─ Step 1: AUDIT — run de_ai_audit.py --json on the target
  │     uv run --with pyyaml python3 ${CLAUDE_SKILL_DIR}/scripts/de_ai_audit.py --json <file>
  │     Read: composite_human_likeness, tic_density, spans[], advisories[]
  │
  ├─ detect-only? → report spans + signals, STOP.
  │
  ├─ Step 2: REWRITE the flagged spans (NOT the whole draft)
  │     - tic spans                 → rewrite the construction (no honest use)
  │     - diction:always_flag       → swap for the listed plain replacement
  │     - diction:cluster           → fix enough of the cluster to drop below 2/para
  │     - style:em_dash             → recast as semicolon / period / comma — but NOT all (see Preserve)
  │     - style:false_precision     → round to a high-level fraction ("1.3771 percent" → "about one
  │                                   and a half percent"); KEEP the exact value if the sentence sits
  │                                   next to the exhibit that reports it
  │     - advisories (burstiness)   → vary sentence length where it reads flat; do NOT chop for chop's sake
  │     PRESERVE already-human passages (no spans) untouched.
  │     PRESERVE quoted material, block quotes, code, footnote citations.
  │
  ├─ Step 3: ONE corrective 2nd pass
  │     Re-run de_ai_audit.py. Fix spans the first pass introduced or missed.
  │     STOP at 2 passes — a 3rd rarely finds more and costs a full regeneration.
  │
  └─ Step 4: REPORT (edits-made + verification), NOT the whole file
        - what changed and why (span → before → after, grouped by scorer)
        - before/after composite + tic-density (must improve or hold; if it dropped, you over-edited)
        - spans deliberately LEFT (author's voice / quoted / domain term) and why

If text and flowchart disagree, the flowchart wins.

Preserve-Human (the other half of Goodhart)

The composite penalizes em-dashes hard, and real legal scholarship — including this user's own published prose — uses them deliberately. Do NOT zero them out.

  • Em-dashes: thin clusters and the clearest default-connector uses; KEEP em-dashes that set off a genuine appositive or a deliberate aside. Target fewer, not zero.
  • dropped-tier diction (significant, robust, leverage, comprehensive, …): NEVER flag or swap — these are legal/finance-normal; the audit already excludes them.
  • Quoted text, block quotes, statutory language, party names, code, citations: flag at most; never rewrite someone else's words or a term of art.
  • Footnotes are auto-excluded: the audit MASKS pandoc inline ^[...] and markdown [^id]: footnotes before scoring, so findings never land inside them (citation/legal-normal text). You will not see footnote spans to triage; if you ever do, do not edit them. (--keep-footnotes disables masking for debugging the raw signal only.)
  • British spelling in a genuinely UK-register document: the check assumes US register. For a UK journal or an English court filing, ignore spelling:british entirely — do not "correct" an author writing in their own dialect.
  • A flagged span the author clearly chose (a fragment for emphasis, a repeated key term over elegant variation): leave it; note it in the report.

Fact rows

  • The synthetic-AI baseline scores composite ~27 and tic-density 100; a real human legal draft scores ~55-65 with em-dashes as nearly the whole signal. So a composite in the 50s is NOT "AI" — it is a human who likes em-dashes. Treating the composite as a pass/fail bar instead of a span guide produces voice-destroying edits and is the exact failure the corpus tiering was built to prevent.
  • diction.yaml dropped tier exists because "significant/robust/leverage" fire on every real law-review article; a linter that flags them is worse than none. The audit omits them — if you hand-flag one anyway, you reintroduced the false positive.
  • The British-spelling map deliberately EXCLUDES words correct in both dialects — analysis, characteristic, basis, emphasis, thesis, hypothesis, and practice/licence as nouns. The -sis nouns are not the -ise verbs. Adding any of them turns the check into a false-positive generator, which is the exact failure the corpus tiering elsewhere in this skill exists to prevent.
  • It matches STRICTLY (\bword\b), not via _word_rx, because every inflected form is enumerated. Using _word_rx made "recognise" also match inside "recognised" — two spans for one word, one carrying the wrong replacement.
  • A 3rd rewrite pass regenerates the whole span set for ~0 new fixes (CAP AT 2). The built-in corrective pass IS pass 2; "iterate to convergence" does not stack on it.
  • Em-dash count near zero after a de-AI pass is over-editing, not success: you optimized the metric and flattened the author's rhythm. Fewer, not none.

Red Flags — STOP

  • About to swap every flagged diction word → STOP. Cluster/density tiers are advisory; fix enough to clear the threshold, keep the ones that read right.
  • About to delete every em-dash → STOP. Target fewer; keep deliberate appositives.
  • About to rewrite a paragraph with zero spans because it "feels AI" → STOP. The audit found it human; trust the corpus over the vibe.
  • About to run a 3rd rewrite pass → STOP. Cap is 2.
  • About to return the whole rewritten file by default → STOP. Return the edits-made report unless the user asked for the full text.
  • About to rewrite quoted/statutory text → STOP. Flag it; never alter someone else's words.

When invoked inside the writing workflow

  • /writing-verify runs ${CLAUDE_PLUGIN_ROOT}/scripts/prose-audit.py on every draft before dispatching its prose reviewers and INJECTS the resulting spans into their prompts as evidence — the reviewer is not asked to run a scorer, and a reviewer that cites none of the hard spans it was handed is recorded as unreliable. Those spans become AI-ism findings (advisory minors unless they cluster into a major).
  • /writing-revise applies this skill (rewrite mode) as a non-optional pass on every edited draft after fixing REVIEW.md issues, then re-audits. The substrate gate is unchanged: AI-prose spans are advisory polish, not blocking criticals.

Deep Research

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Web-grounded deep research via Gemini Interactions API.

IRON LAW: Always Use the Script

NEVER call the Gemini Interactions API manually. ALWAYS use bun deep-research.ts. This is non-negotiable.

cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts "research query"

The script handles model selection, interaction submission, polling with retries, timeout enforcement, and report formatting. Calling the Interactions API by hand means you skip retry logic, miss the 65-minute timeout guard, and lose the saved report file.

When to Use

Deep Research is the LAST RESORT in the librarian's source hierarchy. It costs $1-7 per query, takes minutes not seconds, and produces a broad web-grounded report rather than precise academic citations.

User needs information on a topic
    |
    v
Check curated sources FIRST (all must be exhausted):
  1. Paperpile bib (local library)
  2. Google Scholar (scholar search/lookup)
  3. Consensus (consensus search)
  4. NLM notebooks (existing sources)
  5. Readwise (highlights/full-text)
    |
    v
Are there still significant gaps?
    |
    +-- NO --> Stop. Curated sources are sufficient.
    |
    +-- YES
          |
          v
        Has the user explicitly asked for broader/deeper research?
          |
          +-- NO --> Report gaps, suggest deep research, WAIT for approval.
          |
          +-- YES --> Run deep-research.ts

Trigger signals (use this skill):

  • User says "deep research", "thorough investigation", "comprehensive report"
  • User explicitly asks for web-grounded synthesis beyond academic databases
  • Curated sources have been exhausted and user wants more

NOT-triggers (do NOT use this skill):

  • User wants a specific known paper -- use Scholar or Consensus
  • User wants papers from their own library -- use Paperpile bib or Readwise
  • User wants citation metadata -- use Scholar with --bibtex
  • User has not exhausted curated sources yet
  • User has not explicitly approved the cost

Commands

Basic (thorough report)

cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts "What is the current state of mandatory climate disclosure regulation worldwide?"

Fast (quick scan)

cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts --fast "ESG disclosure enforcement mechanisms"

Resume polling an existing interaction

cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts --status <interaction-id>

JSON output (raw interaction object)

cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts --json "corporate governance reform trends"

Models

ModelFlagQueriesTimeCostUse when
deep-research-max(default)~160 web queries5-20 min~$3-7Thorough reports, literature reviews, multi-faceted questions
deep-research--fast~80 web queries2-10 min~$1-3Quick scans, narrow questions, time-sensitive requests

Cost Warning

$1-7 per query. NEVER auto-invoke. Always ask the user first.

This is the most expensive tool in the librarian's toolkit. Before every invocation:

  1. Confirm the user explicitly requested deep research
  2. Report which curated sources were already checked and what gaps remain
  3. State the estimated cost range ($1-3 for fast, $3-7 for thorough)
  4. Wait for the user's go-ahead

Cost Facts

  • Even --fast costs $1-3 and takes 2-10 minutes; Scholar, Consensus, Paperpile, NLM, and Readwise are free and return in seconds. Reaching for deep research because it seems "faster than searching multiple sources" is counterproductive on its own terms.
  • "The curated sources didn't have enough" is only a fact after ALL five (Paperpile, Scholar, Consensus, NLM, Readwise) have been tried. Before that it is an assumption — and $3-7 spent on an assumption without the user's go-ahead is unauthorized spending.

Integration with Librarian Workflow

Paperpile bib  -->  Scholar  -->  Consensus  -->  NLM / Readwise
       (local)       (free)        (free)          (free)
                                                      |
                                                      v
                                          [GAPS IDENTIFIED + USER ASKS]
                                                      |
                                                      v
                                          Deep Research (THIS SKILL)
                                            $1-7, 2-20 min
                                                      |
                                                      v
                                          Curate results to NLM
                                          (save report, extract key sources)

After deep research completes:

  1. Present the report to the user
  2. Extract key sources cited in the report
  3. If the user wants to keep specific sources, add them to NLM notebooks
  4. The report is also saved to /tmp/deep-research-<id>.md for reference

Red Flags

#ActionWhy WrongDo Instead
1Using before checking curated sourcesScholar, Consensus, Paperpile, NLM, Readwise are free and fastExhaust all five curated sources first
2Using for known paper lookupDeep Research returns web-grounded reports, not citation metadataUse scholar lookup --bibtex or scholar cite
3Launching without user's explicit request$1-7 per query; unauthorized spending violates trustReport gaps, state cost, wait for go-ahead
4Calling Interactions API directly (without the script)Skips retry logic, timeout guard, report file saveAlways use bun deep-research.ts
5Ignoring the report and re-searching the same topicWastes another $1-7 for information you already haveRead the saved report at /tmp/deep-research-<id>.md

Environment

VariableRequiredDescription
GOOGLE_API_KEYYesGemini API key with Deep Research access

Timeout: 65 minutes maximum polling duration (covers the API's 60-min hard limit plus buffer).

Polling interval: 10 seconds between status checks.

Retries: Up to 3 retries on transient errors (503, 429, service_unavailable, rate_limit).

Output

The script prints the research report to stdout as Markdown. It also saves the report to /tmp/deep-research-<interaction-id>.md for later reference.

With --json, it prints the full interaction object as JSON (useful for debugging or extracting structured metadata).

On failure, it prints the error to stderr and exits non-zero.

Limitations

  • Beta API -- the Gemini Interactions API is in preview; behavior may change
  • No structured output -- reports are free-form Markdown, not JSON-structured citations
  • Max 60 minutes -- the API enforces a hard time limit on research sessions
  • Cost per query -- $1-7 depending on model; no free tier
  • Web-grounded, not academic-specific -- results include news, blogs, and general web sources alongside academic content; always cross-reference claims against Scholar/Consensus for academic rigor
  • No incremental results -- the report arrives all at once when polling completes; no streaming

dev — a code change, run through work with a test-first gate

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

The lifecycle is work. Read it and follow it. This file is a delta: it supplies the domain — the CLARIFY axes, the task-row shape, the lens, the mechanical checks, the refs, the authority text. It ships no workflow.js and restates none of work's mechanics.

What makes a run dev rather than plain work is one thing: every implementation task carries a redCommand (the one exception: redDisposition — see below), so work's probes execute the failing test before the implementer and the passing test after it, and the JS reads both exit codes. A task whose acceptance cannot be expressed as one command that fails now and passes later is not ready to plan.

Phase 1 — CLARIFY

work's Phase 1, on these axes. Ask them before any reconnaissance: code says how the system works, not what the user wants.

  1. Outcome — what behaviour exists after this run that does not exist now, and what does done look like for one user.
  2. Exclusions — what must not change: files, interfaces, dependencies, behaviour other callers rely on.
  3. Automated test framework and the exact command — the runner, and the literal command string a probe will execute. No framework in the repo means the first task is test infrastructure, decided here. A manual-only test proposal is a blocker: resolve an automated approach or leave this workflow. Never waive it silently.
  4. The user workflow, production protocol and transport the real test must exercise — the action sequence a user performs, the protocol production actually uses, and the observable result. A test over an alternate path proves nothing about the one that ships.
  5. The first failing test and what RED looks like — which test fails first, and the failure output that counts. A syntax error, an import error, a broken fixture, a missing dependency or an unrelated failure elsewhere in the suite is not the RED this workflow accepts. This is computed, not asserted: red_probe_gate classifies a collection/import error as could-not-run and refuses to dispatch. For a surface that does not exist yet, that means a stub — one that exists and raises, so the failure is an assertion rather than an ImportError — and the stub is named in the plan's scaffoldPaths so the main thread may author it while the run is armed.
  6. Required runtime evidence — what has to be observed at runtime rather than read in source: logs, HTTP/WS exchanges, rendered output, exit codes, screenshots.
  7. Review surface — working tree, commit range, or PR.
  8. Language and its language server — the language this change is written in (observed for an existing repo, decided here for new code), and the LSP that follows from it. Install it before reconnaissance, project-scoped, so recon and every implementer navigate by symbol rather than by grep.

Installing the LSP is two things, and the plugin is only one of them.

claude plugin install <lang>-lsp@claude-plugins-official --scope project

--scope project writes enabledPlugins into .claude/settings.json — a committed file, so it enables the server for everyone who opens the repo; --scope local writes .claude/settings.local.json instead. Official servers: pyright (py), typescript (ts/js), gopls, rust-analyzer, clangd, jdtls, kotlin, csharp, php, ruby, lua, swift, liquid.

The plugin declares a command, never a binary — pyright-lsp is just {"command": "pyright-langserver"}. Enabling it while the server is absent registers something that fails at first use with ENOENT: Executable not found in $PATH.

The binary is global, and that is not a preference. Claude Code resolves command from the process PATH only — never the plugin root, never ${CLAUDE_PROJECT_DIR}, never a project node_modules/.bin or pixi env. A project-local language server is unreachable by construction; install it the way this machine installs system tools (nix here, or the distro package), not with pixi add. What stays project-scoped is the enablement above and the server's own config — pyrightconfig.json points a global pyright at the project interpreter, and a global typescript-language-server still loads the project's own typescript from node_modules.

Registration is separate again: a mid-session install is inert until /reload-plugins. Self-send it rather than asking the user to type it. Then prove it with one LSP call on a real file in the repo — documentSymbol is enough. The install command's exit code says the plugin was enabled, not that a server answers.

work's remaining axes are taken as work states them, with one domain binding: work axis 4 (observable success criteria) is answered with the target project's own test, lint and build commands, and those strings become mechanicalChecks verbatim.

Then reconnoitre — a scouting Explore or Plan subagent, never a read of the whole tree into this conversation — and return: the entry point and data flow, the integration boundaries, the existing test harness and the files a new test extends, and the decisions code cannot answer. Re-ask only what reconnaissance newly raised.

Before planning, present 2–3 feasible architectures with their boundaries, testing implications and trade-offs, and obtain an explicit choice. A sole viable option still needs its trade-off accepted.

Phase 2 — PLAN

work's Phase 2. The plan opens with frontmatter workflow: dev — required, so a context clear at approval resumes here and not in work.

Three domain requirements on the table:

  • redCommand per implementation task — one invocation, no shell operators, failing now for the intended missing behaviour and passing once the task is done. It goes in the plan's Run sizing Test-first: block too, because it costs 2 agents against the fan-out ceiling.
  • redDisposition — the only substitute for redCommand, and plan-lint (dev-red-disposition) refuses any other text: no-behaviour: <reason> (docs, config, rename — nothing to pin) or already-covered: <test that pins it>. A task carrying one is judged on its acceptanceCmd/verification, not on a test that went RED.
  • refs on every task row and on the lens — required, may be empty. work's spine does not validate it; wc-probe P7 refuses an absent key in THIS file, so a live run assembled from an approved plan is unchecked. Write refs: [] to state "no domain rules" rather than omitting the key.
  • Narrow writablePaths — the probe runs a command that loads code the implementer can edit, so a wide writable set lets the implementer reach the thing proving its own RED.
  • scaffoldPaths for greenfield only, naming files and not directories — the test suite needs no entry (no task writes it, so the guard already allows it); the stub does, because it is the implementer's output and a precondition of its own red gate. A scaffold as wide as a task's writable surface is scaffold-swallows-task at plan-lint, and it is the guard turned off with extra steps.

Phase 3 — GOAL

work's Phase 3 unchanged.

Phase 4 — the work call

The args go in the plan's <!-- work:dispatch --> arming block, and the dispatch is work's own work-dispatch.sh — never a hand-written runner line. That script owns the TIER 1 plan-lint gate, which refuses to dispatch on a major/critical plan finding and fails CLOSED on a verdict it cannot count; hand-rolling the invocation silently drops it. work owns the wait, the result handling and the return shape too, and work-result.sh reads the verdict. This run's projectDir is the session repo, so work's own run directory is already inside it and no --run-dir override applies. There is no built-in Workflow call — the guard at ~/.claude/hooks/main-thread-guard.sh denies that tool outright.

bash ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/work-dispatch.sh   # armed plan; or pass one
bash ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/work-dispatch.sh --provider codex   # "run this through codex"

Forward the provider. A provider named in this skill's $ARGUMENTS, however it is spelled — --provider codex, --dispatch codex, "run this on gpt" — is --provider codex on the line above. --provider is the only spelling the scripts take, and both reject the others by name. Omitting it silently runs the user's codex request on claude.

const projectDir = /* the session repo */
const goal = /* one sentence */
const planPath = /* the approved plan's path */
const testCmd = /* the project's test command, from CLARIFY axis 3 */
const lintCmd = /* the project's lint command, or null */
const buildCmd = /* the project's build command, or null */
const taskName = /* the task's name */
const taskWork = /* what to build */
const taskPaths = /* narrow writablePaths — see Phase 2 */
const taskAcceptance = /* the criterion the verifier checks */
const taskRed = /* the exact command from CLARIFY axis 3, scoped to the first failing test */
const args = {
  projectDir,
  goal,

  // The plan's table verbatim. Every implementation task carries redCommand and refs.
  tasks: [
    { id: "T1",
      name: taskName,
      work: taskWork,
      writablePaths: taskPaths,
      acceptance: taskAcceptance,
      redCommand: taskRed,
      refs: ["${CLAUDE_PLUGIN_ROOT}/skills/dev/references/tdd.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/dev/references/writing-good-tests.md"] },
  ],

  // The TARGET project's own commands, collected at CLARIFY. Nothing generic — a check that does
  // not run this repo's suite gates nothing.
  // ONE entry point; three lose one silently. A missing --test-cmd, --lint-cmd, --build-cmd or --plan is exit 2 unless --no-lint-cmd / --no-build-cmd / --no-plan declines it.
  mechanicalChecks: [
    { name: "dev",
      cmd: `bash ${CLAUDE_PLUGIN_ROOT}/skills/dev/scripts/check.sh --project-dir ${projectDir} --plan ${planPath} --test-cmd "${testCmd}"${lintCmd ? ` --lint-cmd "${lintCmd}"` : ' --no-lint-cmd'}${buildCmd ? ` --build-cmd "${buildCmd}"` : ' --no-build-cmd'}` },
  ],
  // check.sh runs every checks/dev/constraints/ script through `workflow-check dev --constraints-only`. work-checks.sh runs ruleChecks as a command (no agent); Jev scores the rules under checks/dev/rules/ (LOOP, MOCK, NET, SHELL, WEAK); a rule with p >= 0.85 lands in rulesThatFailed and blocks, lower ones reach the lens as a ranked checklist.
  ruleChecks: { name: 'jev-dev-rules', cmd: `bun ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/rule-check.ts --workflow dev --project-dir ${projectDir} --plan ${planPath}` },

  // One lens after verification and mechanical checks; its checklist covers all five dimensions.
  lens: {
    agentType: "Explore",
    refs: ["${CLAUDE_PLUGIN_ROOT}/skills/dev/references/lens-security.md",
           "${CLAUDE_PLUGIN_ROOT}/skills/dev/references/lens-performance.md",
           "${CLAUDE_PLUGIN_ROOT}/skills/dev/references/writing-good-tests.md"],
    prompt: "Judge the change in the working tree against the approved plan and the goal. CHECKLIST, every item in scope on every run: (1) CRITERIA vs ARTIFACTS — for each success criterion in the plan and goal, is there an artifact in the tree that satisfies it? Missing or partial satisfaction is a finding, CRITICAL where the deliverable cannot stand without it. (2) SCOPE intent — path containment is check.sh's scope leg, so do not re-derive it; judge only unrequested features and silently skipped plan items (a plan task with no artifact), CRITICAL where a plan task the goal depends on was skipped. (3) SECURITY of the changed code, against the finding classes in the first ref — read it in full first. A finding names a file and line and states the concrete attack vector: the input an attacker controls, the path it travels, and what it reaches. Pre-existing defects this change did not introduce are out of scope. CRITICAL where the vector reaches attacker-controlled input on a production path. (4) PERFORMANCE — the runtime cost of the changed code, against the finding classes in the second ref, read in full first. A finding names a file and line, sits on a path that runs often enough to matter, and states the cost as Big-O over the input that actually grows or as concrete latency/memory; speculation without a growing input is not a finding. CRITICAL where the growth makes a production path unusable at realistic input size. (5) TEST PINS BEHAVIOUR: for each task, does the test that went RED (for a task with `redDisposition`, its acceptanceCmd/verification instead) assert the behaviour its acceptance names, not an incidental property (exit status only, a log line)? A test that would stay green with the behaviour removed is a MAJOR finding. Severity is MAJOR at minimum for all five. MODE — RED (a task was flagged or a mechanical check failed): diagnose EVERY failure in the digest and route it, naming cause and fix, to the task id whose writablePaths own the file, or to 'plan' when no task can own the fix. MODE — GREEN (everything passed): make one open-ended pass over the whole checklist and report each finding with an ownerTask. In both modes, rule every carried finding open or closed against evidence you actually read; silence leaves it open.",
  },

  authorityExtra: [
    "TDD RULE — no implementation before a genuine RED observation.",
    "Write the smallest real test that exercises the production path this task names, run it, and read the output. Accept RED only when it fails for the intended MISSING BEHAVIOUR — never a syntax error, an import error, a broken fixture, an unavailable dependency, or an unrelated failure elsewhere in the suite. If you implemented before observing RED, delete the implementation and restart the task; a test written beside its fix passes whether or not it ever exercised the defect.",
    "A test must do what the user does. Exercising an alternate protocol or transport, or inspecting source and logs instead of runtime behaviour, is not evidence for a task whose acceptance names a user-visible result.",
    "Your task's redCommand is executed by probes outside your control, before and after you work. A self-reported RED is not evidence and never was.",
    "Rules: ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/tdd.md and ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/writing-good-tests.md govern every task — read the latter IN FULL before writing a test. Browser and web work also follows ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/testing-web.md; desktop and native work also follows ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/testing-desktop.md.",
  ].join("\n"),

  verifierAgentType: "Explore",
}

Path containment is mechanical: check.sh's scope leg (work/scripts/scope-check.sh, given --plan) fails a changed path under no task's writablePaths or scaffoldPaths, naming each. Lens item (2) judges intent only.

verifierAgentType and the lens agentType pin Explore because it has no Edit and no Write: a judge that structurally cannot modify the tree beats a prompt asking it not to.

implementerAgentType is deliberately unset: the output is code under TDD, and a custom agent body would replace the software-engineering prompt the tasks depend on.

Add ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/testing-web.md or ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/testing-desktop.md to a task's refs when that task's real test drives a browser or a native app. authorityExtra names them; refs is what makes an implementer read one in full.

Handle the result per work's Phase 4 — including that a red-not-red, red-unproven or green-not-green verdict lands in tasksThatFlagged and is fixed as a task, not as a lens.

Phase 5 — HUMAN REVIEW

work's Phase 5 unchanged, on the review surface from CLARIFY axis 7.

Red flags

SituationWrong moveRight move
Guard denies a pre-dispatch write--abandon to get movingread the denial: an uncovered path is already writable, and a covered one that must pre-exist goes in scaffoldPaths. --abandon unguards the rest of the session and silences the Stop nudge with it
RED is an ImportError on the module being builtcall it red and dispatchthe probe refuses it as could-not-run; write the throwing stub, declare it in scaffoldPaths
LSP enabled, binary never installedtrust the install's exit 0one LSP documentSymbol on a repo file before recon; ENOENT there is the whole failure
No test harness in the repoproceed and test by handtest infrastructure is the first task, decided at CLARIFY — absence of tests is never a waiver
The failing test needs two stepsredCommand: "build && test"work throws on shell operators; put the steps in a script and name the script
Task's real test would drive a browser or an appassert on source or logsthe runtime references are refs on that task; a screenshot with no assertion is not evidence
Adding a domain review dimensiondeclare a second lens beside this onework takes ONE lens; add the dimension to its checklist, or make it a leg of check.sh when an exit code can decide it
RED reported by the implementeraccept itprobes execute redCommand on both sides; the verdict is the JS's, never the doer's
Something work does not obviously dowrite a dev/workflow.jsask which work parameter is missing

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Contents

What Dewey Is

Dewey Data is an academic data marketplace — one institutional Platform Subscription unlocks a catalog of ~300 datasets from ~40 providers (foot traffic, POI, mobility, consumer transactions, real estate, labor). UVA Library and NYU both hold the institutional subscription; SafeGraph and most providers are free under it.

Dewey is not a SQL warehouse like WRDS. Data is delivered as partitioned Parquet/CSV.gz files downloaded via an API key. You discover datasets, read metadata, sample, filter (by date partition + columns), then download. Think "S3 of presigned Parquet links," not "PostgreSQL."

WRDSDewey
DataFinance/accountingPOI, foot traffic, mobility, consumer, real estate
AccessPostgreSQL / SAS on the gridFile download (Parquet/CSV.gz) via API key
Query engineserver-side SQLDuckDB over the files (local or remote presigned URLs)
Licensingper-vendor, negotiatedone platform subscription unlocks the catalog
AI accessnoneMCP server (api.deweydata.io/mcp)

Credential Enforcement

IRON LAW: NEVER GUESS, INVENT, OR HARDCODE THE API KEY

The Dewey API key belongs to the **user's** account (`app.deweydata.io` → Connections → Add Connection → API Key). You cannot derive it. Keys are `akv1_` + 35 chars (40 bytes).
  • ALWAYS look for an existing key first: DEWEY_API_KEY env, then ~/.config/dewey/apikey (also on rjds at the same path; agenix dewey-api-key.age → DEWEY_API_KEY_FILE). Validate before any pull: GET https://api.deweydata.io/api/v1/external/data/<prj_id>/files?page=1 with header X-API-Key; 200 = valid, 401 Invalid API key. = revoked/wrong (Authorization: Bearer is the wrong scheme and gives a misleading 401). Ask the user only when no key exists or it 401s.
  • NEVER write a placeholder like apikey = "your_api_key" and run it — it will 401 and waste a round trip. Read from DEWEY_API_KEY env var or a gitignored file (~/.config/dewey/apikey).
  • NEVER commit the key, echo it back, print it, or paste it into a script that gets committed.

Guessing or hardcoding the key is NOT HELPFUL — every call 401s, and a committed key is a security incident the user must rotate.

Each product (dataset) has its own product path / project ID (prj_…), obtained from the dataset page: Get Data → (Skip filtering) → Connect to API / Bulk API → API URL. One API key, many product paths. If you don't have the product path, discover it via the MCP server (search_datasets) rather than guessing.

Download Enforcement

IRON LAW: NO BULK DOWNLOAD WITHOUT METADATA + SAMPLE + FILTER FIRST

Before downloading ANY Dewey dataset, you MUST:

  1. IDENTIFY the product path and what partitions/columns you actually need
  2. META — call get_meta (deweydatapy) / get_download_info (MCP) to learn partition columns, date range, file count, total size
  3. SAMPLE — pull 100 rows (read_sample / MCP sample_dataset) and INSPECT the schema before committing to a full pull
  4. FILTER — restrict by date partition (partition_key_after/before) AND columns; for selective pulls use DuckDB COPY TO over the presigned URLs, never download the whole catalog
  5. DOWNLOAD the filtered subset, then verify row counts / NULLs / date range on disk

This is not negotiable. Skipping the sample-and-filter step is NOT HELPFUL — Dewey datasets are routinely hundreds of GB to multiple TB; an unfiltered pull burns hours of bandwidth and disk for data you'll immediately throw away.

Dewey Facts

  • SafeGraph Patterns is multi-TB; "download everything and filter in pandas" fills the disk before the filter ever runs — counterproductive on its own terms. Use DuckDB COPY TO with a WHERE clause on the remote parquet to pull only the rows/columns you need.
  • Column names differ by provider and release (naics_code vs NAICS_CODE; opened_on may not exist at all). A full pull against guessed columns is the exact incompetence the sample step exists to prevent — read_sample(nrows=100) BEFORE the full pull.
  • Most datasets are date-partitioned weekly; "all of it" means every weekly file ever shipped. Set partition_key_after/before to the study window.
  • Presigned links expire in 24h (download_files0). For large multi-day pulls use download_files1 (page-by-page, refreshes links) — a long job on download_files0 dies mid-pull.
  • A wrong prj_ product path 404s or returns someone else's data. Get the path from Connect to API or MCP search_datasets; hardcoding a guessed path is an unverified claim presented as fact.
  • Use deweypy.get_dataset_files, not deweydatapy.get_meta/get_file_list — the latter's external-api/v3 endpoint is dead (returns non-JSON / 500 → JSONDecodeError). See references/deweypy-client.md.
  • The download service throws transient HTTP 500s on individual presigned URLs, and one bad file aborts a whole-batch DuckDB COPY read_csv([...]). For filtered pulls: chunk (~20 files), retry per chunk re-minting fresh URLs, fall back to per-file skip; restartable via per-chunk parquet. Set SET http_timeout=120000; SET http_retries=3;. Worked example in references/deweypy-client.md.
  • Some providers gate access behind extra terms (e.g. ConsumerEdge): the web "Get Data" flow shows an "I acknowledge…additional terms" modal you must accept once before the dataset is usable / its prj_ path mints. Don't auto-accept a provider license without the user's OK.
  • Bulk access is per project, with limits (3 active projects, 5 datasets each; tables of one dataset count once; "New Project" disabled at 3/3). Add the table to a project, then read its API URL there. "Issue New Key" revokes the previous key immediately (a click that looked interrupted still went through): never rotate without the user's OK, and after a rotation update every copy (local file, rjds, agenix). The Bulk API panel renders the LIVE key (readonly "API Key" textbox and the deweypy one-liner), so a take_snapshot of a get-data page contains it: never print or grep such a snapshot; capture the key with evaluate_script + filePath straight into the key file. Flow and limits: references/access-options.md.
  • A "Data Request Form ... requires an institutional Dewey license" modal is not a block (BrightQuery Delaware Stock Filings): the table is added anyway. Cancel it, never submit without the user's OK, and check the project's dataset list. references/brightquery-delaware-stock-filings.md.
  • MCP tools load only at session start. After claude mcp add … dewey-prod, the search_datasets/sample_dataset/etc. tools are NOT available in the current session — start a new session to use them.

Red Flags — STOP Immediately If You're About To:

  • Call download_files* without first calling get_meta + read_sample → STOP. Meta + sample first.
  • Download a dataset with no start_date/end_date / partition filter → STOP. Scope the date range.
  • Load a whole remote dataset into a DataFrame → STOP. Use DuckDB COPY TO … (FORMAT PARQUET, PARTITION_BY …) to persist a filtered subset to disk.
  • Run a pull with apikey="your_api_key" or any guessed key → STOP. Read env/file and validate; ask the user only if none exists or it 401s.
  • Write the API key into a script you'll commit → STOP. Env var or gitignored file only.

Access Method Decision Table

NeedMethodReference
Discover/search datasets, check schema, sample — from inside ClaudeMCP server (api.deweydata.io/mcp)references/mcp.md
Scripted Python bulk downloaddeweypy (recommended) or deweydatapy (legacy, product_path API)references/deweypy-client.md
Selective pull — specific columns/rows from huge datasetsDuckDB over presigned URLs (read_parquet($urls) + COPY TO)references/duckdb.md
R workflowdeweyr (download_dewey())references/deweypy-client.md
One-off, dataset < 2.0 GBUI CSV download (platform → project)references/access-options.md
Analyze data already on diskDuckDB / pandas / polars over *.parquet or *.csv.gzreferences/access-options.md

Authentication

Get the key once from app.deweydata.io → Connections → Add Connection → API Key. Store it out of source control:

mkdir -p ~/.config/dewey && echo 'YOUR_KEY' > ~/.config/dewey/apikey && chmod 600 ~/.config/dewey/apikey
# or: export DEWEY_API_KEY=...   (add to .envrc, which should be gitignored)
import os, pathlib
apikey = os.environ.get("DEWEY_API_KEY") or pathlib.Path("~/.config/dewey/apikey").expanduser().read_text().strip()

Institutional login (to browse the catalog / create the key) is via UVA NetBadge (use your UVA email) or NYU SSO. The Platform Subscription is what makes SafeGraph etc. free — see references/datasets.md.

Quick Reference: Featured Datasets

ProviderDataset(s)What it is
SafeGraphGlobal Places (POI), Geometry, Spend, PatternsPOI master, building footprints, card spend, foot-traffic visit patterns
Advan ResearchMonthly/Weekly Patterns, Home PanelFoot traffic aggregated to place & census-block
dataplorPOIGlobal POI, strong emerging-markets coverage
VerasetMovementDevice-level mobility (institutional license only)
PassByFoot TrafficPer-POI foot-traffic analytics
Consumer Edge / PDISpend / transactionsCard & product-level purchasing
GovFilesUS Business EntitySecretary of State registries, 98.6M rows incl. dissolved; uneven by state (NV 37K rows, ~2023 formations only; no DE status)
Exchange Data InternationalGlobal Equity Corporate Actions (WCA/RCAN)Dividends, splits, mergers, tenders — 2001→, global, w/ CUSIP/SEDOL/ISIN/FIGI bridges
LinkUpJob postingsLabor-market activity, scraped from employer career sites (2007→)
ATTOM / Dwellsy / RentHubReal estateProperty records, rentals

Full catalog (all ~250 datasets): references/catalog.md — every dataset grouped by category with time coverage, row count, size, and download access (machine-readable: references/catalog.csv). Featured-dataset detail + discovery workflow: references/datasets.md.

SafeGraph Global Places Quick Reference

Core POI schema — columns are UPPERCASE, NAICS_CODE is a string, BRANDS is a JSON-array string (extract with json_extract_string(BRANDS,'$[0].safegraph_brand_name')). Always sample before filtering.

ColumnMeaning
PLACEKEYStable unique POI id (join key across SafeGraph products)
LOCATION_NAMEPOI name
BRANDSJSON array: [{"safegraph_brand_name":"…"}] — not plain text
STREET_ADDRESS,CITY,REGION,POSTAL_CODE,ISO_COUNTRY_CODEAddress (REGION=US state)
LATITUDE,LONGITUDECoordinates
NAICS_CODE,NAICS_CODE_20226-digit NAICS (string)
TOP_CATEGORY,SUB_CATEGORYCategory labels
OPENED_ON,CLOSED_ON,TRACKING_CLOSED_SINCEOpen/close dates (exist but sparsely populated — NULL for BTMs)

Resolved empirically: crypto/Bitcoin ATMs do exist as standalone POIs under NAICS_CODE='522320'; all major operators are present. But OPENED_ON/CLOSED_ON are NULL for BTMs in the current release → it's a cross-section, not a time series. Full details, the 7 BTM operators, and the worked example: references/safegraph-places.md and examples/btm_safegraph_pull.py.

Additional Resources

Reference Files

  • references/access-options.md — all download methods (UI, deweypy, deweydatapy, DuckDB, MCP, R), projects (limits, add-to-project flow, Bulk API/key, CDP tips), 24h link expiry, partitioning, reading data on disk
  • references/deweypy-client.md — deweypy (modern CLI + auth/download) and deweydatapy (get_meta, get_file_list, read_sample, download_files0/1) function reference; deweyr for R
  • references/duckdb.md — selective remote-Parquet pulls, COPY TO … PARTITION_BY pattern, querying downloaded files
  • references/brightquery-delaware-stock-filings.md — BrightQuery Delaware Stock Filings: 9 tables, measured profile of 4 (rows/cols, schemas, join key, 1990-2024 years, generic class names and no votes, identifier fill, dual-class verdict), Stock Class Get Data disabled, the Data Request Form non-block
  • references/govfiles-business-entity.md — GovFiles US business-entity registry: all 7 table schemas w/ fill rates, Identifiers (documented CIK/LEI bridge is empty in the 2026-10 snapshot; FEIN in 6 states only), the missing officers/parties table, the ungraphable Relationships table (1% counterparty key), FILED_ON sentinel dates, launched Jun 2026 / empty changelog — which sparse fields may backfill and which won't, worked DuckDB join
  • references/linkup-job-postings.md — LinkUp: 12 tables across 2 products (incl. Extracted Salary, Job Descriptions, Structured Fields, Remote Tag — added Jun 2026, NOT in the stale catalog.csv), join keys (COMPANY_ID/JOB_HASH/BASE_HASH/REQID), salary top-coding at 12k/1M, point-in-time ticker joins, the scrape-log structural-break trap, worked firm-quarter vacancy panel
  • references/edi-corporate-actions.md — EDI WCA/RCAN corporate actions: event grain (options=ORs, serials=ANDs), identifier hierarchy + outturn ids, generic label/value slots, future-dated rows, Notes-table coverage gap, worked dividend/CUSIP query + EVENTCD enumeration
  • references/mcp.md — Dewey MCP server URL, JSON config, the 9 tools, discovery → schema → sample workflow
  • references/datasets.md — featured-dataset catalog, UVA NetBadge / NYU institutional access, discovery workflow
  • references/catalog.md + catalog.csv — full enumerated catalog (~250 datasets / 39 partners) by category, with coverage / rows / column count / size / access
  • references/schemas.json — full column schemas for all ~250 datasets (keyed by slug → columns[] with name/type/description; 11,264 columns). Look up a dataset's columns here before pulling, instead of a live get_dataset_schema call
  • references/linkage.md — cross-dataset join-key map (placekey, ticker, cusip/cik, domain, person id, lat/long, fips, zip…) — which datasets combine and on what spine
  • references/safegraph-places.md — Global Places schema, NAICS 522320, BTM operator brands, opened_on/closed_on, the Bitcoin-ATM worked example

Example Files

  • examples/btm_safegraph_pull.py — acceptance test: filter SafeGraph Global Places to the 7 BTM operator brands + NAICS 522320, verify standalone-POI / open-close coverage, export the US subset to ~/projects/batm/

Announce: "I'm using docx-render to convert this document to PDF via doc_render."

DOCX → PDF/PNG rendering

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Office docs → PDF/PNG go through the shared converter ${CLAUDE_PLUGIN_ROOT}/scripts/doc_render.py (convert()), which picks the best engine and applies the right fixes. Do not hand-roll soffice/libreoffice or lean on the generic docx skill's own export — those skip the Word-fidelity path and the x2t kerning/table fixes.

Default to --renderer word (Iron Law)

For any PDF a human will read — a deliverable, an email attachment, a reading-list file, anything you hand to the user — pass --renderer word. The bare/auto command is NOT the safe default: auto deliberately EXCLUDES Word (the check is if avail["word"] and allow_word, and allow_word defaults False) and silently uses x2t/LibreOffice, which reflow the layout (real case: a 5-page doc shipped as 7). Word works even from a background/headless job via cmux dispatch (below), so there is no reason to skip it for a deliverable.

Reserve bare auto for parallel pipeline builds (e.g. many law-review renders at once) where headless/parallel-safety matters more than line-exact fidelity and the docs are pipeline-generated (x2t/soffice already grid-faithful there).

On Linux, --renderer word cannot work — it drives Word.app through AppleEvents. Use --renderer word-remote instead: the same real Word engine in a Windows guest, driven over SSH. See "Word in a Windows guest" below.

Always verify the engine actually used via the PDF Producer before handing off (see table below) — auto can fall back, and --renderer word only raises if Word is truly unavailable.

A legacy .doc goes STRAIGHT to Word — never convert it first

Both Word engines open .doc natively, and WORD_SRC_SUFFIXES admits it:

python3 scripts/doc_render.py IN.doc OUT.pdf --renderer word-remote

Never soffice --convert-to docx a .doc and render the result. That hop silently corrupts the document, and because the damage is in the intermediate it survives into a genuine Word render, so the Producer string still says Word and the output still looks authoritative. Measured on a 23-page expert report:

  • every footnote gained a stray superscript ? — soffice writes the real <w:footnoteRef/> and then a literal <w:t>?</w:t> run beside it, where the legacy footnote mark was;
  • the page count grew by one (23 → 24), so pagination no longer matched what the sender saw.

Rendering the .doc itself reproduced the sender's 23 pages with clean footnote numbers. If a ? opens every footnote, suspect the conversion, not the sender.

A wrong extension fails silently. Word dispatches on the suffix, so a .doc staged under a .docx name makes it bail: the guest's scheduled task completes, writes no PDF, and the only symptom is guest render did not complete. The transport carries the real suffix through — keep it that way.

Entry point

# CLI (faithful Word engine — DEFAULT for anything a human reads):
python3 ${CLAUDE_SKILL_DIR}/../../scripts/doc_render.py IN.docx OUT.pdf --renderer word
# auto engine (LibreOffice/x2t; NO Word) — ONLY for parallel pipeline builds:
python3 ${CLAUDE_SKILL_DIR}/../../scripts/doc_render.py IN.docx OUT.pdf
import sys; sys.path.insert(0, "<plugin>/scripts")
from doc_render import convert
convert("in.docx", "out.pdf", renderer="word", allow_word=True)   # gold standard
convert("in.docx", "out.pdf")                                     # auto (headless)

Agents without the Skill tool (most workflow subagents): run the CLI above directly — you don't need to invoke this skill, just call doc_render.py.

Which engine, and why it matters

EngineFidelityNotes
Word (--renderer word)gold standardnative layout; only engine that keeps an auto-wrapping table as a grid in a hand-authored docx (LibreOffice collapses it to a stacked column). Recomputes Word fields (REF/NOTEREF/PAGEREF/TOC). macOS only — it drives Word.app over AppleEvents.
word-remote (--renderer word-remote)gold standardthe same real Word engine, in a QEMU Windows guest over SSH. The Word path on Linux, where --renderer word cannot work at all. Also usable from macOS against a guest on that host.
x2tgoodOOXML-native; correct per-section footnote restart; doc_render injects GPOS/kern + EB-Garamond so it matches.
LibreOfficegood exceptwrong for per-section/page footnote restart; collapses auto-wrapping tables not pre-broken upstream.

convert() auto-falls-back Word → x2t/soffice. Verify which ran via the PDF Producer: macOS … Quartz PDFContext = Word; LibreOffice … = LibreOffice.

Garamond documents on macOS need a one-time setup. x2t mis-measures the macOS (Monotype) Garamond italic face badly enough to cram every upright Garamond run. ${CLAUDE_PLUGIN_ROOT}/scripts/setup_garamond_render_override.py writes a four-face override to ~/.config/x2t-render-fonts/garamond/ — macOS Garamond for regular/bold, EB Garamond for the slanted faces (--all-eb for the all-EB variant) — then rm -rf ~/.cache/x2t-docfonts to re-stage. Full measurements: docs/investigations/2026-06-19_x2t-kerning-patch.md, Part 2.

Word from a background/headless job (the non-obvious part)

A detached Claude job is in a non-console GUI session without Word's TCC grant, so direct AppleEvents fail with -600. doc_render transparently dispatches the render into a cmux pane (console session, TCC-granted) and falls back to x2t/LibreOffice if that's unavailable. Prereqs + full root-cause: docs/investigations/2026-06-22_word-render-cmux-dispatch.md. Disable with $DOC_RENDER_NO_CMUX=1.

Driving the Mac's Word from another machine over SSH — doesn't work

The cmux rescue above assumes you are on the Mac. Invoking --renderer word over SSH from another host (e.g. a Linux box rendering on mbp) fails differently and has no fallback — cmux dispatch fails too, because there is no console session on the far end to dispatch into:

doc_render: word renderer failed: Word direct render failed
  ([Errno 1] Operation not permitted:
   ~/Library/Containers/com.microsoft.Word/Data/wordrender/<uuid>);
  cmux dispatch also failed (…same…)

Note this is a filesystem permission error on Word's app container, not the AppleEvents -600 of the local case — an SSH session is outside the TCC grant entirely. launchctl asuser $(id -u) … does not rescue it (Could not switch to audit session: Operation not permitted — needs root).

Fixes, in order of preference:

  1. Use word-remote (next section) — a Windows guest is the supported remote path; driving the Mac's Word from off-box is not.
  2. Run the render from a terminal inside the Mac's GUI login session (then the normal local path, incl. cmux dispatch, applies).
  3. Grant Full Disk Access to /usr/libexec/sshd-keygen-wrapper in System Settings → Privacy & Security, after which headless SSH renders work.

Do not paper over this by falling back silently — an explicit --renderer word deliberately raises rather than downgrading.

Word in a Windows guest (word-remote) — the Linux path

--renderer word is macOS-only. word-remote runs the same Word engine in a Win11 guest and drives it over SSH, so Linux gets gold-standard fidelity:

python3 scripts/doc_render.py IN.docx OUT.pdf --renderer word-remote

Provisioned by the programs.wordRender nix module — word-render and word-render-install-fonts on PATH, transport at ~/.local/share/word-render/word_render_remote.sh (override with $WORD_RENDER_REMOTE). Full setup: ~/nix/modules/shared/word-render/README.md.

On Linux the guest is a docker container, not QEMU/qcow2. vm/start-winvm.sh and ~/.local/share/winvm/*.qcow2 are the macOS path. Linux uses dockur/windows — the image Omarchy's omarchy-windows-vm drives — as container omarchy-windows, disk at ~/.windows/data.img, compose at ~/.config/windows/docker-compose.yml. There is no qcow2 to find here.

docker start omarchy-windows        # or: omarchy-windows-vm launch -k
until ssh -o ConnectTimeout=5 -o BatchMode=yes word@winvm exit; do sleep 15; done

A cold boot takes several minutes before sshd answers, and the error changes as it comes up: Connection refused (container down) → Connection reset by peer / timed out during banner exchange (Windows still booting) → success. Only the first means something is wrong. Wait on the until loop rather than concluding the guest is broken; docker logs --tail 20 omarchy-windows shows boot progress.

Selection rules:

  • Explicit --renderer word-remote always runs it (and raises rather than falling back, like every explicit engine).
  • auto picks it only with allow_word=True, and only when local Word is unavailable — i.e. it is the Linux stand-in for renderer="word", preferred over the lower-fidelity engines rather than silently downgrading.
  • auto never reaches it in the fallback cascade. Booting/using a VM is not something best-effort should do behind the caller's back.

Availability is a file check on the transport script, not an SSH probe — a reachability test would cost a round-trip (and can hang on a suspended VM) on every convert(). A down guest surfaces as a render error naming the fix.

A fresh guest silently renders the wrong fonts. Word substitutes Cambria/Calibri for any font it can't resolve and still exits 0, so the render "succeeds" with wrong typography. Run word-render-install-fonts once per guest, then verify with pdffonts — never trust the exit code. (Stock lmodern does not work: Word won't render CFF-flavoured OpenType, and it matches families on name ID 1. The nix module ships a converted set.)

Google Docs exports

A docx exported from Google Docs can carry OOXML package corruption (case-broken customXML part paths) that makes Word pop a "recover unreadable content" modal on open — fatal to a headless render. The Word path auto-repairs it via a preflight (${CLAUDE_PLUGIN_ROOT}/scripts/docx_repair.py); you'll see Word preflight — repaired Google-export package … on stderr. Repair a docx standalone with python3 ${CLAUDE_PLUGIN_ROOT}/scripts/docx_repair.py in.docx [out.docx].

Related skills

Part of the document skill group (extract → create → repair → build → render → verify):

  • law-review-docx — builds a .docx from markdown (template + pandoc), then renders.
  • docx (generic) — edits docx content (tracked changes, comments, text).
  • docx-repair — repairs a cloud-editor-damaged .docx (package/XML wiring + footnote markup).
  • xlsx recalc / pptx-render — spreadsheet recalc / slide inspection.

DOCX Repair (Google Docs / Word Online damage)

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Cloud editors damage a .docx in independent ways. This skill is the front door for all of them; run only the track(s) you need.

Damage classSymptomFix
A. Package / OOXML wiringWord pops "recover unreadable content?" or refuses to open; LibreOffice won't load; phantom blank page${CLAUDE_PLUGIN_ROOT}/scripts/docx_repair.py (plugin root) — references/package-repair.md
B. Footnote & cross-reference markupBios show 1,2,3 not *,†,‡; numbering starts wrong; "supra note N" points to the wrong footnote; missing separator linethe footnote scripts — references/footnote-procedure.md
B1. Cross-references aimed at the wrong sourceLin, supra note 130 where note 130 is a different work; Rebuttal Report, supra note 22 where 22 defines the deposition; a short form surviving a renumber that moved its targetcheck_crossrefs.py <file.docx> — read-only, exit 1 on any problem
C. Document content (boxes + headings + cruft)Visible boxes around freshly-edited text; heading-looking lines not styled as headings; same-style headings rendering differently; blank heading lines; bloated XML full of all-zero rsids, no-op shading, explicit b=0/i=0/u=none, redundant black color & default fontsfix_footnotes.py's document.xml passes — references/content-cleanup.md
D. Presentation hygieneA footnote renders blue/underlined in the PDF but looks normal in Word; a URL prints one address and navigates to another; heading gaps uneven page to page; tracking params (?utm_source=…) behind a clean-looking linkdocx_links.py, docx_spacers.py — references/presentation-hygiene.md

They are decoupled: package repair fixes the part wiring (never touches content); footnote repair fixes the footnote markup; content cleanup strips Google-Docs leftover content controls and normalizes headings. A file can need any, all, or none. If you don't know which, run the package check first (it's a no-op on a clean package), then the footnote pass (it carries the content cleanup).

Heads-up: docx-render's Word path already composes docx_repair.py as a preflight, so a Google export "just renders" without a manual Track A. Run Track A manually when you need the repaired file itself (to hand back, edit, or footnote-fix), not just a PDF.


Reference index — read the file for the track you are running

All procedural detail lives in references/. Read only what the job needs.

  • references/package-repair.md — Track A. The two concrete Google Docs export defects (case-mismatched customXML/customXml OPC part references; leftover <w:evenAndOddHeaders/>), and the CLI + Python API for ${CLAUDE_PLUGIN_ROOT}/scripts/docx_repair.py. Read when Word calls the file corrupt, refuses to open it, or shows a phantom blank page — or before handing back a repaired file rather than a PDF.

  • references/footnote-procedure.md — Track B, the canonical procedure. The exact four-step order for a Google Docs round-trip (accept changes → fix_footnotes.py → create_crossrefs.py --baseline → render with Word), what to verify in the render, the incident-grounded procedure facts (why step 3 must follow step 2, why --baseline is not optional, why LibreOffice lies), the list of symptoms footnote repair applies to, and the quick-start "which script do I want?" routing. Read when running any footnote repair — start here.

  • references/footnote-scripts.md — Track B, script detail. Per-script behaviour and every flag for fix_footnotes.py, create_crossrefs.py (including how --baseline remaps stale numbers) and refresh_noteref_caches.py (why the naive approaches fail, its requirements and intentional scope), plus the footnote numbering-offset fix (numRestart). Read when you need a specific flag, or the exact semantics of one of the three scripts.

  • references/footnotes-reference.md — Track B, OOXML technical reference. (1) Run-level editing gotchas (NBSP, cross-run matching, xml:space); (2) cloud editor damage patterns — what gets destroyed and why; (3) direct ZIP surgery patterns that bypass Document libraries; (4) the numbering-restart details and the critical rule that numRestart goes in settings.xml ONLY. Read when hand-editing footnote XML or debugging behaviour the scripts do not cover.

  • references/content-cleanup.md — Track C. The four content passes fix_footnotes.py carries: stripping goog_rdk content controls (the "boxes", default on), heading normalization (--normalize-headings), the OOXML hygiene / de-cruft rules (default on, with the exact strip/keep lists), body-indent normalization (--normalize-body-indent), and applying the template's body styles (--restyle-body), plus the content-cleanup facts. Read when boxes, inconsistent headings, stray indents or XML bloat are the complaint, or before passing any of those flags.

  • references/presentation-hygiene.md — Track D. docx_links.py (tracking-param stripping, SSRN canonicalization, footnote hyperlink unwrapping — and why display text and relationship Target must both be rewritten) and docx_spacers.py (manual spacer removal, why a text-empty paragraph may not be empty, the title-page collapse hazard), plus why both edit bytes rather than the ElementTree tree. Read when the PDF shows blue/underlined footnotes, a link that navigates elsewhere, or uneven heading gaps.


Related (document skill group)

This skill owns the REPAIR stage — package wiring (Track A), footnote markup (Track B), and document-content cleanup (Track C) — for a .docx damaged by a cloud editor. Adjacent stages:

  • Build a styled .docx from markdown → law-review-docx (its build_docx.py chains this skill's footnote repair + NOTEREF conversion after the pandoc build).
  • Render to PDF/PNG → docx-render / ${CLAUDE_PLUGIN_ROOT}/scripts/doc_render.py (Word path composes Track A's docx_repair.py as a preflight automatically).
  • Check cross-references before believing them. check_crossrefs.py is the only pass that READS the target note: it verifies the short form names a source the target actually contains, on top of existence and direction. create_crossrefs.py --dry-run proves only that note N exists and will report "25 targets, 25 valid" over a document whose cites point at the wrong documents — it is a conversion preflight, not an audit. Run check_crossrefs.py on any draft returned by a coauthor, and after every renumber. It name-matches loosely (any capitalised word in the short form appearing in the target), so it catches a cite aimed at the wrong source, never a wrong pincite.
  • Footnote repair lives only here. fix_footnotes.py is the single canonical Google-Docs / Word-Online footnote fixer; there is no second copy. (Bluebook's create_crossrefs.py + audit_crossref_targets.py remain a deliberate, actively-used cross-reference fork with their own retargeting strategy — a different concern, not a footnote-fix duplicate.)

See the full document skill group.

DOCX ↔ Typst Bridge

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Typst source is the thing the repo keeps. Word is the thing coauthors edit. This skill moves a document across that boundary in both directions and reconciles what comes back.

The load-bearing fact: typ → docx → typ reaches a fixed point after one pass. So the pipe's output is itself valid Typst, the canonical form can be committed, and reconciling a coauthor's returned file collapses from "read two documents side by side" to git merge-file.

   existing.docx ──canonicalize.py --from-docx──> body.typ + media/   (bootstrap, once)
                                                     │
   main.typ ──typst compile──> PDF                   │
      │                                              │
      │ #include                                     ▼
      ▼
   body.typ ──build.py──> paper.docx ──email──> coauthor edits in Word
      ▲                                                    │
      │                                                    │ sends back
      └──── reconcile.py <── merged body.typ <── returned.docx
                                                     │
                                                     └──comments.py──> comments.json

Scripts

All under ${CLAUDE_SKILL_DIR}/scripts/. Each is self-contained and prints --help.

Run them with uv run --script, not uv run python3. Four of the five declare lxml in a PEP 723 header, and uv run python3 <path> ignores that header and fails with ModuleNotFoundError: No module named 'lxml'. --script (or executing the file directly, since the shebang is uv run) reads the header and provisions the dependency.

ScriptDirectionDoes
build.pytyp → docxConvert with --reference-doc styles and stamp provenance, in one step
canonicalize.pydocx → typBootstrap an existing Word manuscript (--from-docx); put a body file on its fixed point; --check gates it; --lint guards the body/main split
reconcile.pydocx → typResolve the ancestor, three-way merge a returned file against the repo source
comments.pydocx or Drive → JSONExtract comments with their anchor text, resolved state, and threading
provenance.py—Read/write the stamp directly (build.py already applies it)
expand_citations.pytyp → typFreeze every computed reference — #cite(...) and @label — into the literal body the docx path needs
make_redline.pydocx × docx → docxRebuild a coauthor's untracked edits as real tracked changes, against a baseline (stdlib + LibreOffice; no PEP 723 header)

The main.typ / body.typ split

main.typ    #import / #let / #show / #set, then #include "body.typ"    ← typst compiles this
body.typ    pure markup: = headings, prose, #emph, #footnote           ← pandoc reads this

Both paths see the same prose and neither degrades the other. The split is not stylistic tidiness — see the first fact row in ${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/facts.md.

References — read the one the task needs

ReadWhen
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/conversion.mdAny actual conversion: bootstrapping a Word manuscript into body.typ, building the .docx, reconciling a returned file, rebuilding untracked edits as a redline, extracting comments
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/citations.mdA manuscript whose numbers must maintain themselves — supra note N, Section IV.B, Id.; the bluebook.typ renderer, bib_to_entries.py, hand-authored short forms, the entries schema, the two-body-file split
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/facts.mdBefore changing the conversion path, or when a build/round trip behaves in a way the commands do not explain — the pandoc and typst defects this skill normalizes around, and why each rule exists
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/maintenance.mdChanging this skill or its scripts: the test suite, the two tests that pin bugs, and what "verified" requires

Red Flags — STOP

ActionWhy wrongDo instead
About to point pandoc at main.typIts #show rules destroy heading semantics in the docxPoint it at body.typ
About to pass --allow-styling to clear a lint errorShips a headingless Word fileMove the directives into main.typ
About to hand-edit the returned .docx and call it reconciledThe repo source still diverges; the next build overwrites the editsRun reconcile.py and merge into the source
About to resolve <<<<<<< markers by deleting one side wholesaleDiscards a coauthor's edit unreviewedRead both sides; ask the user when the prose choice is theirs
About to commit a merge without reading .merged.typ.diffThe merge is a claim about someone else's edits, unverifiedRead the diff, then commit
About to send a .docx built from an uncommitted sourceFallback 3 needs a committed blob; the ancestor is unrecoverableCanonicalize, commit, then build
About to run --from-docx without --media-dir because the error is in the wayEvery figure is dropped and the output still looks completeName the sidecar directory; it is one argument
About to invoke a script with uv run python3 <path>The PEP 723 header is ignored and the lxml scripts die on importuv run --script <path>
About to hand-fix Officers" in a recovered fileThe converter did it, not the source; hand-fixes are re-corrupted next passRe-recover with current canonicalize.py, which restores ’
About to name a custom citation function in the body filepandoc dies with Identifier not found and no docx is produced#show cite: over the built-in #cite(<Key>)
About to put live #cite in the file reconcile.py merges intoIt can never be canonical; the merge churns on every citationKeep the symbolic form in body-src.typ, generate the literal one
About to write #cite(<a>); #cite(<b>) in a stacked footnotetypst groups them and the ; vanishes from the PDF#cite(<a>)#[;] #cite(<b>)
About to hand-edit the generated cite-data.typ to fix a short formIt is regenerated, and --diff will report your fix as a delta foreverFix the .bib or the CSL, then regenerate and diff
About to patch hayagriva to get supra note NIt is statically linked into typst, and note numbers are a layout property a patch cannot reachRender citations in typst with a #show cite: rule
About to typst compile a freshly recovered body and conclude the conversion failedWord's TOC arrives as links to bookmarks that are not labelsDelete the recovered TOC block; #outline() in main.typ replaces it

Scope

Owns typst → docx, and Bluebook citation rendering in typst — the renderer lives here rather than in bluebook because it is co-designed with expand_citations.py, which reads the <bb-out> tag it emits, and that two-file split exists only because of the docx round trip. bluebook owns the RULES; this skill owns making typst emit them.

law-review-docx and law-econ-docx own markdown → docx; different input format, no overlap — a markdown manuscript gets Bluebook from pandoc-citeproc and the bundled CSL, and needs nothing here. docx-repair fixes OOXML damage from a cloud round trip and composes cleanly before reconcile.py when a returned file is also damaged.

DS rules

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

These are binding on every data task, however you were launched. Each entry is the rule statement from its canonical file, printed at load time from the corpus itself; open that file for the rationale, the examples and the checker that enforces it. When this skill is invoked with arguments, they are your task: do it under these rules.

!c=${CLAUDE_PLUGIN_ROOT}/scripts/load-constraints; [ -x "$c" ] && exec "$c" ds --digest; echo "(rule loader unavailable: NO DS rule was loaded — none of C1-C6, V1-V9, A1-A6, E1-E7 is in your context; read them from the plugin's rules/ before any data work)"

ds — a data project, run through work with a computed data-quality gate

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

DS constraints in force — every rule scoped to this workflow, discovered from the corpus so a new one needs no edit here; read the ones your task touches: !c=${CLAUDE_PLUGIN_ROOT}/scripts/load-constraints; [ -x "$c" ] && exec "$c" ds; echo "(constraint index unavailable: NO DS rule is listed here — the four aggregates under skills/ds/rules/ are not the whole set)"

The lifecycle is work. Read it and follow it. This file is a delta: it supplies the domain — the CLARIFY axes, the plan grammar, the lens, the mechanical checks, the refs, the authority text. It ships no workflow.js and restates none of work's mechanics.

What makes a run ds rather than plain work is one thing: every declared data output is verified by a computed runner, not by the agent that produced it. checks/ds/constraints/ds-dq.py reads the approved plan's ## Data Outputs table and opens each artifact itself; work's JS reads its exit code.

Write surface

Main chat clarifies, plans and dispatches. It does not do the analysis, and it never writes a .py, .ipynb, .R, .sas, .sql or .qmd file — not by Write/Edit, and not by Bash (python3 -c, pixi run python, inline pandas/numpy). Analysis runs in a dispatched agent. work's dispatch is already structural and its judges are pinned to Explore, so this line is a rule on you, not a hook; reach for the workflow first rather than after a refusal.

Phase 1 — CLARIFY

work's Phase 1, on these axes.

**ASK BEFORE YOU LOOK; PLAN ONLY FROM EVIDENCE YOU ACTUALLY GATHERED. This is not negotiable.**

Loading a dataset or proposing a method before the user has stated the outcome anchors the work to whatever data happens to be nearby. Skipping a necessary profile replaces evidence with a convenient guess. Neither shortcut is helpful: it creates an analysis that is fast to start and expensive to correct.

AxisEstablish with the user
OutcomeQuestion, audience, decision the analysis informs, and deliverables
UniverseUnit of observation, entity inclusion rule, sample period, and authoritative source when sources disagree
SourcesAvailable data, access/location, expected refresh/vintage, and known restrictions
ConstraintsReplication target, required methods, deadline, compute/cost limits, and reproducibility needs
EvidenceWhat makes each planned output credible: a check, an inspectable exhibit, a source comparison, or user judgment

Ask in one AskUserQuestion call when answers are independent. Ask cascading questions separately: a source choice that determines available variables must be answered before variable questions.

Three facts decide these axes, and each prevents a defect that is expensive to find later:

  • A sample period and an entity universe are research choices, not properties inferred from the first query result. A rate at the wrong grain is a different statistic with a familiar label.
  • The universe predicate belongs where scope is decided. Applying it to a lookup or a denominator can turn available values into nulls.
  • A criterion that cannot name evidence is a wish. The explicit TBD (<phase>) convention is legitimate only when profiling is the scheduled evidence-producing phase; never invent coverage.

work's remaining axes are taken as work states them, with two domain bindings: work axis 4 (observable success criteria) is answered with the DQ runner over the plan's ## Data Outputs plus the target project's own test and lint commands, and those strings become the --test-cmd / --lint-cmd arguments of the single ds mechanical entry (Phase 4); work axis 5 (review surface) is answered by the plan's ## Review Surfaces section. The ds-dq entry is never conditional. CLARIFY records, for each of test and lint, either the project's command or an explicit "none" — an unanswered one is a refusal (check.sh exits 2), never a leg that ran nothing. A "none" becomes --no-test-cmd / --no-lint-cmd; work's probe runs a cmd verbatim, so never leave a placeholder in it.

Then gather planning evidence — a read-only profile, never implementation. For each source that materially affects the plan, collect or dispatch a profile covering: location/access, approximate shape, columns/types, date coverage, and likely row grain/key; nulls, duplicates, type drift, category/distribution anomalies, and likely join risks; raw size and whether source-side filtering, caching, partitioning, or server-side computation is needed; and a small representative sample only when access and project policy permit it. Use direct read-only inspection for one small local source; for independent sources, dispatch read-only profilers in parallel and consolidate. Do not make a profiling agent implement a pipeline, mutate project files, or spawn further agents. Do not pull a full source merely to estimate it. If an answer depends on implementation, state it as a planned evidence task rather than pretending the profile established it.

Four discovery rules run in this same planning step, each governed by its own constraint file:

  • Large sources. If a source may be large (roughly 50M rows, 500 MB to ship, or described as bulk/large/uncertain), read ${CLAUDE_PLUGIN_ROOT}/rules/ds-pull-sizing-decision.md and follow it. The profile must compare filtered raw and candidate aggregate/server-side paths before planning commits to one. Do not pull a full source merely to estimate it.
  • External-skill discovery. When the plan will use another skill or data provider, discover its relevant references and examples before choosing an approach: read ${CLAUDE_PLUGIN_ROOT}/rules/ds-external-skill-decision.md and follow it. Record the resulting ADOPT, PATCH, or GREENFIELD decision in the plan itself.
  • Master datasets. For multi-output work that shares a sample, read ${CLAUDE_PLUGIN_ROOT}/rules/ds-master-datasets.md. Plan the minimal canonical analysis dataset(s), their grain and keys, and which planned outputs consume each one.
  • Parameter transparency. For analytic filters or tunable thresholds, read ${CLAUDE_PLUGIN_ROOT}/rules/ds-parameter-transparency.md; name one configuration location, rationale, and treatment of convenience choices in the plan.

Phase 2 — PLAN

work's Phase 2. The plan opens with frontmatter workflow: ds — required, so a context clear at approval resumes here and not in work.

Five domain requirements on the plan:

  • The source/access strategy and the profile-derived risks. The plan states how each source is reached, and the data-quality and scale risks the read-only profile actually surfaced. A risk found at CLARIFY that reaches no plan section is a risk nothing acts on.

  • Reproducibility decisions, appropriate to the work: source vintages, seeds, environments and config. R1 (fresh re-run reproducibility) is judged by the review lens's data-quality item, so a plan recording none of these leaves R1 with nothing to judge against.

  • A ## Data Outputs table, in exactly this grammar, because checks/ds/constraints/ds-dq.py parses it and an output absent from it is one nothing will check:

    ## Data Outputs
    
    | Path | Grain | Key Columns | Required Window |
    |---|---|---|---|
    | data/processed/panel.parquet | one row per firm-fiscal-year | pk: gvkey, fyear; event: gvkey, datadate | datadate: 2005-01-01..2025-12-31 |
    | data/processed/industry_xwalk.csv | one row per SIC code | sic | n/a |
    
    • Path — the produced artifact, relative to the project root. .parquet, .csv, or .tsv; one row per artifact the plan promises to produce.
    • Grain — the declared row grain, in words. This is the claim DQ3 verifies rather than assumes.
    • Key Columns — the declared primary key DQ3a tests for uniqueness. Write a, b for a bare primary key, or pk: a, b; event: c, d to also declare the coarser business/event key DQ3c needs to catch amendments and restatements. Without an event: clause DQ3c reports N/A, and the runner says why. Add constant: c (one repeated non-null value, e.g. paper in a per-paper file) to exempt that column from DQ1; an all-null column still fails.
    • Required Window — the sample period COV checks, as [column: ]YYYY-MM-DD..YYYY-MM-DD. Write n/a for an unwindowed output; COV then reports N/A with that as its reason. Naming the column is optional only when exactly one date column exists; an ambiguous window is a FAIL, not a pass.
  • A ## Review Surfaces section naming the concrete tables, figures, notebook exports, diagnostics, or decisions the user will inspect during human review.

  • refs on every task row and on the lens — required, may be empty. work's spine does not validate it; wc-probe P7 refuses an absent key in THIS file, so a live run assembled from an approved plan is unchecked. Write refs: [] to state "no domain rules" rather than omitting the key.

  • Batch extraction strategy. A task whose work is per-document LLM coding or extraction is a task that LAUNCHES a batch job on pre-cut excerpts, provider picked by batch-router (with a 1-request synchronous test first). It is never an agent reading the documents. Dual-class share status of a 10-K is already built: ${CLAUDE_PLUGIN_ROOT}/skills/wrds/scripts/dual_class/ (extractor + gated classifier; ${CLAUDE_PLUGIN_ROOT}/skills/wrds/references/dual-class.md).

Phase 3 — GOAL

work's Phase 3 unchanged.

Phase 4 — the work call

The args go in the plan's <!-- work:dispatch --> arming block, and the dispatch is work's own work-dispatch.sh — never a hand-written runner line. That script owns the TIER 1 plan-lint gate, which refuses to dispatch on a major/critical plan finding and fails CLOSED on a verdict it cannot count; hand-rolling the invocation silently drops it. work owns the wait, the result handling and the return shape too, and work-result.sh reads the verdict. This run's projectDir is the session repo, so work's own run directory is already inside it and no --run-dir override applies. There is no built-in Workflow call — the guard at ~/.claude/hooks/main-thread-guard.sh denies that tool outright.

bash ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/work-dispatch.sh   # armed plan; or pass one
bash ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/work-dispatch.sh --provider codex   # "run this through codex"

Forward the provider. A provider named in this skill's $ARGUMENTS, however it is spelled — --provider codex, --dispatch codex, "run this on gpt" — is --provider codex on the line above. --provider is the only spelling the scripts take, and both reject the others by name. Omitting it silently runs the user's codex request on claude.

const goal = /* one sentence */
const planPath = /* the approved plan's path */
// shq single-quotes a command so quotes and $ reach check.sh intact.
const shq = s => `'${s.replaceAll("'", `'\\''`)}'`
const testCmd = /* the project's test command, or null when CLARIFY recorded "none" */
const lintCmd = /* the project's lint command, or null when CLARIFY recorded "none" */
const args = {
  projectDir,
  goal,

  // The plan's table verbatim. Every task carries refs, empty or not. The four constraint
  // aggregates are UNCONDITIONAL — every implementation task carries all four, alongside
  // whichever vendored references that task's own work needs.
  tasks: [
    { id: "T1",
      name: "<data pull or pipeline task>",
      work: "<what to build>",
      writablePaths: ["<narrow>"],
      acceptance: "<the criterion the verifier checks>",
      refs: ["${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-constraints.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-conventions.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-analysis-constraints.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-engineering-constraints.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/references/etl-enforcement.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/references/sql-patterns.md"] },

    { id: "T2",
      name: "<task whose acceptance is evidentiary>",
      work: "<what to build>",
      writablePaths: ["<narrow>"],
      acceptance: "<the criterion the verifier checks>",
      refs: ["${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-constraints.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-conventions.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-analysis-constraints.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-engineering-constraints.md",
             "${CLAUDE_PLUGIN_ROOT}/skills/ds/references/verification-patterns.md"] },
  ],

  // The DS gate: the runner reads the approved plan's ## Data Outputs and opens each artifact
  // itself. ds-dq is never conditional. tests and lint are the TARGET project's own commands,
  // collected at CLARIFY: a command, or an explicit "none" that becomes --no-test-cmd / --no-lint-cmd.
  // ONE entry point: a list of N independent commands loses one silently. The project's own test and
  // lint commands are ARGUMENTS because this skill cannot know them — omit both a command and its
  // opt-out and check.sh exits 2.
  mechanicalChecks: [
    { name: "ds",
      cmd: `bash ${CLAUDE_PLUGIN_ROOT}/skills/ds/scripts/check.sh --plan ${planPath} --project-dir ${projectDir}${testCmd ? ` --test-cmd ${shq(testCmd)}` : ' --no-test-cmd'}${lintCmd ? ` --lint-cmd ${shq(lintCmd)}` : ' --no-lint-cmd'}` },
  ],
  // check.sh runs every checks/ds/constraints/ script through `workflow-check ds --constraints-only`. work-checks.sh runs ruleChecks as a command (no agent); Jev scores the rules under checks/ds/rules/ (A1, A4, DEN, DQ4, DQ6, E7, M1, R1, UNI; DQ1 waits in checks/ds/rules/uncalibrated/ and ds-dq.py decides it); a rule with p >= 0.85 lands in rulesThatFailed and blocks, lower ones reach the lens as a ranked checklist.
  ruleChecks: { name: "jev-rules", cmd: `bun ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/rule-check.ts --workflow ds --project-dir ${projectDir} --plan ${planPath}` },

  // One lens after verification and mechanical checks; its checklist covers all five dimensions.
  //
  // It does NOT pin Explore. Explore is a built-in agent with a PREDEFINED prompt that no preloaded
  // skill reaches, and it skips the CLAUDE.md hierarchy — so a constraint-indexed judgement
  // dispatched there is graded from memory of constraint ids it was never given. ds-reviewer's body
  // is a file this repo controls, and it is read-only by tools allowlist AND by
  // tests/agent-contract.test.mjs — the same structural property Explore is pinned for, in an agent
  // whose prompt can be told what it is grading. The four aggregates are index tables in skills/ds/rules/ that point at the
  // canonical bodies under ${CLAUDE_PLUGIN_ROOT}/rules/ and are not copies of them. They reach the lens as refs,
  // with the rule bodies its checklist grades; ds-reviewer has no Skill tool, so refs are its only guaranteed route.
  lens: {
    agentType: "ds-reviewer",
    refs: ["${CLAUDE_PLUGIN_ROOT}/skills/ds/references/ds-checks.md",
           "${CLAUDE_PLUGIN_ROOT}/skills/ds/references/verification-patterns.md",
           "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-constraints.md",
           "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-conventions.md",
           "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-analysis-constraints.md",
           "${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-engineering-constraints.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-se-matching.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-join-diagnostics.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-p-hacking-prevention.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-sample-selection.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-assumption-over-evidence.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-failure-loudness.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-nondeterminism-sources.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-rerun-equivalence.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-misleading-charts.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-companion-figures.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-sample-coverage.md",
           "${CLAUDE_PLUGIN_ROOT}/rules/ds-pull-sizing-decision.md"],
    prompt: "Judge the datasets, tables and figures in the working tree against the approved plan and the goal. Read every file in refs in full before judging the items that name one. The binding constraint statements are the rule bodies under rules/ in refs; the four aggregates in refs are the index to them, not the statements. CHECKLIST, every item in scope on every run: (1) CRITERIA vs ARTIFACTS — for each success criterion in the plan and goal, is there an artifact in the tree that satisfies it? Missing or partial satisfaction is a finding, CRITICAL where the deliverable cannot stand without it. (2) SCOPE fidelity — did the changes stay inside the plan's task table and writable paths? Out-of-scope edits, unrequested features and silently skipped plan items are findings, CRITICAL where an edit landed outside every declared writable path. (3) DATA-QUALITY JUDGEMENT, against ds-checks.md. A1, A4, DEN, DQ4, DQ6, E7, M1, R1 and UNI are the jev-rules verdicts in the digest: do not re-grade them, except to rule on a verdict the digest ranks below the block line. A rule in rulesThatFailed is not re-judged either: diagnose it and route it like a mechanical failure. Jev does not score DQ1 or DEL; ds-dq.py decides DQ1. DEL (coverage improved because the base shrank) is yours and MODEL-EVALUATED: report it with the evidence you actually read, never as PASS, which presents a judgement as a computation, and a coverage improvement not attributable to the base shrinking is a finding. Two judgements reach past the wired rule texts and stay yours, stated as broader than any verdict in the digest: whether the sources the outputs combine admit the same entities, which one shared filter predicate does not establish, and whether the vintages, environment and config the plan records would let a fresh re-run reproduce the result, which seeded draws alone do not. Plan criterion compliance is item (1). A judgement you cannot support with evidence you actually read is a finding, never a pass. (4) METHODOLOGY, against verification-patterns.md: a statistic computed at a grain other than the one the plan declared, a universe predicate applied where scope was not meant to be decided, a claim whose evidence is read from code rather than observed from a run (you cannot run anything: report it as unverified), and an output the plan promised that nothing checks. (5) DS-CONSTRAINTS — grade the implemented code and outputs against the rule bodies in refs (the binding statements behind C1-C6, V1-V9, A1-A6, E1-E7); the four aggregates in refs are their index. For a constraint whose body is not in refs, open its file under ${CLAUDE_PLUGIN_ROOT}/rules/ with Read before grading it. Grade every id except the ids item (3) reserves and the ids a line from a checks/ds/constraints/ script in the digest has already decided: those are not re-graded. Grade only the constraints the task actually touches; an engineering constraint applied to a pure analysis task is a wrong finding that costs a round. Report every constraint finding with the file, the line and the quoted code, naming the constraint id, and list every id you considered including those you judged satisfied. NEVER report a constraint judgement as a computation — it is MODEL-EVALUATED. Severity across the whole checklist: `major` at minimum, `critical` where the defect invalidates the output's stated grain, universe or inference, never `minor`, which would leave the gate passing over a real universe defect. MODE — RED (a task was flagged or a mechanical check failed): diagnose EVERY failure in the digest and route it, naming cause and fix, to the task id whose writablePaths own the file, or to 'plan' when no task can own the fix. MODE — GREEN (everything passed): make one open-ended pass over the whole checklist and report each finding with an ownerTask. In both modes, rule every carried finding open or closed against evidence you actually read; silence leaves it open.",
  },

  authorityExtra: [
    "THE DS CONSTRAINT CORPUS IS ONE COMMAND AWAY. Run `${CLAUDE_PLUGIN_ROOT}/scripts/load-constraints ds` for the index — the C/V/A/E ids the ds-rules digest prints plus any further rule scoped to this workflow, each with an openable path — then read the ones your task touches. Do not work from a paraphrase of them in a brief: a paraphrase is a lossy copy of a corpus that moves.",
    "IRON LAW OF DS PLANNING — ask before you look; plan only from evidence you actually gathered.",
    "IRON LAW OF DS VERIFICATION — no check result without the runner's own output. A mechanical check reported from reading the code, or an N/A justified by a reason the model composed, is the model certifying its own enumeration. Every mechanical result is a line ${CLAUDE_PLUGIN_ROOT}/checks/ds/constraints/ds-dq.py emitted, quoted as emitted.",
    "Never report M1, UNI, DEN, DEL or R1 as PASS. The runner lists them as MODEL-EVALUATED; Jev scores M1, UNI, DEN and R1, and DEL is a judgement reported with the evidence read.",
    "DQ4 and DQ6 are `N/A` from the runner on a present output (FAIL, like every check, when the output is absent), and that `N/A` is not a third kind of pass — the runner emits a line for them only because ENUM requires one, and an N/A never sets its non-zero exit. Jev's DQ4 and DQ6 verdicts score them against the input → transform → output count chain and the before/after shape. Never read their N/A as `the runner checked this`.",
    "An artifact absent from the plan's ## Data Outputs table is one nothing will check and cannot be claimed as verified. Do not verify an output the table never declared.",
    "The analysis is done by dispatched agents. Main chat writes no .py, .ipynb, .R, .sas, .sql or .qmd file, by any tool.",
    "Standing DS doer authority — every implementation task follows the indexed constraints in the four aggregates under ${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/: ds-common-constraints.md (C1-C6), ds-common-conventions.md (V1-V9 — assumption-over-evidence, deferred verification, statistical validity, P-hacking prevention, sample-selection documentation), ds-analysis-constraints.md (A1-A6 — robustness checks, standard-error specification, visualization integrity, table-figure pairing, chart typography, chart colour) and ds-engineering-constraints.md (E1-E7 — determinism and seeds, schema contracts, join audits with row counts and match rates, idempotency, loud error handling, native document input, network politeness), each indexing the self-contained files under ${CLAUDE_PLUGIN_ROOT}/rules/. The binding statements are the ds-rules digest. If it is not in your context, invoke `Skill(ds-rules)` before writing any code — the aggregates are index tables, not the statements. Open a rule's file under ${CLAUDE_PLUGIN_ROOT}/rules/ when the task turns on its detail. All four are named in every implementation task's refs, and refs are contractual reads, not a reading list: read all four in full before writing code. ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/ds-checks.md stays a separate load.",
    "Rules: ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/ds-checks.md defines all thirteen checks; ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/etl-enforcement.md governs pipelines; ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/sql-patterns.md governs data pulls; ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/verification-patterns.md governs evidence; ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/competing-hypothesis.md governs debugging.",
  ].join("\n"),

  implementerAgentType: "ds",   // the doer's own prompt replaces Claude Code's software-engineering one, which frames a dataset as a codebase
  verifierAgentType: "Explore",
}

implementerAgentType names ds because the default agent carries Claude Code's software-engineering system prompt, and the deliverable here is a dataset, a table and a figure a human reads — not a codebase. An agent is justified only by a custom prompt, hooks or preloaded skills; ds earns it on the first alone — the constraint aggregates reach it as task refs, not as a preloaded skill.

verifierAgentType pins Explore because it has no Edit and no Write: a judge that structurally cannot modify the tree beats a prompt asking it not to. The lens buys the same property a different way — ds-reviewer is read-only by tools allowlist — because Explore's prompt is predefined and no preloaded skill or CLAUDE.md reaches it; the lens carries the four aggregates (the index) in its own refs and the rule bodies its checklist grades in the same refs. The lens returns schema-validated JSON through StructuredOutput, and the runner writes lens.json. The lens has no Bash, so it cannot re-run anything: a claim that needs a run is reported as unverified, never graded from the code.

Add ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/competing-hypothesis.md to a task's refs when that task is a diagnosis rather than a build. authorityExtra names it; refs is what makes an implementer read it in full.

The runner's own contract suite, ${CLAUDE_PLUGIN_ROOT}/skills/ds/scripts/ds_dq_test.py, is the ds-dq-tests leg of the one mechanicalChecks entry.

Phase 5 — HUMAN REVIEW

work's Phase 5 unchanged, over the plan's ## Review Surfaces. A clean technical verification is evidence for that conversation, not human acceptance.

Red flags

SituationWrong moveRight move
Sizing a sourcepull the whole thing to see how big it isthe transfer is the failure you are meant to prevent — profile filtered counts and aggregate candidates read-only
Profiling a sourcetreat a head sample as the profilenulls, grain failures and type drift live outside the head — profile shape, tail/coverage, keys and quality signals
Naming a produced artifactname it in prosea runner given prose cannot open it, key it, or window it — add its row to ## Data Outputs before approval
The DQ resultlet the implementer report itthe doer cannot see the assumption it made in both places — the runner is a mechanicalCheck and the JS reads its exit code
M1/UNI/DEN/DEL/R1report them as PASSthat presents a judgement as a computation — Jev's verdicts decide M1, UNI, DEN and R1; DEL is MODEL-EVALUATED with the evidence read
DQ4/DQ6 reported N/Aread the N/A as the runner having checked theman N/A on a present output is not a third kind of pass (an absent one is FAIL) — the runner computes neither and an N/A never sets its non-zero exit; Jev's DQ4 and DQ6 verdicts decide them
Per-document LLM coding or extractionfan out agents to read each documentlaunch a batch-router-picked batch job on pre-cut excerpts (with a 1-request synchronous test first)
A judgement that depends on the constraint indexdispatch a built-in agent (Explore, Plan, general-purpose)their prompts are predefined, no preloaded skill reaches them and they skip the CLAUDE.md hierarchy, so the constraints are graded from memory — dispatch a custom agent whose body you control, like ds-reviewer
Handing a doer the constraint aggregatesname the four paths in the task prompt's prose, or copy the aggregates into a skillprose is discretionary and a copy is a second source of truth tests/constraints-no-duplication.test.ts fails on — put the four canonical paths in the task's refs, which work defines as reads the doer owes in full
Project statewrite a SPEC.md, STATE.md or LEARNINGS.mdcompeting state makes progress ambiguous — the approved plan is the authority and work hashes it
Something work does not obviously dowrite a ds/workflow.jsask which work parameter is missing — mechanicalChecks is what makes the DQ runner the gate

elide-case — a court opinion cut into a student reading, on the work spine

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Typst rules in force here — read the one that governs what you are writing; they are independent: !k=notes,prose; command -v typst-rules >/dev/null 2>&1 && exec typst-rules "$k"; echo "(typst corpus unavailable: NO Typst rule is listed here — install the typst plugin, or start a new session so its bin/ reaches PATH)"

The lifecycle is work. Read it and follow it. This file supplies work's spine with the domain and nothing else — the three CLARIFY questions, the plan grammar, the task rows, the two scored checks, the one mechanical entry point, the refs and the authority text. It ships no workflow.js and no .js of any kind, and restates none of work's mechanics.

The domain rules live in references/ and reach every dispatched agent through refs:

refwhat it carries
references/authenticity.mdthe two Iron Laws — authentic reporter text, and compile-before-table
references/retrieval.mdworkflows:westlaw as the first route and the .westlaw.docx as the ONLY stored source, the corpus fallback (opinions vs RECAP, the 404, token gating, telling a clean source) and what that fallback's PDF cannot carry, and the two reasons to run on Gemini — a scanned source or a filter-tripping subject
references/editing-marks.mdone elision mark per addendum, [ ] conventions, never altering subject/tense/sentence boundary, the editors'-note rule (a note is the EXCEPTION, for background), where the mechanical conventions live (the preamble, once), the layout-only widow rule, and the source-mapping marker // elide-source: that check.sh reads
references/verification.mdwhat each gate leg decides, and how to read a miss rate
**The Iron Law of Authentic Text.** Every word of the excerpt is authentic reporter text, retrieved this run and saved under `docs/`. Never drafted from memory, from a Teacher's Manual summary, or from a casebook note. A plausible paraphrase handed to students as a quoted opinion is fabricated authority in a course reader.

The Iron Law of Compile-Before-Table. The summary table is written AFTER the compile, from the checker's computed output. Never before, never from an estimate.

Both are stated in full in references/authenticity.md, which is the refs entry every implementer and the advisory fidelity reviewer must read. Refactoring this file never weakens them.

Phase 1 — CLARIFY: three questions, asked with AskUserQuestion, never self-answered

This interview is the point of the workflow. Ask exactly these three, with AskUserQuestion, and record every answer verbatim in the plan:

  1. What is the doctrinal thread, in your own words — and which assigned reading does it cut against? One thread, named by the instructor, plus the case or casebook page it is in tension with. This is what decides every sentence that survives the cut.
  2. Is this case taught for its HOLDING or for its REASONING? Holding means the disposition and the rule statement survive and the analytical march may be compressed. Reasoning means the analytical steps survive and the disposition may be a sentence.
  3. What is the page target for each reading? One MIN-MAX per reading, offered with 2-6 as the default option — the skill's established range — since a reading whose length he does not care about takes the default. A lead case and a short blog post reasonably differ, which is why the plan carries one target per row. When the request or the schedule already gives a target, pre-fill the options from it rather than posing the question blind; nothing is inferred silently, and nothing is typed at a command line.

Do not ask anything else. Insertion point and length are not open-ended interview axes. The per-reading page target is question 3 above, set in the interview and enforced from the plan, because a number typed at the command line is a number an agent can invent. Insertion point and class number still come from the request, or off the schedule if the request omits them.

THE INTERVIEW, AND EXACTLY WHAT ENFORCES IT

Never self-answer either question, never state an assumption and continue, never infer the thread from the Teacher's Manual or from the case itself. work's CLARIFY is conversational and a dispatched agent cannot call AskUserQuestion — a run started from a farmed-out agent has nobody to ask, and its only correct move is to stop and report that the interview is unanswered. Elevating the run to a session that can reach the user is the fix; guessing is not.

A doctrinal target the model states to itself is a silent guess; nothing makes it ask. This workflow exists to remove that path.

The chain, with each link's real strength named. Three things carry the interview, and only one of them is software:

  1. The answers are recorded in the plan, under ## Doctrinal target, on the three labelled lines Phase 2 specifies.
  2. The instructor reads and approves that plan before the run arms, and the spec hash binds what he approved. This is the link that makes the answers his. It is a human gate, and it is the only one there is: an instructor who approves a plan whose ## Doctrinal target holds answers he never gave defeats the whole chain, and no software in this design closes that.
  3. ${CLAUDE_PLUGIN_ROOT}/skills/elide-case/scripts/check.sh --plan <path> refuses a plan whose section is missing or unfilled — absent section, or Doctrinal thread: / Cuts against: missing, empty or a placeholder, or Taught for: missing or saying neither holding nor reasoning. The leg fails closed: passing neither --plan nor --no-plan is itself a FAIL naming the missing flag, and --no-plan (for fixture and dev runs that check an addendum alone) prints a loud NOT CHECKED line.

What that leg proves, and what it does not. It proves the section exists and every required answer is non-empty. It is a backstop against an orchestrator that skipped the question — nothing more. The plan file is written by the orchestrator, so no grep of it can tell the user's answers from the orchestrator's own, and work's mechanicalChecks run at gate time, after IMPLEMENT, so no mechanical leg can refuse to proceed past CLARIFY. Do not close the gap with a second mechanicalChecks entry — P10 refuses one, and the leg belongs inside the single entry point.

Phase 2 — PLAN

work's Phase 2. Domain requirements on the plan:

  • A ## Readings In Scope section, one row per reading: caption | source docs/*.westlaw.docx (a docs/*.txt from an earlier run, or from the fallback route, still resolves) | page target. This section is the source of truth for the page target, and the --plan leg enforces it exactly as it enforces the three Doctrinal target lines: an absent section, a row with no page target, or a placeholder one (TBD, <…>, N/A) FAILS the leg naming the offending row. The target is per reading, from interview question 3; check.sh maps each .typ caption to its row and passes that row's own MIN-MAX to check-addendum.py. The caption is the identifier the .typ and the mechanical entry point key on — the entry point derives its per-reading checks from the captions in the .typ. Each row also generates one cut-<slug> task, which is how a per-reading re-run reaches work's onlyTasks.
  • The first two CLARIFY answers, quoted (the third is the page target column above), under ## Doctrinal target, on three labelled lines the --plan leg parses — Doctrinal thread:, Cuts against:, Taught for: (holding or reasoning). A placeholder (TBD, <…>, N/A) reads as absent and fails the leg.
  • A ## Non-court readings section whenever a reading is not court text — one - <caption> — <why> bullet each, and omitted entirely when every reading is an opinion. Declaring a reading non-court text is the INSTRUCTOR's call, recorded in the plan, never a marker the implementer writes into the .typ: a self-exempting marker would let fabricated text switch the verbatim check off on itself, which is the first Iron Law defeated by the file it governs. The quotes leg reads this list from --plan and reports those captions DECLARED-UNCHECKED; a // elide-unchecked: marker in the .typ is no longer honoured and its presence is itself a FAIL naming the reading.
  • Run sizing naming the one mechanical entry point, the two scoredChecks keys on the Scored checks: line marked ADVISORY, the criteria/scope checklist on the Review lens: line, and the task rows. R readings give R + 2 tasks — one retrieve, R cut-<slug>, one compile-table — and 2R scored items; with one lens and one mechanical check the floor is 2(R + 2) + 1 + 1 + 2R = 4R + 6 agents — 10 for a single reading, 22 for four readings.
  • The wave assignment, stated: retrieve in wave 1, every cut-<slug> in wave 2, compile-table in wave 3. work-dispatch.sh prints the wave graph it computes; if it does not match those three lines, the dependsOn edges are wrong, not the prose.

Writable paths for a generated run are addenda/, output/addenda/ and docs/ only. notes/, slides/ and templates/ are excluded, and so is everything else in the course tree.

Phase 3 — GOAL

work's Phase 3, on the PASS phrasing: a run that writes can turn a FAIL into a PASS, and work's outer fix loop is what does it.

Condition: work-result.sh returned 0 for the plan path, and the tuicr gate returned approved.

/goal work-result.sh has returned 0 for <plansDirectory>/<slug>.md at its current hash, and the
tuicr gate has returned approved, or stop after N turns

Name the plan by path, never by a pinned digest — the FAIL loop amends and re-hashes the plan, so a named hash self-invalidates. Name work-result.sh's exit code, which is the run's verdict: overallPass in result.json is not the verdict and must never be read directly (work/SKILL.md's mechanicalChecks row). Naming the returned verdict is also what makes a gate FAIL reach a stopping condition at all — including one carried by a lens finding, which otherwise reaches none.

The condition names no legibility or fidelity finding. A clause like "no critical|major finding survives" would reinstate the non-terminating prose review this design exists to exclude: round n's fix gives round n+1 new things to object to. Those two run as scoredChecks, which cannot reach overallPass at all, so their output sits in result.json and is retrievable at Phase 5 by a human who runs the command there. Nothing carries it to him. Both named verdicts get printed, so both are judgeable from the transcript.

Phase 4 — the work call

The args go in the plan's <!-- work:dispatch --> block and the dispatch is work's own work-dispatch.sh — never a hand-written farm.sh --workflow line, which drops the TIER 1 plan-lint gate and both probe gates:

bash ~/.claude/skills/workflows/skills/work/scripts/work-dispatch.sh \
  --run-dir /home/eh/.local/state/work "$PLAN"     # ABSOLUTE — never the course tree

Forward the provider. A provider named in this skill's $ARGUMENTS, however it is spelled, becomes --provider codex on that line. Omitting it runs the user's codex request on claude. The content-filter facts that make this matter are in references/retrieval.md, alongside the other reason a run belongs on --provider gemini: a source whose text came from OCR.

--run-dir is not optional: work resolves .work/<run-id> against $PWD, which on an elide run is the course directory — a tree this run writes only addenda/, output/addenda/ and docs/ in.

The args object, complete as it stands.

One task per reading, in three waves. The ## Readings In Scope table generates one cut-<slug> row each — cut that reading and nothing else — above one shared retrieve row and below one compile-table row. A second reading therefore adds exactly one task, and adds no lens and no mechanical entry: the entry point derives its per-reading legs from the .typ itself. This granularity is not cosmetic. It is what makes a per-reading re-run expressible in work's own vocabulary — see The FAIL loop below.

The wave shape is forced by work's disjointness rule, not chosen for tidiness. Same-wave tasks must have pairwise-disjoint writablePaths, checked prefix-aware and refused at arg-validation, before any agent is dispatched. Every reading needs its source in docs/, so if each reading row claimed docs they would all land in wave 1 claiming the same directory and a two-reading addendum would be refused before dispatch. One row owns docs/ instead:

waverowswritable pathswhy disjoint
1retrieve (one row, however many readings)docsalone in its wave
2one cut-<slug> per reading, each dependsOn: ["retrieve"]addenda/NN-addendum-<slug>.typone file each, slugs distinct
3compile-table, dependsOn every cut-<slug>output/addenda, addenda/NN-addendum.typalone in its wave

retrieve writes docs/ and nothing else; the cut rows read it and never write it. A single-reading run keeps all three rows — collapsing them re-introduces the overlap the moment a second reading is added.

retrieve stays one serial row (P17 exempted): every Westlaw export shares the user's one signed-in browser, one delivery queue whose top item #0_qitem is clicked, and one Downloads directory, so concurrent retrievals can take each other's file.

const addendum = /* /home/eh/areas/secreg/addenda/NN-addendum.typ with the addendum's number */
const pdf = /* /home/eh/areas/secreg/output/addenda/NN-addendum.pdf, same number */
const planPath = /* the approved plan, /home/eh/areas/secreg/.claude/plans/<slug>.md */
const goal = /* one sentence */
const args = {
  projectDir: "/home/eh/areas/secreg",          // the COURSE directory — the tree being edited
  goal,

  // ── The deterministic floor: ONE entry point, whose exit code IS the mechanical verdict. ──
  // Seven legs — the plan's interview answers (check-plan.py), typst compile, check-quotes.py per derived caption,
  // check-addendum.py --target, the strays leg (canonical widows.py/orphans.py/runts.py
  // plus check-stranded-headings.py), check-marks.py (one elision mark per addendum; every
  // retained footnote renumbered 1..k and paired), and `bin/workflow-check elide
  // --constraints-only`, which runs every constraint under checks/elide/constraints/ (where the six
  // checkers live) over the addendum and the plan — none short-circuiting. A second entry here would spread the
  // verdict over two commands and lose one silently; P10 refuses it.
  // The plan leg FAILS CLOSED: pass `--plan <md>` on a real run or `--no-plan` on a fixture run,
  // and passing neither is a FAIL naming the missing flag. `--plan` also carries the plan's
  // `## Non-court readings` declarations. It checks that the answers are RECORDED and non-empty;
  // that they are the instructor's is carried by his approval of the plan, not by this command.
  // `--plan` ALSO supplies each reading's page target, from the plan's `## Readings In Scope`
  // rows, so a real run passes NO `--target`. The plan wins if one is passed anyway — the flag is
  // ignored with a loud line naming the plan's values. `--target MIN-MAX` / `--no-target` remain
  // the fail-closed pair on `--no-plan` fixture and dev runs, where passing neither is a FAIL
  // naming the missing flag and the leg must not report length.
  // The WIDOWS leg is ON BY DEFAULT, which is its fail-closed form: an absent flag RUNS the
  // check, so no invocation can quietly certify page breaks nobody looked at. `--no-widows`
  // is the loud waiver for fixture and dev runs; passing it with `--widows` FAILS. Its fix
  // vocabulary is LAYOUT-ONLY — spacing or pagination, never the court's words, and when the
  // two conflict fidelity wins and the widow stands.
  mechanicalChecks: [
    {
      name: "elide-check",
      cmd: `bash ${CLAUDE_PLUGIN_ROOT}/skills/elide-case/scripts/check.sh --addendum ${addendum} --pdf ${pdf} --plan ${planPath}`,
    },
  ],

  // Jev scores the calibrated excerpt rules under checks/elide/rules/: an elision mark cutting
  // inside a holding or rule sentence, a heading or ordinal sequence that skips with nothing
  // marking it, editors' voice set as the court's, stress inside a quotation with no emphasis
  // notation, and a reading's note that is only mechanical boilerplate. p >= 0.85 lands in
  // rulesThatFailed and blocks; lower ones reach the lens ranked. It scores the assembled
  // addendum by --files and names no --project-dir, so it scores that one file WHOLE: the run's
  // baseline scoping (--since) is appended only to a cmd that names --project-dir. Earlier
  // readings already in the same file are therefore graded too; earlier addenda are other files.
  ruleChecks: { name: "jev-elide-rules", cmd: `bun ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/rule-check.ts --workflow elide --files ${addendum} --plan ${planPath} --project secreg` },

  tasks: [
    // ── WAVE 1: ONE row, whatever the reading count. It is the only row that writes `docs/`,
    // which is what keeps wave 2 pairwise-disjoint. See the wave table above.
    {
      id: "retrieve",
      name: "retrieve every reading's reporter text into docs/",
      work: "For EVERY row of the plan's `## Readings In Scope` table, retrieve the opinion FIRST through the `workflows:westlaw` skill, which owns the export procedure and the .westlaw.docx naming convention and yields publisher-keyed text needing no OCR adjudication. THE DOCX IS THE ONLY ARTIFACT: it is the golden copy, its text projection is in memory, and you write no .txt beside it — a stored twin drops the docx's italic runs, so a case name the court italicised would compile in roman. Fall back to ${CLAUDE_PLUGIN_ROOT}/skills/elide-case/scripts/fetch-opinion.sh — saving both docs/<name>.pdf and docs/<name>.txt under the naming convention already in docs/ — only when Westlaw does not cover the case or no Westlaw credential is available; that route yields a court PDF with NO formatting layer, so its italics are unrecoverable and the gate says so on every reading cut from one. On that fallback, a citation 404 means 'not in the opinions corpus', not 'no such case' — take the RECAP docket-entry route. Retrieve and nothing else: write no addenda file, cut no text, and do not edit an excerpt. A reading whose source cannot be retrieved is reported as such — never substituted with a related opinion, a Teacher's Manual summary or a previous excerpt.",
      writablePaths: ["docs"],
      acceptance: "Every `## Readings In Scope` row has its retrieved source on disk under /home/eh/areas/secreg/docs/ — <name>.westlaw.docx ALONE on the Westlaw route, or <name>.pdf plus <name>.txt on the corpus fallback — and each is the reporter text of that row's caption — `check-source-identity.py` (a leg of check.sh's constraints leg) fails a `.docx` source that does not contain the caption's reporter citation; a Route 2 `.txt` lacking it is reported IDENTITY NOT CHECKED (exit 0), because a slip opinion or RECAP entry carries no reporter citation, so the Phase 5 reader confirms that source by eye. Writing a <name>.westlaw.txt for a source retrieved THIS RUN fails this row — the projection is in memory. A .westlaw.txt already in docs/ from an earlier run is left alone: an addendum whose plan or `// elide-source:` names one still resolves and still passes, and deleting it would break a gate that was green.",
      redDisposition: "Retrieval writes no excerpt, so there is no behaviour a redCommand could hold red: the run's mechanical entry point cannot run at all until wave 2 has produced a .typ to check. The per-task Verify channel adjudicates the files this row is responsible for.",
      refs: [
        "${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/authenticity.md",
        "${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/retrieval.md",
      ],
    },
    // ── WAVE 2: ONE ROW PER READING. Copy this row for each `## Readings In Scope` line,
    // substituting the slug, caption, source basename and page target. Each row cuts its own
    // reading, so `tasksThatFlagged` on a FAIL names exactly the readings to re-cut. Each writes
    // ONE fragment and never `docs/` — that is what makes the wave disjoint.
    {
      id: "cut-<slug>",
      name: "cut <caption>",
      work: "ONE READING, CUT ONLY. Write the excerpt into addenda/NN-addendum-<slug>.typ, from the docs/<source> the `retrieve` task saved (the file the plan's source column names: the .westlaw.docx, or the .txt on the Route-2 fallback) and from nothing else; check-quotes.py projects a docx to text in memory, so read a docx the same way rather than writing one out. THE EXCERPT INHERITS THE COURT'S ITALICS FROM THE SOURCE and must neither invent nor drop them: a case name the projection carries as `_Weinberger_` is set as `_Weinberger_` in the .typ, and emphasis the source does not carry is not added. You do not write docs/; if the source is missing or unreadable, stop and report it rather than retrieving it yourself. Keep only the transaction structure, the analysis of the ONE thread the plan's Doctrinal target section quotes, and the reasoning that puts the case in tension with the assigned reading it names. Holding-taught means the disposition and rule statement survive; reasoning-taught means the analytical steps do. Caption block and judge line follow addenda/01-addendum.typ. EDITORS' NOTE — the default is NO note (instructor ruling). Write one only when this reading needs BACKGROUND the student cannot read the case without: prior history of the same litigation, an earlier related decision, or real-world context. The mechanical conventions — the elision mark, footnotes omitted except as indicated and renumbered, record citations dropped without notation — are stated ONCE in the addendum's preamble and cover every reading; do not repeat them per reading. Elision marks and the disclosure rule are in editing-marks.md — an elided passage may instead be replaced by `#emph[[Summary: ...]]` (<= 40 words, between sentences, no quotation, figure or holding), whose form is in editing-marks.md and which check-marks enforces; record citations, party-brief citations, string cites and internal cross-references may go silently; a dropped substantive footnote, qualifier, alternative holding or dissent may not, and a disclosure specific to THIS reading (e.g. it retains footnote 16 renumbered as 1) goes in this reading's own note. SOURCE MAPPING: check.sh derives each reading's source from its caption and resolves it to the .docx. When that derivation is wrong or ambiguous, declare the file with `// elide-source: <basename.westlaw.docx>`, whose syntax and placement are in editing-marks.md — never point it at a loosely related file to quiet a NOSOURCE, and note that it names a file that must exist, so it cannot exempt anything. You may NOT declare a reading non-court text: `// elide-unchecked:` is no longer honoured and writing one is itself a FAIL, because the agent writing the excerpt cannot be the one who switches the verbatim check off on it. A reading with no reporter source stops and reports to the instructor, who declares it in the plan's `## Non-court readings` section.",
      writablePaths: ["addenda/NN-addendum-<slug>.typ"],
      dependsOn: ["retrieve"],
      // Task-specific: check-quotes.py for THIS caption alone. Not the run's entry point — a task
      // graded by the shared command can be reported done because a DIFFERENT task's work made it
      // pass, which leaves `work`'s per-task Verify channel nothing task-specific to judge.
      acceptance: "python3 ${CLAUDE_PLUGIN_ROOT}/checks/elide/constraints/check-quotes.py /home/eh/areas/secreg/addenda/NN-addendum-<slug>.typ /home/eh/areas/secreg/docs/<source> --caption '<this reading's caption>' --skip-editorial exits 0, having checked a non-zero number of sentences for that caption. <source> is the file the plan's source column names for this reading: the .westlaw.docx, or the .txt on the Route-2 fallback.",
      redDisposition: "The behaviour this task must satisfy is already executed by the run's one mechanical entry point, whose check-quotes.py leg is red before the excerpt exists and green after; a redCommand here would be that same command run a second time by a second agent.",
      refs: [
        "${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/authenticity.md",
        "${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/editing-marks.md",
      ],
    },
    // ── WAVE 3: alone, so its two writable paths collide with nothing.
    {
      id: "compile-table",
      name: "compile, then write the table from the computed rows",
      work: "Assemble the per-reading fragments into addenda/NN-addendum.typ in the plan's Readings In Scope order, compile via teaching:generate-addendum, then run check-addendum.py --target and paste ITS computed rows into the summary table. THE PREAMBLE CARRIES THE MECHANICAL CONVENTIONS FOR THE WHOLE DOCUMENT — one sentence covering the elision mark, footnotes omitted except as indicated and renumbered, and record citations dropped without notation. This row owns that sentence, because it owns the preamble; a per-reading note repeating it is the boilerplate the no-default-note rule removes. Never hand-edit a range to make the checker pass — the checker is reporting the truth about the PDF. On a length or arity miss, fix the offending READING's fragment and recompile; do not rewrite a fragment this row did not break.",
      writablePaths: ["output/addenda", "addenda/NN-addendum.typ"],
      // Task-specific: check-addendum.py's own arity, table-truth and length legs against the
      // compiled PDF. The run's entry point stays the WHOLE mechanical verdict without also being
      // this task's only grader.
      acceptance: "python3 ${CLAUDE_PLUGIN_ROOT}/checks/elide/constraints/check-addendum.py /home/eh/areas/secreg/addenda/NN-addendum.typ /home/eh/areas/secreg/output/addenda/NN-addendum.pdf --target <the page target the plan's `## Readings In Scope` rows carry; when the rows differ, run the run's entry point instead, which reads each row's own target> exits 0 — summary-table rows equal readings found (arity), every stated page range matches the compiled PDF (table truth), and each reading is inside its target (length).",
      dependsOn: ["cut-<slug>" /* , one id per reading row; `retrieve` is reached transitively */],
      redDisposition: "Work whose whole acceptance is the run's mechanical entry point; `work` re-runs that command in a shell to adjudicate it, so a redCommand would duplicate the gate rather than gate the task.",
      refs: [
        "${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/authenticity.md",
        "${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/verification.md",
      ],
    },
  ],

  // ── legibility and fidelity are `scoredChecks`, and that is WHY they are advisory. ──
  // The `lens` gates by construction: the work skill's workflow.js computes overallPass FROM the
  // blocking lens findings that stand, so folding these two into the lens prompt and calling them
  // advisory in prose would be a claim about the spine rather than a parameter to it. `scoredChecks` is the mechanism
  // that carries the property: overallPass is computed without reading any scored value, there is no
  // threshold and no blockBelow, it opens no selector channel, and a dead agent does not flip the
  // verdict (work/references/scored-checks.md, S6/S7). That is ~/.claude/CLAUDE.md rule 9 satisfied
  // structurally — an open-ended prose review cannot gate a loop, because round n's fix gives round
  // n+1 new things to object to.
  //
  // ONE ITEM PER READING, so a score reads per reading exactly as a task does. Each agent returns
  // RAW COUNTS and never sees the weights below, which is what stops a self-graded score.
  scoredChecks: [
    {
      key: "legibility",
      items: ["<caption>" /* , one per `## Readings In Scope` row */],
      agentType: "Explore",
      refs: ["${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/editing-marks.md"],
      prompt: "Read this reading's excerpt in addenda/NN-addendum.typ as a student would, with no access to the full opinion. Count ONLY these five defect kinds, and report COUNTS plus the offending sentences verbatim: (1) unparsable — a sentence that does not parse as a sentence; (2) fusedElision — an elision that joins two fragments into something the court did not write; (3) danglingReferent — a pronoun or 'it'/'that' whose referent was cut; (4) unattributedQuote — a block quote whose speaker is not identifiable from the excerpt alone; (5) undefinedTermOfArt — a term of art used before it is introduced. Anything outside those five is not counted here. `itemsChecked` is the number of sentences you actually read; report 0 only if you read none, which marks the item unreliable rather than clean.",
      schema: {
        type: "object",
        additionalProperties: false,
        required: ["itemsChecked", "unparsable", "fusedElision", "danglingReferent", "unattributedQuote", "undefinedTermOfArt"],
        properties: {
          itemsChecked: {type: "integer"},
          unparsable: {type: "integer"},
          fusedElision: {type: "integer"},
          danglingReferent: {type: "integer"},
          unattributedQuote: {type: "integer"},
          undefinedTermOfArt: {type: "integer"},
          offendingSentences: {type: "array", items: {type: "string"}},
        },
      },
      passthrough: ["offendingSentences"],
      components: [
        {name: "reads", weight: 1.0, base: 10, penalties: {unparsable: 2.0, fusedElision: 2.0, danglingReferent: 1.0, unattributedQuote: 1.0, undefinedTermOfArt: 0.5}},
      ],
    },
    {
      key: "fidelity",
      items: ["<caption>" /* , one per `## Readings In Scope` row */],
      agentType: "Explore",
      refs: ["${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/authenticity.md"],
      prompt: "Verbatim matching cannot see misrepresentation. Judge whether the CUT misrepresents the opinion, reading this reading's excerpt against its full source docs/<source>, the file the plan's source column names. A .westlaw.docx cannot be opened by Read: project it to text with `python3 ${CLAUDE_PLUGIN_ROOT}/skills/westlaw/scripts/westlaw-docx-text.py docs/<source>` (Bash) and read that output; never write it to disk. A .txt (Route-2 fallback) is read directly. Count ONLY these five, and report COUNTS plus the excerpt passage and the source line for each: (1) quotedAsOwnReasoning — a passage the court QUOTES FROM ANOTHER CASE presented as the court's own reasoning  (e.g. a block quote the court cites only to reject, rendered as running text); (2) unqualifiedHolding — a holding stated without the qualifier that limits it; (3) omittedAlternativeHolding — an omitted alternative holding that changes what the case stands for; (4) dissentAsMajority — a dissent's language read as the majority's; (5) misstatedSummary — an editorial `#emph[[Summary: ...]]` that misstates, slants or adds to the passage it stands in for (trace it to the source text cut at that point). `itemsChecked` is the number of excerpt passages you traced back to the source; 0 marks the item unreliable rather than clean.",
      schema: {
        type: "object",
        additionalProperties: false,
        required: ["itemsChecked", "quotedAsOwnReasoning", "unqualifiedHolding", "omittedAlternativeHolding", "dissentAsMajority", "misstatedSummary"],
        properties: {
          itemsChecked: {type: "integer"},
          quotedAsOwnReasoning: {type: "integer"},
          unqualifiedHolding: {type: "integer"},
          omittedAlternativeHolding: {type: "integer"},
          dissentAsMajority: {type: "integer"},
          misstatedSummary: {type: "integer"},
          offendingPassages: {type: "array", items: {type: "string"}},
        },
      },
      passthrough: ["offendingPassages"],
      components: [
        {name: "faithful", weight: 1.0, base: 10, penalties: {quotedAsOwnReasoning: 3.0, unqualifiedHolding: 2.0, omittedAlternativeHolding: 2.0, dissentAsMajority: 3.0, misstatedSummary: 2.0}},
      ],
    },
  ],

  // One lens after verification and mechanical checks; legibility and fidelity remain advisory above.
  lens: {
    agentType: "Explore",
    refs: ["${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/editing-marks.md"],
    prompt: "Judge the delivered artifacts against the approved plan and the goal. CHECKLIST: (1) CRITERIA vs ARTIFACTS — for each success criterion in the plan and goal, is there an artifact that satisfies it? Missing or partial satisfaction is a finding, CRITICAL where the deliverable cannot stand without it. (2) SCOPE FIDELITY — did the changes stay inside the plan's task table and writable paths, limited to addenda/, output/addenda/ and docs/? Out-of-scope edits, unrequested features and silently skipped readings are findings, CRITICAL where an edit landed outside every declared writable path. Severity is MAJOR at minimum. Legibility and fidelity are advisory scoredChecks, not additional checklist items here. Holding-cutting elisions, unmarked heading or ordinal skips, editors' voice in the court's text, un-noted emphasis in quotations and boilerplate notes are the jev-elide-rules verdicts in the digest: do not re-grade them, except to rule on a verdict the digest ranks below the block line. MODE — RED (a task was flagged or a mechanical check failed): diagnose EVERY failure in the digest and route it with cause and fix to its owner task id, or to 'plan' when no task can own the fix. MODE — GREEN (everything passed): make one open-ended pass over the checklist and report each finding with an ownerTask. In both modes, rule every carried finding open or closed against evidence you actually read; silence leaves it open.",
  },

  authorityExtra: "DOMAIN RULE — every word of the excerpt is authentic reporter text, cut from the .westlaw.docx retrieved THIS RUN into docs/ and projected to text in memory — never from a .txt written beside it, which is not produced. Never from memory, a Teacher's Manual summary, a casebook note, or a previous excerpt. DOMAIN RULE — the summary table is written after the compile, from the checker's computed rows; never hand-edit a range to make the checker pass. WRITE SURFACE — addenda/, output/addenda/ and docs/ only. notes/, slides/ and templates/ are excluded, and notes/07-security.typ may be open in the user's editor. THE TYPST CORPUS IS ONE COMMAND AWAY, AND IS NOT IN YOUR REFS. Run `typst-rules notes,prose` for the index, then read the ones your edit touches; it is on PATH and needs no path or plugin variable. Do not work from a paraphrase of these rules.",

  maxAgents: 50,
}

Phase 5 — HUMAN REVIEW

work's Phase 5, on the compiled PDF: the reviewer reads the reading, not the diff. Open the PDF beside human-review-gate.sh -w.

Where the legibility and fidelity scores actually are

They are in result.json under scores[], and nothing delivers them. work returns each item's computed score, its raw counts, and — because passthrough declared them — the verbatim offending text under evidence. Phase 5 is a human step, so no exit code can compel anyone to look. Print them with one command:

jq -r '.scores[] | "\(.key)  \(.item)  composite=\(.composite // "null")  itemsChecked=\(.itemsChecked // "n/a")  \(.reason // "")",
       "  components: \(.components // {} | tojson)",
       "  evidence:   \(.evidence   // {} | tojson)"' \
  /home/eh/.local/state/work/<run-id>/result.json

The field is composite, not score — work emits one scores[] entry per (key, item) pair as {key, item, components, composite, itemsChecked, evidence, reason?}, and there is no cross-item mean or total to read.

This is an INSTRUCTION, not a gate, and the distinction is the whole design. The scores cannot fail the run by construction — that is exactly why they are scoredChecks and not the lens — and therefore a reviewer who does not run that command does not see them. That is the accepted cost of their not gating, and it is stated here rather than papered over with a sentence claiming they are "reported".

If that cost is unacceptable on a given run, the remedy is NOT to fold them into the lens prompt — that reinstates the non-terminating prose gate rule 9 forbids — it is for the instructor to read the excerpt himself, which is what Phase 5 is for.

When the reviewer does want them in the findings file, write result.json's scores[] there before opening the PDF, one section per reading, naming the fields:

## Advisory scores — legibility and fidelity (these gate nothing)

### <caption>
legibility  composite <n>/10   itemsChecked <n>
  unparsable <n> · fusedElision <n> · danglingReferent <n> · unattributedQuote <n> · undefinedTermOfArt <n>
  evidence.offendingSentences:
    - "<verbatim sentence>"
fidelity    composite <n>/10   itemsChecked <n>
  quotedAsOwnReasoning <n> · unqualifiedHolding <n> · omittedAlternativeHolding <n> · dissentAsMajority <n> · misstatedSummary <n>
  evidence.offendingPassages:
    - "<verbatim excerpt passage, and the source line it was traced to>"

Three rules on the rendering, because each is a way the output silently stops meaning anything:

  • evidence.offendingSentences and evidence.offendingPassages are the whole point of the passthrough declarations — the counts say how many, the evidence says which. Render the verbatim text. A section showing counts with no evidence gives the reviewer nothing to act on.
  • A null composite is rendered as null WITH its stated reason — never as 0, never as a blank, never omitted. null means the item was unreliable or its agent died; 0 means it was measured and scored zero. Collapsing them turns "nothing checked this reading" into "this reading is bad", and the reviewer cannot tell.
  • scores: [] with scoresRun/scoresReported as null is n/a, not clean. Say the phase did not run rather than printing an empty section that reads as nothing found.

The reviewer decides what to do with all of it. A low score re-runs nothing by itself — see The FAIL loop.

The FAIL loop — work's selectors, and where the per-reading granularity comes from

The selector is work's own, all four channels, consumed together: tasksThatFlagged, mechanicalThatFailed, lensesThatFlagged and planFindings. A scoped re-run is onlyTasks + priorResults, derived by work-redispatch.sh from the result. There is no caption channel — work scopes by task id and nothing else, so a re-run instruction phrased as "re-cut the failing caption" names a dimension work does not have and cannot be executed.

Per-reading re-cutting falls out of one task per reading, not out of any caption machinery. Because each reading is its own task row, a reading that fails puts its own id in tasksThatFlagged, and onlyTasks re-runs exactly that reading. A four-reading addendum keeps the three readings the gate already adjudicated — re-cutting them manufactures new surface for the next round.

mechanicalThatFailed names elide-check, whose leg lines name the failing reading. The lens routes each failure to its owner task or to plan; task routes narrow the re-run, while planFindings require amending and re-hashing the plan. Two domain rules on the fix, both in references/verification.md:

  • A miss rate above roughly a quarter is the wrong source, not bad sentences. Re-cut the reading from docs/<name>.westlaw.docx; never patch sentence by sentence.
  • Never trust a corruption tell you have not grepped for in both files.

lensesThatFlagged is ["lens"] whenever a blocking finding of the criteria/scope lens still stands, and [] otherwise. The legibility and fidelity scores appear in no selector at all: scoredChecks opens no channel, which is the same fact as their not gating. A low score is read by the human at Phase 5 and acted on there or not at all; it never re-runs anything by itself. A null score with a stated reason means the item was unreliable or its agent died — not clean.

The rule for whoever edits this next

Any property this file asserts about work must be traceable to a parameter work reads or to a command's exit code. Otherwise it is decoration. Three times in this skill's construction a property was written as a sentence and bound nothing, and a fourth is recorded below: a re-run selector declared to be a "caption", a dimension work has no channel for; two review dimensions declared ADVISORY while sitting in the gating review lens, which work computes overallPass from; and an interview refusal declared absolute when no mechanism could carry it — the plan is written by the orchestrator, so a grep of it cannot identify the answerer, and mechanicalChecks run after IMPLEMENT, so none of them can stop a run at CLARIFY. The first two now rest on a parameter — the task-id selector and scoredChecks. The third was scoped down instead: check.sh --plan proves only that the answers are recorded and non-empty, and the instructor's approval of the plan is named as the step that carries the rest.

The fourth was the scored checks being "reported to the human". work's Phase 5 is a human step, so no exit code can compel a human to read a score, and three review rounds said so before the claim came out. It too was scoped down rather than re-promised: the scores are retrievable at a named jq command over result.json, a reviewer who does not run it does not see them, and that is the accepted cost of their not gating.

So: when no parameter and no exit code can carry a property, SCOPE THE CLAIM DOWN to what the mechanism does prove, and name the human step that carries the remainder. An honest smaller claim beats a bigger one nothing enforces. Before writing "this workflow does X", name the parameter or the command that makes X true — and if there is none, say so plainly where the claim would have gone.

Red flags — STOP if you catch yourself

About toDo instead
Answer either CLARIFY question yourselfStop and ask, or report that the interview is unanswered. Nothing mechanical can catch a self-answer — the plan is written by the orchestrator, so only the instructor's approval distinguishes his answers from yours
Ask about a passage to keep, or about the insertion pointBoth were declined; insertion point and class number come from the request or the schedule. The page target IS asked — question 3
Type a --target on a real run, or invent a range to fit the excerptThe target lives in the plan, per reading. With --plan the flag is ignored and said to be ignored; --target is the --no-plan fixture override
Leave a ## Readings In Scope page target blank or TBDThe --plan leg FAILS naming the row. Ask question 3, or take the 2-6 default
Let a legibility or fidelity finding block the runThey are scoredChecks, which work's overallPass never reads — the deterministic gate is the entry point's exit code
Add legibility or fidelity to the lens prompt because it "reads like a lens"The lens gates by construction; the advisory property has to be a parameter, not a sentence. Keep them in scoredChecks
Omit lens or pass {} to turn review offboth select work's generic default. Keep the explicit criteria/scope checklist, with legibility and fidelity only in scoredChecks
Add a threshold or blockBelow to a scored checkNo such knob exists, and adding one chases minors. Read the score; gate on the entry point
Add a second mechanicalChecks entryThe verdict is one command; P10 refuses a list, and a list drops a check without reporting it
Write a workflow.js for this skillwork's spine takes a parameter for it. Name the missing parameter and generalize work
Give every reading row docs in its writable pathswork refuses a wave with overlapping writable paths before dispatch. retrieve owns docs/ in wave 1; the cut rows write one fragment each
Write "this workflow does X" with nothing reading XName the parameter or the exit code that makes it true, or record that none does — see The rule for whoever edits this next
Write that the scores "are reported to the human"Nothing reports them. They sit in result.json under scores[]; Phase 5 names the jq command that prints them, and a reviewer who does not run it does not see them
Re-cut every reading after one FAILRe-run onlyTasks on the flagged reading's task id — one task per reading is what makes that possible
Scope a re-run "by caption"work has no caption channel; consume tasksThatFlagged, mechanicalThatFailed, lensesThatFlagged and planFindings
Silently skip a reading that has no reporter sourceStop and report it. The instructor declares it in the plan's ## Non-court readings section; an undeclared one FAILS, which is the point
Write // elide-unchecked: into the .typ to quiet a readingThat marker is self-exempting and is no longer honoured — its presence FAILS. Non-court text is declared in the plan, by the instructor
Omit both --plan and --no-plan from the entry point's cmdThat is a FAIL naming the missing flag — the leg fails closed. Pass --plan <md> on a real run; --no-plan is the fixture escape and prints a loud NOT CHECKED line
Say the workflow "cannot arm" or "does not proceed" without the interviewNo mechanism carries that. Say what does: the answers are recorded in the plan, the instructor approves the plan before the run arms, and check.sh --plan refuses a plan whose section is missing or unfilled
Write an editors' note on every readingThe default is NO note (instructor ruling). One exists only for BACKGROUND — prior history, an earlier related decision, real-world context — or a disclosure specific to that one reading
Repeat the elision-mark and footnote conventions in each reading's noteThey are stated ONCE in the preamble and cover the whole document. That repetition is the boilerplate the reversal removed
Reword the court's text to close a widowThe fix vocabulary is LAYOUT-ONLY — spacing, or where the page breaks. Rewording defeats the verbatim gate; if the two conflict, fidelity wins and the widow stands
Pass --no-widows on a real runIt waives the page-break leg and prints a loud NOT CHECKED line. The leg is on by default; leave it on
Write a .westlaw.txt beside the docx, or cut from oneThe docx is the only stored source and its projection is in memory. A stored projection drops the docx's italic runs, so every case name reads in roman
Add emphasis the source does not carry, or set a source italic in romanThe excerpt inherits the court's typography. Invented emphasis and dropped emphasis are both misquotation, and neither is caught by matching words alone
Re-create a .westlaw.txt for a case whose docx is in docs/Both gates run green against the docx, so nothing needs one. A .txt naming survives only where the fallback route produced it, and // elide-source: X.westlaw.txt resolves to the .docx rather than failing
Read a CourtListener/RECAP .txt and assume its typographyThat route is a court PDF with no formatting layer — italics are unrecoverable there, not merely unchecked. The gate prints ITALICS NOT CHECKED for it; the route stays, because Westlaw does not carry every case
Hand-edit a table range so the checker passesFix the source and recompile

farm-out

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Delegation runs in a separate process on a CLIProxyAPI wrapper, not in this session. This session keeps its own auth, Remote Control, and connectors; the work runs on proxy models.

There is no default runner: each --tasks row is routed by its kind (see Provider routing below). Name a provider only when the task calls for a specific model family, or when cross-checking one family against another.

Supersedes the built-in tools

Instead ofUse
Agent tool / subagentsscripts/farm.sh (single or fan-out)
Workflow toolscripts/farm.sh --workflow <abs script> --args <file> --out <file>
in-session agent teamscripts/farm-team.sh
persistent / remote agentsthe agent-spawn skill (it reaches other machines; this skill does not)

A description cannot carry this on its own — three rewrites all plateaued near 40% recall, so the routing is enforced by ~/.claude/hooks/main-thread-guard.sh (PreToolUse on Agent|Task|Workflow|Edit|Write|NotebookEdit). It denies with a reason that names the runner, which a bare permissions.deny rule cannot do. The hook exempts FARM_OUT_CHILD=1 (our own runners' children, else farm-team.sh blocks itself) and named agent types like Explore/Plan/librarian. Add to its case to exempt another one.

Delegating is a choice the hook cannot make for you — the Agent|Workflow branch only redirects delegation you already chose. To make a project refuse main-thread implementation outright, set "farmOutOnly": true in its committed .claude-workflows.json; every Edit/Write there is then denied unless it targets that file, .claude/plans/, or .work/. Without it the Edit branch allows unconditionally whenever no work dispatch is owed.

Two rules that are not optional

1. A returned result is not evidence. A delegated run reports is_error: false and a confident summary whether or not it did the work. Observed twice: two runs quoted invented teammate output, with zero tool calls and zero filesystem trace. Always give the task a checkable artifact and pass --expect <path>; the runners exit non-zero when an expected artifact is missing, empty or older than the row (not written this run: <path>), or when the child ended with a background job still running (the row's failure names it: ended with background job running: <cmd>; a Stop hook in the child refuses that turn end first). Never relay a delegated summary you did not verify.

2. Every delegated prompt carries the anti-simulation clause. The runners append it automatically. Do not hand-roll a delegation that skips it.

One watcher per SESSION, not one per dispatch

Do NOT arm a Monitor or an until test -s loop to be woken. The watcher mod (hooks/watch/watcher.ts) reads every farm run, work round and grind loop this session launches from $TMPDIR/farm-events/<session>/ and wakes YOU — not the user — ONCE per run on DONE and on a run whose process is gone with no verdict (GONE), naming its report path. farm-status, run as a statusLine command, prints the running line from the same files (farm: 2 running (ds-rules 43m, …) · work <run> round 2/3 2/5 checks); set it with {"statusLine":{"type":"command","command":"farm-status","refreshInterval":5}}, or a composing script that calls it. /farm prints the table: label, state, elapsed, artifact, report. Its timer restarts with every session start and reload, which the retired farm-runs monitor never did — a finished run once woke nobody for hours.

Arm a task-specific wait ONLY when a run's finish must trigger a follow-up COMMAND — a wake is not your next step. Background that step: until test -s <expect>; do sleep 20; done; <next step>. Never with an unbracketed pgrep -f; the v6.26.3 guard denies a pattern matching its own checker.

A --workflow run prints an hourly CronCreate backstop — it survives --resume/--continue and a session that was not running (--no-cron opts out; --tasks prints none). Arm it the turn you see it. When you launch that run DETACHED (setsid nohup … > log) the printout lands in the log, not in front of you, so create the cron yourself at launch: 7 * * * *, recurring, non-durable, prompt and? (farm <run-dir name>).

Pick the shape first: sealed worker, or orchestrator

--agent <name> runs the delegation AS one of your agents — its real system prompt, its preloaded skills, its declared toolset. That last part is the whole decision, because every persona agent is sealed against DELEGATION: ds is Read/Grep/Glob/Edit/Write/Bash/Skill with no Agent and no Workflow, so it can do the work and load a domain skill, but it cannot fan out or dispatch a run.

Skill is deliberately NOT withheld. Withholding Agent/Workflow prevents recursive fan-out; withholding Skill only blinded the persona to its own domain library — ds gets the constraint aggregates it is graded against as task refs, while wrds, crsp-v2, dewey, bmll, marimo and ten more sat unreachable. teaching always carried Skill; the others not carrying it was drift, not policy.

JobShape
one well-scoped piece of worka one-row --tasks file with "agent": "ds" — the persona's prompt and its narrow toolset are the point
needs to plan, fan out, dispatch subagents, or run a work workflow end-to-endsame, but omit "agent" — a persona has Skill but no Agent/Workflow, so it cannot fan out or dispatch
several independent jobs at oncemore rows; each carries its own "agent"

Row count is orthogonal to the persona question: a fan-out of five can be five ds workers. "More than one job" does not mean "generic".

An orchestrating child is a full Claude Code session, so it has Agent and Workflow and can dispatch the persona itself — Agent(subagent_type: "ds") gives the subagent the same real persona plus the same deliberate restriction. That layering is the design the work skills already use (skills/ds/SKILL.md sets implementerAgentType: "ds"), not a workaround.

Agents live in ~/.claude/agents/: ds, writing, writing-econ, writing-legal, workshop, teaching (workers); ds-reviewer, workshop-reviewer, writing-reviewer (read-only).

A --workflow run takes no --agent — it picks agents PER LEG, which is the point: agent(prompt, {agentType: "ds"}) inside the script, or implementerAgentType / verifierAgentType / lens.agentType in a work args file. One top-level persona could only apply to every leg, when what you want is ds implementing and ds-reviewer or Explore judging. Note workflow.js strips the Agent tool from every leg regardless of agentType, so legs cannot nest further delegation; fan-out is the workflow's own parallel() / pipeline().

Use

There is one task mode, --tasks, and it always takes a JSON array — one row or fifty. There is no inline --task: the only caller is a model reading this file, so an inline prompt saved nobody anything, and a machine-written prompt passed as a shell argument has to survive quoting (backticks, nested quotes, $) that a JSON file sidesteps.

S=~/.claude/skills/workflows/skills/farm-out/scripts

# Build the task file with jq, never by hand-quoting a heredoc.
jq -n '[{prompt:"…", expect:"/repo/out.md", label:"count", kind:"script", agent:"ds"}]' > /tmp/t.json
bash $S/farm.sh --tasks /tmp/t.json --cwd /repo --detach

# Rows run in PARALLEL. Per row: prompt (required), workflow (required unless the persona
# implies it), kind (or provider/model), expect (string or array), label, agent.
# Omit "agent" when the row must orchestrate.
# schema (JSON Schema object, inline JSON or a path) -> --json-schema; the CLI-validated value is
# written to the row's ONE expect unless the child wrote it. A persona needs StructuredOutput in tools.
# sandbox: true -> the row's Bash runs in a throwaway bwrap overlay of the tree, its data/ targets and
# $HOME, no network (scripts/sandbox.sh); Edit/Write denied; the agent runs as <agent>-sandboxed.

# workflow script — --out is REQUIRED (the structured return is the result, not the
# summary), and paths resolve against OUR cwd, not --cwd, so pass them absolute.
bash $S/farm.sh --workflow /abs/wf.js --args /abs/args.json --out /abs/result.json --cwd /repo --provider claude

# Long runs: never foreground (a Bash-tool call caps out and kills the run mid-flight).
# Detach, then wait on the artifact:
setsid nohup bash $S/farm.sh --workflow /abs/wf.js --out /abs/result.json --cwd /repo --provider claude \
  > /abs/run.log 2>&1 < /dev/null &

# team: named teammates that message each other, one result back
$S/farm-team.sh --prompt-file t.txt --cwd /repo --expect /repo/a.txt --expect /repo/b.txt

In the main thread, run --tasks with --detach and never with run_in_background: it returns at once with the log path and the watcher's wake on the run's DONE is its one wake; with no watcher ticking it prints farm: --detach ignored and runs in the foreground.

--provider claude|codex|gemini is optional on --tasks (omit it and every row is routed), required on --workflow, and defaults to claude on farm-team.sh.

Budget defaults: FARM_TASK_BUDGET=4000000, FARM_RUN_BUDGET=20000000 (FARM_SESSION_BUDGET a deprecated alias), FARM_TASK_ESTIMATE=750000 per task (capped at its budget). FARM_MAX_TURNS=250 on farm and grind. The pre-launch check refuses a run whose own estimate reaches the run cap; the printed session spend is informational and never enforced.

Read reference.md before changing a runner, debugging a 429, or hand-writing a proxy call — it holds the verified model-routing and failure-mode details.

Provider routing

Every --tasks row carries a kind — script, judgement, review, bulk or vision. farm.sh hands each row to ${CLAUDE_PLUGIN_ROOT}/scripts/lib/route.ts, which picks the provider and a pinned model from the committed ${CLAUDE_PLUGIN_ROOT}/scripts/lib/routing.json: the kind's pick, else its first available fallback. A row may name provider/model itself instead. A row with neither a kind nor a provider/model is refused — exit 2, no row runs, and it never falls through to a default.

--provider is the legacy whole-run override: that wrapper for every row, route.ts never consulted.

Each row prints farm: ROW <label> <rowId> on stderr and appends one line to ~/.local/state/workflows/farm-outcomes.jsonl (FARM_OUTCOMES overrides). A row whose expect artifact is missing, whose child ended with no result (GONE), or that left a background job running (background-orphaned) is labelled wrong automatically. Once you have verified any other result, label it: farm.sh --verdict <rowId> correct|wrong "<why>". route.ts --outcomes prints rows, wrong rate and top failing checks per kind x model.

bun route.ts --refresh, run in scripts/lib/ of a checkout of this repo and never the plugin cache, updates availability (proxy catalog) and prices (OpenRouter) and never touches kinds. Review its routing.json diff before committing. route.ts --propose reads the refreshed signals and prints suggested kinds changes with reasons, plus newer same-family models the proxy serves that beat a candidate at <= its price (candidate C: model <old> -> <new>), writing nothing: you approve by editing kinds, or that candidate's model and openrouter then --refresh. Design: docs/DESIGN-routing.md.

Red flags

SituationWrong moveRight move
Per-document LLM coding/extraction over many filesUse --tasks fan-out to have agents read each documentUse batch-router, which picks the Claude, OpenAI or Gemini batch job
Writing style (fonts, hex colours, "X-style look") into a chart briefThe child follows the brief over its rules; farm.sh refuses the rowThe subagent's rules own style; the brief says what to show. Only the user's verbatim style words go in a row's userStyle
A row with no "workflow"farm.sh refuses the run at launch (exit 2) and nothing spends: a row's output is gated by its workflow's checks, so it must say whichSet "workflow": "<wf>" (any workflow bin/workflow-check <wf> --list resolves: ds, dev, writing, writing/legal, …) or "workflow": "none" out loud. A persona implies one: its name with / for - (writing-legal is writing/legal); a read-only persona implies none
Treating a farmed row as done on its own reportThe child can ship a join with no audit, a figure in the wrong face, a test that asserts only on a mock or a meta-commentary topic sentencefarm.sh runs bin/workflow-check <wf> after the row on the files it touched: every constraint and Jev rule under checks/<wf>/ and its parents'; findings or an unrunnable check mark the row wrong

Contents

Fuzzy Name Matching

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Fast many-to-many fuzzy entity matching: char n-gram TF-IDF + sparse top-k cosine similarity (the ING banks recipe). Scales to ~10⁵ × 10⁵ on a laptop in seconds, ~10⁶ × 10⁶ with chunking.

The full recipe — code, threshold guide, gotchas, alternatives considered — is in references/fuzzy-name-matching.md. Read it before writing match code; a runnable template is in examples/fuzzy_name_match_sample.py.

Match Enforcement

IRON LAW: NO NAME MATCHING WHILE A SHARED IDENTIFIER EXISTS

Before normalizing anything, ask what identifier both sides already carry. Filings from one regulatory regime usually share a filer key, and it is often sitting in a field nobody read — a path, an accession, an exhibit header.

Example (npx-reconcile): a full session built a name-based ISS→13F crosswalk reaching 59.45% of the target universe, while the registrant CIK already parsed from the filing path reached 99.11% with no name matching at all. Both filings were '40 Act filings from the same registrant. The linker was not badly built; it was the wrong instrument.

Ask first: are these the same filer under one regime? If yes, find the key. Fuzzy matching is for identifiers that genuinely do not cross.

IRON LAW: NO FUZZY MATCH WITHOUT NORMALIZATION FIRST

Fuzzy matching is the last step of a linkage, never the first:

  1. NORMALIZE both sides (uppercase, punctuation → space, strip entity suffixes/titles) — see the normalize() function in the reference
  2. JOIN exactly on the normalized name, scoped by a secondary key
  3. MEASURE the exact-join hit rate
  4. FUZZY-MATCH only the residual rows
  5. INSPECT a sample of accepted pairs at the chosen threshold
  6. CLAIM a hit rate only after inspecting matched pairs

Skipping straight to TF-IDF is NOT HELPFUL — every deterministic rule you skip upfront comes back as false positives the user has to find later, in a join they now believe is clean.

Fuzzy Matching Facts

  • If the normalized-name exact join yields <70% on ground truth, normalization is leaving money on the table — fix that before reaching for TF-IDF. Fuzzy matching a badly normalized field buys false positives, not coverage.
  • IDF weights are corpus-dependent: fitting a fresh vectorizer per scoped subset makes scores incomparable across subsets. "GABELLI MARIO JOSEPH" ↔ "GABELLI MARIO J" scores 0.69 in a 2-row toy fit and 0.84 in a 600K-corpus fit — same pair, different context. Fit once on pd.concat([left_all, right_all]), reuse .transform() everywhere.
  • Scoping is where the signal lives. On the 13,663-row blockholder bridge, the scoped fuzzy pass (≥0.80, per issuer_cik) converted 70% of unmatched rows; the global pass afterward (≥0.90) added 11. Scope by issuer/year/geo and the smaller candidate pool lets you lower the threshold safely.
  • TF-IDF on characters is not semantic. "IBM" ↔ "International Business Machines" will never match at any threshold — that needs a name dictionary or an embedding model.
  • sp_matmul_topn(A, B, ...) wants B already transposed (R.T). sort only orders the hits within each row — top-k selection always keeps the largest values, so top_n=1 returns the argmax with sort=False too (verified on 1.2.0: 60/60 rows matched a dense argmax either way). Pass sort=True when top_n>1 and you care about the order.
  • Duplicate normalized names on the right side with different ids make top_n=1 pick one of them with no defined rule — sort does not disambiguate an exact tie (tested on 1.2.0: it kept the last duplicate in every layout, under both sort settings, but nothing documents that). Whichever it is, the id you get is an accident of row order. Deduplicate the right side first.

Red Flags — STOP If About To:

  • Run sp_matmul_topn before an exact normalized join has been tried and measured → STOP. You cannot tell false positives from real coverage without the exact-join baseline.
  • Fit a TfidfVectorizer inside a per-key loop → STOP. Scores become incomparable across keys; fit once on the full corpus outside the loop.
  • Accept matches below 0.75, or below 0.90 on an unscoped global pass → STOP. The threshold guide auto-accepts 0.85–0.95 only when scoped; the global pass floor is ≥0.90.
  • Report a hit rate without eyeballing matched pairs near the threshold → STOP. Reporting an unverified hit rate is presenting an unverified claim as fact.
  • Reach for TF-IDF on <5K × 5K, or on strings under ~3 characters → STOP. RapidFuzz edit distance is better there; n-grams collapse on short strings.

When to Use

Reach for this when linking two datasets whose only shared field is an entity name — bridging identifiers (CIK ↔ permno ↔ gvkey ↔ wficn ↔ EIN ↔ TR personid), deduping filer/fund/insider names, or resolving vendor records that share no key at all.

Don't use it for exact joins (pandas merge), for semantic equivalence, or below ~5K × 5K rows where RapidFuzz is simpler. The reference's "Alternatives considered" table covers rapidfuzz, recordlinkage, dedupe, and edit-distance metrics and when each wins.

The Pipeline

normalize both sides
    ↓
exact scoped join (issuer/year/geo + norm_name)   ← most of your hits
    ↓
exact global join (unambiguous norm_names only)
    ↓
fuzzy SCOPED pass   — threshold ≈0.80, per-key candidate pools
    ↓
fuzzy GLOBAL pass   — threshold ≥0.90, residuals only, dedup right side first
    ↓
residual: synthetic ids / leave unmatched — never force a match

Code for each stage, and the threshold table that governs what to accept at each one, is in references/fuzzy-name-matching.md.

Packages

# pixi.toml
[dependencies]
scikit-learn = "*"
[pypi-dependencies]
sparse_dot_topn = ">=1.2"      # ING's fast sparse top-k matmul

conda-forge tops out at sparse_dot_topn 0.3.1, which predates the v1 sp_matmul_topn API this recipe uses (checked against the conda-forge channel). pixi add sparse_dot_topn therefore installs a version without the function — use pixi add --pypi sparse_dot_topn to get ≥1.2.

Additional Resources

Reference Files

  • references/fuzzy-name-matching.md — the full recipe: minimal code, threshold guide, normalize-first rule, scoped + global two-pass pattern, seven gotchas, alternatives considered, end-to-end results table

Example Files

  • examples/fuzzy_name_match_sample.py — runnable template: normalize(), fuzzy_match(), fuzzy_match_scoped(), plus a toy demo linking insider names to reporting-owner CIKs

Related

  • skills/wrds/examples/blockholders_pipeline/redo_bridge.py — production pipeline this recipe came out of (TR personid → SEC rptOwnerCik, 97.4% hit rate)
  • skills/wrds/references/linkage.md — try a real link table first; fuzzy matching is for identifiers that genuinely don't cross vendors

Gemini production batch

## IRON LAW: NEVER use AI Studio for production runs

NEVER USE AI STUDIO / THE GEMINI DEVELOPER API FOR PRODUCTION RUNS. Production batch runs use Gemini Enterprise Agent Platform (formerly Vertex AI): genai.Client(vertexai=True, project=..., location=...), Application Default Credentials (ADC), and GCS input/output. Substituting API-key/Files batch is not a shortcut: it sends the production workload to the wrong service.

  • Developer API: a 2,111-row grounded batch returned 1,307 code 9 "Precondition check failed" rows; its retry queue stalled over 40 minutes. Tier 1's gemini-3.8-flash queue cap was 3M tokens. Repeating that production route recreates the failure, not a cheaper solution. These are Developer limits, not Cloud quotas.

Product name; current SDK docs use enterprise=True / GOOGLE_GENAI_USE_ENTERPRISE=True; vertexai=True remains compatible. Check SDK spelling and precedence before changing pins. aiplatform.googleapis.com and IAM roles/aiplatform.* retain their identifiers. Developer examples retained in references are not for production.

Choose the Cloud tier

Route by row count: ≤10 documents/rows → Cloud Flex PayGo; more → Cloud Batch. This is a judgment default from Flex throughput (~9 rows/hour at peak, measured on Developer Flex). Flex gives the same 50% discount as one direct generate_content call per row: no GCS JSONL, no queue, no import-format failures. A bulk-guard redirect for a few documents lands on Flex, not a Batch job; recipe in Flex.

TierUsePrice / availabilityRequest path
Cloud Batch (Recommended for >10 rows)Independent extraction/classification and tested grounded lookups50% off real-time; shared capacity; up to 72h queued, then most jobs finish within 24h runningGCS JSONL → client.batches.create; config.dest → GCS
Cloud Standard PayGoSame-model synchronous smoke tests; interactive searchStandard Cloud tariff; capacity/model-dependentclient.models.generate_content with ADC
Cloud Flex PayGo (Preview) (Recommended for ≤10 rows)Few-document extraction; small, synchronous, latency-tolerant lookups50% off Standard; higher throttling; global only; timeout up to 30 minVertex header X-Vertex-AI-LLM-Shared-Request-Type: flex
Cloud Priority PayGoLatency-sensitive work, not cheap bulkHigher model-specific tariff; global and supported us/eu multi-regions, not regional endpointsSame Vertex header, value priority

Sources: Batch, Flex, Priority, Cloud pricing. Cloud Flex/Priority are documented equivalents, not the Developer Interactions service_tier recipe or its quotas; read tier request patterns.

Cloud Interactions is Preview; generateContent remains fully supported. Production batch uses GenerateContentRequest JSONL, not interactions.create or an Interactions background task.

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Before writing code

READ EXAMPLES BEFORE WRITING ANY CODE. NO EXCEPTIONS.

NO BULK SUBMISSION WITHOUT A SAME-MODEL CLOUD SMOKE TEST AND A 5–10-ROW CLOUD END-TO-END TEST. Skipping these scales bad prompts, lost identifiers and ungrounded answers into a bad dataset.

  1. Read Cloud batch, the ADC/GCS setup runbook, and examples/batch_processor.py or examples/icon_batch_vision.py. Verify project, API enablement, ADC, IAM and readable input/writable output GCS paths. gcloud user login alone is not ADC. Flex route (≤10 rows): read Flex instead of the GCS runbook.
  2. Fetch current Cloud model cards, locations, batch support, thinking and pricing. New-project extraction defaults: gemini-3.5-flash-lite for cheap extraction, gemini-3.8-flash for harder extraction/search. Preserve existing model pins; Flash/Flash-Lite is the measured extraction default, not Pro. See model-selection evidence.
  3. For Gemini 3.x, omit temperature, top_p and top_k. Use exact-model-supported thinking levels: 3.8 Flash supports low/medium/high, not minimal. See Cloud model guidance.
  4. Send relevant native PDF pages via GCS fileData.fileUri; preserve layout/scans. For text-native filings, pre-cut relevant sections without truncating target evidence. Historical PDF cost observations are in Files, not a universal tokens/page tariff.
  5. Run one synchronous Cloud request with the exact model/input/schema, then 5–10 rows through the actual Cloud batch/tier. Inspect content, per-row status errors, finish reasons, usage, identifier round-trip and grounding if required. Follow scale-up testing. A ≤10-row Flex job has no separate pilot: its first row is the smoke test.

Production request pattern

from google import genai

client = genai.Client(vertexai=True, project="your-project-id", location="global")
job = client.batches.create(
    model="publishers/google/models/gemini-3.5-flash-lite",
    src="gs://your-bucket/requests.jsonl",
    config={"display_name": "extraction", "dest": "gs://your-bucket/outputs/"},
)

Keep this client alive through submission, polling and retries. dest is a GCS prefix inside config, not a filename or top-level kwarg; after completion use the job's returned destination. Cloud input has request plus the examples' scalar correlation metadata, not Developer Files semantics. Validate with scripts/validate_jsonl.py --backend cloud; see request/output format and schema.

Cloud batch limits

LimitCurrent Cloud Gemini batch
Requests/job200,000
GCS JSONL inputOne file, up to 1 GB
Concurrent Gemini jobs / enqueued tokensNo predefined quota; dynamically shared model capacity (not Developer Tier 1 caps)
Queue expiryUp to 72h before starting
Running timeMost complete within 24h; incomplete jobs cancelled after 24h running, charged for completed requests
EndpointGlobal for base models; supported regional endpoints for residency; global does not satisfy residency
Unsupported batch featuresProvisioned Throughput, explicit caching, RAG; tuned Gemini 3+ models

Sources: batch limits, quotas. Embedding batch has separate Cloud limits/schema. BigQuery is a documented alternative, with regional constraints; GCS is this skill's production default.

Grounding and acceptance

  • Configure Google Search through the Cloud GenerateContentRequest tool shape and inspect candidates[].groundingMetadata per row. Cloud grounding and model support are not a promise that every batch+search+schema combination works; the exact Cloud end-to-end sample must prove it. Read measured Cloud gotchas before grounded submission: empty googleSearch fails import; M100 grounded 97/100 on 3.8 Flash/global. Cloud batch excludes RAG/File Search.
  • Measured on Developer API, not a Cloud availability claim: past-tense founder framing grounded 0/138; fresh framing (“Search the web NOW for current pages… report what they say TODAY”) grounded 10/10, 1.9 searches/row. Flex shed load with 503s at peak, about 9 rows/hour. Prefer fresh framing and verify the Cloud sample; do not route bulk to Flex merely because an old prompt failed. More evidence: gotchas.
  • Bulk (>10-row) per-document extraction is one Cloud Batch job over pre-cut inputs, never an interactive-agent fan-out. A 297-prospectus coder run consumed all three shared Claude accounts; it belonged in Batch.
  • Reconcile every input identifier, duplicate/missing output and row error; preserve IDs on retry. A succeeded job or valid JSON is not correctness evidence. Budget input/output and search separately; a prompt search cap is not an enforced quota. Before a full grounded run, measure about 100 Cloud rows and project query cost from usage, not the requested cap.
  • NO GROUNDED RUN OVER ~100 ROWS WITHOUT A USER-APPROVED COST PROJECTION: rows × pilot-measured queries/row × $14/1,000, minus the remaining free allowance; set a budget alert first. Searches were $390 of a $421 bill and every alert arrived after the spend. Formula, SKUs and the Billing → Reports URL: search cost gate.
  • Use the harness's background notification mechanism for long monitoring, not a model session repeatedly narrating status. Do not switch backend, model, schema or location to clear an error without retesting.

Red flags — STOP

About toDo instead
Use genai.Client() without vertexai=True for a production batchSTOP. Use explicit Cloud client, project/location and ADC
File API upload for a production jobSTOP. Use private GCS input and output
Pass dest= to create, or use a filename as outputSTOP. Put the GCS output prefix in config.dest; retrieve this job's actual destination
Use a bare model resource after a Cloud batch 404STOP. Use publishers/google/models/<id> and verify the exact endpoint
Infer batch availability from models.list() or downgrade to clear 404STOP. Check Cloud support and run the same-model Cloud batch sample
Put Batch requests into interactions.createSTOP. Batch uses GenerateContentRequest JSONL and client.batches.create
Switch platforms or abandon a batch approachSTOP. Cancel every job and runner of the old approach — list its batches, stop the poller/farm — and confirm none are pending. An unstopped AI Studio retry runner billed searches for ~8h after the move to Vertex
Accept a succeeded job without row reconciliationSTOP. Inspect status, content, finish reason, identifiers and grounding

Operations: GCS, CLI, troubleshooting, production patterns. Historical Developer File Search and Files/Interactions/embedding examples are not for production; never use them as production fallbacks.

Google Scholar CLI (scholar)

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Search Google Scholar for academic papers via the scholar command-line tool.

Requires: scholar on PATH (~/.local/bin/scholar → ~/projects/google-scholar-cli/scholar)

Check: command -v scholar || echo "MISSING: scholar CLI not installed"

IRON LAW: No Hallucinated Metadata

NEVER fabricate paper titles, authors, journals, years, or abstracts from memory.

When the user asks about a specific paper or needs citation details:

User mentions a paper
    ↓
Do you have the exact metadata from a scholar command in this session?
    ↓
YES → Use that data
NO  → Run scholar search/lookup --bibtex FIRST, then report

Metadata Facts

  • Training-data paper metadata is unreliable even for famous papers — wrong years, missing co-authors, garbled titles. Presenting remembered metadata is an unverified claim presented as fact; run scholar lookup "title" --bibtex first, then report.
  • --bibtex adds seconds, not minutes, and the BibTeX includes the abstract — reconstructing either from memory is counterproductive on its own terms.

Red Flags

  • About to type a paper title from memory → STOP. Run scholar lookup.
  • About to list authors without a source → STOP. Run --bibtex.
  • Saying "published in" without verification → STOP. Check the BibTeX.
  • Writing an abstract from memory → STOP. BibTeX includes the abstract.

Authentication

Before first use, authenticate by extracting cookies from an active Chrome session:

# Chrome must be running with remote debugging enabled
scholar auth --port 9222

Cookies are stored in ~/.google-scholar/cookies (mode 0600).

Core Commands

Scholar Labs Search (AI-Enhanced)

Natural language search using Google Scholar Labs API:

# One-shot search
scholar search "what are the key papers on attention mechanisms"

# JSON output for parsing
scholar search "corporate disclosure and information asymmetry" --json

# With BibTeX citations (includes abstracts)
scholar search "attention is all you need" --bibtex

# Download PDFs for results with full-text links
scholar search "transformer architectures" --download

# Interactive multi-turn mode (follow-up questions)
scholar search --interactive

Traditional Keyword Search

Standard Google Scholar full-text search:

# Keyword search
scholar lookup "machine learning transformers"

# Author search
scholar lookup "author:shleifer disclosure" --json

# With BibTeX
scholar lookup "asset pricing" --bibtex

# With PDF download
scholar lookup "deep learning" --download

Cite (BibTeX by Cluster ID)

Fetch BibTeX directly when you already have a cluster ID from search results:

# Single paper
scholar cite 5Gohgn6QFikJ

# Multiple papers, JSON output
scholar cite 5Gohgn6QFikJ 8409835334886051453 --json

Download (Single PDF)

Note: --download works for open-access PDFs (arXiv, NBER, etc.) but is unreliable for papers behind library link resolvers (institutional access). For paywalled papers, return the URL and let the user download manually.

scholar download "https://arxiv.org/pdf/1706.03762" --output attention.pdf

Quick Reference

NeedCommand
Natural language questionscholar search "question"
Keyword/author searchscholar lookup "keywords"
BibTeX citationsAdd --bibtex to search/lookup
BibTeX by cluster IDscholar cite <clusterId>
Download PDFsAdd --download to search/lookup
Download single PDFscholar download "url" --output file.pdf
JSON outputAdd --json to any command
Interactive follow-upsscholar search --interactive
Re-authenticatescholar auth

Decision Tree: Which Command?

Do you have a cluster ID already?
  YES → scholar cite <clusterId>
  NO  ↓
Do you have a natural language research question?
  YES → scholar search "question"
  NO  ↓
Do you need keyword-exact or author-specific results?
  YES → scholar lookup "author:name keyword"
  NO  ↓
Do you want follow-up refinement?
  YES → scholar search --interactive

When to add flags:

Need citation metadata (authors, journal, year, abstract)?
  YES → Add --bibtex
Need to download the PDF (open-access only)?
  YES → Add --download (unreliable for paywalled papers — return the URL instead)
Need machine-readable output?
  YES → Add --json

Verified Paper Information Workflow

When presenting paper information to the user, follow this workflow:

1. Run scholar search/lookup with --bibtex
2. Parse BibTeX fields for authoritative metadata:
   - title, author, journal/booktitle, year, abstract
3. Present ONLY the fields returned by BibTeX
4. If BibTeX is missing a field, say "not available" — do NOT fill from memory

BibTeX includes abstracts: The --bibtex flag injects the snippet as an abstract field in the BibTeX entry. Use this instead of generating abstracts.

Output Format

Table output (default): Columns for #, Title, Authors, Year, Cited, Journal, followed by snippets and URLs.

JSON output (--json): Array of ScholarResult objects:

{
  "title": "Paper Title",
  "authors": "Author A, Author B",
  "journal": "Journal Name",
  "year": "2024",
  "citations": 150,
  "snippet": "Abstract excerpt...",
  "url": "https://...",
  "pdfUrl": "https://... or null",
  "clusterId": "12345",
  "position": 1
}

BibTeX output (--bibtex): Standard BibTeX entries with abstract field:

@article{key,
  title={Paper Title},
  author={Author, A and Author, B},
  journal={Journal Name},
  year={2024},
  abstract={Abstract text from Google Scholar snippet...}
}

Journal filtering

scholar lookup --journal "<exact name>" restricts results server-side (repeat the flag for more than one; they OR together). It is the only way to reach the student-edited business law reviews — Chicago BLR, Penn JBL, Harvard BLR, Columbia BLR, NYU JLB, Virginia L&B, Berkeley BLJ, Journal of Corporation Law, Delaware JCL — none of which Consensus indexes.

scholar lookup "shareholder voting" --journal "Virginia Law & Business Review"

Division of labor with consensus: consensus filters peer-reviewed venues via --journals-file and covers the T14 general-interest flagships; scholar covers the business specialties via --journal. Same trusted-journal list feeds both.

--journal is lookup-only. scholar search (Labs) rewrites the query and silently ignores the filter, so it errors out instead. Journal names go inside q, which Scholar truncates past ~256 chars — a few journals per search, not the whole list.

Trusted Journals (shared resource)

When searching Google Scholar, ALWAYS consult the shared trusted-journal list first:

File: ${CLAUDE_PLUGIN_ROOT}/references/trusted-journals.local.md

Shared with the consensus and research skills and the librarian agent — it lives at the plugin root, not under this skill. It is the user's curated list of trusted journals: one exact journal name per line, # comments. That format is load-bearing (consensus search --journals-file reads the same file), so every non-comment line must be a journal name — do not add authors or notes as bare lines. Use it to:

  1. Prioritize results from known-good journals and authors
  2. Flag unfamiliar sources - if a result is from an unknown journal, note it
  3. Suggest related searches - use known authors to refine queries
  4. Assess quality - weight results higher when they appear in trusted venues

How to Use Domain Knowledge

User asks: "find papers on corporate disclosure"
    ↓
1. Read `${CLAUDE_PLUGIN_ROOT}/references/trusted-journals.local.md`
2. Run scholar search/lookup
3. Cross-reference results against trusted journals/authors
4. Present results with quality signals:
   - ★ = from trusted journal or by trusted author
   - Results from unknown sources shown without star

Operational Rules

  1. No hallucinated metadata — NEVER cite title/author/journal/year/abstract from memory. Use --bibtex or scholar cite to get verified data.
  2. Scholar is for discovery — Use it to find new papers, not to read them
  3. Always use --json when results will be processed programmatically
  4. Use --bibtex when presenting papers — It provides verified author, journal, year, and abstract fields
  5. Cross-reference the trusted-journal list — Always check results against ${CLAUDE_PLUGIN_ROOT}/references/trusted-journals.local.md
  6. Auth required — If search fails with auth errors, re-run scholar auth
  7. Rate limits — Google Scholar may rate-limit; space out rapid queries

grind — a loop with no session, whose only memory is one append-only journal

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

A bash while loop that calls a fresh model process per iteration and keeps nothing between them. Cost inside a chat session is wakes × context, and the context only grows: measured on this project, one orchestrator spent 1.05 billion cache-read tokens in a day across 1,816 turns on a ~575K-token context, and 208 of those wakes were status reports that needed no judgement at all. Here nothing accumulates between iterations because nothing survives them.

The loop is scripts/grind.sh. Nothing session-scoped can be load-bearing in it — no CronCreate, no Stop hook, no Monitor, no session id — so the journal is the whole interface. A human, or a chat session peeking in, tails it and has no other handle on the run.

Three rules, and each one is a defect this loop already hit

1. The supervisor is a shell command, so no model call is spent while it is red. --gate is a cheap precondition — a queue check, a file that appears when a round lands. While it exits non-zero the loop appends a wait, sleeps --sleep, and never invokes the runner. A verdict that takes ten hours otherwise buys ten hours of model calls whose only finding is that the job is still queued. Wake on decisions, not on events. Each wait record carries the gate's last output line as why, so a gate should print the reason it is shut rather than just exiting non-zero. A streak of consecutive waits is announced through the notification channel while the run continues (--wait-alert N, default 6, 0 disables), because a gate stuck closed looks identical to legitimate waiting from outside.

2. The keep decision is --check's exit code, never the agent's opinion. An iteration reports success whether or not it did anything, and no downstream check catches a confidently wrong answer. The agent's only write channel is append, and no record it can write makes the loop declare success.

3. The floor list is read from the journal at the top of every pass, before the prompt is built. A fresh agent handed the same dead pool re-diagnoses it every iteration, for a week. A key filed as a floor is injected into every later prompt as an exclusion, which is the only reason an amnesiac loop converges at all.

## IRON LAW: NO STOP THE LOOP DID NOT COMPUTE

done is written only after --check exited 0, in the loop's own process.

A model asked whether the goal is met answers from the pass it just ran, which is the one view that cannot see the goal. Every other terminal record — stalled, budget, stopped — is likewise a count or an exit code, never a judgement. Pick a check that names the state the WORK reaches: a suite passing, a rate under a number, a count at zero. test -f report.md is true while the objective is unmet, and the loop exits on pass 1.

Start it detached

setsid nohup bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh run \
  --journal /abs/run.jsonl \
  --prompt-file /abs/PROMPT.md \
  --check 'bash /abs/measure.sh --lost-below 120000' \
  --gate  'test -f /abs/out/round-ready' \
  --sleep 300 --stall-after 5 --max-iters 200 \
  >/abs/grind.log 2>&1 </dev/null &

Never foreground it: a Bash tool call caps out and takes the run with it, and a session held open to watch is the cost this skill exists to remove. --runner defaults to claude-code and --model is passed through to it; a test or a dry run points --runner at a stub instead. Every ending — done, stalled, budget, stopped — is announced by default through the launching session's watcher (the farm-events stream); agent-msg goes only to a --notify-to other session, or when there is no stream or no live watcher there. A herdr popup too, where herdr is installed. --notify CMD replaces that and gets GRIND_STATE, GRIND_EXIT and GRIND_JOURNAL; --notify none silences it. A failed notification never changes the exit code. A loop killed hard reaches no ending, so it also files itself in the launching session's farm-events stream, whose monitor reports a pid gone with no DONE (--no-events opts out).

If this session will ACT on the result (restart the loop, read the floors, ship the fix), CronCreate an hourly heartbeat in the same turn as the launch — cron 7 * * * *. The prompt names the session's standing objective or journal directory, never one batch, so it never goes stale; each tick runs grind.sh status there. The farm-events monitor is the primary wake; the cron is the backstop that survives a --resume, which monitors and background tasks do not. NEVER delete the heartbeat when a loop ends (done, stalled, budget, stopped): the session's mandate (an overnight or autonomous instruction, or queued next items) outlives any single loop, and a deleted heartbeat leaves the session with no wake. If the prompt must change, create the new one first. Deleting asks the user (permissions.ask), so a delete happens only when the user ends the mandate.

Then, from anywhere and at any time:

J=/abs/run.jsonl
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh status --journal "$J"      # state, pid, iterations, floors, stall
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh tail   --journal "$J" -n 40
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh floors --journal "$J"
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh stop   --journal "$J" --why 'grid down for maintenance'

stop is a record, not a signal: the loop honours it at its next boundary, after the pass in flight finishes. Once the loop answers it with stopped, the same run command resumes the journal, floors and all. Exit codes are 0 done, 2 refused, 3 stalled, 4 budget exhausted, 5 stopped. The per-iteration watchdog defaults to FARM_MAX_TURNS=250.

That subcommand is refused from inside an iteration, and the refusal names its reason — the loop marks the environment it invokes the runner in, and an operator's own shell carries no such mark, so an iteration that reads this file and concludes the work looks impossible cannot end a run the operator started. The guard closes the documented path and claims nothing beyond it: an iteration runs bash and can still kill the pid recorded in start, so this prevents a good-faith mistake rather than guaranteeing the loop cannot be stopped from inside.

Against the work hold — the difference is whether anything stays alive

the work holdgrind
what stays alivea chat session, held open by a Stop hook and re-woken by a cron heartbeatnothing — a bash loop outside every session
the memorythat session's context, which grows with every turnone journal; each iteration is a fresh process that starts empty
what decides to continuethe hook, running the check when the session tries to stopthe loop, running --check at the top of every pass
what it costswakes × a context that only growsone model call per decision, none while the gate is red
reach for it whenthe work needs that session's own tools, connectors or approvalsthe work is a command plus a prompt, and it will take days

The journal is the only file

There is no pidfile, no state file, no lock and no cache. Every fact is derived by reading the journal — the iteration counter is the largest i, the floor set is every floor record, the pid comes from start. Two files that can disagree about one fact are a bug generator.

kindwritten bymeans
startthe loopa run began: pid, check, gate, runner, model, prompt file
iter / iter_endthe loopiteration N began; the runner exited with this code
waitthe loopthe gate was red, so the pass cost nothing
progressthe agentthe goal moved; this is what --stall-after counts back from
floorthe agentthis key is dead, and every later prompt is handed it until a reopen names its key
reopenthe agent or operatorwith subject: an exhausted subject is worth another pass; with key: that floor is retracted (a later floor on the key closes it again). Always with a non-empty rerunReason saying what changed
donethe loop--check exited 0 — the only record that means success
stalled / budgetthe loop--stall-after passes in a row with no progress (default 3, 0 disables; an attempt or note is not progress); --max-iters spent
stop / stoppedthe operator / the loopa stop was requested; the loop honoured it

That split is enforced, not merely described. The agent may append only progress, floor, attempt, note and reopen: append refuses every loop-owned kind with exit 2, so nothing an iteration writes can end a run or report a finish the loop did not compute. A floor is refused unless it carries a non-empty key, because one accepted without a key is dropped by the reader and the loop re-diagnoses that family forever.

Any record may also carry a subject — the family the work is about, a CIK or a shard rather than a free-text key — and that is what groups one iteration's attempt with the earlier ones. Every prompt carries a GRIND_SUBJECTS block listing each subject's attempts since its last reset, its lifetime progress count and an excerpt of its newest attempt, so a fresh iteration can see what has already been tried instead of re-diagnosing it under a new key. A subject with --exhaust-after attempts (default 3, 0 disables) and no progress since is marked EXHAUSTED and is excluded from the iteration's choices the way a floor excludes a key. The only way back is a {"kind":"reopen","subject":"...","rerunReason":"what changed"} record, which resets that subject's count and is refused without both a subject and a non-empty rerunReason; a progress record on the subject resets it too. Recording is never refused for exhaustion — an attempt on an exhausted subject still lands, because the journal must not lie about the work that was done.

A floor is lifted the same way, by key: {"kind":"reopen","key":"<floor key>","rerunReason":"..."} drops that floor from floors, status and GRIND_FLOORS, ordered by journal position; a floor filed again afterwards is live again. A note cannot lift a floor — an operator withdrawing one appends this reopen, or the passes keep honouring it.

Every prompt carries GRIND_JOURNAL, GRIND_SH, GRIND_ITER, GRIND_FLOORS, GRIND_SUBJECTS and GRIND_NOTES, so an iteration needs nothing from outside itself:

bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh append --journal "$GRIND_JOURNAL" \
  '{"kind":"progress","key":"2004-noseries","note":"heading index resolved 41K rows"}'
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh append --journal "$GRIND_JOURNAL" \
  '{"kind":"floor","key":"1934-paper-scans","why":"no machine-readable text in the source"}'

Iterations run with FARM_OUT_CHILD=1, so an iteration writes and commits directly; farm out only work the prompt itself asks to delegate.

An operator steers a running loop with the same channel: append {"kind":"note","key":"...", "note":"..."} and the newest ten notes are printed as GRIND_NOTES in every later prompt, newest last, outranking the prompt's own ranking. No iteration opens the journal file, so a note that is not injected is a note nobody reads.

A record must be one JSON object on one line and must fit in a single atomic write, or it is refused — never split, never truncated — because a split record leaves a log nobody can replay. Readers skip lines that do not parse, which is what makes a journal cut short by a crash resumable rather than fatal. Anything larger than a key and a note belongs in a file the record names.

Reading it by hand is the same one pass:

grep -c '"kind":"iter"' "$J"                                  # iterations attempted
grep '"kind":"floor"' "$J" | jq -r '"\(.key)\t\(.why)"'       # the set every prompt is handed
grep '"kind":"iter_end"' "$J" | jq -r 'select(.exit != 0) | .i'

Red flags — STOP

About toWhy wrongDo instead
Let an iteration's own report decide the run is finishedthe loop would be grading itself, and one confident wrong answer ends a week of work with the goal unmetonly --check exiting 0 writes done; the agent writes records, never verdicts
Point --check at a milestone — a file existing, a report written, an agent having reported backit is true while the objective is unmet, so the loop stops on pass 1name the state the work reaches: a suite passing, a count under a number
Write a --check that passes once the inputs are fetched or builtthe loop records done with the deliverable missing — 209 exhibit PDFs fetched and none read, 1,404 order PDFs fetched and 0 outcomes coded (board-structuring, twice)fail the check until the final deliverable exists and is non-empty: coded rows > 0, or every fetched file extracted; print the count so done shows it
Run without --gate when the verdict takes hoursevery pass spends a model call to learn the job is still queued — that is the whole billa shell precondition; while it is red the loop records wait and sleeps for free
Hand an iteration a key an earlier one closedit re-diagnoses the same dead family every pass, and the loop never convergesread GRIND_FLOORS before choosing work; file a floor the first time a key dies
Append a whole report as one recordover the single-write bound it is refused, and splitting it would leave a log nobody can replaya key and a one-line note; write the report to a file and name its path
Add a pidfile, a progress file or a notes file beside the journaltwo files that can disagree about one fact, and the tiebreak rule is the bugappend a record; derive the pid, the counter and the floors from the journal
Arm a hold in a session while a grind loop works the same objectivethe hold blocks that session's stop while the grind's gate waits for no round in flight: each waits for the other and neither movesone driver per objective — grind owns a long loop, and no session holds alongside it
Foreground the loop from a chat sessionthe Bash tool call caps out and kills the run mid-flight, and the live session is the cost grind removessetsid nohup … &, then grind.sh status when you want to know
kill -9 the loop to end itthe journal then ends on iter, and nothing can tell a kill from a crashgrind.sh stop, honoured at the next boundary
About to write a pass prompt that says "do one chunk, commit, exit"each pass re-reads the investigation doc and the journal, so an hour of analysis became 10+ passes (docket-outcomes reached pass 59, ~20M weighted tokens per 14 passes)tell the pass to finish the deliverable when it can, committing at checkpoints rather than exiting at them
About to ask a pass for regression tests, "contracts" or edge-case hardening beyond what --check readspasses polished instead of finishing, and operator "estimate now" notes took 2-3 passes to land (s220-deterrence)the pass prompt names the deliverable and the check, nothing else
Read iter_end with exit 0 as progressit says the process ran, not that anything movedprogress records, which are what --stall-after counts
Launch a loop you will act on without a heartbeata --resume restores no monitor and no background task, so a resumed session never hears it endCronCreate an hourly tick at launch, naming the standing objective; keep it past the ending
About to delete the heartbeat cron because a loop/batch endedthe mandate outlives the batch; a loop that lost its only wake this way sat idle for 3 h until the user found itleave it; the user ends the mandate, and the delete asks them
Reach for the work hold to keep this goingthe hold keeps a SESSION alive, which is the thing this loop exists to avoidthe hold while a session must live; grind when none should

Headline Cards for Typst Presentations

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Typst rules in force here — read the one that governs what you are writing; they are independent: !k=slides; command -v typst-rules >/dev/null 2>&1 && exec typst-rules "$k"; echo "(typst corpus unavailable: NO Typst rule is listed here — install the typst plugin, or start a new session so its bin/ reaches PATH)"

An editorial recreation of a newspaper clipping: light paper stock, the real masthead logo, a top rule carrying a category kicker and dateline, a serif headline with a Last-Week-Tonight yellow highlighter swipe over one phrase, and a short standfirst.

Read templates/theme.typ before writing a card. The implementation is the authority; if it disagrees with this file, believe it and fix this file.

Architecture

presentation/
├── templates/theme.typ        ← headline-card() lives here; read it first
├── data/headlines-NN.json     ← card data, NN = the lecture number
├── assets/logos/*.svg         ← DARK / full-colour logos (see below)
└── slides/XX-topic/NN.typ     ← loops over the JSON, one slide per card

Step 1: the JSON entry

{
  "venue": "WSJ",
  "date": "June 24, 2026",
  "kicker": "Tech",
  "headline": "The headline exactly as published",
  "highlight": "phrase to swipe in yellow",
  "quote": "The article's own standfirst/dek.",
  "logo": "../assets/logos/wsj.svg"
}

highlight must be a substring of headline, matching character for character — the function splits on it, so a straight quote where the headline has a curly one silently produces no swipe. kicker and quote are optional.

Quote the article, do not summarise it. The standfirst should be the publication's own dek or a vivid line from the piece. "Discusses the impact of proxy advisors" is not a card; "Anyone who gives them money — shame on you" is.

Nothing on the card may be composed. Headline, dek, outlet and date are the article's own or they do not go on the slide — a card asserts that a publication printed this. Pull them from the record, and if a field cannot be sourced, omit it rather than inventing it. Note that Reader's reader-get-document-details returns null for published_date/source_url while reader-list-documents --id <id> returns them populated — check both before concluding a date is unavailable.

Step 2: paper stock is per publication

headline-card picks the stock from venue. FT is famously salmon and the broadsheets are white; one generic cream for everything reads as wrong to anyone who knows the paper.

venuestock
Financial Times, FT#FFF1E5 salmon
WSJ, NY Times, Bloomberg, ABC News#FFFFFF
X, Twitter#FFFFFF — light-mode surface; the mark is black
Bluesky, Mastodon, Threads, LinkedIn#FFFFFF — same embed layout; accents #01A5FF / #563ACC / black / #0A66C2
anything else#F7F4EC cream fallback

Microblog cards are embeds, not clippings

A post takes the same function and the same stock table, but an embed layout, because the masthead layout is what made it read as a clipping of a tweet:

{
  "venue": "X",
  "name": "Matt Levine",
  "handle": "@mattlevine",
  "avatar": "../assets/avatars/mattlevine.jpg",
  "date": "3:42 PM · Aug 22, 2026",
  "headline": "The post text, verbatim."
}
slotembednewspaper
marktop rightleft, as the masthead
identityavatar + name over @handle, leftvenue wordmark
datebelow the text, over a hairlinetop right, with the kicker
textsans regular 21ptserif bold 28pt
border#cfd9de (X's own)#ded7c7, radius 0.3em

name and avatar are optional — with no name the handle carries the slot alone rather than leaving a bold gap, and with no avatar the row simply starts at the name. Secondary text is #536471, X's own. Supply logo to use a real X mark; otherwise a bold sans X stands in.

Bluesky and Mastodon take the same layout. All three microblog embeds have one shape — avatar + name over handle at the left, mark top right, timestamp below the text — so this is a venue LIST (X, Twitter, Bluesky, Mastodon, Threads, LinkedIn), not a branch per platform. Only the mark and its accent differ, and the handle format is data rather than code: @user (X, Threads), @user.bsky.social, @user@instance.social. Every microblog ships its real mark inline. post-mark-svg in theme.typ carries the official path for X, Bluesky, Mastodon, Threads and LinkedIn, each filled with the platform's own colour, so no deck has to ship a logo asset. The non-X paths are the simple-icons set (MIT); X's is the official 300×271 mark. A caller-supplied logo still wins over all of them. The letter fallbacks (b., m, @, in) now only fire for a venue with no path at all.

X/Twitter is a venue, not a second function. The card suppresses the newspaper-only chrome for it — top rule, kicker, serif headline face, highlighter swipe — and takes handle: ("@elonmusk") beside the mark. Everything else, the stock table included, is shared.

Add a venue to the stocks dictionary in theme.typ rather than passing a one-off, so the next card for that outlet inherits it. stock: rgb("…") overrides per card when one genuinely differs.

Legibility does not constrain this choice — ink, dek and dateline land within ~1:1 of each other across every stock — so it is purely identity.

Step 3: the logo

Dark or full-colour, on a light card. Not the -white.svg variant; those are for dark backgrounds and are invisible here. Use wsj.svg, nyt-dark.svg, ft-dark.svg, abc-news-dark.svg.

IRON LAW: real vector logos only

If you catch yourself writing an SVG with <text> elements spelling out the publication name — stop. That is a placeholder, not a logo, and it ships with the wrong font, weight and spacing.

grep '<text' assets/logos/publication.svg && echo "FAKE — download a real logo"

Download from Wikimedia Commons, the brand's press page, or a logo repo. Wikimedia has SVGs for essentially every major publication; "I can't find one" means search harder.

Sizing needs no per-logo tuning: the function normalises by aspect ratio, so a narrow mark and a wide wordmark carry similar visual weight.

Step 4: render

#{
  let cards = json("../../data/headlines-NN.json")
  for card in cards {
    slide[
      #headline-card(
        venue: card.venue,
        date: card.date,
        headline: card.headline,
        quote: card.at("quote", default: none),
        logo: card.at("logo", default: none),
        phrase: card.at("highlight", default: none),
        kicker: card.at("kicker", default: none),
        stock: card.at("stock", default: auto),
      )
    ]
  }
}

One card per slide, and no === heading on it. Grid layouts truncate quotes and shrink logos below recognition; a heading eats the vertical space the card fills. Put any section header on the preceding slide.

Step 5: look at it

Compile, get the page from find-slide-page, render it, and read it. What matters: the logo renders and is dark enough; the swipe covers the intended phrase; nothing is clipped; no single orphaned word ends the headline or standfirst.

Fix widows in the JSON text, never in Typst — tighten the wording, or join the last two words with  . Never pad with filler to fix a widow.

Red flags — STOP

SymptomCauseFix
Logo invisiblea -white.svg on the light carduse the dark/full-colour variant
Logo missing entirelyJSON path taken as relative to slides/paths resolve from theme.typ: ../assets/logos/…
No yellow swipehighlight is not an exact substring of headlinematch curly vs straight quotes character for character
Card doesn't read as an object on white stockborder derived from the stock is near-whitethe border is a fixed grey; do not re-derive it from newsprint
An FT card looks like every other cardvenue missing from the stocks dictionaryadd it
An X post wears a serif headline, a top rule or a yellow swipevenue not spelled X/Twitter, so it took the newspaper branchfix the venue string — never add a second card function
Dek is genericsummarised instead of quoteduse the publication's own standfirst
A date is inventedit wasn't in the sourceomit the field; never fabricate one
<text> in the logo SVGplaceholder, not a logodownload the real one

Facts

  • The dateline is the smallest text on the card, so it carries its own darker grey rather than the dek's mute, which lands ~5.3:1 on newsprint and washes out under projector gamma.
  • Black on the yellow swipe measures ~13.7:1 — the highlight helps readability rather than hurting it, so spend it on the phrase that carries the point, never decoratively.

What this skill carries — grep references/ for any subject the names below miss: !d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"

Contents

When to Use What

Three compute environments, each with a clear role:

EnvironmentUse ForExamples
Local / RJDSExploration, prototyping, notebooksEDA, quick plots, marimo/Jupyter, test on small samples, iterate on code
WRDS (SGE)Data access, SAS ETL, file parsingSAS jobs against WRDS libraries, SEC filing parsers on /wrds/sec/, scan_covers, ad-hoc SQL
UVA HPC (Slurm)Scale computeModel estimation (PIN), large polars pipelines, anything needing >10 cores or >1 hour

The Workflow

1. EXPLORE (local/RJDS)     →  Prototype code, test on 5-10 items
2. BUILD DATA (WRDS)        →  SAS ETL or PostgreSQL queries (data lives there)
3. ESTIMATE AT SCALE (HPC)  →  sbatch when you need 100+ cores
4. ANALYZE RESULTS (local)  →  Pull results back, notebooks, regressions, tables

Decision Rules

  • Does it need WRDS filesystem access? (/wrds/sec/, SAS libraries) → WRDS
  • Is it CPU-intensive and embarrassingly parallel? → HPC
  • Is it exploratory or iterative? → Local / RJDS
  • Is it a quick SQL query? → Either WRDS or HPC (both have PostgreSQL access)

HPC Interactive Partition

The interactive partition (42 nodes, 12h max) is for testing sbatch scripts on one chunk before submitting 176 tasks, not for replacing local dev work:

salloc -p interactive --cpus-per-task=4 --mem=16G --time=1:00:00
# test your script, then exit and sbatch the real job

Why This Split Matters

PIN estimation proved it: WRDS SGE has 10 concurrent slots and took 8+ hours without starting OWR. UVA HPC ran 70+ OWR tasks simultaneously and finished in 30 minutes. But WRDS is still the right place to build the data — the SAS libraries and SEC filings live there.

Login Node Enforcement

IRON LAW: NEVER RUN COMPUTE ON THE LOGIN NODE

The login node is shared infrastructure. Running estimation, bulk processing, or any CPU-intensive work directly via SSH will get the account flagged and the process killed.

ALWAYS write a Slurm submission script and submit via sbatch. No exceptions.

  • ssh uva-hpc 'python3 est.py owr 2020' → WRONG. Use sbatch.
  • ssh uva-hpc 'nohup ./process &' → WRONG. Still the login node. Use sbatch.
  • ssh uva-hpc 'for year in 2003..2024; do python3 ...; done' → WRONG. Use sbatch --array.
  • sbatch run_est.sh owr → CORRECT.

The login node is for: sbatch, squeue, scancel, sinfo, scp, ls, head, short queries.

Login Node Facts

  • Tests go through the scheduler too: write the sbatch script first and test with --array=1-1. The login-node "quick test" is the run that flags the account — one stock becomes 5,000 when the args change, and you don't know it "only takes 30 seconds" until it runs.

Red Flags — STOP If You're About To

  • Write ssh uva-hpc 'python3 ... > output' → STOP. Write a submit script.
  • Write ssh uva-hpc 'nohup ... &' → STOP. Use sbatch.
  • Run a loop over years/permnos interactively → STOP. Use --array.

Cluster Reference

UVA HPC (Rivanna/Afton)

  • SSH: ssh uva-hpc (configured with ProxyJump through Mac via tailnet)
  • User: vwh7mb
  • Home: /home/vwh7mb (GPFS, 12PB shared, no per-user quota displayed)
  • Scratch: /scratch/vwh7mb/ (Weka, 12TB)
  • Allocation: 10M SUs (service units ≈ weighted CPU-core-hours)

Partitions

PartitionNodesCPUs/NodeRAM/NodeMaxTimeMinNodesMaxNodesUse For
standard30140+384GB+7 days01Single-node jobs, array tasks
parallel17996768GB3 days264Multi-node MPI jobs only
gpu4436+257GB+3 days——GPU workloads
interactive4232+128GB+12 hrs——Interactive/debugging

CRITICAL: Partition Selection

**Use `standard` for embarrassingly parallel array jobs** (PIN estimation, file processing, per-year/per-stock tasks).

The parallel partition requires MinNodes=2 — it will reject single-node jobs with "Node count specification invalid". It is designed for MPI jobs that span multiple nodes.

Wrong: #SBATCH --partition=parallel for array jobs → submission fails Right: #SBATCH --partition=standard for array jobs → 301 nodes available

When to Use Each Partition

standard (default choice for most research computing):

  • Embarrassingly parallel work: array jobs where each task is independent (PIN estimation, file parsing, per-stock/per-year processing)
  • Single-node Python/R with ProcessPoolExecutor, multiprocessing, mclapply
  • Any job where tasks don't communicate with each other
  • MaxNodes=1, so each array element runs on exactly one node

parallel (multi-node distributed computing):

  • MPI jobs where processes on different nodes exchange messages (mpi4py, OpenMPI, MVAPICH)
  • Dask distributed or Ray clusters spanning multiple nodes
  • Large linear algebra / matrix factorizations that exceed single-node RAM (ScaLAPACK, PETSc)
  • Simulations with inter-process communication (CFD, molecular dynamics)
  • Key requirement: your code must explicitly coordinate across nodes (MPI, Dask scheduler, etc.) — ProcessPoolExecutor and multiprocessing are single-node only
  • MinNodes=2, 96 CPUs and 768GB RAM per node — use when one node isn't enough

gpu (GPU-accelerated workloads):

  • Deep learning training/inference (PyTorch, TensorFlow, JAX)
  • GPU-accelerated linear algebra (CuPy, RAPIDS)
  • LLM inference or fine-tuning

interactive (debugging and development):

  • Testing job scripts before full submission: salloc -p interactive --cpus-per-task=4 --mem=16G --time=1:00:00
  • Debugging segfaults or data loading issues
  • 12-hour max — not for production runs

Python/R Environment

  • pixi: Install to $HOME/.pixi/bin/pixi via curl -fsSL https://pixi.sh/install.sh | bash
  • Project envs: $HOME/projects/<name>/.pixi/envs/default/bin/python
  • Modules (alternative): module load python — but pixi preferred for reproducibility
  • NEVER install Jupyter kernels globally on HPC

Slurm Job Submission

Basic Submit Script

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --partition=standard
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=3:00:00
#SBATCH --output=logs/job-%A_%a.log

mkdir -p logs

export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

PYTHON=$HOME/projects/my-project/.pixi/envs/default/bin/python
$PYTHON -u my_script.py --workers ${SLURM_CPUS_PER_TASK:-8}

Submission

sbatch script.sh              # submit
sbatch script.sh arg1 arg2    # args passed to script as $1, $2

Note: unlike SGE's qsub run.sh <model>, Slurm passes arguments after the script name directly. Use ${1:?Usage: sbatch script.sh <arg>} to enforce.

Array Jobs

Pattern

#SBATCH --array=1-176           # tasks 1 through 176
#SBATCH --array=1-176%50        # max 50 concurrent tasks
#SBATCH --array=1,5,9,13        # specific tasks only

Year × Chunk Sharding (PIN estimation pattern)

#SBATCH --array=1-176
# 22 years × 8 chunks = 176 tasks
# Decode: year = START_YEAR + (id-1)/NCHUNKS, chunk = (id-1)%NCHUNKS

NCHUNKS=8
START_YEAR=2003

idx=$((SLURM_ARRAY_TASK_ID - 1))
year=$((START_YEAR + idx / NCHUNKS))
chunk=$((idx % NCHUNKS))

Task List Sharding (file processing pattern)

# Equivalent to SGE's sed -n "${SGE_TASK_ID}p" pattern
ITEM=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$TASK_LIST")

Re-running Failed Tasks

# Re-run specific tasks
sbatch --array=5,12,87 script.sh

# Re-run a range
sbatch --array=10-20 script.sh

SGE to Slurm Translation

Directives

SGESlurmNotes
#$ -N job_name#SBATCH --job-name=job_name
#$ -cwd(default behavior)Slurm runs from submit dir by default
#$ -l m_mem_free=4G#SBATCH --mem=4GPer-node memory
#$ -pe onenode N#SBATCH --ntasks=1 --cpus-per-task=NSingle-node parallel
#$ -j y(default behavior)Slurm merges stderr into stdout by default
#$ -o logs/out-$TASK_ID.log#SBATCH --output=logs/out-%A_%a.log%A=job, %a=array task
#$ -t 1-176#SBATCH --array=1-176
(no equivalent)#SBATCH --partition=standardRequired — no default partition
(no equivalent)#SBATCH --time=3:00:00Default 5h, max 7d on standard

Environment Variables

SGESlurmDescription
$SGE_TASK_ID$SLURM_ARRAY_TASK_IDArray task index
$JOB_ID$SLURM_JOB_IDJob ID
$NSLOTS$SLURM_CPUS_PER_TASKAllocated CPUs
$HOSTNAME$SLURM_NODELISTAssigned node(s)
$SGE_TASK_FIRST$SLURM_ARRAY_TASK_MINFirst array index
$SGE_TASK_LAST$SLURM_ARRAY_TASK_MAXLast array index

Commands

SGESlurmDescription
qsub script.shsbatch script.shSubmit job
qstat -u $USERsqueue -u $USERList running jobs
qdel job_idscancel job_idCancel job
qstat -j job_idscontrol show job job_idJob details
qacct -j job_idsacct -j job_idJob accounting
(no equivalent)sinfo -p partitionPartition info

Conversion Checklist

When converting an SGE script to Slurm:

  1. Replace #$ directives with #SBATCH equivalents (see table above)
  2. Add #SBATCH --partition=standard (SGE has no equivalent — partition is implicit)
  3. Add #SBATCH --time= (SGE defaults to unlimited on WRDS)
  4. Replace $SGE_TASK_ID → $SLURM_ARRAY_TASK_ID
  5. Replace $NSLOTS → $SLURM_CPUS_PER_TASK
  6. Replace $JOB_ID → $SLURM_JOB_ID
  7. Remove #$ -cwd and #$ -j y (Slurm defaults)
  8. Update log path variables: $TASK_ID → %a, $JOB_ID → %A
  9. Update data paths from WRDS scratch to HPC scratch

Monitoring & Debugging

Check Job Status

squeue -u $USER                              # all my jobs
squeue -j 12345678                           # specific job
squeue -j 12345678 -t R | wc -l             # count running tasks
squeue -j 12345678 -t PD                     # show pending tasks + reasons
squeue -u $USER --format='%.10i %.9P %.12j %.2t %.10M %.4C %R'  # detailed

Common Pending Reasons

ReasonMeaning
(Priority)Lower priority than other queued jobs — will run eventually
(Resources)Not enough free nodes/CPUs — waiting for running jobs to finish
(QOSMaxCpuPerUserLimit)Hit per-user CPU limit on this QOS
(AssocMaxJobsLimit)Hit max concurrent jobs for this account

Job Accounting (after completion)

sacct -j 12345678 --format=JobID,State,ExitCode,Elapsed,MaxRSS,NCPUS
sacct -j 12345678 -a --format=JobID,State,ExitCode  # all array tasks

Log Files

Output goes to --output path. With %A_%a pattern:

  • logs/est-12345678_1.log — job 12345678, array task 1
  • Check for errors: grep -rl 'Error\|Traceback' logs/est-12345678_*.log

Resource Billing

UVA HPC bills in Service Units (SUs), which are weighted CPU-core-hours:

SU = (CPU_cores × 4.6369 + Memory_GB × 0.2842) × hours

Cost Examples (standard partition)

ConfigSU/hour176 tasks × 3 hrs
1 CPU, 4GB~5.8~3,062
8 CPU, 32G~46.2~24,404
40 CPU, 160G~231~121,968

With 10M SUs allocated, even aggressive usage (8 CPU × 176 tasks × 3 hrs = ~24K SUs) is negligible (<0.25% of allocation).

Check Balance

allocations                    # show allocation balance
allocations -a myallocation    # specific allocation

WRDS Data Access

WRDS PostgreSQL is accessible from HPC compute nodes. Use polars + connectorx for fast data pipelines that replace SAS entirely.

Connection

  • Host: wrds-pgdata.wharton.upenn.edu:9737
  • Credentials: ~/.pgpass (chmod 600)
  • User: edwin_hu (UVA account)

Quick Start

from wrds_conn import read_wrds

# WRDS SQL → polars DataFrame in one line
df = read_wrds("SELECT * FROM crsp.msf WHERE date >= '2020-01-01'")

# Write to Parquet for reuse
df.write_parquet("/scratch/vwh7mb/data/crsp_msf.parquet")

wrds_conn.py (see examples/wrds_conn.py) parses .pgpass and builds a connectorx-compatible URI — connectorx doesn't read .pgpass natively.

Pipeline: SQL → polars → Parquet (replaces SAS)

Old: WRDS SAS → .sas7bdat (7GB) → Python HDF5 conversion → .h5 (390MB)
New: WRDS PostgreSQL → polars/connectorx → .parquet

No SAS license needed. Single step. Portable output.

See references/wrds-polars-pipeline.md for full examples (joins, partitioned output, Slurm submission for large queries).

Additional Resources

Reference Files

  • references/sge-to-slurm.md - SGE → Slurm migration: directive-by-directive translation table for converting WRDS grid jobs to UVA HPC
  • references/wrds-polars-pipeline.md - WRDS PostgreSQL → polars → Parquet pipeline (joins, partitioned output, Slurm submission for large queries)

Example Files

Copy these rather than writing a submit script from scratch — both use --partition=standard, which is the choice array jobs actually need:

  • examples/array_job_filelist.sh - Array job sharded over a file list (one task per chunk of files)
  • examples/array_job_year_chunk.sh - Array job sharded over year × chunk, with multiple workers per task (the PIN-estimation shape)
  • examples/wrds_conn.py - Parses .pgpass into a connectorx-compatible URI (connectorx doesn't read .pgpass natively)
jev-rulesjupytextlaw-econ-docxlaw-review-docxlexislook-atlseg-datamarimomarimo-servenlmnotebook-debugnpx-ownership-panelobsidian-organizeopenai-batchpaperpilepattern-capturepermaccpinciteplugin-creatorpptx-renderprose-rules-econprose-rules-legalprose-rules-reviewprose-rulesreadwise-chatreadwise-docsreadwise-prunereadwise-searchreadwisereplicate-paperreplication-packageresearchsetupskill-creatorsource-verifytypst-test-slidetuicrui-json-captureusing-skillsvisual-mockupvisual-verifyweb-scrapingwestlawworkworkflow-creatorworkshopwrdswriting-econwriting-generalwriting-legalwriting

Sponsors

Details

Marketplace
edwinhu/workflows
Category
Development