
workflows
One loop — work — for code, data, writing and workshop presentations: an approved plan as the run's only authority, delegated implementation, independent verification against computed gates, and human review
- version
- 6.62.0
- GitHub stars
- 27
Install
/plugin install workflows@edwinhu-pluginsRun it inside Claude Code after adding the marketplace with /plugin marketplace add edwinhu/workflows.
Contents
Discovered from sourceSkills 91
afk — a standing mandate until the armed ceiling
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
!bash ${CLAUDE_SKILL_DIR}/scripts/arm.sh "${CLAUDE_SESSION_ID}" "$ARGUMENTS"
The line above armed (or declined to arm) the hold when this skill loaded. Its output is the authority on what is armed; if it says "Not armed", say so, do not arm by hand, and still follow steps 1-5: the heartbeat is then the only keep-alive.
**NO QUESTIONS UNTIL THE CEILING** — the time the armed line prints: 18:00 for a daytime /afk, otherwise 09:00. The user is away. Asking is not careful, it is an idle session and lost hours. Take the Recommended branch, state the choice in one line, and keep going.- Heartbeat. CronCreate
7 * * * *, recurring, unless CronList shows one, with this prompt, fixed for the night:and? (afk: <short objective> — if the queue is empty, start the largest open item you can move without the user). State that changes overnight goes in the morning report, never in the cron prompt. Never CronDelete it: CronDelete is inpermissions.ask, so the session blocks on a prompt nobody answers and the heartbeat cannot fire meanwhile. - Branches. Take the Recommended one instead of AskUserQuestion.
- Commit locally. Explicit paths. Never push, never delete data.
- Work the queue. Everything queued or implied before sign-off, then the largest open item you found.
- Morning report before the ceiling: what landed, what is blocked, what needs the user.
- Release is the user's:
work-hold.sh --disarm.
| About to | Why wrong | Do instead |
|---|---|---|
| End a turn announcing the next step | A check-less afk hold allows every stop, so nothing re-enters the session until the next tick | Take the step in the same turn |
| Launch a grind loop on this objective | A hold beside a grind deadlocks both | One driver per objective; the heartbeat covers the grind |
| CronDelete or re-create the heartbeat to change its prompt | The permission prompt blocks the session and silences the heartbeat | Keep the prompt fixed; write state into the morning report |
| Answer a tick with "nothing new" / "no change" | That is the idle night this skill exists to prevent | Take step 4: start the largest open item |
About to start a Bash command with cd <dir>; or splice quotes with '"'"' | Claude Code can show a permission prompt for these even in bypass mode, and nobody answers | Use absolute paths and git -C; pass long text through a file (jq --rawfile) |
AI Writing Anti-Patterns
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Field guide for detecting and revising AI-generated content indicators based on Wikipedia's "Signs of AI writing" guide.
When to Use
Invoke this skill:
- Before finalizing ANY AI-assisted writing
- When reviewing text for AI writing indicators
- When editing content to sound more natural
- After completing writing tasks (automatic via hooks)
The Iron Law
Check every piece of AI-assisted writing against these patterns before submission.
This is not optional. AI writing patterns are detectable and undermine credibility.
Quick Screening Order
Start with the most objective indicators:
| Priority | Section | What to Check |
|---|---|---|
| 1 | ChatGPT Artifacts | turn0search0, oaicite, contentReference |
| 2 | Citation Problems | Hallucinated DOIs, dead links, non-existent sources |
| 3 | Prompt Refusals | "As an AI language model...", "I hope this helps" |
| 4 | Puffery | "stands as a testament to", "rich tapestry", "nestled" |
| 5 | Structure | Section summaries, "Despite challenges", rule of three |
| 6 | Mannered Prose | Metaphor where a literal phrase was available (see below) |
Critical Patterns to Avoid
CRITICAL Severity (Immediate Revision Required)
These patterns are unambiguous AI artifacts:
ChatGPT-Specific Artifacts:
turn0search0,turn1search2(internal search references)oaicite:X(citation placeholders)contentReference[oaicite:X](unresolved references)- JSON attribution blocks in output
Prompt Refusals:
- "As an AI language model..."
- "I cannot provide..."
- "I hope this helps!"
- "I hope this email finds you well"
HIGH Severity (Strong Revision Recommended)
THE SHIPPED LIST IS ~/.claude/skills/ai-tic/linter/tics.yaml, NOT THIS TABLE. That file holds
both halves — the tics that cleared the ~0-human-rate gate against 14,294,148 sentences, and a
rejected: block naming the phrases that did not, so they are not re-proposed. Six phrases were
listed here as AI tells while sitting in that rejected: block: serves as (141.35/M),
it is important to note that (44.42/M), plays a crucial/vital/key role bare (21.94/M),
delve into bare (11.69/M), underscores the importance (5.93/M), and Of course, (386.94/M).
They are gone from the tables below and from every executable table in the repo. This is the same
correction the User-Voice section thirty lines down already made for a different list — "Keep
these as voice preferences if you like them … Do not present them as AI detection." — applied to
this one. To check a phrase, run /ai-tic <phrase>; do not add a row here.
Puffery and Exaggeration (those that cleared the gate):
- "stands as a testament to"
- "rich tapestry of"
- "nestled in/among"
- "the landscape of"
Promotional Language:
- "groundbreaking", "transformative", "revolutionary"
- "unparalleled", "unprecedented"
- "cutting-edge", "state-of-the-art"
MEDIUM Severity (Review and Consider)
Structural Patterns:
- Section summaries that repeat the heading
- "Despite [challenge], [positive outcome]" formula
- Negative parallelisms: "However... Nevertheless..."
- Rule of three: exactly three examples every time
- Weasel wording: "some experts say", "it is believed"
Stylistic Quirks:
- False precision — a summarising figure carried to spurious decimal places
("a rate of 1.3771 percent", "covering 85.63 percent of the universe").
Prefer a high-level fraction in the abstract and introduction, where a figure
summarises rather than reports; keep the exact value next to the exhibit that
backs it. Enforced as
style·false_precision. - Elegant variation (synonym cycling to avoid repetition)
- False ranges ("from X to Y" without real data)
- Title Case In All Headings
- Em dash overuse (—)
- Excessive boldface for emphasis
User-Voice Preferences (NOT AI tells — corpus-checked)
This section is a personal style preference, not a linter. Nothing here is in an executable
table, so a writing-verify pass knows none of these phrases. Three of the four were measured against 14,294,148 sentences of human
scholarship (8.73M finance/accounting + 5.56M law review) and they are normal human prose:
| phrase | finance | law | verdict |
|---|---|---|---|
has bite / have more bite | 1.9/M | 5.2/M | human — incl. a law review title, "Do the SEC's New Rating Agency Rules Have Any Bite?" |
the cut (regulatory reduction) | 8.7/M | 4.7/M | human — "the cut in the corporate tax rate" |
Of course, | 299.9/M | 523.7/M | human |
To be sure, ← not an AI tell; do not replace it | 11.5/M | 194.0/M | human, and 3× commoner in law reviews than what it replaces |
Admittedly, | 15.5/M | 63.3/M | human |
cuts against | 0.9/M | 13.1/M | human |
Keep these as voice preferences if you like them — they are unfalsifiable and that is fine. Do not
present them as AI detection. Full record: docs/investigations/2026-08-05_emphasis-enforcement.md.
What DID survive the gate, and is now enforced as a span:
ai-tic·sev3·rule-bites— "the reform should bite hardest". The verb is unattested (1/14.29M, and that hit is a cited Financial Times headline in a footnote). The noun above is not. That noun/verb split is the whole rule.ai-tic·sev2·sharpest-version— "the sharpest version of the objection". 0/14.29M, while all four hits of<superlative> version ofare "the strongest version of" andthe sharpest <noun>at large is 86 hits. The word is ordinary; the collocation is the tell.
Bridge repetitions remain the one genuinely enforced entry from the old list —
skills/writing-verify/scripts/bridge_repetition_check.py, invoked from
workflows/writing-verify.js. It is real logic over section openings, not a phrase table.
To check a phrase yourself: /ai-tic <phrase> — it runs the FP-hunt against both corpus halves
and refuses to add anything over the eligibility gate.
How to Revise
For Puffery
| AI Pattern | Human Alternative |
|---|---|
| "stands as a testament to" | "shows" or "demonstrates" |
| "rich tapestry of" | describe specifically what it contains |
| "nestled in the heart of" | "in" or "located in" |
For Structure
| AI Pattern | Human Alternative |
|---|---|
| Section summary of heading | Start with substance, not meta-commentary |
| "Despite challenges..." | State the reality directly without formula |
| Exactly three examples | Use the number that fits: 2, 4, 5, or just 1 |
For Mannered Prose
Definition (Anthropic guidance — quoted, do not paraphrase):
Mannered prose substitutes metaphor and flourish for direct statement. Instead of "a parameter worth varying," the mannered writer produces "a dial worth turning." Instead of "this point still matters," they write "this point earns its keep." The phrases exist to display the writer, not to convey the idea, and readers can tell. That is why mannered prose irritates: it makes the reader work harder so the writer can perform. It is also imprecise. Metaphors drag in connotations the writer did not choose and cannot control. The fix is to say what you mean. When a literal phrase is available, use it.
The short form — "Please remove all mannered prose" — works on its own and is often enough.
The mechanism is metaphor-for-literal substitution. That is narrower than "flourish" generally: the test is whether a literal phrase was available and passed over. "A dial worth turning" for "a parameter worth varying" is mannered; a long sentence is not, and an ornate one is purple, a different fault. Puffery inflates ("stands as a testament to"); mannered prose performs.
The imprecision is the load-bearing objection, not the irritation. A metaphor imports connotations the writer did not choose: "the bar routed around the holding" implies coordinated evasion, where "lawyers filed the same claims as mootness-fee requests" makes no claim about motive. Revising for taste alone misses that the mannered version often asserts something the writer cannot support.
The revision, in order:
- Replace every mannered phrase with the literal thing it avoided.
- Table each swap: the flourish, the plain version, and the connotation it smuggled in.
- Name the habit the writer repeats most.
The three constraints — violating any makes the pass worse than not running it:
- Keep the meaning and the structure. A diction pass, not a rewrite.
- Leave direct sentences untouched. A sentence with no metaphor has nothing to swap.
- Don't shorten for its own sake. Literalness is the goal, not brevity. A plain replacement that runs longer than the flourish is still the right replacement.
Why the table is the deliverable, not just the edits. The third column is the point. Listing swaps without naming what each smuggled in hides the substantive change. And naming the repeated habit matters more than any single swap: individual metaphors are cheap to fix and recur immediately, while the habit is the thing the writer can act on.
A related failure, one level up (supporting research, not the definition). Density, not the device, is what makes rhetorical figures read as machine-written: "The problem is not that LLMs use these techniques: it's that they're so robotically consistent in how they use them that it becomes an abuse… What the LLM lacks is not technical ability, but taste" (Colin Gorrie, Why ChatGPT writes like that). The figures to watch are antithesis ("not X, but Y"), tricolon, isocolon, and unearned profundity. A balanced two-part closer at the end of every section is this failure, not metaphor substitution — so it is out of scope for the swap table above, and needs naming as a habit instead.
Model note. Claude Fable 5.1 writes denser prose than Fable 5 — longer sentences, fewer paragraph breaks. Defining the anti-pattern in a user message (preferred) or the system prompt is the documented mitigation.
For False Precision
A figure in the abstract or introduction is SUMMARISING; a figure beside its table is REPORTING. Only the second earns its decimal places. Carrying four of them into a summary implies a precision the estimate does not have and reads as machine output.
| AI Pattern | Human Alternative |
|---|---|
| "a rate of 1.3771 percent" | "about one and a half percent" |
| "roughly ten times the 2.7361 percent rate" | "roughly ten times the abstention-wide rate" |
| "6,612 of 575,553 testable items — 1.15 percent — fall below" | "about one percent of testable items fall below" |
| "covering 85.63 percent of the testable universe" | "covering roughly six in seven of the testable universe" |
| "the 28.3328 percent figure" | "that figure" |
Exceptions the rule already carries, and which must stay exceptions: exact values inside a table, figure caption, code block or footnote; years, statutory and rule cites, docket, page and version numbers; and money where the cents are the point.
For Promotional Language
| AI Pattern | Human Alternative |
|---|---|
| "groundbreaking" | describe what it actually does |
| "revolutionary" | compare to what came before |
| "cutting-edge" | specify the technology |
| "transformative" | show the transformation with evidence |
Reference Files
For detailed patterns and extensive examples, consult:
THE "ENFORCED BY" COLUMN IS THE POINT OF THIS TABLE. Eleven chapters listed as reference
material reads as eleven chapters of coverage; it is not. Four of them have no executable module
and are marked reference-only here so nobody has to rediscover the gap by shipping a draft through
a clean review pass. (That is exactly how chapter 05's boldface entry went unenforced until
v5.134.0 — see docs/investigations/2026-08-05_emphasis-enforcement.md.)
| File | Contents | Enforced by |
|---|---|---|
references/_index.md | Overview and quick screening guide | — |
references/01-puffery-and-exaggeration.md | "Stands as", superficial analyses | wikipedia-puffery |
references/02-promotional-language.md | "Rich tapestry", disclaimers | wikipedia-promotional |
references/03-structural-patterns.md | Section summaries, negative parallelisms | wikipedia-structural |
references/04-stylistic-quirks.md | Elegant variation, false ranges | reference only — regexable, but must clear the ai-tic corpus gate first |
references/05-formatting-and-typography.md | Boldface, em dashes, emojis | emphasis, formatting, em-dash (boldface/emoji since v5.134.0); inline-header lists partly |
references/06-communication-patterns.md | Subject lines, "I hope this helps" | wikipedia-communication |
references/07-template-artifacts.md | Mad Libs patterns, placeholders | wikipedia-template-artifacts |
references/08-markup-issues.md | Markdown vs wikitext confusion | reference only — wikitext-specific; no in-repo analogue |
references/09-chatgpt-specific-artifacts.md | turn0search, oaicite | wikipedia-chatgpt-artifacts |
references/10-citation-problems.md | Hallucinated DOIs, dead links | reference only — covered better by skills/cite-check + skills/source-verify and the cite-fidelity-* constraints, which check citations against sources rather than pattern-matching them |
references/11-meta-indicators.md | Abrupt cutoffs, style discrepancies | reference only — judgement; lives in the user-agents/writing-reviewer.md rubric |
Every named system is a ${CLAUDE_PLUGIN_ROOT}/scripts/prose-audit.py system, so a finding from it carries a span id.
A chapter marked reference only produces no span, which by THE READER'S RULE means anything it
surfaces is judgement, not a deterministic finding.
Automatic Detection
This plugin includes PostToolUse hooks that automatically scan Write/Edit output for anti-patterns. When patterns are detected:
- Hook emits a warning with specific patterns found
- Claude immediately revises the content
- Revision removes or replaces flagged patterns
The hook checks for all CRITICAL and HIGH severity patterns automatically.
Review Facts
- A user's style request ("make it punchy", "professional tone") is a request for an outcome, not for AI-smell — delivering it with puffery, hedges, or bold-emphasis patterns intact ships text that reads as obviously AI-generated and damages the user's credibility.
- Skimming is not checking. The check is sentence-by-sentence against the pattern list; a skim that lets puffery pass because flagging it felt pedantic presents unreviewed text as reviewed — an unverified claim.
Key Principles
From Wikipedia's guide:
- These are signs, not proof - Multiple indicators strengthen the case
- Context matters - Some patterns appear in human writing too
- Focus on deeper issues - Surface defects point to synthesis and quality problems
- Don't rely on detection tools - Human judgment required
Related Skills
/writing- Core writing principles from Elements of Style/writing-legal- Legal writing (Phase 2)/writing-econ- Economics writing (Phase 2)
Batch router
Picks the provider for one batch job and hands off. It never submits: the handoff is to
claude-batch, openai-batch or gemini-vertex, which own the request format, the pilot and the
submission.
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Every bulk LLM job goes through a provider's Batch API (half price), never a loop of synchronous calls
and never an interactive-agent fan-out; one synchronous verify per job is the only sync call. The full
run waits for the pilot's results --input exit 0 and a message from Eddy approving the cost this
router printed. A sync loop that "already works" still costs double, and a fan-out of agents bills the
shared Claude accounts per document.
Route
uv run ${CLAUDE_SKILL_DIR}/scripts/route_batch.py --requests 5000 --in-tokens 3000 --out-tokens 300 \
[--tier cheap|mid|top|frontier | --model <id>] [--needs vision|pdf|web-search|long-context] \
[--balance uva=200 --balance nyu=200] [--json]
uv run ${CLAUDE_SKILL_DIR}/scripts/route_batch.py --jsonl requests.jsonl --out-tokens 300 # measured
It prints a ranked table with a reason per row and the next command. The ranking:
- Prepaid credit that expires soonest, covers the estimated cost +20%, and has a usable key.
- Other prepaid credit (short balance, or expiry within 24h).
- Pay-per-token, cheapest model that fits; Gemini Vertex wins a tie and is the default fallback.
- Blocked: no key, or the batch path lacks a needed capability. Listed, with why.
Over a provider's per-batch limit (Claude 100k / 256 MB, OpenAI 50k / 200 MB, Gemini 200k) the row says how many shards. Details, prices and limits: routing rules.
Then invoke the skill the top row names (workflows:claude-batch, workflows:openai-batch or
workflows:gemini-vertex) and follow it: validate exit 0 → one verify → 5–10 row pilot whose
results --input exits 0 → Eddy's yes to the printed cost → full run → results --input exit 0.
Facts
- Balances are never stored. There is no balance API for Claude credits; the Console is the only
source. Pass
--balance <account>=<usd>from what Eddy or the Console reported today; without it the router assumes at most the monthly credit and printsbalance UNVERIFIED. Writing a balance intoaccounts.tomlcreates a number that is silently wrong a day later. - Accounts are config, at
$BATCH_ACCOUNTS(default~/.config/batch-router/accounts.toml; shape inreferences/accounts.example.toml). This repo is public: org ids, emails and key paths never go in it. - OpenAI is pay-per-token until its account says
credit_linked = true: the ChatGPT Pro credit page never names Batch. Its batches can queue for hours; not for same-day results. - Credits expire at cycle end with no rollover. An org whose credit lapses in four days with $200 unspent is worth more than any per-token discount, which is why expiry outranks price.
- Web search is priced per search, not per row: Claude and OpenAI $10/1k, Gemini $14/1k after 5k free a
month. The router assumes 2 searches per row (
--searches-per-row); a grounded Gemini run once cost $390 in searches against $30 in tokens.
Red flags — STOP
| About to | Do instead |
|---|---|
| Send bulk work straight to gemini-vertex out of habit | Run the router; a prepaid Claude credit may be expiring unused |
| Loop synchronous API calls over the rows | Batch; the only sync call is the one verify |
| Fan out agents or farm rows, one per document | One batch job over pre-cut inputs |
| Submit on the router's word alone | The router never submits; follow the provider skill's pilot and get the go-ahead |
| Quote an UNVERIFIED balance as the real one | Say it is assumed, or ask Eddy for the Console figure |
Create an API key for an org that shows no key | Ask Eddy; he adds the key source to accounts.toml |
Bib-manage
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Owns the bibliography as an index: every entry pointing at the right PDF, and that PDF being the version of record.
pincite CONSUMES this index — it resolves footnote → source through
file = {...} and through nothing else. So the two failures this skill exists to
prevent are both silent in pincite: an entry with no file is a pincite that is
never attempted, and an entry pointing at a preprint is a pincite that is
confidently wrong.
audit entries with/without file, files missing on disk, entries without a
DOI, PDFs no entry claims <- run this first
link citekey -> PDF, written back as file = {...}
version is each entry's PDF the published version, or a preprint/proof?
doi backfill missing DOIs from Crossref, verified against the bib's pages,
and the PAGE RANGE that same verified record already carries
reconcile
rewrite the entry's OWN FIELDS from a better source:
Paperpile > Crossref > the extracted value. Never `file`, never a key
fedreg fetch the Federal Register PDFs the entries cite
CLI
B="${CLAUDE_SKILL_DIR}/scripts/bibman.py"
python3 "$B" audit --root . # start here
python3 "$B" link --root . --dry-run # inspect the mapping first
python3 "$B" link --root . --backup # then write
python3 "$B" version --root .
python3 "$B" doi --root . --dry-run --only aggarwal2015
python3 "$B" reconcile --root . --dry-run # read the full set FIRST
python3 "$B" reconcile --root . --backup --fields doi,pages
python3 "$B" fedreg --root . --dry-run
Every path is a flag — --bib, --pdf-dir, --fedreg-dir — defaulting to the
layout pincite uses, so the tool works on any manuscript. --only scopes
version and doi to named citekeys; --dry-run reports without writing;
--backup leaves a .bak; --strict makes version exit 1 on any entry that
is not the version of record.
--fields scopes reconcile to a subset of author,title,journal,volume,pages, year,doi,publisher; --pp-index points at the Paperpile CLI's index cache.
audit, link and version never touch the network. doi and fedreg do.
reconcile reads Paperpile locally and reaches Crossref only for what Paperpile
does not cover.
Facts
-
The link threshold depends on the YEAR, and that is the whole of its accuracy. Two matchers run in order: PDF stem == citekey (48 of 79 on the test bibliography), then Paperpile-style
Author Year - Title.pdfon first-author surname plus title similarity (the other 31). Same year, the year corroborates and 0.55 is enough. DIFFERENT year — a working paper against its published version,Lund 2017→lund2018— demand ≥ 0.95, because only the title separates two papers by one author. At 0.55 with the year ignored,alexander2010("Interim News and the Role of Proxy Voting Advice") claims the PDF "The role of advisory services in proxy voting" at 0.52. That is a wrongfilein the bib, which is the one input pincite trusts absolutely — a mis-link there sends a footnote to another paper and can verify CONFIRMED while doing it. -
Never force a match; REPORT what stays unmapped. On the test bibliography four PDFs stay unclaimed and every one is correct:
Alexander et al. 2009(0.52),Cremers and Romano 2011whose filename is truncated mid-title by the filesystem (...proxy voting on co ... ion plans..., best match 0.89 against a 2007 entry), and two whose filenames are notAuthor Year - Titleat all. An unmapped PDF costs a minute of human attention. A forced one costs a wrong citation nobody looks at again. -
Paperpile writes SURNAMES ONLY, so the first-author surname is the FIRST token.
Barzuza et al.,Bebchuk and Hirst,Rock. Taking the last token — the correct habit for a bibtexauthorfield — reads "Barzuza et al." as an author named "al." and unmapped 14 of 83 PDFs here before it was caught. -
A 6-digit
pagesis an Elsevier ARTICLE NUMBER, not a page.J. Fin. Econ. 154 (2024) 103810is paginated 1–N in its own PDF and IS the version of record. Range-testing it reports every such entry as a preprint —shu2024andcabezon2025are both genuinely published and both false alarms without this exception. -
A publisher's own PDF can be an ADVANCE-ACCESS PROOF. OUP served
iliev2021as pages 1–49 instead of 5581+,v 00 n 0, internal nameRFS-OP-….tex— authenticated, current, and useless for a pincite. Flag when the range test fails AND page 1 carries no nonzero volume designation. A successful fetch never means the pagination problem is solved. -
Do NOT identify the version of record by matching the journal name against page 1. Bib journal names are Bluebook abbreviations ("Rev. Fin. Stud.") that never appear in a masthead ("The Review of Financial Studies"). That test flagged 46 of 61 entries and is worthless. The derived printed-page range against the bib's start page is the test that works: 5 of 79 flagged, all five real preprints.
-
The page derivation is pincite's, imported not copied.
pages_of,page_numbersandpage_offsetare pure functions and pincite does nothing at import time. Copying them lets the two skills' notion of "what page is this" drift, and aversionverdict that disagreed with pincite's own would be worse than no verdict. -
A bibliographic title query returns the SSRN PREPRINT DOI. Measured on
aggarwal2015: title alone returns10.2139/ssrn.2120797,10.2139/ssrn.1688993,10.2139/ssrn.2023480— the top three hits are all SSRN, none carries a page. Query with title + journal + volume + pages, then VERIFY the returned record's start page equals the bib'spagesbefore accepting: that yields10.1111/jofi.12284, page 2309–2346 against the bib's 2309. A DOI that fails the page check is reported, never written — it is usually the preprint of the very work the entry means. -
The verified Crossref record already carries the END page;
doiwrites it. The match test was alwaysrecord start page == bib start page, so the range that passes it is the same work's own span —10.1111/jofi.12284returns2309-2346against a bib holding2309. Written aspages = {2309--2346}. Four guards, and each blocks a different way of destroying a value that was already right:@articleonly (a@miscpress release'spages = {1}is not an article span); a bare start page only, so a Bluebook pincite someone recorded there survives — on OPVeasterbrook1991(66-67) andjackson2019(849, 851--52) are the only range-like values in 197 entries and BOTH are pincites, not spans; never anARTICLE_NUMBER; and the Crossref range must START at the bib's page, re-checked at write time so the DOI test and the range test can never disagree. Measured on OPV: 55 of the 64 bare start pages were eligible, 30 resolved, and the 25 that did not are law reviews, which Crossref indexes thinly. -
An end page turns
versionfrom one test into three. With only a start page the derived span can be asked one question,lo <= start <= hi. With the end page it answers whether the span's BOTH endpoints fall inside, and whether the PDF's page count can hold the printed span at all —n < spanis impossible for a version of record, andn > span + 2(beyond a cover sheet) is a different artifact.aggarwal2015is a 57-page PDF claiming a 38-page span,ertimur201864 pages for 21,hayne201959 for 43; each now fails on three counts instead of one, with no header parsing. Entries with no end page keep the start-only path unchanged — on OPV that is still most of them, and all 77 verdicts were byte-identical before and after the backfill. -
reconcile's precedence is Paperpile > Crossref > the extracted value, and it is implemented as source ORDER, not a merge. An entry confidently matched in Paperpile is never queried against Crossref, so the stronger source cannot be contradicted by the weaker one. On OPV — a bib whose header readsAuto-extracted from sources.md via Gemini Vertex Batch, where every field was written by a model reading a markdown list and no entry carried a DOI — Paperpile covered 33 of 197, Crossref 24, and 140 had neither. Every change prints ascitekey.field: old -> new [source];--dry-runwrites nothing and is byte-identical run to run. -
A CONFIDENT Paperpile match needs the surname, the year AND the title, and an entry with no author needs the year EXACTLY. Surname equal, |Δyear| ≤ 1, title ≥ 0.90 same-year / ≥ 0.95 across a year — the same asymmetry
linkuses. With NO author there is nothing to corroborate with, and 1.00-vs-0.97 is not enough:secproxyadvice2019(the SEC's own 2019 release, 84 Fed. Reg. 66,518) matched a 2020 COMMENT LETTER about it at 0.98, whose title is the release's title withRE:in front. Requiring Δyear == 0 there kills it; a document with no author has a fixed date and no working-paper drift to forgive. Two candidates within 0.02 of each other are AMBIGUOUS — reported, never picked, because a wrong library match rewrites author, title, journal and DOI at once. -
Crossref's start-page verification is
@articleONLY, because a pincite is shaped exactly like a page range. Nothing in the VALUE66-67says whether it is pages 66 to 67 of an article or a Bluebook pincite into a book. Readingeasterbrook1991's 66 as a start page made Crossref accept10.1111/j.1468-0319.1991.tb00155.x, titled "Discussion Papers", because it happens to begin on page 66 — a proposal to overwrite the title of The Economic Structure of Corporate Law. The entry TYPE is what separates them, andrange_blockedalready said so. This guard lives incrossref_pick, sodoigets it too. -
A Paperpile record can be the SSRN PREPRINT of a published article, and taking its fields REGRESSES the citation.
choi2009(S. Cal. L. Rev. 82:649) andbebchuk2019a(B.U. L. Rev. 99:721) are both published; both library records carryjournal = SSRN Electron. J.and a10.2139/ssrn.*DOI. Precedence orders sources by reliability — it does not authorise replacing a real journal name with a preprint server's placeholder. When the source record is SSRN-shaped and the entry ALREADY names a journal,journal,volume,pages,year,publisheranddoiare all refused and reported. A genuine working paper has no journal to protect, so it is unaffected. -
Precedence does not authorise a THINNER rendering of a name. Paperpile answered
J Fisch and A Hamdani and S D Solomonforfisch2019'sJill E. Fisch and Assaf Hamdani and Steven Davidoff Solomon, and Crossref answeredVICENTE CUÑAT and MIREIA GINE and MARIA GUADALUPEforVicente Cuñat and Mireia Giné and Maria Guadalupe— initials for given names, and full caps with the accents gone. Both are refused and reported. Note thatstr.isupper()on the whole author string is ALWAYS False: theandseparators are lowercase, so the test must run on the names alone. -
reconcilenever touchesfileand never touches a citekey.fileis not inRECONCILE_FIELDSand is refused by name infield_blocked— the PDF mapping belongs tolink, which owns the matcher and its thresholds. The keys are load-bearing for something outside this repo:paper/typst/opv-body.typreferences entries through#ref(<key>)and that project'sscripts/resolve_refs.pyresolves supra numbers through them, so a renamed key silently breaks the manuscript. After writing, the file is re-parsed and the key LIST compared to what it was; a mismatch exits non-zero pointing at the.bak. -
federalregister.gov bot-blocks every request and lands on unblock.federalregister.gov. Fetch from govinfo's link service,
https://www.govinfo.gov/link/fr/<vol>/<page>?link-type=pdf, into--fedreg-dirasfedreg-<vol>-<page>.pdf. Pre-2000 volumes return the WHOLE ISSUE, 200+ MB — trim withqpdf --empty --pages IN a-b -- OUT. The SEC's own release PDF of the same document has different pagination, so it is the wrong source for a Fed. Reg. pincite.
Red flags — STOP
| About to | Why wrong | Do instead |
|---|---|---|
| Run pincite on a bib you have not audited | Entries with no file are skipped silently; here that was 118 of 197 | audit first |
| Lower the 0.95 cross-year threshold to map "just one more" PDF | At 0.55 alexander2010 claims the wrong paper at 0.52 | Leave it unmapped and link it by hand |
Hand-add a file to make an unmapped PDF go away | The unmapped list is the output, not a failure | Read the reason it prints first |
Report a 6-digit pages entry as a preprint | It is an Elsevier article number; the PDF is the version of record | Trust the article-number skip |
| Check the version of record by finding the journal name on page 1 | Bluebook abbreviations never match mastheads — 46 of 61 flagged | Range-test the derived pages against the bib's start page |
| Accept a Crossref DOI because the title matched | Title-only returns the SSRN preprint; 6 of 7 test entries | Require the start page to equal the bib's pages |
Write a page range over a pages that already holds one | It is a Bluebook pincite, not a span — both such values on OPV were | Leave it; doi reports it as SKIPPED |
Give a @misc or @book entry an article span | pages = {1} on a press release is not page 1 of an article | The @article guard; read the SKIPPED list |
| Trust a freshly fetched publisher PDF | It can be an advance-access proof paginated 1–N | Re-run version on anything fetched |
Use the SEC release PDF for a Fed. Reg. cite | Different pagination from the one the cite uses | fedreg via govinfo link/fr/<vol>/<page> |
Run link without --dry-run on a bib you have not inspected | It writes into 197 entries | --dry-run first, then --backup |
Run reconcile without reading the full --dry-run set | It rewrote 163 fields across 47 entries on OPV | Read every line, then --backup; --fields to scope |
Take a Paperpile record's journal when the entry already names one and the record is SSRN | SSRN Electron. J. over B.U. L. Rev. is a regression, not a correction | The SSRN guard; read the CONFLICTS list |
| Accept a Paperpile match on a no-author entry a year apart | The SEC's 2019 release matched a 2020 comment letter at 0.98 | Δyear == 0 when there is no surname to corroborate |
Let Crossref verify a non-@article by its start page | 66-67 is a pincite, and "Discussion Papers" also starts on page 66 | @article only — the type is what separates them |
Add file to RECONCILE_FIELDS "since it is a field too" | The PDF mapping is link's, matcher and thresholds included | Leave it; field_blocked refuses it by name |
Bloomberg Law EDGAR search
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
A Bloomberg-specific profile of ui-json-capture: drive the logged-in browser at human pace and
keep the JSON the page already receives. Load browser-automation (port 9222,
mcp__chrome-devtools__*) and read ui-json-capture for the procedure and its Red Flags.
Why this route
Measured against SEC EFTS, WRDS web and wrds_sec_search: Bloomberg matches per
exhibit (option "Exhibits Only"), runs unstemmed (stemmed_search: false), tolerates
extraction glitches ("Non -Competition" still matched), and has Terms & Connectors: "phrase",
^"case-sensitive", ! and * wildcards, N/x, NP/x, S/, P/, ATLx(), ATMx().
Coverage from 1994; ~19.6M EDGAR documents.
Login
The browser is often signed out. Credentials are the 1Password item Bloomberg Law (UVA) (vault
"Shared with Agents"); handle them only inside a dispatched agent, never in the main thread.
- Fields:
input#username,input#password. A bare#usernamematches the<m-text-field>wrapper first. EnterInput.insertTextafter focusing; React ignores.value. - Enter does not submit. Click the button whose text is
Sign In; it lands onbloomberglaw.com/start.
Search
- Form:
https://www.bloomberglaw.com/start#advanced-search/features_edgar_all_search/features_edgar_all_search(Keywords; Apply To = Forms & Exhibits / Forms Only / Exhibits Only; Forms; Exhibits; Exhibit title; filer, CIK, company, location, SIC, file number). The favorite link may need one reload. - Install
scripts/01-install-hook.jsbefore clicking Search, so page 1 is captured.
Wire facts (observed, not for calling)
POST /product/blaw/api/v1/search/criteria, body{"criteria":{"model":"features_edgar_all_search","term":…,"keywords_apply_to":"edgar_exhibit","page_size":50,"page":N}}. Page 1 is the form submit; page N comes from the pager. Filter clicks hit the same endpoint with new criteria; the hook keys each page by criteria-minus-page, so check__cap.queriesholds exactly one query before assembling.- Response:
results_page.components.documents.items[](50 per page;id,titlelike"WOLFSPEED, INC.; Form 10-K; EX-10.21",date,metadatafiler,subtitlesfile number / filed / period / form,extractswith<mark>hits);.remote_counttotal (the UI shows "1000+");components.facets.items[](form family, form type, exhibit type, company, SIC) with counts. - Paging is an in-app route change (URL
/product/blaw/search/results/<criteria_id>, same document), so the hook persists. Next control:svg[data-testid="search-results-next-page"],disabled="true"on the last page; click it with a dispatchedMouseEvent.
Capture
scripts/01-install-hook.js and scripts/02-run-pager.js here; 03-status.js, 04-dump.js,
05-stop.js and assemble.py from ui-json-capture/scripts/ unchanged:
python3 ~/.claude/skills/workflows/skills/ui-json-capture/scripts/assemble.py \
--batches scratch/blaw --glob 'batch_*.json' --out data/raw/blaw_hits.jsonl \
--expect-total <remote_count> --page-size 50 --id-field id
From hits to EDGAR (accession, full text)
A hit carries no accession or CIK. scripts/resolve_exhibits.py resolves an assemble.py file:
uv run --script ${CLAUDE_SKILL_DIR}/scripts/resolve_exhibits.py --hits data/raw/blaw_hits.jsonl \
--out-dir data/processed/blaw --cache-dir data/raw/edgar_sgml \
--phrase "consulting agreement" --phrase "non-competition" # phrases optional
- Join on WRDS: keys go up as parameter arrays (pgdata is a hot standby: no TEMP tables) and
join server-side to
registrant(file no.),filing_view(date, form prefix) andwrds_forms(wrdsfname), so only ids and paths come back, neverfilingtext. - Files via rclone: raw submissions from
wrds:/wrds/sec/archivesbywrdsfname, cached; over 10,000 files it usesedgar.md's server-side tar.--ignore-checksumis deliberate: the default md5 check opens an ssh session per file on the login node, and exhausted it at 16 transfers. - Parse locally: one pass over the SGML
<DOCUMENT>blocks gives TYPE, DESCRIPTION, FILENAME and text for every hit, with no per-hit query.
hits_resolved.jsonl / .parquet: Bloomberg fields, match_status (unique / resolved_by_exhibit /
ambiguous / unmatched), resolved_by (type → exact title form → DESCRIPTION → extract text),
doc_match, accession, cik, filename, sec_url, text_path, chars, extract_found,
phrase_verification. Title quirks it handles: normalized form drops /A (the title keeps it);
EX-99: EX-99.1 is TYPE: DESCRIPTION; EX-10.6 2 is the DESCRIPTION of the second EX-10.6;
Rule 425 Communications = 425.
Measured on 295 hits: WRDS join 1–27 s; 277 submissions = 2.62 GB by rclone in 27 s;
parse + extract ~20 s. 294 resolved, 1 unmatched (CB/A: absent from the WRDS tables). Exhibit
text equals sec.gov's document minus its SGML header line in 275/275 checked.
Run a first pass on extracts alone; send only positives and unsure rows to a full-text pass.
Filters
Results-page facets are <label> elements (click the label, not the a11y checkbox node); each
click is a new query. Form type lands in criteria as
"facets":{"normalized_form_type":["Form 8-K"], "form_family":[], "display_exhibit_type":[], "cik":[], "sic":[]}.
A facet slice under 1,000 needs no date filter: capture it whole and filter locally. Date presets
sit under [data-testid="date-selector-preset-dropdown-button"] (Any, Single date, Date range,
Last 7/30 days, Last 12 months, Last 5 years) and land as e.g. "date_type":"last_5_years".
Verified: three-phrase query, Exhibits Only, Form 8-K → 356 hits, 8 pages captured by
the pager in ~4 min, assemble.py exit 0 (no gaps, 356 unique ids).
Not yet verified
Whether paging continues past result 1,000; the CSV export's row cap; the field names for a custom date range.
Bluebook Footnote Audit Workflow
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Systematic Bluebook 21st edition compliance audit for law review manuscripts in DOCX format.
Announce: "Using bluebook-audit to run a systematic Bluebook compliance check."
Overview
Seven-phase linear workflow: Extract -> Check -> Report -> Correct -> Verify -> Archive -> Cross-Refs
/bluebook-audit -> extract -> check -> report -> [USER REVIEWS] -> correct -> verify -> archive -> crossrefs
/bluebook-audit-fix -> diagnose -> route to {re-check, re-correct, re-verify}
Phase Summary
| Phase | Responsibility | Gate |
|---|---|---|
| Extract | Parse DOCX -> structured JSON with formatting | footnotes_data.json exists, all FNs extracted |
| Check | Mechanical checks → Gemini Batch per-footnote → Claude cross-footnote review | audit_findings.json exists, ALL FNs covered, three-layer merge complete |
| Report | Present findings to user for review | AUDIT_REPORT.md exists, user acknowledges |
| Correct | Apply fixes to DOCX via lxml | Corrected DOCX exists, fix counts match |
| Verify | Re-scan to confirm fixes applied | Zero remaining issues in re-scan |
| Archive | perma.cc URL archiving | All URLs archived, links written to DOCX |
| Cross-Refs | Convert supra/infra notes to NOTEREF fields | All cross-refs are auto-updating fields |
How to Start
- User provides a DOCX file path
- Workflow creates
scratch/directory for intermediate artifacts - Proceeds through phases sequentially
Next Step
Read the entry command:
Read("commands/bluebook-audit.md") # relative to this skill's base directory
## Iron Law: ALL Footnotes Must Be Checked
Every footnote in the document must be audited. No subsets. No sampling.
Auditing only "major-severity" footnotes or a random sample guarantees missed errors. The formatted Gemini audit must cover ALL footnotes, not just previously flagged ones.
Skipping footnotes is NOT HELPFUL — missed errors go to publication and embarrass the user.
## Iron Law: Formatted Text for Gemini AuditNEVER send plain text to Gemini for typeface auditing. Always include formatting markup.
Plain text produces 10-20x false positives because Gemini cannot see what is already italic/small caps/roman. Inline markup (*italic*, [SC]small caps[/SC]) reduces false positives from ~400 to ~20 for a 239-footnote document — measured on gemini-2.5-flash, so on the current judgment role it is an estimate, not a measurement.
After applying corrections, ALWAYS re-run the scanner to verify fixes were applied.
NBSP characters, run boundaries, and cross-run text cause silent failures. A fix that "applied" in code may not have actually changed the DOCX. Re-scanning is mandatory.
## Iron Law: Mechanical Findings Override GeminiNever drop a mechanical finding because Gemini didn't flag it.
Deterministic checks (signal italic, terminal periods, Id. chains) are 100% reliable. Gemini misses ~30% of signal formatting issues because it focuses on citation-level analysis and lacks a dedicated signal-checking output field. During merge/dedup, mechanical findings are authoritative for their rule categories. Gemini adds value only for judgment calls (source type classification, abbreviation tables).
Previous failure: Gemini reported FN103 as having only typeface issues on article titles, completely missing that "See also" was not italicized — which the mechanical checker caught.
## Iron Law: Source Type Classification Requires Human ReviewGemini consistently misclassifies non-standard source types. Never auto-fix Gemini's typeface suggestions for SEC releases, executive orders, working papers, or regulatory materials.
The hardest part of a Bluebook audit is determining the correct typeface for non-standard sources. Gemini defaults to "everything should be italic or small caps" but many source types are correctly roman:
- SEC releases/rules/concept releases → roman (regulatory material, Rule 14.6)
- Executive order titles → roman (Rule 14.7)
- Working paper series designations → roman (parenthetical)
- Company names in no-action letters → roman
The audit report MUST separate "verified fixes" (clear violations) from "judgment calls" (source type dependent) and include a "correct as-is" section documenting why Gemini's suggestions were rejected. See references/audit-patterns.md for the full source type reference table.
Previous failure: Gemini flagged 10+ items as needing italic/small caps that were actually correct as roman (SEC releases, exec orders, working paper designations). Without the source type reference table, these would have been incorrectly "fixed."
## Audit Facts- Gemini hallucinates citation formats and fabricates citation details. Its output is a lead sheet, not a verdict — run the mechanical checker independently, merge results, and spot-check before applying anything. A Gemini suggestion applied unverified is an unverified claim applied as a fix.
- Fixes cascade: a correction in one phase creates new errors in dependent phases, and every phase catches a different error type. Re-run the affected scanner after every correction pass — "earlier phases were clean" says nothing about errors the fixes just introduced.
- A hardcoded journal list and a previously-extracted
sources.bibare both LAGGING records, and each one hid real errors in the same document:J. Corp. L.was not inJOURNAL_NAMES, and 19 of 53 article cites were missing from a bib extracted four months earlier. Validating a manuscript against either has the dependency backwards — it hides precisely the citations most recently reworked, which are the ones most likely to be wrong. Runbib_gaps.pyBEFORE the small-caps check so the bib is current when the check reads it. - The bib's
journalfield only exists on@articleentries, so it covers article periodicals and nothing else — 23 of one document's 54 small-capped names were newspapers, magazines, blogs and BOOK titles, which live in@misc/@book. Treating "scan the bib" as complete coverage silently exempts every book and newspaper in the manuscript. - One wrong supra reference invalidates the reader's trust in ALL footnotes. Sampling, eyeballing, or marking a footnote correct without checking the reporter/volume audits a subset and asserts compliance for the whole — a claim the audit never verified. Every footnote, every phase, regardless of document length.
If you sent plain text to Gemini instead of formatted text with footnote markers, DELETE the results and START OVER with properly formatted input. Gemini cannot audit what it cannot parse.
Phase 6: Archive
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Archive all non-permanent URLs in footnotes via perma.cc API.
What This Phase Does
- Extract URL inventory from footnotes data
- Deduplicate URLs (same URL across multiple footnotes)
- Archive each URL via perma.cc API
- Write perma.cc links back to DOCX footnotes
- Save progress after each successful archive
Prerequisites
- Perma.cc API key in
.envfile (project-level or user home directory) - For institutional accounts: organization folder ID (unlimited archives)
- Free accounts: 10 links/month limit
Script
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/permacc_archive.py" --docx <path> --data scratch/footnotes_data.json
Institutional Account Setup
# Find your organization and folder ID
curl -H "Authorization: ApiKey YOUR_KEY" https://api.perma.cc/v1/organizations/
Use the folder parameter when creating archives:
requests.post("https://api.perma.cc/v1/archives/", json={
"url": url,
"folder": FOLDER_ID, # enables institutional limits
})
Gate: Exit Archive
- All non-perma.cc URLs archived
- perma.cc links written to DOCX
-
scratch/permacc_archives.jsoncontains all mappings
Next Phase
Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-crossrefs/SKILL.md and follow its instructions.
Phase 2: Check (Mechanical + AI Audit)
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Two-stage checking: Python mechanical checks catch definite errors; Gemini batch audit catches judgment-call issues.
Stage 2a: Mechanical Checks (Python)
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/scan_formatting.py" --docx path/to/file.docx
Checks performed on ALL footnotes:
- Journal name small caps - Comprehensive pattern list (law reviews, finance journals, newspapers, periodicals, forums)
- Book title small caps - Detect italic book titles that should be small caps
- Id. chain validation - Rule 4.1 mechanical check (single-source predecessor)
- Signal italic consistency - All signals (see, cf., e.g.) must be italic
- Terminal period - Every footnote must end with a period
- Hereinafter consistency - Defined at first citation, used consistently after
- Author name supra format - Text before
*supra*should be roman, not italic (unless it's a case name short form). Catches*Manne, supra*→ should beManne, *supra* - Italic spillover - Trailing/leading spaces inside italic or small caps runs (e.g.,
*supra *should be*supra*). These don't affect Word display but cause Gemini misparses
Stage 2a-ii: Bibliography drift
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/bib_gaps.py" \
--docx path/to/file.docx --bib paper/references/sources.bib --out scratch/bib_additions.bib
make_bib_from_docx.py bootstraps a bib for a manuscript that has none; this is
the counterpart for every round after the first. It reports article cites present
in the footnotes and absent from the bib, and emits BibTeX for review.
Parsing is deterministic, not model-driven — a fabricated bib entry is worse than a missing one because it looks answered. A title containing a comma ("The Millennial Corporation: Strong Stakeholders, Weak Managers") cannot be separated from an author list positionally, so review the emitted entries before merging.
NBSP Handling
DOCX uses non-breaking spaces (\xa0) in abbreviations. ALL search functions must handle both \x20 and \xa0:
No.\xa02106,Feb.\xa07,Oct.\xa021Wall St.\xa0J.,Corp.\xa0Governance
Stage 2b: Gemini Batch Formatted Audit
Default: Use Gemini Batch API (50% cheaper, handles all footnotes in one job). Fallback to sync calls if batch is unavailable.
Step 1: Extract formatted footnotes
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/gemini_audit.py" --docx path/to/file.docx --extract-only
This outputs a JSON file mapping footnote numbers to formatted text with inline markup:
*text*= italic[SC]text[/SC]= small caps- Plain text = roman
Step 2: Submit Gemini Batch Job
Build a JSONL file with one request per footnote, then submit via Batch API:
# Build JSONL (one line per footnote)
for fn_num, formatted_text in footnotes.items():
request = {
"custom_id": f"fn-{fn_num}",
"body": {
"contents": [{"parts": [{"text": PROMPT.format(fn_num=fn_num, formatted_text=formatted_text)}]}],
"generationConfig": {"responseMimeType": "application/json", "temperature": 0.1}
}
}
# Submit batch job (see /gemini-vertex skill for full pattern)
# Use examples/batch_processor.py pattern — DO NOT guess API parameters
IMPORTANT: Follow the /gemini-vertex skill's Iron Law — read examples/batch_processor.py before writing batch code.
Fallback (sync): If batch is unavailable, use:
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/gemini_audit.py" --docx path/to/file.docx
Gemini Prompt Focuses On:
- Source type classification (case, statute, article, book, newspaper, working paper, hearing, letter, regulation)
- Typeface correctness per Rule 2 (italic vs small caps vs roman)
- Abbreviation correctness per T6/T13
- Short form validity (cases must not use supra)
The Gemini audit MUST cover every footnote, not a subset. Auditing only "major" or "flagged" footnotes guarantees missed errors.
Previous failure: Auditing 45 of 239 footnotes missed 41 journal names needing small caps.
Stage 2c: Claude Cross-Footnote Review
Never trust a single agent. After Gemini's per-footnote audit, Claude reviews the full set for cross-footnote patterns that per-footnote analysis misses.
Claude receives:
- ALL formatted footnotes (fits in 1M context at ~20-40K tokens)
- Gemini's per-footnote findings
- Mechanical check results
Claude reviews for:
- Supra chain validity — does
supra note 42actually point to the right source? - Hereinafter consistency — defined at first citation? Used consistently after?
- Repeated source type errors — if Gemini misclassified one SEC release, it likely missed them all
- Id. chain context — predecessor footnote analysis that per-footnote Gemini can't see
- Gemini false positive filtering — flag Gemini suggestions that are likely wrong (SEC releases, exec orders, working papers)
Output: Annotated version of Gemini findings with cross-footnote issues added and false positives flagged.
Red Flags
- Sending plain text to Gemini → always include inline markup; plain text produces 10-20x false positives without formatting info.
- Auditing a subset of footnotes → audit ALL footnotes; subsets guarantee missed errors.
- Skipping NBSP variants in mechanical checks → always try both space types; NBSPs cause silent search failures.
- Trusting Gemini results without the Claude cross-check → always run Stage 2c; per-footnote review misses cross-footnote patterns.
- Trusting Claude review without mechanical checks → mechanical checks are authoritative for their categories; Claude misses deterministic patterns.
- Skipping any stage → run all three (mechanical → Gemini → Claude); each catches a different error class.
Merging Three-Layer Findings
Priority order: Mechanical > Claude cross-review > Gemini per-footnote
Mechanical checks are authoritative for deterministic rules:
- Signal italic formatting → trust mechanical checker (regex on run-level XML)
- Terminal periods → trust mechanical checker
- Id. chain validation → trust mechanical checker
- Journal/book small caps patterns → trust mechanical checker
Claude cross-review is authoritative for:
- Cross-footnote consistency (supra chains, hereinafter definitions)
- Gemini false positive filtering (source type misclassifications)
- Patterns across footnotes that per-footnote analysis misses
Gemini per-footnote is authoritative for:
- Individual source type classification (when not overridden by Claude)
- Abbreviation correctness (T6/T13 tables)
- Short form validity
Never drop a mechanical finding because Gemini or Claude didn't flag it. The mechanical checker catches 100% of signal issues by design.
Gate: Exit Check
Before proceeding to Report phase:
-
scratch/audit_findings.jsonexists - Mechanical check results cover ALL footnotes
- Gemini audit results cover ALL footnotes (verify count matches extract)
- Claude cross-footnote review complete
- Findings merged (mechanical > Claude > Gemini priority)
-
bib_gaps.pyrun FIRST; any emitted entries reviewed and merged -
scan_formatting.py's journal list reconciled against the bib'sjournalvalues (see below)
Order matters. bib_gaps.py is what keeps sources.bib current, and the
small-caps check is only as good as the list behind it. Run the gap check first
and the bib supplies the article periodicals; run it last and the small-caps
check is validating against a record that has already fallen behind — which is
how J. Corp. L. survived a full audit.
Note what the bib does NOT cover: its journal field exists only on @article
entries, so newspapers, magazines, blogs and BOOK titles are absent — 23 of this
document's 54 small-capped names. Those stay the name list's job.
Next Phase
Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-report/SKILL.md and follow its instructions.
Phase 4: Correct
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Apply approved corrections to the DOCX file via lxml XML manipulation.
What This Phase Does
- Back up the original DOCX
- Apply corrections in order:
a. Cross-reference resolution (fill
[_]placeholders) b. Small caps for journal/periodical names (run-splitting) c. Small caps for book titles (italic -> small caps) d. Signal italic fixes e. Id. chain corrections f. Terminal period additions g. Other typeface fixes - Write corrected DOCX
Run-Splitting Approach
When formatting a substring within a larger run:
- Find the run containing the target text
- Split into 3 runs: prefix (original) + target (new format) + suffix (original)
- Clone
rPrfrom original run viadeepcopy - Add new formatting only to target run
- Set
xml:space="preserve"on all<w:t>elements
Critical Gotchas
NBSP Variants
All search operations MUST handle \xa0 (non-breaking space):
def find_in_run(text, target):
if target in text:
return text.find(target)
nbsp_target = target.replace(' ', '\xa0')
if nbsp_target in text:
return text.find(nbsp_target)
# Try regex with [\s\xa0] for mixed
import re
pattern = re.escape(target).replace(r'\ ', r'[\s\xa0]')
m = re.search(pattern, text)
return m.start() if m else -1
footnoteRef Space Run Bug
The run after <w:footnoteRef/> often contains the full footnote text, not just a space. When replacing entire footnote content, keep ONLY the footnoteRef run and add an explicit space run.
Cross-Run Text
supra note 10 spans italic + roman runs. Target the specific run containing the text you need to change (e.g., just "note 10" in the roman run).
Multi-Split Footnotes
Some footnotes need multiple formatting changes in the same run (e.g., FN91: italic the letter title AND small-caps the annual report title, both in one roman run). Process splits sequentially left-to-right:
- First split creates 3 new runs from the original
- Second split finds target text in one of the NEW runs and splits again
- The
find_run()search re-scans the footnote element each time, so it finds the new runs
Example: Jamie Dimon, Chairman & CEO Letter to Shareholders, in JPMorgan Chase & Co., 2023 Annual Report 1 (2024)
- Split 1: "Chairman & CEO Letter to Shareholders" → italic (creates 3 runs)
- Split 2: "JPMorgan Chase & Co., 2023 Annual Report" → small caps (splits the third run from Split 1)
Italic Spillover Cleanup
After all substantive fixes, clean up trailing/leading spaces in italic runs. Word displays these fine, but they cause Gemini annotation issues on re-audit:
# Find italic runs with trailing spaces
if text.endswith(' ') and is_italic:
t.text = text.rstrip(' ')
# Insert a new roman space run after
## Iron Law: Verify Every Fix
After each category of corrections, verify the fix was applied by reading back the modified XML. Silent failures from NBSP, run boundaries, or wrong-run targeting are common.
Skipping read-back verification is NOT HELPFUL — silent failures from NBSP or run boundaries mean the user's document still has errors.
Correction Facts
- NBSP characters and run boundaries cause silent failures and wrong-run targeting — code that "applied" is not code that worked. A fix reported without XML read-back is an unverified claim presented as a correction.
- Verification batched to the end can't attribute a failure to the fix that caused it; verify after each category.
- Every footnote has a unique run structure — a pattern that worked on the last footnote proves nothing about this one.
- One XML corruption destroys the whole document; skipping the backup gambles the user's manuscript to save seconds.
Gate: Exit Correct
Before proceeding to Verify phase:
- Backup DOCX exists (original preserved)
- Corrected DOCX exists
- Per-category fix counts logged
- Spot-check verification passed (at least 5 random fixes checked)
Next Phase
Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-verify/SKILL.md and follow its instructions.
Phase 7: Cross-References
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Convert hardcoded supra/infra note numbers to NOTEREF field codes that auto-update when footnotes are renumbered, then tie each cross-reference to a bibkey from the document project's references/sources.bib so the bibliography becomes the semantic identity layer.
Canonical 3-script pipeline
sources.bib ← make_bib_from_docx.py # BOOTSTRAP (once per paper, Gemini)
↓
docx ← create_crossrefs.py # ONE-TIME conversion of hand-typed text
↓
docx ← audit_crossref_targets.py --grep --apply # DRIFT CORRECTION (deterministic)
↓
docx ← bib_integrate.py --bib references/sources.bib # BIBKEY-TAG SYNC (deterministic)
(also pre-computes cached display values)
All four live in ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/.
Maintenance one-liner
After a Word session that added/edited footnotes (including new hand-typed
<X>, supra note N text), run:
"${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/sync_crossrefs.sh" \
draft.docx references/sources.bib
The shell wrapper runs create_crossrefs → audit_crossref_targets → bib_integrate in sequence. All three are idempotent and fully deterministic — re-running on a clean doc does nothing; ~5 seconds total runtime.
| Stage | Script | Gemini? | When |
|---|---|---|---|
| Bootstrap | make_bib_from_docx.py | Yes (one-off batch) | Paper has hand-typed footnotes and no sources.bib yet |
| Initial conversion | create_crossrefs.py | No | Convert hardcoded supra note N text → NOTEREF fields |
| Drift correction | audit_crossref_targets.py --grep | No | After footnote edits — fix any supras whose target drifted |
| Drift correction (ambiguous) | audit_crossref_targets.py --batch | Yes | Only for refs grep can't resolve uniquely |
| Bibkey-tag sync | bib_integrate.py | No | After cross-refs settle — rename bookmarks to _RefBib_<bibkey> and pre-compute cached display |
Initial conversion (one-time)
Convert hardcoded <X>, supra note N text to NOTEREF cross-reference fields.
# Preview
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/create_crossrefs.py" --docx <path> --dry-run
# Apply
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/create_crossrefs.py" --docx <path>
Workflow
- Bootstrap the bib (only if no
references/sources.bibyet):
Walks the docx footnotes, skips supra-only / bio fns, sends each first-cite candidate to Gemini Vertex Batch, emits BibTeX withuv run --with lxml --with google-genai --with google-cloud-storage python3 \ "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/make_bib_from_docx.py" \ --docx <path> --out references/sources.bibnote = {fnN}linking each entry to its source footnote. - Dry run create_crossrefs — review the cross-reference map and bookmark plan
- Apply create_crossrefs — the script backs up the original before writing
- Audit + retarget — run
audit_crossref_targets.py --grep --apply(see below). Hand-typedsupra note Nnumbers go stale as footnotes are added or reordered; the create_crossrefs script faithfully bookmarks the literal N, which can be silently wrong for every reference. Grep resolves the easy cases deterministically; only ambiguous refs need--batch. - Bibkey-tag — run
bib_integrate.py --bib references/sources.bib --apply. Renames bookmarks to_RefBib_<bibkey>, retargets supras by bibkey, and pre-computes cached display values so the doc renders correctly on first read (no F9 needed). - Verify in Word — open the DOCX, spot-check 5-10 supras. Cmd+A →
fn+F9(Mac) orCtrl+A → F9(Windows) to force-refresh fields if needed. - Renumber test (optional) — add a footnote before a referenced target and confirm the supra numbers update.
audit_crossref_targets.py (drift correction, also runnable standalone)
# Mechanical audit only — flags refs whose surnames don't appear in the target footnote
uv run --with lxml python3 \
"${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/audit_crossref_targets.py" \
--docx <path>
# Deterministic first pass — grep each reference for its first cite. No LLM cost.
# On a 138-ref test set: resolved 62 refs unique with 98% accuracy, deferred
# 76 truly ambiguous cases. Always run this BEFORE invoking Gemini/Batch — it
# strips the easy cases and saves LLM calls.
uv run --with lxml python3 \
"${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/audit_crossref_targets.py" \
--docx <path> --grep --apply
# Grep + production LLM batch for the residue (recommended for full pipeline)
uv run --with lxml --with google-genai --with google-cloud-storage python3 \
"${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/audit_crossref_targets.py" \
--docx <path> --grep --batch --apply --location global
# LLM-only paths (skip grep — coverage-risky for --gemini):
uv run … --docx <path> --batch --apply # --model overrides the resolved role
uv run … --docx <path> --gemini --apply
bib_integrate.py (bibkey-tag sync, fully deterministic)
uv run --with lxml python3 \
"${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/bib_integrate.py" \
--docx <path> --bib references/sources.bib --apply
What it does:
- Parses
sources.bib. - For each bib entry, derives signature tokens (author surnames + distinctive title words + institutional bibkey prefix) and finds the docx footnote whose body text contains all of them. This re-derives the
fn_id → bibkeymap from current docx content — sidesteps any drift between bib'snote={fnN}tag and current numbering. - For each supra/infra NOTEREF, picks the best matching bibkey (target-fn bonus + global token scoring).
- Creates
_RefBib_<bibkey>bookmarks wrapping body footnoteReferences. Multi-cite footnotes get multiple bookmarks (one per source) — each pointing at the same body footnoteReference, so the displayed number stays the same; only the bookmark name differs by source. - Rewrites each NOTEREF instr text to point at its bibkey-named bookmark.
- Pre-computes cached NOTEREF display values (using
--bio-count, default 3 for*, †, ‡author bios that don't consume display numbers whennumRestart=eachSectis set). - Emits
SUPRA_BIB_AUDIT.mdlisting any supras whose surnames don't match the picked bibkey's haystack (author + title + howpublished + bibkey).
Fully deterministic — no LLM calls. Idempotent on re-runs.
make_bib_from_docx.py (bootstrap, Gemini)
uv run --with lxml --with google-genai --with google-cloud-storage python3 \
"${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/make_bib_from_docx.py" \
--docx <path> --out references/sources.bib
Walks the docx footnotes, splits multi-cite footnotes on ;, sends each first-cite candidate to a Vertex AI Batch job (one independent request per citation; the bulk role by default). Emits BibTeX entries with bibkey conventions firstauthorlastYEAR for academic works and short slugs (gao2017, crs2024, secReg2020) for institutional sources.
Each entry includes note = {fnN} linking back to the source footnote.
Once sources.bib exists, maintain it directly — do not regenerate. Edits to the bib (typos, missing fields, new sources) should be in-place.
Vertex AI batch requires:
gcloud auth application-default loginfor ADC--project(default$GOOGLE_CLOUD_PROJECToractivist-defense-nal)--gcs-bucket(default$GEMINI_BATCH_BUCKETornal-batch-extraction)- Location:
us-central1(default)
For gemini-3.x models, thinkingLevel: MINIMAL is set automatically — without it, batch responses silently return empty content (see /gemini-vertex SKILL gotcha 12).
Output written to <docx-dir>/scratch/:
crossref_audit.json— every cross-reference with current target + match statuscrossref_remap.json— Gemini's proposed corrections (with confidence)CROSSREF_AUDIT.md— human-readable diff report
The audit is safe to run standalone without re-running the rest of the bluebook-audit workflow. Use it any time create_crossrefs has been run and you want to validate that hand-typed supra/infra numbers were correct.
Recommended pipeline order
--grepfirst. Cheapest, deterministic, ~98% precision onuniquematches. Resolves ~half the refs at $0 cost.--batch(Vertex AI, per-ref independence) on the deferred set.- Manual first-cite verification on remaining mechanical flags (Step 4 below).
How --grep works
A cross-reference reads <short-form identifier>, supra/infra note N. The
identifier can be any Bluebook short form, not just author surnames:
- author group:
Hu, Malenko & Zytnick - hereinafter tag:
GAO Report,CRS Report,Rosenberg,2020 SEC Regulation - institutional doc:
Best Practice Principles,Policies and Procedures,Senate Banking Committee Letter - case name:
ISS v. SEC - title fragment:
Choi, Fisch & Kahan, Power of Proxy Advisors
For each reference the script:
- Tokenizes the identifier (filters Bluebook signals like
See,But see,E.g.). - Strips every
<X>, supra/infra note <N>phrase from each candidate footnote, so a footnote that merely cites the same identifier via supra doesn't count as the first cite. - Searches every footnote — except the source — for one whose post-supra-strip text contains all the identifier tokens with word-boundary matches.
- If exactly one footnote matches →
unique, apply deterministically. - If multiple match →
ambiguous, defer to LLM (picking the earliest is wrong ~17% of the time — typically when the identifier appears in passing in one footnote and as a first cite in another). - If none match →
no_match, defer to LLM (which has the catalog and can guess, or correctly say "not in document").
Step 4 (mandatory): Manual first-cite verification on residuals
Iron Law: Neither Gemini pass is fully trustworthy on its own. A two-batch consensus is high-confidence, but the residual disagreements ALWAYS require human first-cite verification. Do not stop after --apply.
The failure modes we observed on a real 248-footnote draft:
- Single-batched call: model returns uniform "medium" confidence — a smoking-gun signal it didn't reason per-reference.
- Per-reference Vertex Batch: model returns uniform "high" with detailed reasoning, but still confidently picks wrong targets when:
- the cited work isn't in the doc (model hallucinates a plausible-sounding nearby match)
- the first cite is buried mid-footnote in a
See alsoor multi-cite list (model misses it and picks a different paper by the same author) - the reference is to a self-referencing pattern (model returns the source footnote itself)
- Mechanical surname check: only inspects the leading 250 chars of the target footnote, so first cites buried after
See/hereinafter/inline-discussion lead-ins are flagged as mismatches even when correct.
Recipe
- Compare both Gemini passes, taking the intersection where they agree as the high-confidence consensus. Apply that subset first.
- For each disagreement, search the doc for the first non-supra cite of the disputed surname / title fragment. The script's regex below catches mid-footnote occurrences:
for fn in fns.findall('.//w:footnote'): txt = ''.join(t.text or '' for t in fn.iter('w:t')) if re.search(r'<surname-pattern>', txt) and 'supra' not in txt[:300].lower(): print(fn.get('w:id'), txt[:200]) - Build a manual remap of
(source_fn, surnames, current_bookmark) → correct_fn_id, save ascrossref_remap_manual.json, and apply with--remap PATH --apply. - For the "no first cite in document" residue (works the author cites without ever having introduced):
- Search broadly first — the first cite is often mid-footnote in a
See also/multi-cite list. - Only if truly absent: flag for an author edit. Write
SUPRA_UNRESOLVED.mdlisting the source FN, surnames, and the search patterns tried.
- Search broadly first — the first cite is often mid-footnote in a
- Final re-audit will still show "mechanical mismatches" for the false-positive cases — verify each by spot-checking the target footnote content. Do not retarget these.
Gate
- Consensus retargets applied (intersection of
--geminiand--batch) - Manual remap built and applied for disagreements
- Each surviving "mismatch" flag spot-checked against actual target content
- No supra/infra reference points at an unrelated footnote
-
SUPRA_UNRESOLVED.mdwritten (or empty if all resolved)
What Gets Converted
| Pattern | Example | Result |
|---|---|---|
| Single supra | supra note 42 | NOTEREF to FN42 bookmark |
| Single infra | infra note 188 | NOTEREF to FN188 bookmark |
| Range | infra notes 209-210 | Two NOTEREFs with separator |
| With pincite | supra note 42, at 15 | NOTEREF + roman , at 15 |
| Existing NOTEREF | (already converted) | Skipped |
What Is NOT Converted (Phase 2 — Future)
- Part/Section references (
supra Section I.A.,infra Part III) - These require a heading-to-bookmark mapping strategy since heading numbering is partially auto-generated
Gate: Exit Cross-References
- Dry run reviewed — cross-reference map is correct
- NOTEREF fields created for all supra/infra note references
- Target audit passed —
audit_crossref_targets.pyrun + Step 4 manual verification complete; no NOTEREF points at an unrelated footnote - Backup DOCX exists
- Word field update (Ctrl+A, F9) confirms correct numbers
Workflow Complete
Present final summary to user:
- Total formatting corrections applied (from Phase 4)
- Total URLs archived (from Phase 6)
- Total NOTEREF fields created (from Phase 7)
- Final DOCX file path
Phase 1: Extract Footnotes
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Parse the DOCX file and build structured data for all subsequent phases.
What This Phase Does
- Parse
word/footnotes.xmlvia lxml - Extract each footnote's runs with formatting flags (italic, small caps, bold)
- Parse
word/_rels/footnotes.xml.relsfor hyperlink URLs - Build citation registry (hereinafter definitions, author-to-first-cite mapping)
- Resolve cross-references (
supra note [_]placeholders) - Extract all URLs for archiving inventory
Script
uv run python3 "${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/scripts/extract_footnotes.py" --docx <path>
Output: scratch/footnotes_data.json
Gate: Exit Extract
Before proceeding to Check phase:
-
scratch/footnotes_data.jsonexists - Contains entries for ALL footnotes in the document (verify count)
- Each entry has
formatted_textfield with inline markup - Citation registry has hereinafter definitions
- URL inventory extracted
If footnote count doesn't match document: STOP. Investigate missing footnotes before proceeding.
Next Phase
Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-check/SKILL.md and follow its instructions.
Phase 3: Report
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Generate a human-readable audit report and present to the user for review before applying corrections.
What This Phase Does
- Merge three-layer findings: mechanical + Gemini per-footnote + Claude cross-footnote review (mechanical > Claude > Gemini priority — see audit-check merge rules)
- Categorize by issue type and severity
- Flag items needing manual review (low-confidence cross-refs, ambiguous citations)
- Generate
scratch/AUDIT_REPORT.md - Present summary to user
Three-Layer Merge Priority
Mechanical > Claude cross-review > Gemini per-footnote
- Mechanical findings (signal italic, terminal periods, Id. chains, small caps patterns) are deterministic and must never be dropped
- Claude cross-review adds cross-footnote patterns (supra chains, hereinafter consistency) and filters Gemini false positives
- Gemini per-footnote adds individual source type classification and abbreviation checks
Report Structure
# Bluebook Audit Report
## Summary
- Total footnotes: N
- Clean: N (XX%)
- Issues found: N across M footnotes
## Fix Counts by Category
| Category | Count | Auto-fixable |
|----------|-------|-------------|
| Journal name small caps | N | Yes |
| Book title small caps | N | Yes |
| Cross-reference resolution | N | Yes (high confidence) |
| Id. chain errors | N | Partial |
| Signal formatting | N | Yes |
| Terminal periods | N | Yes |
| Typeface errors (other) | N | Manual |
## Issues by Footnote
[sorted by footnote number]
## Items Needing Manual Review
[low-confidence cross-refs, ambiguous citations, judgment calls]
## Correct As-Is (Gemini False Positives)
[Items Gemini flagged but are actually correct, with reasoning]
[Group by source type: SEC releases (roman), exec orders (roman), etc.]
[Reference: audit-patterns.md Source Type Typeface Reference table]
Why "Correct As-Is" Matters
Many Gemini suggestions are wrong — especially for non-standard source types (SEC releases, exec orders, working paper designations). Documenting WHY these are correct:
- Prevents re-flagging if someone re-runs the audit
- Forces the reviewer to consciously evaluate each judgment call
- Creates a record of the source type classification decisions
Gate: Exit Report
This is a user gate. The workflow pauses here.
-
scratch/AUDIT_REPORT.mdexists - User has reviewed the report
- User approves proceeding to corrections
Do NOT proceed to corrections without user acknowledgment.
Next Phase
After user approval:
Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-correct/SKILL.md and follow its instructions.
Phase 5: Verify
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Re-scan the corrected DOCX to confirm all fixes were applied and no new issues were introduced.
What This Phase Does
- Re-extract all footnotes from the corrected DOCX
- Re-run mechanical checks (small caps scanner, signal checker, etc.)
- Compare findings against the original audit - all flagged issues should now be clean
- Report any remaining issues
Verification Checks
- Formatting scanner: Re-run
scan_formatting.py --docx corrected.docx- expect 0 findings (or only known false positives like "Institutional Investors" in titles) - Cross-reference check: All
[_]placeholders should be resolved - Signal formatting: All signals should be italic
- Terminal periods: All footnotes should end with periods
- Id. chains: All id. references should have single-source predecessors
- Gemini re-audit on fixed footnotes: Re-run
gemini_audit.py --subset [all previously flagged FNs]to catch issues introduced by fixes or missed in the initial pass
Re-Scan Catches Real Issues
In the "Other People's Votes" audit, the Gemini re-scan on fixed footnotes caught a Wells Fargo press release title (FN206) that wasn't in the original "judgment call" list — it was in the same footnote as other fixes but wasn't flagged initially. The verify phase is not ceremonial; it finds real issues.
## Iron Law: Re-Scan Is Not OptionalThe verify phase MUST re-run the scanner on the corrected DOCX. Skipping verification misses small-caps fixes.
If the re-scan finds issues, go back to Correct phase. Do NOT proceed to Archive with unresolved issues.
Red Flags
- Skipping the re-scan because "all fixes applied" → run the scanner; silent fix failures are common.
- Dismissing remaining findings as false positives → investigate each one; some "false positives" are real.
- Proceeding to Archive with >0 real issues → fix them first; uncorrected issues persist forever.
Gate: Exit Verify
Before proceeding to Archive phase:
- Re-scan completed on corrected DOCX
- Zero remaining issues (or all remaining are confirmed false positives)
- If issues found: returned to Correct phase and re-verified
Next Phase
Read ${CLAUDE_SKILL_DIR}/../../../../skills/bluebook-audit/skills/audit-archive/SKILL.md and follow its instructions.
Bluebook Citation (22nd edition; 21st still reachable)
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Citation formatting for law reviews and legal scholarship per The Bluebook: A Uniform System of Citation (21st ed. 2020).
Announce: "I’m using the bluebook skill for citation formatting."
When to Use
Invoke this skill for:
- Formatting case citations (federal, state, foreign)
- Statutory and regulatory citations
- Secondary sources (books, articles, treatises)
- Short form citations (id., supra, hereinafter)
- Introductory signals and parentheticals
- Citation sentences vs. citation clauses
For legal writing style: Use /writing-legal skill (Volokh)
For general writing: Use /writing skill (Strunk & White)
To RENDER these rules in a Typst manuscript: this skill states the rules; it does
not implement them. docx-typst carries the implementation — assets/bluebook.typ is a
#show cite: rule supplying the three things typst's built-in bibliography cannot
(supra note N, small-caps reporters, shortjournal → container-title-short), because
hayagriva is statically linked into the typst binary and renders none of them. Reach for
it whenever a Typst document needs the short forms in references/short-forms.md to
renumber themselves rather than be typed by hand.
If you haven’t verified EVERY element of a citation, DO NOT write it.
Before writing ANY citation:
- Verify case name spelling and procedural posture
- Verify reporter volume and page numbers
- Verify court and year
- Verify pinpoint page exists
Guessing reporter volumes or page numbers is NOT HELPFUL — the user publishes with wrong citations that fail verification. Period.
## IRON LAW #2: NO SHORT FORMS WITHOUT FULL CITATION FIRSTId., supra, and hereinafter REQUIRE a preceding full citation.
Before using ANY short form:
- Locate the full citation in the document
- Verify no intervening citations (for id.)
- Verify the supra reference is unambiguous
Using id. after intervening citations creates ambiguity. Delete and cite in full.
## IRON LAW #3: FOOTNOTE VS. TEXT CITATION FORMATLaw review citations use footnote format (Rule 1). Court documents use text format (Bluepages).
FOOTNOTE (law reviews): Smith v. Jones, 500 U.S. 1, 5 (1991).
TEXT (court documents): Smith v. Jones, 500 U.S. 1, 5 (1991)
FOOTNOTE (statutes): 18 U.S.C. § 1001 (2018).
TEXT (statutes): 18 U.S.C. § 1001 (2018)
If writing for a law review and using text format conventions, DELETE and reformat.
The Gate Function
Before writing ANY citation:
1. IDENTIFY → What type of source? (case, statute, article, book)
2. LOCATE → Find the correct rule in Bluebook
3. VERIFY → Confirm ALL elements (volume, page, court, year)
4. FORMAT → Apply correct typeface and punctuation
5. CHECK → Does this match examples in the rule?
6. WRITE → Only after steps 1-5
Skipping any step produces unreliable citations.
Citation Facts
- An intervening citation breaks id. — id. after an intervening cite is ambiguous and must become a full short form. Supra only works when the full citation it points to actually exists earlier in the document.
- Signals are checked against Rule 1.2 examples, not intuition — a wrong signal misleads the reader about how the source supports the proposition.
- Parentheticals explain the source's relevance; pinpoints prove the specific claim. A cite deferred ("I'll add the pinpoint later") ships without one.
- Typeface (Rule 2) is mandatory, not stylistic. Abbreviations come from tables T6, T10, T12 — "common" or "obvious" abbreviations that don't match the tables fail cite-check.
- "Pretty sure" about a reporter volume or page number means unverified — a guessed element presented as a citation is an unverified claim, and exact pinpoints are required.
Quick Reference: Common Citation Forms
Cases (Rule 10)
Full citation:
Brown v. Board of Education, 347 U.S. 483, 495 (1954).
Short form (same footnote or five footnotes with no intervening):
Id. at 496.
Short form (different footnote, no intervening):
Brown, 347 U.S. at 497.
Short form (intervening citations):
Brown v. Board of Education, 347 U.S. at 498.
Statutes (Rule 12)
Full citation:
42 U.S.C. § 1983 (2018).
Multiple sections:
42 U.S.C. §§ 1983-1985 (2018).
Short form:
§ 1983 or id. § 1984
Law Review Articles (Rule 16)
Full citation:
Cass R. Sunstein, *On the Expressive Function of Law*, 144 U. Pa. L. Rev. 2021, 2030 (1996).
Short form:
Sunstein, supra note 12, at 2035.
Books (Rule 15)
Full citation:
Richard A. Posner, Economic Analysis of Law 45 (9th ed. 2014).
Short form:
Posner, supra note 5, at 52.
Typeface Rules (Rule 2)
| Source Type | Law Review Format |
|---|---|
| Case names | Italics: Brown v. Board |
| Book titles | Small caps: ECONOMIC ANALYSIS OF LAW |
| Article titles | Italics: On the Expressive Function |
| Journal names | Small caps: U. PA. L. REV. |
| Periodical names (non-consecutively paginated) | Italics: N.Y. Times |
| Statutes | Roman: 42 U.S.C. § 1983 |
Introductory Signals (Rule 1.2)
| Signal | Meaning | Use When |
|---|---|---|
| [no signal] | Direct support | Source directly states proposition |
| See | Implicit support | Source supports but doesn’t directly state |
| See, e.g., | One of several | Multiple sources support; citing representative |
| Cf. | Analogous support | Source supports by analogy |
| Compare ... with | Comparison | Sources illustrate through contrast |
| See generally | Background | Source provides helpful background |
| But see | Contradiction | Source contradicts proposition |
| Contra | Direct contradiction | Source directly contradicts |
Signal Order (Rule 1.3)
Within a single citation sentence, signals appear in this order:
- [no signal]
- E.g.,
- Accord
- See
- See also
- Cf.
- Compare
- Contra
- But see
- But cf.
- See generally
Common Errors Checklist
Case Citations
- Party names shortened properly (omit "Inc.", "Ltd." unless only identifier)
- "United States" abbreviated to "U.S." (as party, not "United States of America")
- Reporter abbreviation matches T1
- Court identifier included unless obvious from reporter
- Year is decision year, not argument year
- Pinpoint included for specific propositions
Statutory Citations
- Current official code used (not session laws for current statutes)
- Section symbol (§) used, not "Section"
- Space between § and number
- Year is code edition year, not enactment year
- Supplements cited when applicable
Short Forms
- Full citation appears earlier in same document
- Id. used only when no intervening citation
- Supra refers to footnote number where full cite appears
- Hereinafter defined in first full citation
Progressive Disclosure
For detailed rules, consult:
Reference Files
references/cases.md- Complete case citation rules (R. 10)references/statutes.md- Statutory and regulatory citations (R. 12-14)references/secondary-sources.md- Books, articles, treatises (R. 15-17)references/short-forms.md- Id., supra, hereinafter rules (R. 4)references/quotations.md- Block quotes, alterations, ellipses (R. 5)references/signals-parentheticals.md- Signals, parentheticals, order (R. 1)references/audit-patterns.md- Citation audit patterns and validationreferences/abbreviations.md- Bluebook abbreviation tablesreferences/editions-21-to-22.md- What changed in the 22nd edition, from the publisher's preface plus rule-by-rule checks
Every reference file below is checked against the rule text. editions-21-to-22.md is
not in the table because it is the SOURCE that verification was done against, not a subject
of it.
| file | rules | status |
|---|---|---|
quotations.md | 5 | verified vs 21e scan AND 22e; 6 errors corrected |
signals-parentheticals.md | 1 | rebuilt from 22e; carries the new contrast signal |
short-forms.md | 4 | rebuilt from 22e |
cases.md | 10 | rebuilt from 22e |
statutes.md | 12 (13-14 NOT in corpus) | rebuilt from 22e; 13-14 flagged in place |
secondary-sources.md | 15, 16, 17 | rebuilt from 22e |
abbreviations.md | 10.2.2, 15.1(e), 16.1, T6 | rebuilt from 22e; T6 reproduced from the capture, table gaps flagged in place |
audit-patterns.md | cross-cutting (1, 4, 10, 12, 15-18, T6) | rebuilt from 22e; every check names its rule, untraceable checks marked in place |
Five were then adversarially re-verified by separate agents against the same corpus, with
instructions to find errors rather than agree: signals-parentheticals.md and short-forms.md
came back with zero contradictions (every imperative traced to a governing sentence);
cases.md, statutes.md and secondary-sources.md each had defects, all now corrected.
quotations.md, abbreviations.md and audit-patterns.md had no second pass. Reports:
scratch/bb22/verify/.
All eight were checked against subsection pages extracted verbatim from the official Bluebook
Online 22nd edition; page cites in them are 22e pages. Claims the corpus did not cover are
marked UNVERIFIED in place rather than left looking checked. Per-file reports:
scratch/bb22/reports/. Rule 5 verdicts: scratch/bluebook-verify/REPORT.md.
What is still unchecked, and why. "Rebuilt from 22e" means every claim traces to the extracted corpus or is marked in place — it does not mean the whole rule surface was available. These gaps are real and are flagged inside the files rather than papered over:
| gap | state of the capture | consequence |
|---|---|---|
| Table T10 (geographical terms) | captured in full — T10.1 (22e pp. 340-42), T10.2 and T10.3 as separate subtable pages | its values may be stated |
| Table T13 (institutional names in periodical titles, 22e pp. 346-48) | captured in full — the earlier empty capture used a wrong slug (t13-institutional-names-... rather than t13-periodicals) | its values may be stated |
| Tables T1, T7, T11, T12 and the rest | never fetched | cross-references to them are reported; their contents are not |
| Rules 6, 7, 8, 9, 11, 19 | never fetched | checks resting on them are marked, not asserted. Rules 3, 5, 20, 21 and 23 have since been extracted |
Nothing from these gaps has been filled from training knowledge. A claim depending on one is either absent or carries a marker within a line of itself.
Looking a rule up in the actual book
Best source: the official Bluebook Online, 22nd edition. UVA Law provides institutional access and the browser on CDP 9222 is already signed in. This is live publisher text — no OCR, no auth expiry, no retrieval tricks — and it is the source of record.
https://www.legalbluebook.com/bluebook/v22/rules/<n>-<slug>
https://www.legalbluebook.com/bluebook/v22/rules/<n>-<slug>/<n>-<m>-<slug>
e.g. /bluebook/v22/rules/5-quotations/5-2-alterations-and-quotations-within-quotations
Swap v22 for v21 to read the 21st edition; the site's own dropdown does the same. Drive it
with the browser-automation skill (Linux → mcp__chrome-devtools__* on 9222). If that MCP
server is not connected this session, raw CDP over http://127.0.0.1:9222/json/list works — and
open a new tab rather than navigating the user's.
EDITION MATTERS, IN TWO WAYS. The 22nd is now live and is the default.
references/editions-21-to-22.md is the guide — the publisher's preface (which names changes
in rules 1.2, 10.8.3, 12.4(f), 14.4, 15.1(d), 15.8, 18, 20.2.4, new 22 and 23, and tables T1.3,
T1.5, T2, T6, T10) plus the rule-by-rule differences verified here. Read it before trusting
anything in this skill against a 22e manuscript. Three things that bite immediately:
- Rules 22 and 23 are NEW; the 21st stops at 21. Rules 1-21 keep their numbers.
- Pagination moved — Rule 5 is 22e pp. 87-91 against 21e-scan pp. 103-108. A bare page cite is ambiguous; name the edition.
- The preface's list is partial. Rule 5.1(a)(i) gained "single spaced" in the 22nd and the preface never mentions it. Absence from the preface is not evidence a rule is unchanged.
Pinpoint — the 21st edition, full scan
Pinpoint holds the full 21st edition — use it. A complete 394-page scan is in the Bluebook
collection (b7425c3f3368f9c9), OCR'd by Google, searchable to the page. This is the authoritative
lookup for any rule in this skill, and it is what references/quotations.md was finally verified
against.
pinpoint search Bluebook "<terms>" --pages --order density --limit 60
pinpoint generate ask Bluebook "<question>" # locates fast; do NOT quote its paraphrase
Four things learned the hard way, all of which cost a pass:
--order density, not the default. Document order means a bounded run stops early — Rule 5 sits at scan pp. 103-108 and a default-ordered run never reached past p. 71.- It OR-matches; a quoted phrase returns zero. Cast wide and filter locally with
awk. --no-dedupewhen a page you know exists will not come back. Rule 1.5(b) at p. 86 returned nothing across four term sets at limits up to 120;--no-dedupe --limit 200returned it in full.- Quote from
--pages, never fromgenerate ask. It locates well and paraphrases confidently — it produced a fluent synthesis of a capitalization test that no line in the book supports.
Auth expires within the hour (API error 7 (PermissionDenied)); pinpoint auth re-lifts the
session from the CDP browser. Re-auth and retry — do not read an auth failure as "the rule is not
in the book."
NotebookLM — narrow, and mostly superseded
The notebook (f70a9976-b443-43d5-b5fd-43ff86b2b700) holds a 53-page excerpt, not the book.
Verified: the only full rule text is Rule 1 and 1.1-1.4 (pp. 51, 53) plus the Quick
Reference tables. Everything else is cover scans and cropped strips of the printed book's thumb
tabs — so "rule 10", "rule 12" appear as tab labels with nothing behind them, and a model asked
what the PDF contains will read those tabs and name rules it cannot quote. It did exactly that
before retracting under stricter questioning. It cannot answer on Rule 5 at all.
Refuse its web-research offer. On a rule it does not hold it replies "Would you like me to perform some web research?" — observed three times. Accepting turns a web search into something formatted as a source-grounded answer, which is how an unverified reference file gets written by an author who believes they consulted the book.
Prefer Pinpoint above. Reach for the notebook only for Rule 1 or the Quick Reference tables:
When to query the notebook:
- Rule wording is ambiguous in reference files
- Formatting international or specialized materials
- Checking obscure abbreviations not in quick reference
- Resolving conflicts between rules
- Understanding historical changes from previous editions
When to Load References
Load the specific reference when:
- Formatting an unfamiliar source type
- Encountering edge cases (unpublished cases, foreign sources)
- Checking state-specific reporter requirements
- Working with complex statutory schemes
- Formatting international materials
Integration
Use with /writing-legal for complete legal scholarship workflow:
/bluebookformats citations correctly/writing-legalensures argument structure and evidence handling/ai-anti-patternscatches AI writing indicators before submission
Delete & Restart Pattern
When to delete and restart:
- Citation uses guessed page numbers → Delete, verify source, cite with real numbers
- Id. follows intervening citation → Delete id., use full short form
- Wrong signal used → Delete, reread Rule 1.2, apply correct signal
- Typeface incorrect → Delete, apply Rule 2 typeface
- Abbreviation doesn’t match Bluebook tables → Delete, use table abbreviation
How to restart:
Old: See Smith v. Jones, 500 U.S. at 15. Id. at 20. [intervening cite] Id. at 25.
New: See Smith v. Jones, 500 U.S. at 15. Id. at 20. [intervening cite] Smith, 500 U.S. at 25.
The third cite cannot use id. after an intervening citation.
Citation Verification with Gemini File Search
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Scan pandoc-flavored markdown drafts for citations, upload source PDFs to a Gemini File Search store, and verify each citation is grounded in its source. Produces a structured REVIEW-CITES.md report.
Prerequisites
GOOGLE_API_KEYenv var set (Google AI Studio; on this machine:export GOOGLE_API_KEY="$(cat $GEMINI_API_KEY_FILE)")- Bun runtime
rclonewith agoogle-drive:remote configured (used to bypass Google Drive FUSE deadlocks)python3withpymupdf4llminstalled (used for PDF text extraction in passage grounding)readwiseCLI installed and authenticated (for Readwise article export in source materialization)- One or more
.bibfiles withfilefields mapping bibkeys to PDF paths (e.g., Paperpile'spaperpile.bib)
Source Materialization
Before running cite-check, materialize all sources locally:
cd ${CLAUDE_SKILL_DIR}
bun materialize-sources.ts \
--bib ~/Google\ Drive/My\ Drive/resources/Paperpile/paperpile.bib \
--bib ./references/sources.bib \
--refs ./references \
--drafts ./drafts \
--debug
This populates references/ with local copies of all cited sources:
- Paperpile PDFs → batch
rclone copyfrom Google Drive →references/<bibkey>.pdf - Readwise articles (reports, news, speeches without PDFs) → search by title, export markdown →
references/<bibkey>.md - Gaps → printed at the end for manual action (Obsidian web clipper or manual sourcing)
After materialization, cite-check operates purely locally.
Usage
cd ${CLAUDE_SKILL_DIR}
bun install # first time only
# Single bib file
bun cite-check.ts --bib ~/Google\ Drive/My\ Drive/resources/Paperpile/paperpile.bib --drafts <path-to-drafts>
# Multiple bib files (Paperpile + project-local; first bib wins on duplicate keys)
bun cite-check.ts \
--bib ~/Google\ Drive/My\ Drive/resources/Paperpile/paperpile.bib \
--bib ./references/sources.bib \
--drafts <path-to-drafts>
CLI Flags
| Flag | Required | Default | Description |
|---|---|---|---|
--bib <path> | Yes* | -- | Path to .bib file (repeatable; first wins on duplicate keys) |
--store <id> | No | auto-create | Use existing File Search store ID |
--drafts <dir> | No | ./drafts | Directory with markdown draft files |
--out <path> | No | <drafts>/REVIEW-CITES.md | Output report path |
--limit <n> | No | all | Check only first N citations (smoke test) |
--dry-run | No | false | Print prompts without querying |
--sequential | No | false | Run queries one-at-a-time instead of Batch API (default: batch) |
--retry-model <model> | No | the pro role | Retry UNSUPPORTED results with a stronger model |
--audit | No | false | Audit source availability without querying (checks Paperpile PDFs) |
--debug | No | false | Verbose logging |
*Either --bib or --store is required.
Ask Mode: Targeted Source Queries
Ask a specific question about a single source:
# Does Bebchuk2019 support a specific claim?
bun cite-check.ts ask @Bebchuk2019-uq "do expense ratios fall since 2010?" --bib paperpile.bib
# What does a source say about a topic?
bun cite-check.ts ask @Brav2022-ht "what are retail turnout rates?" --bib paperpile.bib --bib sources.bib
The ask mode uploads the single source PDF via the legacy Files API (with manifest caching, 48h TTL), queries Gemini with inline file references, and prints the answer with supporting passages to stdout. No File Search store is created and no report is generated.
Cross-Directory File Resolution
When multiple --bib files are provided, file paths are resolved across all bib directories. This handles the common case where a project-local sources.bib has file = {All Papers/...} paths that are relative to the Paperpile folder rather than the project's references/ directory. The tool tries each bib directory as a fallback when the primary path doesn't exist on disk.
How It Works
-
Extract citations from markdown using pandoc
[@bibkey]syntax -
Parse bib file to map bibkeys to PDF file paths via
filefields -
Sync the File Search Store, per file — PDFs for cited bibkeys are imported into a persistent Gemini File Search store with bibkey metadata. Google Drive FUSE paths are copied locally via
rcloneto avoid EDEADLK deadlocks. Stores persist across runs (no 48h TTL). "Changed" means the SHA-256 of the bytes of a cited source differs — not just its bibkey or path — so swapping a PDF at the same path under the same bibkey (a working paper replaced by the published version) invalidates it. A missing or unreadable source hashes to a<missing>sentinel rather than erroring; coverage is reported separately.State (
.cite-check-store.json, in the drafts directory) records a per-bibkey hash, so invalidation is surgical rather than whole-store: one swapped PDF among 46 re-imports one document, not 46. The run takes one of three paths and always logs which and why on stderr:[store] REUSE— no cited source changed; zero API calls.[store] SURGICAL— changed and removed keys have their document deleted, then changed and added keys are imported. Delete comes first, so a failed import leaves the key absent from the store and it reports NOT_IN_STORE rather than being verified against stale bytes. Documents are located by paginatingdocuments.listand matchingcustomMetadata.bibkey— neverdisplayName, which the store replaces with a random id on import.[store] FULL REBUILD— the store is deleted and everything re-imported. This is the fallback for anything that leaves the store's contents unknowable: no prior state, state written before per-key hashes existed (a one-time migration), adocuments.listfailure, a key the state claims was imported but no document carries, or a failed delete. A silent partial update is the failure mode this guards against, so the reason is always printed.
-
Query Gemini with structured prompts for each citation, using the
fileSearchtool with metadata filtering to scope each query to the relevant source documents -
Classify each citation as SUPPORTED / PARTIAL / UNSUPPORTED / NOT_IN_STORE / ERROR
-
Verify grounding — for SUPPORTED/PARTIAL results, extract source PDF text via
pymupdf4llmand run token-level LCS alignment to confirm the passage Gemini quoted actually exists in the source. Ungrounded passages are flagged[UNGROUNDED]in the report. -
Write report to REVIEW-CITES.md
Bib File Format
The --bib flag expects a .bib file where entries have a file field with a path relative to the bib file's directory. Paperpile's exported paperpile.bib follows this convention:
@article{Hu2024-bm,
author = {Edwin Hu and ...},
title = {{Custom proxy voting advice}},
file = {All Papers/H/Hu et al. 2024 - Custom proxy voting advice.pdf},
year = {2024}
}
All bib entries are parsed. Entries with a file field (~95% of Paperpile entries) are imported into the File Search store. Only sources for bibkeys that are actually cited in the drafts are imported.
Citation Features
- Bracketed
[@key]and in-text@keycitations - Locators:
[@key, p. 42] - Compound cites:
[@a; @b](queried together) - Footnote indirection: citations in
[^id]:footnote bodies - Bluebook signals:
see,cf.,see also, etc. (softens verification) - Parenthetical extraction:
[@key] (holding that X)
Output
REVIEW-CITES.md with:
- Summary counts (supported/partial/unsupported/not in store/error/ungrounded)
- Details table: status, file:line, bibkey, claim, response
[UNGROUNDED]flag on any SUPPORTED/PARTIAL result whose passage failed grounding verification
Batch Mode (Default)
By default, all citation queries are submitted as a single Gemini Batch API job using the File Search tool with metadata filtering. Each query is scoped to the relevant source documents via bibkey metadata, so there is no cross-contamination between queries.
# Default (batch)
bun cite-check.ts --bib paperpile.bib --drafts ./drafts
# Sequential (one query at a time, useful for debugging)
bun cite-check.ts --bib paperpile.bib --drafts ./drafts --sequential
The --sequential flag runs each query as an individual generateContent call instead of a batch job. This is useful for debugging or when batch jobs hit rate limits.
Audit Mode
Run --audit before checking citations to see which sources are available and which need to be added:
bun cite-check.ts --bib paperpile.bib --bib sources.bib --drafts ./drafts --audit
The audit checks each cited bibkey for PDF availability on disk (via bib file field with cross-directory resolution). Missing sources should be added to Paperpile.
Exit code is 1 if any sources are missing, 0 if all sources are available. No Gemini store is created and no queries are sent.
Passage Grounding
After Gemini returns a SUPPORTED/PARTIAL result with a supporting_passage, the tool verifies the passage actually exists in the source PDF text using token-level LCS alignment (ported from langextract's WordAligner). Two gates reject bad matches:
- Coverage gate (default 0.75): at least 75% of passage tokens must appear in the matched source span
- Density gate (default 0.33): matched tokens must be at least 33% of the source span length (rejects scattered matches)
Signal cites (see, cf., etc.) use relaxed thresholds (0.5 coverage / 0.2 density) since they only need conceptual alignment.
Grounding requires extracting text from the source PDF. This uses pymupdf4llm (via extract-pdf-text.py) which preserves document structure, footnotes, and tables as clean markdown. Extracted text is cached in <drafts>/.cite-check-text/.
The Gemini File Search API behind the primary grounding signal — store creation, metadata filtering, and the
groundingMetadatashapegrounding.tsparses — is documented inskills/gemini-vertex/references/file-search.md.
Google Drive FUSE Bypass
PDF files stored on Google Drive Desktop's FUSE mount (~/Google Drive/My Drive/) are subject to EDEADLK deadlocks when accessed concurrently or when not locally cached. The tool detects Google Drive paths — including through symlinks (e.g., references/All Papers → ~/Google Drive/.../Paperpile/All Papers) — and uses rclone copyto to fetch them to a local cache (~/.cache/cite-check-pdfs/) before upload or text extraction. Requires rclone with a google-drive: remote configured.
Architecture
cite-extract.ts -- Pure citation extraction (no I/O)
gemini.ts -- Gemini API wrapper (File Search store CRUD, query, legacy upload for ask mode, rclone FUSE bypass)
grounding.ts -- Post-hoc passage grounding (tokenizer, LCS aligner)
extract-pdf-text.py -- PDF text extraction via pymupdf4llm
materialize-sources.ts -- Copy Paperpile PDFs + Readwise articles to references/
cite-check.ts -- CLI orchestrator (extract -> import -> query -> ground -> report)
Claude Message Batches
Paid from the prepaid API credit an Ultra plan grants one linked Console org. Every action goes through
one script; the accounts it can bill are in the batch-router accounts config
($BATCH_ACCOUNTS, default ~/.config/batch-router/accounts.toml, shape in
../batch-router/references/accounts.example.toml).
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
The full run waits for: validate exit 0 on the full file, verify exit 0, the 5–10 row pilot's
results --input exit 0, and a message from Eddy approving the cost route_batch.py printed. A batch
that is wrong is wrong 100,000 times, and the credit it burns expires at cycle end with no rollover:
skipping the pilot spends the month's credit on a bad dataset.
The sequence
S=${CLAUDE_SKILL_DIR}/scripts/claude_batch.py
uv run $S accounts # which orgs have a key; free
uv run $S validate requests.jsonl # schema, custom_id, limits; free
uv run $S verify requests.jsonl --account gmail # ONE sync request at full price; prints its cost
uv run $S submit pilot.jsonl --account gmail --run # 5-10 rows first; refuses without --run
uv run $S status <batch_id> --account gmail --wait # run in the background, not a narrated poll
uv run $S results <batch_id> --account gmail --input pilot.jsonl --out pilot.out.jsonl
Input is one {"custom_id": "...", "params": {<Messages API body>}} per line. results writes the
succeeded rows and prints every errored, canceled, expired, missing or unexpected custom_id; it exits 1
unless all of them reconciled. Pick the model from the current
models overview at use time,
never from memory; the price table the script uses is in limits and prices.
Facts
- Credit: $200/month per Ultra plan to ONE linked org; covers the Claude API including Message Batches, not Claude Code. Expires at the end of each billing cycle, no rollover; included credit spends before purchased credit. Spending the soonest-expiring org first is free money; spending a purchased card balance while an idle credit expires is the waste this skill exists to stop.
- No balance API: balance and expiry show only in Console Billing > Promotional credits. Never write a balance into config; report it as UNVERIFIED unless Eddy or the Console gave a number today.
- Exhausted credit fails the request with
Your credit balance is too low to access the Anthropic API. Record that text verbatim and stop; switching to another org's key without saying so bills an account Eddy did not pick. - Keys are per org, and batches are per workspace:
status,resultsandcancelmust use the account that submitted. A UVA or NYU account withkey = "none"exits with "no key"; never create a key. - Limits: 100,000 requests or 256 MB per batch; most finish within 1h; unfinished requests expire at
24h; results kept 29 days; only
succeededrows are billed.stream,speed,fallbacksandmax_tokens: 0are rejected in batch. PDFs past the cap:uploadunder the SAME--accountassubmitand reference thefile_id(proven in a batch 2026-11-05; another org's key gets 404) — Files API. - Shared prefix: use 1-hour prompt caching (
cache_controlwithttl: "1h"); a 5-minute cache entry expires before most batch rows run. - Spend counts toward the org's monthly spend cap (resets on the 1st), and a batch can overshoot the workspace spend limit slightly.
Red flags — STOP
| About to | Do instead |
|---|---|
submit --run without a verify and a 5–10 row pilot on the same model and params | Run them; a paid full batch needs Eddy's go-ahead |
| Create an API key for the UVA or NYU org | Ask Eddy; record the key source in accounts.toml once he has made one |
Loop messages.create over many rows | Build JSONL and submit a batch: 50% off and no rate-limit fight |
Accept processing_status: ended as success | results --input; done only on exit 0, else resubmit the custom_ids it lists |
| Poll status from the model turn by turn | status --wait in a background Bash, notified on exit |
Consensus CLI
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Search Consensus.app for academic papers via the consensus CLI tool.
Binary: ~/projects/consensus-cli/consensus
Requires: a Chrome/Chromium signed in to consensus.app with CDP on port 9250 (override with CONSENSUS_CDP_PORT, or CDP_PORT for the whole CLI family).
Check: ls ~/projects/consensus-cli/consensus || echo "MISSING: consensus binary not built"
Core Command
consensus search "<query>" [options]
consensus journals [<query>] # exact journal names accepted by --journal
consensus publishers # values accepted by --publisher
Flags
| Flag | Description |
|---|---|
--n <int> | Result count (default 20, max 100) |
--type <csv> | Study types: rct,systematic,meta,non_rct,observational,lit_review,case,animal,in_vitro |
--years <range> | Year range: 2018-2024 or past N years (e.g. 5) |
--min-citations <int> | Minimum citation count |
--rank <q1|q2|q3|q4> | Journal quartile filter (SJR) |
--human | Human studies only |
--rct | Shorthand for --type rct |
--open-access | Open access papers only |
--domain <csv> | Fields of study (e.g. Medicine,Chemistry) |
--country <csv> | Country filter (e.g. USA,UK) |
--journal <name> | Restrict to one journal; repeat the flag for more. Names must match the index exactly |
--journals-file <path> | Read journal names from a file, one per line (# comments and blanks skipped) |
--publisher <name> | Restrict to one publisher; repeat for more |
--page <int> | Page number (default 0) |
--sort <field> | Client-side sort: citations (descending) |
Output Fields (per paper)
{
"title": "...",
"authors": ["..."],
"year": 2023,
"journal": "...",
"doi": "...",
"citations": 150,
"study_type": "rct",
"takeaway": "One-sentence finding...",
"open_access_pdf_url": "https://... or null",
"url": "https://consensus.app/papers/..."
}
Journal Filtering — the primary quality gate
File: ${CLAUDE_PLUGIN_ROOT}/references/trusted-journals.local.md
A shared resource — the google-scholar and research skills and the
librarian agent read the same file. It is the user's curated list of trusted
journals, one exact name per line, with # comments. It is the argument to --journals-file — pass the path
directly, do not re-type the names:
consensus search "<topic>" --n 50 --sort citations \
--journals-file ~/projects/workflows/references/trusted-journals.local.md
When the user asks for "journals I like", "relevant journals only", "top
journals", or a field they clearly work in, filter SERVER-SIDE with
--journals-file (or --journal for a narrower subset). Server-side
filtering means all N results are from trusted venues, instead of filtering a
mixed result set down to two or three afterwards.
For a narrower cut, pass the subset explicitly — one flag per journal, since journal names contain commas:
consensus search "insider trading enforcement" --n 20 \
--journal "Journal of Finance" \
--journal "Journal of Financial Economics" \
--journal "Review of Financial Studies"
Adding a journal to the list
A journal name that is not in Consensus's index silently matches nothing — it does not error. Verify before adding:
consensus journals "review of financial"
Then append the exact name string to trusted-journals.local.md, under the
right # section.
Publishers
--publisher is a coarser cut over a fixed vocabulary (consensus publishers
lists it: Elsevier, Wiley, Springer Nature, OUP, CUP, JAMA, NEJM, ...). Use it
when the user wants a house rather than a venue; journal filtering is otherwise
strictly better.
Still mark, still resolve
--journals-file also makes the ★ pass trivial: every returned paper is from a
trusted venue, so mark them all ★ and note the filter in the preamble. Without
a journal filter, mark ★ per-paper against the same file, and run the SSRN DOI
resolution below.
SSRN Label Detection & DOI Resolution
SSRN label patterns (journal field is NOT the real venue):
- Contains "eJournal", "Topic)", "SSRN Electronic Journal"
- Starts with a subject code:
PSN:,ERN:,ERPN:,SRPN:,POL:,LSN:
Note that SSRN labels cannot appear when --journal/--journals-file is in
play — the filter matches on the indexed journal name, so working-paper labels
are excluded by construction. This section applies to unfiltered searches.
When a paper has an SSRN-label journal AND a non-null doi:
curl -s "https://api.crossref.org/works/<doi>" | uv run python3 -c "
import json, sys
d = json.load(sys.stdin)
msg = d.get('message', {})
ct = msg.get('container-title', [])
print(ct[0] if ct else 'NOT FOUND')
"
Use the resolved journal name to re-check against the trusted list. If it matches, mark ★ with a note: ★ (resolved via DOI from SSRN label).
If doi is null or CrossRef returns no container-title: leave as unresolved SSRN label.
Presentation Format
★ [Title](url) — Authors (Year), *Journal*, N citations
> Takeaway: ...
★ [Title](url) — Authors (Year), *Resolved Journal* (resolved via DOI), N citations
> Takeaway: ...
[Title](url) — Authors (Year), *Journal* [SSRN label, unresolved], N citations
> Takeaway: ...
Trusted papers first (confirmed then resolved), then unresolved, then non-trusted.
IRON LAW: Always Use the CLI Binary
NEVER use mcp__consensus__search. ALWAYS use the ~/projects/consensus-cli/consensus binary. This is not negotiable.
The MCP tool is rate-limited to 3 results per search and requires a free account. The CLI binary drives the signed-in enterprise session in the CDP browser and returns up to 100 results.
Red Flags
| Action | Why Wrong | Do Instead |
|---|---|---|
Using mcp__consensus__search instead of the CLI | MCP is rate-limited to 3 results; CLI has no limit | Always use ~/projects/consensus-cli/consensus |
| Presenting results without reading trusted-journals.local.md | User expects journal quality signals on every search | Read the shared trusted-journal list first, always |
| Filtering trusted journals client-side after an unfiltered search | Wastes most of the result set — 50 results collapse to 3 | Pass --journals-file and get 50 trusted results |
| Passing a journal name you did not verify | An unindexed name matches nothing, silently — you get zero papers and blame the query | consensus journals "<partial>" first |
Comma-separating journals in one --journal | Journal names contain commas; the whole string is treated as one name | Repeat the flag, or use --journals-file |
| Treating SSRN topic labels as real journals without checking DOI | The paper may be in JF or JAE — you'd miss a trusted hit | Run CrossRef DOI lookup first |
| Skipping DOI resolution because there are many SSRN-labeled papers | High-citation SSRN-labeled papers are often published in top venues | Resolve all of them — it's one curl per paper |
Using --rank q1 as a journal quality filter | The API maps SSRN working papers under Q1 labels — it is not reliable | Use --journals-file — it is now an explicit server-side filter |
Passing --n > 100 | CLI validates and rejects — exits non-zero | Max is 100 |
Decision Tree
User wants papers on a topic
↓
Read trusted-journals.local.md
↓
Does the user want only their journals / "relevant" / "top" journals,
or is the topic squarely in a field the file covers?
↓
YES → consensus search "<topic>" --n 50 --sort citations \
--journals-file <path to trusted-journals.local.md> [other filters]
→ every result is trusted: mark all ★, say which filter was applied
→ zero results? The filter may be too narrow for the topic —
rerun unfiltered and mark ★ per-paper instead of widening silently
↓
NO → consensus search "<topic>" --n 50 --sort citations [other filters]
→ per paper:
journal matches the file? → ★
SSRN label + doi present? → curl CrossRef → re-check → ★ if match
else → unresolved / non-trusted
→ present ★ confirmed, ★ resolved, then rest
Common Patterns
# Basic search — sort by citations to surface highest-impact papers first
consensus search "mandatory disclosure effects" --n 50 --sort citations
# Restrict to RCTs
consensus search "aspirin cardiovascular" --rct --n 10
# Recent papers, high-citation
consensus search "ESG disclosure" --years 5 --min-citations 50
# Systematic reviews only
consensus search "minimum wage employment" --type systematic
# Only the user's journals — the default for their own fields
consensus search "corporate governance" --n 30 --sort citations \
--journals-file ~/projects/workflows/references/trusted-journals.local.md
# A narrower cut: the finance top three
consensus search "payout policy" --n 20 \
--journal "Journal of Finance" \
--journal "Journal of Financial Economics" \
--journal "Review of Financial Studies"
# Check a name before adding it to the trusted list
consensus journals "review of financial"
# By publisher house rather than venue
consensus search "machine learning in radiology" --n 20 --publisher Elsevier --publisher Wiley
Operational Notes
- Chrome must be running with CDP and signed in. Exit codes follow sysexits: 69 = browser unreachable, 77 = not signed in, 75 = CAPTCHA/rate limit — branch on the code, do not parse stderr
consensus journalsis rate-limited hard (429 → exit 75) after ~30 rapid calls. Verifying a batch of names needs ~1.5s between calls and a cool-off after a 429--rank q1is imprecise (SSRN papers slip through) —--journals-fileis the reliable quality gate- Journal filters are exact-match on the indexed name and fail silently, not loudly — an empty result set usually means a wrong name, not a dry topic
- Law reviews ARE indexed — all fourteen T14 flagships resolve. The student-edited business specialties (Journal of Corporation Law, Delaware JCL, Harvard/Columbia BLR, Penn JBL, NYU JLB, Virginia L&B, Berkeley BLJ) are NOT; reach those with
scholar lookup --journal "<name>" study_typecomes from Consensus badges and may benullfor many papersopen_access_pdf_urlisnullwhen no PDF is available (notundefined)
Continuous Learning Skill
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Extract reusable patterns from sessions and save them as learned skills for future use.
How It Works
This skill analyzes the session transcript to identify extractable patterns:
- Session Evaluation: Checks if session has enough messages (default: 10+)
- Pattern Detection: Identifies error resolutions, workarounds, debugging techniques
- Skill Extraction: Saves useful patterns to
~/.claude/skills/
When to Use
- At end of long sessions with multiple problem-solving cycles
- After resolving complex errors that might recur
- When you develop project-specific conventions
Invocation
Use /learn command or invoke directly:
/continuous-learning
Configuration
Edit config.json to customize:
{
“min_session_length”: 10,
“extraction_threshold”: “medium”,
“auto_approve”: false,
"learned_skills_path": "~/.claude/skills/learned/",
“patterns_to_detect”: [
“error_resolution”,
“user_corrections”,
“workarounds”,
“debugging_techniques”,
“project_specific”
]
}
Pattern Types
| Pattern | Description |
|---|---|
error_resolution | How specific errors were resolved |
user_corrections | Patterns from user corrections |
workarounds | Solutions to framework/library quirks |
debugging_techniques | Effective debugging approaches |
project_specific | Project-specific conventions |
Learned Skills Format
Extracted skills are saved following the standard skill directory structure:
~/.claude/skills/learned/
├── fix-marimo-import-error/
│ └── SKILL.md
├── debug-pixi-environment/
│ └── SKILL.md
└── wrds-connection-pattern/
└── SKILL.md
Each learned skill follows standard SKILL.md format with:
- Frontmatter (name and description)
- Problem context
- Solution pattern
- Example usage
Integration
The skill reads from CLAUDE_TRANSCRIPT_PATH (JSON conversation transcript)
which is automatically set by Claude Code during sessions.
Related
/learncommand - Manual pattern extraction mid-session/checkpointcommand - Save session state- Session-end hook - Auto-evaluates sessions for patterns
Court dockets: retrieving filings from Lex Machina, Docket Alarm and Bloomberg Law
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Three commercial platforms, one job: turn a list of cases into a directory of PDFs plus a manifest you can defend. Verified end to end on 1,597 Delaware Chancery cases — ~4,000 documents retrieved across all three.
Auth is the user's live browser session on CDP 9222. None of these needs a password. Read the
cookie with scripts/lm_cookie.py; never log in, and never touch stored credentials unless the
user explicitly says to.
Full endpoint chains, request/response shapes and per-vendor quirks:
${CLAUDE_SKILL_DIR}/references/endpoints.md — read it before writing any fetch code.
The scripts are the interface
Do not hand-roll a harvester. These are working, resumable and manifest-writing:
| script | does |
|---|---|
scripts/lm_cookie.py | reads the live session cookie for any host over CDP |
scripts/harvest_complaints.py | Lex Machina: docket-search → tag-257 complaint → signed PDF |
scripts/da_grab.py | Docket Alarm: serial, boring, survives hanging documents |
scripts/fetch_opinion_pdfs.py | opinion/decision PDFs from a plan parquet |
scripts/bl_docket_list.py | Bloomberg Law: every docket matching facets + term, as a list (no PDFs) |
Copy them into the project and adapt. Rewriting from the reference doc reproduces bugs that took hours to find.
## The Iron Law of Identification vs RetrievalA document that is IDENTIFIED is not a document that is RETRIEVED. Never mark a substitute ok.
Every vendor names the complaint far more often than it serves the file. Measured on the same 1,597 cases: Lex identified 99.6% and served 81.0%; Docket Alarm identified 96.4% and served 81.7%. A fallback that takes "the first document that has a file" silently returns a Rule 5.1 certification, a cover letter or a verification page — and reports success.
Conflating the two overstates coverage by 15-20 points. That is not a rounding error: it is a fabricated denominator under every downstream rate, and shipping it is more harmful than returning nothing, because nobody downstream can see it happened.
## The Iron Law of AbsenceNever record a document as unavailable because YOUR SELECTION RULE failed. Enumerate the docket entries and look for a later substantive version before concluding it is not public.
This is the mirror of the law above. That one catches substitutes passed off as complaints; this one catches real complaints recorded as absent — and it is the more expensive failure, because a substitute is visible in the data while an absence looks like a fact about the world.
Delaware Chancery files the complaint confidentially, then files a redacted PUBLIC VERSION days later as a SEPARATE docket entry (Ct. Ch. R. 5.1). Every vendor rule that stops at the first complaint-tagged entry sees no document and reports failure. The public version sits typically around entry 10-15, measured as deep as entry 91, on dockets running to 277 entries.
Measured on two independent Delaware populations:
| population | "unavailable" cases | had a public version | rate |
|---|---|---|---|
| DGCL 220 complaints | 253 | 229 | 90.5% |
| fiduciary-duty follow-on suits | 80 | 73 | 91.2% |
Recovering them moved coverage 82.2% -> 95.3% and added 195 confirmed cases to a study population. The pre-recovery write-up asserted "~18% of complaints are not publicly retrievable — a finding about practice, not a collection defect." That sentence was wrong in both halves.
The correct statement is "~18% are filed confidentially, and ~90% of those are publicly available as a redacted version elsewhere on the same docket." Genuine unavailability is ~4%.
Two defects that bite while recovering them
- Match the public version to a COMPLAINT, not to any document mentioning one.
Public Version of Answer to Verified ComplaintandMotion to Extend Deadline to File Public Version of the Verified Complaintboth contain "public version" and "complaint". Requiring only those two terms returned the wrong document on 10.6% (23/218) and 10.0% (7/73) of cases across two populations — a stable rate, so expect it. Excludeanswer|affirmative defense|motion|letter| certificat|opposition|brief|reply|response|stipulation|notice|transcript|order. - A resume guard keyed on filename alone silently preserves a known-bad file. After fixing the
regex, the re-fetch skipped every corrected case as
already_present, because the wrong document from the first pass sat on disk under the same name. Quarantine before re-fetching.
Redaction does not degrade extraction. Measured on the recovered set: misconduct_category,
plaintiff_law_firm and requester_type all came back at 0% missing, identical to the
unredacted baseline. Captions, parties, statutory citations, prayer for relief and signature
blocks survive; body allegations may be blacked out. Carry a source flag and check the
not_stated rate before using them for content coding.
Listing Bloomberg dockets by court, county or search term
scripts/bl_docket_list.py collects the docket LIST (ids, parties, judge, case type) for any
facet + term search, sliced past both silent caps and asserted equal to the API's remote_count.
Run --discover first to learn the facet codes (Nevada=29, Clark=84, Washoe=85), then
--count-only, then collect; --csv adds the 41-column export and asserts its id set matches.
Flags and examples are in the script header.
--term is sent verbatim, and quoting changes the result set: "Business Court" returns
3,690 dockets, Business Court 17,180. Write the quoted form into the project's methods note.
Vendor facts
- Lex Machina lists the complaint with an empty
files[]on ~18.7% of cases (298/1,597) — which means it was filed confidentially, NOT that it is unavailable; see the Iron Law of Absence above. The document id and title are present; only the file is withheld. That isfile_withheld, a recoverable fact worth recording — not "not found", and not a reason to substitute another document. - Docket Alarm returns HTTP 500 with a ~40 KB HTML body when it does not hold the file. It is reproducible on a fresh cookie and never recovers, so retrying it is pure waste; at a ~15% failure rate the retry backoff was the dominant cost of a run until it was removed.
- Some Docket Alarm documents never respond at all. One hanging request plus a 300 s timeout
plus an ordered
ThreadPoolExecutor.mapis indistinguishable from a deadlocked process. Use a 45-90 s timeout. - Bloomberg's
accessTokenis HttpOnly with an ~8 minute TTL, refreshed by the page's own SPA on a timer — not by your requests. The tab must stay open; a cookie snapshot into any external store authenticates for minutes, not hours. - Bloomberg's CSV export silently truncates at 1,000 rows — no warning in the file or the UI.
Always cross-check an export against the API's own
remote_countand date-slice below the cap. - Docket Alarm's search endpoint is bot-protected: HTTP 400 outside the browser, 200 via
same-origin
fetch()in a logged-in tab. Per-docket pages are fine outside. Do not fight it. - ~18% of Delaware Chancery §220 complaints are unavailable from all three vendors, and they fail on the same cases — adding two vendors to the best single one bought 17 cases out of 1,597. If coverage stalls near 82%, that is the record, not your pipeline.
Finding decisions in a docket
- The docket line names the DOCUMENT, not the holding. Entries read
Memorandum Opinion; the grant/deny is inside the PDF. Only ~15/152 opinion entries mention an outcome word at all. - Chancery disposes of cases through a Magistrate's Final Report, then an order implementing
it:
Granted ([Proposed] Order Implementing the Court's Post-Trial Bench Ruling). Searching only forMemorandum Opinion/Letter Decisionmissed 34% of cases with a decision (169 found vs 257 that exist). Always include:final report,master's report,post-trial ruling,bench ruling,opinion and order,final order and judgment. - Never search for opinions only within cases you already classified as having one. That is circular and it hides exactly the misclassifications you are trying to find. Scan the whole population.
Red Flags — STOP If You Catch Yourself:
| About to | Why wrong | Do instead |
|---|---|---|
| About to record a complaint as unavailable because the first entry has no file | The Rule 5.1 public version sits further down the docket ~90% of the time | Enumerate the entries and search for a later substantive version |
| Matching a public version on "public version" + "complaint" | Also matches answers, motions and briefs ABOUT the complaint — 10% wrong-document rate on two populations | Exclude answer/motion/brief/letter/certificate terms |
| Re-fetching after a selection fix without clearing the old files | A filename-keyed resume guard reports already_present and keeps the known-bad document | Quarantine, then re-fetch |
| Report "session expired" because the cookie read came back empty | An empty cookie name (['']) is a parsing failure, not an absent session. Picking the first CDP page target returns nothing when that tab is blob:/chrome:// — this cost hours and triggered a needless credential hunt | Use scripts/lm_cookie.py, which selects a page on the requested host; verify by fetching one document |
Pin the session cookie into a Cookie: header | It goes stale mid-run when the vendor rotates it, and the resulting 401s look exactly like rate limiting | Put cookies in the requests cookie jar; re-read from the browser on a 401 |
| Retry a Docket Alarm HTTP 500 | Reproducible "document not held"; the backoff dominates the run | Record it and move on |
Take documents[0] when no title matches the entry text | That is how a sealing motion becomes a "complaint" | Mark is_substitute, never ok |
Run pdftotext/pdfinfo inside the download loop | A long timeout on one malformed PDF stalls a worker and looks like a hang | Compute page/text metrics in a post-pass |
kill $! after setsid, or pkill -f a pattern matching your own shell | Kills the wrong PID and leaves crawlers running — three concurrent copies hit one vendor this way | ps -eo pid,cmd | awk '/[d]a_grab/' then kill the real pid |
| Trust a vendor's case-type tag as ground truth | Tags put a judicial letter, a §225 petition and a notary page in a books-and-records population | Screen from the document text |
| Conclude coverage is complete without checking the whole population | Every widening of the net raised the count: 6.5% → 10.7% → 16.1% written decisions | Scan all cases, all document-type spellings |
CourtListener for empirical legal research
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Free Law Project's corpus. Three access routes with very different properties, and one distinction that people get wrong before they start.
The two corpora are not the same thing
**"CourtListener is federal-only" is true of RECAP and FALSE of opinions.** Getting this wrong sends you to a paid vendor for state-court data that is sitting here free.| corpus | source | coverage | contains |
|---|---|---|---|
| Opinions (case law) | scraped from court websites | federal and state | written decisions, full text |
| RECAP | PACER | federal only | docket entries, motions, filings |
The tell is on the court record itself: pacer_court_id: null + has_opinion_scraper: true
means a scraped state court, not a PACER one. Verified on Delaware Chancery:
GET /api/rest/v4/courts/delch/
"full_name": "Court of Chancery of Delaware"
"jurisdiction": "SA" <- state
"has_opinion_scraper": true
"pacer_court_id": null <- NOT RECAP
Pick the route before writing any code
| route | auth | best for | cost |
|---|---|---|---|
| Bulk data files | none | any corpus-scale analysis | quarterly snapshot, big files |
| REST API v4 | search/courts unauth; dockets needs a token | targeted queries, current data | rate-limited, polite pacing on you |
| DB replication / MCP | membership or commercial | continuous sync | contact FLP |
Default to bulk for research. It is public domain, needs no account, and does not rate-limit you. Reach for the API only when you need data newer than the last quarterly snapshot, or when you want a few hundred records rather than a corpus.
Bulk data
No authentication. Public Domain Mark. Regenerated quarterly on the last day of March, June, September and December. Snapshots, not deltas — each file is the whole table.
https://com-courtlistener-storage.s3-us-west-2.amazonaws.com/?list-type=2&prefix=bulk-data/
aws s3 ls s3://com-courtlistener-storage/bulk-data/ --no-sign-request
Measured sizes, 2026-06-30 snapshot:
| file | size | carries |
|---|---|---|
opinions-YYYY-MM-DD.csv.bz2 | 54.6 GB | opinion full text |
opinion-clusters-… | 2.46 GB | court, date_filed, case name, docket_number |
dockets-… | 5.01 GB | docket metadata |
courts-… | <10 MB | the court table |
citation-map-… | 0.52 GB | citation graph |
Also published: financial disclosures, judges, oral arguments, and ~2 TB of ModernBERT opinion
embeddings at s3://com-courtlistener-storage/embeddings/opinions/.
Filter clusters first, then stream opinions. opinion-clusters is 22× smaller and carries
the court and date. Select the cluster ids you want, then stream-decompress the 54 GB opinions
file and keep only matching rows. Never materialise the opinions CSV.
Files are PostgreSQL COPY TO output: CSV, UTF-8, header row. Schema dumps are published
alongside; field definitions are in the CourtListener GitHub models.py, or via an OPTIONS
request to the matching API endpoint.
REST API v4
https://www.courtlistener.com/api/rest/v4/
searchandcourtswork unauthenticated — 3,358delchopinions were pulled in 206 unauthenticated calls.docketsreturns{"detail":"Authentication credentials were not provided."}without a token.- Token lives at
/profile/api/— which sits behind a "Let's confirm you are human" interstitial. That gate is the site's bot check and belongs to Free Law Project; the user clears it, not you. - Search type params:
type=oopinions,type=rRECAP dockets.
Pace politely and back off on 429/5xx. This is a nonprofit's free service; without a token there is no quota to raise, so courtesy is the entire budget.
API Facts
suitNatureis not normalised. The same NOS 160 appears asStockholders Suits,160 Contract: Stockholders Suitsand160 Stockholders Suits. Query the text (suitNature:"Stockholders Suits"), not the numeric code, or you silently lose most of the set.- Derivative suits are docketed under more than one NOS.
"shareholder derivative"returns 1,670 under NOS 160 and 2,020 under 850 (Securities/Commodities). Filtering to 160 alone drops more than half. docketNumbercarries the C.A. number, which joins directly to a Delaware Chancery docket from Lex Machina or CourtConnect — measured at 100% match on 2,350 rows. That join is what lets a filing universe from one source meet a ruling from this one.- Opinions are written decisions only, and that is a selection, not a gap. Measured against a verified Chancery filing universe: 82% of caption-derivative filings never produce a written opinion, because most dispositions are bench rulings from the transcript. An outcome variable built from opinions describes the selected fifth. Reporting a dismissal rate without that denominator is an unverified claim dressed as a finding.
- The bulk snapshot lags up to a quarter. The 2026-06-30 file lands two weeks after a 2026-06-15 event, so a study of anything recent gets a truncated post-period until the next drop. Check the snapshot date against your event date before designing the test.
Coverage and what is coming
Delaware Chancery: 3,358 opinions since 2012 (delch). Texas currently covers the Supreme
Court, Court of Criminal Appeals and the 15 civil courts of appeal.
From the FLP wiki, verbatim:
"In 2026, we are collecting content from high and appellate courts at the five most populous states: California, New York, Texas, Florida, and Pennsylvania."
"In 2027 we aim to continue this with the Delaware Chancery Court and the next most populous states."
The 2027 item is filings, not opinions — Chancery opinions already exist. Docket-level state coverage would reach the bench rulings the opinion corpus structurally cannot see.
Red Flags — STOP if you catch yourself
| About to | Why wrong | Do instead |
|---|---|---|
| Tell the user CourtListener is federal-only | True of RECAP, false of opinions — state case law is there | Check pacer_court_id / has_opinion_scraper on the court |
| Page the REST API for a corpus-scale pull | Rate-limited, slow, and rude to a nonprofit when the same data is a free download | Bulk files |
| Clear the human-verification interstitial to get a token | It is the site's bot check, and you are a bot | Ask the user; or use the unauthenticated endpoints, which cover most needs |
| Decompress the 54 GB opinions CSV to disk | You need a few thousand rows of it | Filter opinion-clusters first, then stream and match |
| Filter derivative suits on NOS 160 alone | 2,020 of them are docketed 850 | 160 OR 850 plus a text filter, then hand-check |
| Report a dismissal rate from opinions without the selection denominator | 82% of filings never get a written opinion; the rate describes the selected fifth | Quantify selection against a filing universe and lead with it |
| Assume the bulk snapshot is current | Quarterly, and the boundary may fall days after your event | Read the snapshot date; use the API for the tail |
Related
ui-json-capture— when the source is a paywalled UI rather than an open corpuswrds— the SEC-filing side of the same research questions
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Contents
- The Problem
- Splice Enforcement
- Symbology Enforcement
- Measured Coverage
- The Pipeline
- Additional Resources
The Problem
CRSP ships annually. The CIZ daily file stops at the last December close and does not error when you query past it — it returns a short panel. LSEG is current to T-1 but is keyed on RIC, not PERMNO, so the join runs through CUSIP.
Verified on WRDS PostgreSQL:
| Source | Last date | Gap |
|---|---|---|
crsp.stkdlysecurityprimarydata (CIZ) | 2025-12-31 | — |
crsp.stkmthsecuritydata (CIZ) | 2025-12-31 | — |
LSEG TRDPRC_1 / TR.TotalReturn1D | T-1 (2026-07-27) | 141 trading days |
This skill assumes crsp-v2 (CIZ table and column names) and lseg-data
(session, quota, entitlements). It covers only what is specific to joining them.
Splice Enforcement
IRON LAW: NO LSEG PRICE LEVEL SPLICED ONTO A CRSP PRICE LEVEL
**LSEG's `get_history` price series is back-adjusted to the CURRENT share basis. CRSP's `DlyPrc` is the price as traded on the day.** For any security that split after CRSP's cutoff, the two series are on different bases and the seam between them is a fabricated return.Measured on the Dec-2025 overlap, where both sources have the same 22 trading
days: 8 of 300 sampled securities disagree on price by >5%, at a ratio that is
constant to std = 0.0000 across all 22 days — the signature of an adjustment
factor, not a data error. STRO.O ratio 0.1000 (1:10 reverse split),
VISN.O ratio 2.0493.
pd.concat([crsp_prc, lseg_prc])→ WRONG. A 1:10 reverse split after the cutoff prints a +900% one-day return on the seam.- Chain the gap off CRSP's last
DlyPrcwith LSEG's daily total returns → CORRECT.p_t = p_cutoff * cumprod(1 + ret).
A fabricated ±90% return on a known date, in a panel that otherwise validates, is
worse than a missing tail — it survives every summary statistic and lands in the
event window. scripts/crsp_lseg_splice.py::rebuild_price() does the chaining.
The chained series is a total-return index anchored at CRSP's last price, not
a quoted price: total-return chaining reinvests dividends into the level. The raw
LSEG quote is carried alongside as lseg_prc for anyone who needs the level.
Return Facts
TR.TotalReturn1Dis theDlyRetanalogue and it validates. On the Dec-2025 overlap, 99.12% of permno-days agree to within1e-5, median absolute difference2.45e-07, correlation 0.991. This is the field to splice on.- It is returned in PERCENT.
-6.369427means −6.37%. CRSPDlyRetis a decimal. Divide by 100 at the boundary or every return in the gap is off by 100×, which is loud in a mean but silent inside a signal that gets standardized. - It is CALENDAR-PADDED, and that breaks any coverage count taken from it.
Every instrument comes back with the identical number of return days —
measured
std = 0.0000across 3,384 RICs — including the 19 that delisted mid-gap. 1.14% of return rows have no trade price that day and 38% of those areret == 0. Build the panel by inner-joining returns against the price series: otherwise a delisted stock keeps emitting flat rows to the end of the window, and the fill rate reads 99.8% because every security looks complete. - It is dividend-inclusive, like CIZ
DlyRet, so no distribution merge is needed — and, percrsp-v2, no delisting-return merge either. DlyCapmust scale on the PRICE relative, not the return chain. Chaining market cap onTR.TotalReturn1Dreinvests every dividend into shares outstanding and inflates cap by the cumulative dividend yield over the gap.lseg_prc_t / lseg_prc_cutoffis the right multiplier: both legs sit on LSEG's single adjusted basis, so splits cancel (as they must for cap, which is split-invariant) and dividends are excluded.TR.CompanyMarketCapexists but is not on CRSP'sDlyCapdefinition or units (CIZDlyCapis $ thousands).- Start the LSEG pull ON the CRSP cutoff date, not the day after. That one
overlapping day is the anchor.
crsp_prc / lseg_prcon the same day measures each security's adjustment-basis ratio directly: ~1.0 means the two agree, anything else is a post-cutoff corporate action LSEG has back-adjusted for (or a bad link that survived the venue screen). The pipeline carries it asadj_ratioandcoveragereports it — it is what makes the splice auditable per security rather than trusted in bulk. - LSEG serves data for still-listed instruments. A security that delists
inside the gap stops on its delist date, and its RIC gets a
^-stamp (IROQ.O^C26= delisted March 2026) — which is how you learn the delist happened, since CRSP has not published it yet. There is no delisting return from this path.
Symbology Enforcement
IRON LAW: NO RIC ACCEPTED WITHOUT A US-VENUE CHECK
**LSEG resolves a US CUSIP to a foreign cross-listing's RIC for ~3% of the universe, and that RIC returns prices in a FOREIGN CURRENCY with no error.**Measured: 101 of 3,485 resolved RICs (2.76%) carry a non-US venue suffix —
.TRE (Tradegate), .MU (Munich), .SG (Stuttgart), .TBEA, .MX, .BCU.
| CRSP | LSEG RIC | effect |
|---|---|---|
TPH TRI POINTE HOMES | T86f.TRE | prices in EUR; CRSP/LSEG ratio 1.3294 (≈ USD/EUR) |
VERO VENUS CONCEPT | 0RR0.MU | Munich listing; 18 of 22 days differ >1pp |
CIVI CIVITAS RESOURCES | US17888H1032.TRE | ISIN-form RIC, Tradegate |
- Taking
RICstraight fromsymbol_conversion→ WRONG. An FX series enters the panel as a price series. - Filter on the suffix before pulling → CORRECT. US venues are the bare RIC
(composite) and
.O .N .A .P .K .PK .OQ.
A EUR price series is plausible — right order of magnitude, right shape, moves with the stock. It fails no null check and no range check. It is only caught at the venue, which is why the check belongs before the pull, not after.
Symbology Facts
- Use CRSP's date-effective
cusip9fromcrsp.stksecurityinfohist, neverstksecurityinfohdr.hdrcusip. CIZ inverted the SIZ naming — CIZcusipis the historical value andhdrcusipis the header one — so header-CUSIP code attaches a security's most-recent CUSIP to its entire history. For a fill-forward anchored at the cutoff date the two usually agree, but the same script pointed at an earlier as-of date silently mis-links. - Pass the 9-character CUSIP to
SymbolTypes.CUSIP. CRSP CIZ carries bothcusip9and the 8-charactercusip; LSEG returns 9-char. Trimming to CUSIP8 to join is fine as a post-hoc key, but resolve on the 9. - Unresolved CUSIPs come back absent, not wrong. Misses land as explicit nulls. The precision risk in this pipeline is the foreign venue and the entity mismatch, not the miss.
- Entity agreement is the cheap guard: CRSP
tickervs LSEGTickerSymbolagrees for 97.31%, RIC root vs CRSP ticker for 95.06%, either for 97.56%. Most residual disagreement is a rename CRSP has recorded and LSEG reports under the current name (COMM→VISN.O), not a mis-link — flag it, do not drop it. symbol_conversionchunks fine at 200 symbols; the session cap is 500 requests/minute and the binding limit isget_dataat 10,000 data points andget_historyat 3,000 rows per request.
Measured Coverage
Full numbers, denominators, and the validation method: references/coverage.md.
Universe = the 3,657 CRSP CIZ common stocks (5-column SHRCD 10/11 equivalent,
primaryexch IN ('N','A','Q')) trading on 2025-12-31.
| Stage | count | share of universe |
|---|---|---|
| CRSP common stocks at cutoff | 3,657 | 100% |
| CUSIP9 → RIC resolved | 3,485 | 95.30% |
| — of those, US venue | 3,384 | 92.53% |
| — of those, entity-agreeing | — | 97.56% of resolved |
Agreement on the Dec-2025 overlap (300-security sample, 6,244 permno-days):
| check | result |
|---|---|
price exact (reldiff ≤ 1e-4) | 96.28% |
| price within 5% | 98.13% |
daily return within 1e-5 | 99.12% |
| return available where CRSP has a day | 98.34% |
The 1.87% of price rows outside 5% are the split-adjustment and foreign-venue cases above — both handled by the two Iron Laws, neither by a tolerance.
The Pipeline
scripts/crsp_lseg_splice.py — five cached steps, all runs them in order:
set -a; . $XDG_RUNTIME_DIR/agenix/lseg-credentials; set +a
export RDP_APP_KEY=$LSEG_APP_KEY RDP_USERNAME=$LSEG_USERNAME RDP_PASSWORD=$LSEG_PASSWORD
python scripts/crsp_lseg_splice.py all --out data/ --start 2020-01-01
| step | does | writes |
|---|---|---|
universe | CRSP CIZ common stocks alive at max(dlycaldt) + their daily panel | universe.parquet, crsp_panel.parquet, asof.txt |
map | CUSIP9 → RIC, venue + entity screen, link_status per row | link.parquet |
pull | gap-period TRDPRC_1/ACVOL_UNS + TR.TotalReturn1D | lseg_hist.parquet, lseg_ret.parquet |
splice | return-chained continuous panel, source column marks provenance | panel_spliced.parquet |
coverage | coverage table + seam sanity check | stdout |
Output panel is permno × date with dlyprc, dlyret, dlycap, dlyvol, source,
plus RIC, lseg_prc (the raw quote) and adj_ratio on the LSEG rows.
source ∈ {CRSP, LSEG}. Keep source in anything downstream — the two halves
have different provenance and the LSEG half is unaudited by CRSP.
Red Flags — STOP If You're About To:
pd.concata CRSP price and an LSEG price → STOP. Different adjustment bases; chain on returns (Iron Law 1).- Use a RIC without checking its suffix → STOP. 4.3% are foreign-currency venues (Iron Law 2).
- Use
TR.TotalReturn1Dwithout/100→ STOP. It is percent; CRSP is decimal. - Join on
hdrcusip→ STOP. Header CUSIP, not date-effective. - Query
crsp.dsf/crsp.msffor the recent panel → STOP. Legacy SIZ, frozen at 2024-12-31 — a year before the CIZ cutoff. Seecrsp-v2. - Open a second LSEG session while a pull runs → STOP. One concurrent platform session; the second fails on quota rather than queueing.
- Count coverage from the return series → STOP.
TR.TotalReturn1Dis calendar-padded; every security looks complete. Count from the price series. - Report the panel as "CRSP data through today" → STOP. It is CRSP through the cutoff and LSEG after, at 91% of the universe. Say so.
Additional Resources
references/coverage.md— full measured coverage, per-failure-mode breakdown, validation method, and the reproduction commandsscripts/crsp_lseg_splice.py— the pipeline${CLAUDE_SKILL_DIR}/../crsp-v2/SKILL.md— CIZ tables, the 5-column universe filter, delisting returns, CUSIP inversion${CLAUDE_SKILL_DIR}/../lseg-data/SKILL.md— session setup, quota, entitlements, rate limits${CLAUDE_SKILL_DIR}/../lseg-data/references/symbology.md—SymbolTypesenum names and the conversion API${CLAUDE_SKILL_DIR}/../wrds/SKILL.md— WRDS Postgres connection and.pgpass
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Contents
- Format Enforcement
- Universe Enforcement
- Return Enforcement
- What Changed at a Glance
- Table Map
- Canonical Queries
- Additional Resources
Format Enforcement
IRON LAW: NO LEGACY SIZ TABLE FOR ANY DATA AFTER 2024-12-31
CRSP's legacy Stock & Indexes Flat File Format 1.0 (SIZ) was **discontinued after the December 2024 data release**. The legacy tables still exist on WRDS and still answer queries — they just stop.Verified on WRDS PostgreSQL:
| Table | Format | max(date) |
|---|---|---|
crsp.dsf | legacy SIZ | 2024-12-31 |
crsp.stkdlysecurityprimarydata | CIZ | 2025-12-31 |
crsp.stkmthsecuritydata | CIZ | 2025-12-31 |
SELECT ... FROM crsp.dsf WHERE date >= '2025-01-01'→ WRONG. Returns zero rows, silently.SELECT ... FROM crsp.msf WHERE date >= '2025-01-01'→ WRONG. Same silent truncation.SELECT ... FROM crsp.stkdlysecurityprimarydata→ CORRECT.
A legacy query does not error when it runs off the end of the data. It returns a short panel, the regression runs, and the sample period is quietly wrong.
Handing back a silently truncated panel is not helpful — it is worse than an error,
because the user ships it. You reach for crsp.dsf because the legacy names are in
your weights and in every paper you have read. The weights are stale. Confirm the
format before writing the first SELECT.
Red Flags — STOP Immediately If You're About To:
- Write
crsp.dsf,crsp.msf,crsp.dse,crsp.dsedelist, orcrsp.stocknames→ STOP. Legacy SIZ, frozen at 2024-12-31. Use the CIZ table (references/tables.md). - Write
WHERE shrcd IN (10,11)→ STOP.shrcddoes not exist in CIZ. Five columns replace it (below). - Write
WHERE exchcd IN (1,2,3)→ STOP.exchcddoes not exist in CIZ. Useprimaryexch. - Write
abs(prc)orWHERE prc > 0→ STOP. CIZ prices are always positive. Usedlyprcflg. - Add a delisting return to
dlyret/mthret→ STOP. CIZ already embeds it. You are double-counting. - Assume
mthretreproduces legacyret→ STOP. Different methodology (compounded daily). Seereferences/known-differences.md. - Guess what a flag value means → STOP.
references/flags.mdhas all 774 values;crsp.metaflaginfois live. - Substitute
comp.secd(or any Compustat price table) because CIZ ran out at 2025-12-31 → STOP. Use thecrsp-lseg-spliceskill.secdcarries no delisting returns, nulltrfdon ~48% of firm-days, andajexdithat goes stale through corporate actions (one gvkey prints +17,700% on an unadjusted reverse split). It also applies none of CIZ's universe screens, so its cross-section is not comparable: measured on one event-study design, swapping CRSP forsecdleft Delaware flat at ~2,075 firms while doubling non-Delaware from 968 to 1,923.
Format Facts
- The CIZ format shipped in July 2022 and became the only updated format in February 2025 (December 2024 data was the last SIZ release). Both formats sit in the same
crspschema in Postgres and the samecrsplibrary in SAS — the schema name does not tell you which format you are in. The table name does. Reporting "I queried thecrspschema, so this is v2 data" is an unverified claim presented as fact. - WRDS-built convenience tables keep their legacy names with a
_v2suffix:crsp.dsf_v2,crsp.msf_v2,crsp.stocknames_v2. CRSP-built tables use the new CIZ names (crsp.stkdlysecuritydata). Both are CIZ; the_v2ones are pre-joined and wider. Treating the absence of a_v2suffix as evidence a table is legacy sends you back to the frozen data. - On the WRDS Cloud filesystem, SIZ stays at
/wrds/crsp/sasdata/a_stockand CIZ is at/wrds/crsp/sasdata/a_stock_v2. - Column names carry their frequency as a prefix:
Dly,Mth,Qtr,Ann. There is no frequency-agnosticretorprcin CIZ —DlyRetandMthRetare different columns in different tables, not the same column read from two files. - The
62suffix (crsp.stkdlysecuritydata62) is the 1962-start subset product; the_indsuffix (crsp.stkindmembership_ind) is the full Index database. Stock-only products carry just 4 index series;crsp.indseriesinfohdr_indcarries 274. Querying the wrong one returns zero rows rather than an error, so "that index isn't in CRSP" is usually the wrong suffix, not a missing index.
Universe Enforcement
IRON LAW: NO CIZ COMMON-STOCK SAMPLE WITHOUT ALL FIVE COLUMNS
Legacy `SHRCD IN (10, 11)` was unpacked into **five** CIZ columns across two tables. No single column reproduces it.sharetype = 'NS' -- No Special Share Type
AND securitytype = 'EQTY'
AND securitysubtype = 'COM'
AND usincflg = 'Y'
AND issuertype IN ('ACOR', 'CORP')
WHERE sharetype = 'COM'→ WRONG. Returns 0 rows.COMlives at the SecuritySubType level;ShareTypeis never'COM'in CIZ. CRSP support confirmed this in writing.WHERE securitysubtype = 'COM'alone → WRONG. Picks up non-US-incorporated firms, ADRs (sharetype='AD'), and REITs (issuertype='REIT') that legacySHRCDexcluded.- All five, ANDed → CORRECT.
A universe that silently differs from SHRCD IN (10,11) breaks comparability with
every prior paper in the literature — that is an anti-helpful result dressed as a
working query. The one-column version is faster to type and returns plausible row
counts, which is exactly why it survives review.
Verified distribution in crsp.stksecurityinfohist: EQTY/COM/NS/CORP/Y
= 90,515 rows and EQTY/COM/NS/ACOR/Y = 21,682 rows are the two SHRCD 10/11 cells;
EQTY/COM/AD/CORP/N (6,279 ADR rows) and EQTY/COM/NS/REIT/Y (2,347 REIT rows) are
what the sloppy filter lets in.
Exchange Facts
EXCHCDis gone.primaryexchis a single letter:N(NYSE),A(NYSE American),Q(NASDAQ),R(NYSE ARCA),B(BATS),I(IEX),C(Consolidated),X(Unknown),N/A.EXCHCD IN (1,2,3)maps toprimaryexch IN ('N','A','Q')— butprimaryexchalone also carries the halted and suspended records that legacyEXCHCDsplit into-2and-1. To reproduce legacyEXCHCD IN (1,2,3)exactly, addconditionaltype = 'RW' AND tradingstatusflg = 'A'.- Legacy
EXCHCD = -2(halted) →tradingstatusflg = 'H'. LegacyEXCHCD = -1(suspended) →tradingstatusflg = 'S'. - The universe columns live on the history table (
stksecurityinfohist, one row per attribute-change interval), not only the header. Filtering onstksecurityinfohdrapplies today's classification to the whole 1925–2025 panel and back-fills survivorship into the sample.
CUSIP Facts
- CIZ inverted the CUSIP naming.
CUSIPis now the historical CUSIP (legacyNCUSIP);HdrCUSIPis the header/most-recent CUSIP (legacyCUSIP). - Code ported from SIZ that joins on
cusiptherefore changes meaning silently — it starts joining on the historical value. For a point-in-time match to Compustat/IBES this is usually what you wanted; for a header match it is a bug.
Return Enforcement
IRON LAW: NEVER ADD A DELISTING RETURN TO A CIZ RETURN
In SIZ, delisting returns lived only in `crsp.dsedelist`/`msedelist` and every researcher hand-merged them in. **CIZ embeds the delisting return directly in the daily and monthly return series.**Verified for PERMNO 10002 (delisted 2013-02-15):
| dlycaldt | dlyprc | dlyprcflg | dlyret | dlydelflg |
|---|---|---|---|---|
| 2013-02-14 | 2.92 | TR | -0.010170 | N |
| 2013-02-15 | 2.98 | TR | 0.020548 | N |
| 2013-02-19 | 0.00 | DA | 0.010906 | Y |
The 2013-02-19 row is the delisting return. crsp.stkdelists still exists, but it is
for the delisting reason and event detail — not for patching the return series.
coalesce(dlyret,0) + coalesce(dlret,0)→ WRONG. Double-counts.(1+dlyret)*(1+dlret)-1→ WRONG. Same double-count, compounded.- Use
dlyretas-is → CORRECT.
Silently inflating delisting-month returns reintroduces exactly the survivorship artifact the merge was supposed to fix. The old merge is muscle memory and the result looks normal — the bias only shows up in the delisting tail, which is where the identification usually lives.
Return Facts
MthRetis a compound of daily returns within the month, with dividends reinvested on the ex-date. LegacyRETwas a month-end-to-month-end holding period return with dividends reinvested at month-end. These are different estimators, not a renaming. WRDS found 90 stock-months differing by >100% and 3,479 differing by >5%.- The same change applies to delisting returns (
DelRet), where the divergence can be larger. DLRETX(delisting return without dividends) does not exist in CIZ. There is no substitute.DlyRetMissFlgandDlyRetDurFlgexplain missing and multi-period returns.DlyRetDurFlg = 'D1'is the ordinary adjacent-trading-day case;P1–P9mean the return spans 2–10 trading periods;MRmeans missing. Filtering on this flag replaces the old ad-hoc "drop returns after a gap" heuristics.DlyPrcis always positive. The bid-ask-average case that legacy encoded as a negative price is nowDlyPrcFlg = 'BA'; a real closing trade is'TR'; a delisting amount is'DA'. Never callabs().
What Changed at a Glance
| Concept | Legacy SIZ | CIZ (v2) |
|---|---|---|
| Daily price | prc (negative = bid/ask avg) | dlyprc (always positive) + dlyprcflg |
| Daily return | ret | dlyret (delisting return included) |
| Monthly return | ret from msf | mthret (compounded daily) |
| Common stock | shrcd IN (10,11) | 5 columns (see above) |
| Exchange | exchcd IN (1,2,3) | primaryexch IN ('N','A','Q') |
| Historical CUSIP | ncusip | cusip |
| Header CUSIP | cusip | hdrcusip |
| Delisting code | dlstcd (3-digit) | delactiontype, delstatustype, delreasontype, delpaymenttype |
| Distribution code | distcd (4-digit) | distype, disfreqtype, dispaymenttype, disdetailtype, distaxtype, disorigcurtype, disordinaryflg |
| Adjustment factors | cfacpr, cfacshr in dsf | crsp.stkdlycumulativeadjfactor (separate table) |
| Market index | vwretd in dsf | indno=1000200 in crsp.inddlyseriesdata |
| Issuer attributes | mixed into stocknames | crsp.stkissuerinfohdr / stkissuerinfohist (PERMCO-keyed) |
Table Map
Most-used CIZ tables. Full catalog with verified columns: references/tables.md.
| Table | Grain | Use for |
|---|---|---|
crsp.stkdlysecurityprimarydata | permno × day, 12 cols | Daily returns/prices/cap — default daily table, ~7 GB |
crsp.stkdlysecuritydata | permno × day, 32 cols | Adds bid/ask, high/low, open, prev-price, dividend amounts — ~20 GB |
crsp.stkmthsecuritydata | permno × month, 37 cols | Monthly aggregates + identifiers |
crsp.stkqtrsecuritydata / stkannsecuritydata | permno × qtr / year | Pre-aggregated panels; check qtrcompflg/anncompflg |
crsp.stksecurityinfohist | permno × interval | Universe filters, historical CUSIP/ticker |
crsp.stksecurityinfohdr | permno | Current/header attributes only |
crsp.stkissuerinfohdr / stkissuerinfohist | permco | Issuer-level SIC/NAICS/ICB, non-duplicated issuer counts |
crsp.stkshares | permno × interval | Shares outstanding history |
crsp.stkdistributions | permno × exdt × seq | Dividends, splits, factors |
crsp.stkdelists | permno | Delisting reason/status detail (not for returns) |
crsp.stkdlycumulativeadjfactor | permno × day | dlycumfacpr, dlycumfacshr, dlyshrout |
crsp.inddlyseriesdata / indmthseriesdata | indno × period | Index returns and levels |
crsp.stkindmembership_ind | permno × indno | Index constituents (S&P 500 = indno 1000500) |
crsp.dsf_v2 / msf_v2 / stocknames_v2 | WRDS-built | Pre-joined convenience tables (identifiers + data + shrout + factors) |
crsp.metafileinfo, metaiteminfo, metaflaginfo, metasiztociz | metadata | Self-documenting schema — query these instead of guessing |
Canonical Queries
All queries below were executed against WRDS PostgreSQL. Full set with
output: references/queries.md.
Common-stock daily panel (the SHRCD 10/11 + EXCHCD 1/2/3 equivalent):
SELECT d.permno, d.dlycaldt, d.dlyprc, d.dlyret, d.dlycap, d.dlyvol
FROM crsp.stkdlysecurityprimarydata d
JOIN crsp.stksecurityinfohist h
ON h.permno = d.permno
AND d.dlycaldt BETWEEN h.secinfostartdt AND h.secinfoenddt
WHERE d.dlycaldt BETWEEN %(start)s AND %(end)s
AND h.sharetype = 'NS' AND h.securitytype = 'EQTY' AND h.securitysubtype = 'COM'
AND h.usincflg = 'Y' AND h.issuertype IN ('ACOR','CORP')
AND h.primaryexch IN ('N','A','Q')
AND h.conditionaltype = 'RW' AND h.tradingstatusflg = 'A';
The BETWEEN secinfostartdt AND secinfoenddt join is mandatory — it is what makes the
classification point-in-time. Dropping it applies the security's final classification
to its entire history.
Market capitalization — do not compute it. dlycap (and mthcap) is CRSP's own
capitalization in $ thousands, already on the row. dlyprc * shrout reintroduces
the precision and rounding differences CRSP documented.
Market index return — join on indno, do not look for vwretd:
SELECT m.permno, m.mthcaldt, m.mthret, i.mthtotret AS vwretd, i.mthprcret AS vwretx
FROM crsp.stkmthsecuritydata m
JOIN crsp.indmthseriesdata i ON i.mthcaldt = m.mthcaldt AND i.indno = 1000200;
1000200 = CRSP NYSE/NYSEMKT/Nasdaq/Arca Value-Weighted (vwretd/vwretx),
1000201 = Equal-Weighted (ewretd/ewretx), 1000502 = S&P 500 Composite
(sprtrn = dlyprcret/mthprcret).
Compustat merge — the CCM link is unchanged by CIZ. crsp.ccmxpf_lnkhist is
still keyed on lpermno, so existing CCM code ports as-is once the CRSP side is CIZ.
Additional Resources
references/tables.md— full CIZ table catalog, verified column lists, grain and key columnsreferences/siz-to-ciz.md— column-by-column crosswalk from every legacy SIZ filereferences/flags.md— all 774 flag values across 62 flag types, dumped fromcrsp.metaflaginforeferences/known-differences.md— WRDS/CRSP-documented value discrepancies and the monthly-return methodology changereferences/queries.md— verified query recipes (universe, delisting, factors, indexes, CCM, MSE-style rebuilds)references/indexes.md— INDNO/INDFAM conventions, the +400 monthly rule, S&P 500 series, decile statisticsexamples/ciz_panel.py— end-to-end Python pull of a CIZ common-stock panel${CLAUDE_SKILL_DIR}/../crsp-lseg-splice/SKILL.md— CRSP updates annually, so the CIZ daily file stops at the last December. That skill carries the panel forward to T-1 with LSEG via CUSIP→RIC, and has the measured coverage (~91% of the common-stock universe) and the two splice hazards (adjustment basis, foreign-venue RICs).${CLAUDE_SKILL_DIR}/../../skills/wrds/SKILL.md— connection,.pgpass, WRDS Cloud/SGE rules (this skill assumes them)- CRSP source PDFs (WRDS login required): User Guide, Cross-Reference Guide, Metadata Guide, Executive Summary
- WRDS transition pages: Announcement, Transition FAQ, Index Overview, Recreate MSE Tables
Data Context Extractor
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Extract tribal knowledge about a dataset or database and generate a reusable data context skill.
## The Iron Law of Data ContextYOU MUST interview the user before generating ANY skill content. This is not negotiable.
You MUST NOT:
- Generate a data context skill from schema alone
- Skip entity disambiguation
- Assume metric definitions without confirmation
- Output a skill without user review of each section
If you're about to write a skill based on your assumptions, STOP. Interview first.
Two Modes
1. Bootstrap Mode
Trigger: No existing data context skill for this project/dataset.
Create a new data context skill from scratch by interviewing the user about their data.
2. Iteration Mode
Trigger: Existing data context skill exists, user wants to add a new domain or update.
Read existing skill, identify gaps, interview for the new domain, merge into existing skill.
Integration with Existing Data Access Skills
Before interviewing, check for existing data access skills that already encode tribal knowledge:
1. READ existing skills: /wrds, /lseg-data, or any project-local data skills
2. IDENTIFY what they already cover (table names, filters, field mappings, gotchas)
3. DO NOT re-document what existing skills handle
4. FOCUS the interview on the project-specific layer:
- Which specific tables/fields from WRDS or LSEG does THIS study use?
- How are identifiers linked across sources? (permno ↔ gvkey, RIC ↔ ISIN ↔ cusip)
- What sample filters define the study universe? (date range, exchange, firm type)
- What derived variables or transformations are project-specific?
The generated data context skill should reference existing skills rather than duplicate them:
## Data Sources
| Source | Skill | Tables/Fields Used |
|--------|-------|--------------------|
| WRDS | `/wrds` | comp.funda (at, lt, ceq), crsp.msf (ret, prc) |
| LSEG | `/lseg-data` | TR.F.TotRevenue, TR.GICSSector |
| Local | DuckDB | data/processed/merged_panel.parquet |
For connection details and critical filters, see the referenced skills.
The Interview Process
1. DISCOVER data sources
→ What databases/files/APIs? Connection details?
→ For each: what dialect? (PostgreSQL, DuckDB, SQLite, Snowflake, etc.)
→ IMPORTANT: If user mentions WRDS or LSEG, read the corresponding skill first
and ask only about project-specific usage, not general access patterns
2. MAP entities
→ What are the core entities? (users, transactions, products, etc.)
→ How do they relate? (foreign keys, join paths)
→ CRITICAL: Disambiguate entity names
- "user" vs "account" vs "customer" — are these the same?
- "order" vs "transaction" vs "purchase" — clarify overlaps
- Document the canonical name and any aliases
3. DEFINE key metrics
→ What are the business-critical metrics?
→ For each metric:
- Exact definition (SQL or formula)
- Known edge cases
- Common misinterpretations
- Time grain (daily, monthly, etc.)
4. DOCUMENT data hygiene
→ Known data quality issues
→ Fields that lie (e.g., "created_at" that's actually "imported_at")
→ Nulls that mean something specific
→ Enums/codes that need translation
→ Date ranges with reliable data vs backfill periods
5. CAPTURE common gotchas
→ Joins that explode (many-to-many lurking as one-to-many)
→ Filters that are always needed (e.g., "WHERE is_deleted = false")
→ Time zones and their traps
→ Slowly changing dimensions
→ Tables that look useful but aren't (deprecated, partial, test data)
6. COLLECT common query patterns
→ Frequently needed aggregations
→ Standard date filters or cohort definitions
→ Boilerplate CTEs that everyone copies
Interview Style
Ask questions in batches of 3-5. Don't overwhelm with everything at once.
Round 1: Data Sources
- What data sources does this project use? (WRDS, LSEG, local files, other databases?)
- For known sources (WRDS/LSEG): which specific tables and fields do you pull?
- For local data: what formats (parquet, CSV, SQLite, DuckDB)? Where stored?
- How do you link identifiers across sources? (permno, gvkey, cusip, ISIN, RIC)
Round 2: Core Entities (after Round 1 answers)
- What are the main entities in this data?
- Walk me through how [Entity A] relates to [Entity B]
- Are there any entity names that are ambiguous or have aliases?
Round 3: Metrics & Definitions (after Round 2 answers)
- What metrics does your team track most?
- For [Metric]: what's the exact calculation? Any edge cases?
- Are there metrics people commonly get wrong?
Round 4: Data Quality & Gotchas (after Round 3 answers)
- What are the known data quality issues?
- Any fields that are misleading or have surprising behavior?
- What's the most common mistake new analysts make with this data?
Round 5: Common Patterns (after Round 4 answers)
- What queries do you find yourself writing repeatedly?
- Any standard filters or cohort definitions everyone uses?
- What boilerplate would save the most time?
Output: Generated Skill
After the interview, generate a skill with this structure:
project-name/
├── .claude/
│ └── skills/
│ └── data-context/
│ ├── SKILL.md # Main skill file
│ └── references/
│ ├── entities.md # Entity definitions and relationships
│ ├── metrics.md # Metric definitions with SQL/formulas
│ └── gotchas.md # Data quality issues and common pitfalls
SKILL.md Template
---
name: [project]-data-context
description: "Data context for [project]. Entity definitions, metric calculations, data quality notes, and common patterns for [data domain]."
---
# [Project] Data Context
## Data Sources
| Source | Skill/Dialect | Tables/Fields Used |
|--------|---------------|--------------------|
| [WRDS] | `/wrds` | [specific tables and fields for this project] |
| [LSEG] | `/lseg-data` | [specific fields for this project] |
| [Local] | [DuckDB/CSV/Parquet] | [file paths or database] |
For connection details and critical filters, see the referenced skill. This context covers only project-specific usage.
## Entity Map
[Entity relationship summary — which entities exist, how they connect]
See `references/entities.md` for full definitions.
## Key Metrics
[Top 3-5 metrics with brief definitions]
See `references/metrics.md` for exact calculations and edge cases.
## Critical Gotchas
[Top 3-5 gotchas that catch analysts]
See `references/gotchas.md` for full list.
## Common Patterns
[Frequently used query snippets or data access patterns]
Generating the References
For each reference file, include:
- entities.md: Entity name, aliases, primary key, important fields, relationships, common joins
- metrics.md: Metric name, definition (SQL/formula), edge cases, common misinterpretations, time grain
- gotchas.md: Issue description, why it happens, how to detect, how to handle
Exit Gate
Before writing skill files, execute this gate:
- IDENTIFY: All 5 interview rounds completed
- RUN: Read back each generated section to the user
- READ: Verify entities, metrics, gotchas sections all have substantive content (not placeholders)
- VERIFY: User approved each section via explicit confirmation
- CLAIM: Only generate skill files after ALL checks pass
Skipping this gate produces a skill based on your assumptions, not the user's knowledge. That skill will mislead every future analysis.
Verification
Before finalizing the skill:
- Read back each section to the user for confirmation
- Test at least one metric definition against actual data if possible
- Check entity relationships match what the user described
- Confirm the gotchas list captures what the user considers most important
Iteration Mode Details
When adding to an existing data context skill:
- Read the existing skill files
- Identify what's already documented
- Interview for the NEW domain only (don't re-interview existing content)
- Merge new content into existing files, maintaining consistent formatting
- Flag any conflicts with existing definitions (e.g., metric redefinition)
Where to Place the Output
- Project-specific:
.claude/skills/data-context/in the project root - The generated skill becomes available to all future Claude sessions in that project
- Subagents spawned by
ds-delegatewill have access to it automatically
Data Context Facts
- Schema and column names carry structure, not semantics:
revenuemay be gross, net, or recognized; auser_idforeign key does not say whether users can hold multiple active accounts. A metric definition or relationship cardinality inferred from names is an unverified claim presented as fact — every downstream analysis that loads the generated skill inherits the error. The interview takes 10 minutes; weeks of wrong analysis is what it prevents. - Gotchas are the most valuable section of the generated skill and the easiest to defer. "Fill in later" means never — capture them during the interview while context is fresh, or the skill ships as a hazard that misleads every future analysis.
- There is no "standard SQL" in practice. When the user says "standard SQL", get the exact dialect — definitions in metrics.md must be runnable as written.
- WRDS connection mechanics are covered by
/wrds, and LSEG field prefixes by/lseg-data. Re-documenting them here duplicates and drifts; document only the project-specific tables, fields, and filters THIS project uses.
de-ai-revise — make prose read less AI-generated
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
A writing-improvement tool. It audits a draft with three corpus-validated scorers, then rewrites only the flagged spans so the prose reads less like an LLM wrote it — plainer diction, burstier rhythm, fewer machine tics — while leaving already-human passages untouched.
This is the GENERATION side of the AI-writing apparatus, not detection. Detecting polished AI was proven near-impossible (60%+ false-positive rates on real human writing); this skill never renders a verdict on authorship. It improves readability for a human reader. The scorers GUIDE which spans to revise; they are not a target to maximize.
This is a BACKSTOP, not the main event. The primary lever for human-reading prose is the GENERATION contract upstream — writing-draft now drafts topic-sentence-led and proportional (varied paragraph/sentence length), which is what produces human burstiness in the first place. A draft generated well needs little here. If de-ai-revise is finding a lot, the fix usually belongs upstream (the outline's POINTs aren't real topic sentences, or the draft padded uniformly), not in a heavy span-by-span rewrite here. Use this to catch residue, not to manufacture rhythm a flat draft never had.
## The Iron Law of GoodhartTHE SCORERS GUIDE; THEY DO NOT GRADE. NO EDIT THAT IMPROVES A NUMBER BUT NOT THE READING. This is not negotiable.
A human reads the output. Mechanically maxing burstiness (chop every sentence), nuking every em-dash, or swapping every flagged word degrades prose to win a composite — that is the failure this skill exists to prevent. Revise a span only when the rewrite reads better to a person. Leave a flagged span alone when the author's choice is the right one (see Preserve-Human below).
The three scorers (all corpus-gated — do NOT re-derive)
scripts/de_ai_audit.py folds them into one line-anchored span list. Every signal
was gated against a 14.3M-sentence law+finance corpus, so flags are AI defaults
real scholars don't write — not generic "fancy word" lint.
The scorers themselves now live in ${CLAUDE_PLUGIN_ROOT}/scripts/prose-audit.py, the plugin's single deterministic
prose audit, and de_ai_audit.py is a thin wrapper over its --profile de-ai view. The output
shape below is unchanged and will stay that way — this skill needs the REWRITE view (a worklist of
spans with plain replacements), which is a different shape from the audit's severity-ranked,
id-bearing span list. Use prose-audit.py directly for anything that is not a de-AI rewrite: it
also carries the wikipedia AI-tell tables, the domain style guides, and the provenance-leak class
this profile is blind to.
| Scorer | Catches | Remedy |
|---|---|---|
Scored AI-tics (ai-anti-patterns/constraints/scored-tics-patterns.py) | phrase/structure tics that passed the ~0-human-rate gate (sev1-5) | rewrite the construction; these have no honest use |
Tiered diction (references/diction.yaml) | fancy→plain words, tiered by corpus rate | always_flag → swap on sight; cluster → fix when 2+/para; density → vary at saturation; dropped → never touch (legal-normal) |
British spelling (BRITISH in de_ai_audit.py) | locale mismatch in US-register prose (recognise, behaviour, whilst, labelled) — LLMs emit these into US documents from mixed training corpora | swap for the US form; drop the check for a UK-register document |
Stylometrics (ai-anti-patterns/scripts/style_metrics.py) | rhythm/structure: composite_human_likeness 0-100, em-dash, metronomic runs, opener transitions, nominalization, false precision, burstiness/passive advisories | vary sentence length toward bursty; em-dash → semicolon/period; plainer Latinate→Anglo-Saxon; round a summarising figure to a fraction |
Modes
| Mode | Trigger | Behavior |
|---|---|---|
| rewrite (default) | "de-AI this", "make it less AI" | audit → rewrite flagged spans → one corrective 2nd pass → return an edits-made + verification report (NOT the whole file) |
| detect-only | "just flag", "scan", "what AI tells are in this", "audit only" | audit only; report flagged spans + composite/tic-density; no edits |
| edit-in-place | "fix draft.md directly", "clean the file in place" | minimal targeted Edits to the file; preserve already-human paragraphs; re-audit after |
Default to rewrite when unspecified.
Process (the spec)
START
│
├─ Step 1: AUDIT — run de_ai_audit.py --json on the target
│ uv run --with pyyaml python3 ${CLAUDE_SKILL_DIR}/scripts/de_ai_audit.py --json <file>
│ Read: composite_human_likeness, tic_density, spans[], advisories[]
│
├─ detect-only? → report spans + signals, STOP.
│
├─ Step 2: REWRITE the flagged spans (NOT the whole draft)
│ - tic spans → rewrite the construction (no honest use)
│ - diction:always_flag → swap for the listed plain replacement
│ - diction:cluster → fix enough of the cluster to drop below 2/para
│ - style:em_dash → recast as semicolon / period / comma — but NOT all (see Preserve)
│ - style:false_precision → round to a high-level fraction ("1.3771 percent" → "about one
│ and a half percent"); KEEP the exact value if the sentence sits
│ next to the exhibit that reports it
│ - advisories (burstiness) → vary sentence length where it reads flat; do NOT chop for chop's sake
│ PRESERVE already-human passages (no spans) untouched.
│ PRESERVE quoted material, block quotes, code, footnote citations.
│
├─ Step 3: ONE corrective 2nd pass
│ Re-run de_ai_audit.py. Fix spans the first pass introduced or missed.
│ STOP at 2 passes — a 3rd rarely finds more and costs a full regeneration.
│
└─ Step 4: REPORT (edits-made + verification), NOT the whole file
- what changed and why (span → before → after, grouped by scorer)
- before/after composite + tic-density (must improve or hold; if it dropped, you over-edited)
- spans deliberately LEFT (author's voice / quoted / domain term) and why
If text and flowchart disagree, the flowchart wins.
Preserve-Human (the other half of Goodhart)
The composite penalizes em-dashes hard, and real legal scholarship — including this user's own published prose — uses them deliberately. Do NOT zero them out.
- Em-dashes: thin clusters and the clearest default-connector uses; KEEP em-dashes that set off a genuine appositive or a deliberate aside. Target fewer, not zero.
dropped-tier diction (significant, robust, leverage, comprehensive, …): NEVER flag or swap — these are legal/finance-normal; the audit already excludes them.- Quoted text, block quotes, statutory language, party names, code, citations: flag at most; never rewrite someone else's words or a term of art.
- Footnotes are auto-excluded: the audit MASKS pandoc inline
^[...]and markdown[^id]:footnotes before scoring, so findings never land inside them (citation/legal-normal text). You will not see footnote spans to triage; if you ever do, do not edit them. (--keep-footnotesdisables masking for debugging the raw signal only.) - British spelling in a genuinely UK-register document: the check assumes US
register. For a UK journal or an English court filing, ignore
spelling:britishentirely — do not "correct" an author writing in their own dialect. - A flagged span the author clearly chose (a fragment for emphasis, a repeated key term over elegant variation): leave it; note it in the report.
Fact rows
- The synthetic-AI baseline scores composite ~27 and tic-density 100; a real human legal draft scores ~55-65 with em-dashes as nearly the whole signal. So a composite in the 50s is NOT "AI" — it is a human who likes em-dashes. Treating the composite as a pass/fail bar instead of a span guide produces voice-destroying edits and is the exact failure the corpus tiering was built to prevent.
diction.yamldroppedtier exists because "significant/robust/leverage" fire on every real law-review article; a linter that flags them is worse than none. The audit omits them — if you hand-flag one anyway, you reintroduced the false positive.- The British-spelling map deliberately EXCLUDES words correct in both dialects —
analysis,characteristic,basis,emphasis,thesis,hypothesis, andpractice/licenceas nouns. The -sis nouns are not the -ise verbs. Adding any of them turns the check into a false-positive generator, which is the exact failure the corpus tiering elsewhere in this skill exists to prevent. - It matches STRICTLY (
\bword\b), not via_word_rx, because every inflected form is enumerated. Using_word_rxmade "recognise" also match inside "recognised" — two spans for one word, one carrying the wrong replacement. - A 3rd rewrite pass regenerates the whole span set for ~0 new fixes (CAP AT 2). The built-in corrective pass IS pass 2; "iterate to convergence" does not stack on it.
- Em-dash count near zero after a de-AI pass is over-editing, not success: you optimized the metric and flattened the author's rhythm. Fewer, not none.
Red Flags — STOP
- About to swap every flagged diction word → STOP. Cluster/density tiers are advisory; fix enough to clear the threshold, keep the ones that read right.
- About to delete every em-dash → STOP. Target fewer; keep deliberate appositives.
- About to rewrite a paragraph with zero spans because it "feels AI" → STOP. The audit found it human; trust the corpus over the vibe.
- About to run a 3rd rewrite pass → STOP. Cap is 2.
- About to return the whole rewritten file by default → STOP. Return the edits-made report unless the user asked for the full text.
- About to rewrite quoted/statutory text → STOP. Flag it; never alter someone else's words.
When invoked inside the writing workflow
- /writing-verify runs
${CLAUDE_PLUGIN_ROOT}/scripts/prose-audit.pyon every draft before dispatching its prose reviewers and INJECTS the resulting spans into their prompts as evidence — the reviewer is not asked to run a scorer, and a reviewer that cites none of the hard spans it was handed is recorded as unreliable. Those spans become AI-ism findings (advisory minors unless they cluster into a major). - /writing-revise applies this skill (rewrite mode) as a non-optional pass on every edited draft after fixing REVIEW.md issues, then re-audits. The substrate gate is unchanged: AI-prose spans are advisory polish, not blocking criticals.
Deep Research
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Web-grounded deep research via Gemini Interactions API.
IRON LAW: Always Use the Script
NEVER call the Gemini Interactions API manually. ALWAYS use bun deep-research.ts. This is non-negotiable.
cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts "research query"
The script handles model selection, interaction submission, polling with retries, timeout enforcement, and report formatting. Calling the Interactions API by hand means you skip retry logic, miss the 65-minute timeout guard, and lose the saved report file.
When to Use
Deep Research is the LAST RESORT in the librarian's source hierarchy. It costs $1-7 per query, takes minutes not seconds, and produces a broad web-grounded report rather than precise academic citations.
User needs information on a topic
|
v
Check curated sources FIRST (all must be exhausted):
1. Paperpile bib (local library)
2. Google Scholar (scholar search/lookup)
3. Consensus (consensus search)
4. NLM notebooks (existing sources)
5. Readwise (highlights/full-text)
|
v
Are there still significant gaps?
|
+-- NO --> Stop. Curated sources are sufficient.
|
+-- YES
|
v
Has the user explicitly asked for broader/deeper research?
|
+-- NO --> Report gaps, suggest deep research, WAIT for approval.
|
+-- YES --> Run deep-research.ts
Trigger signals (use this skill):
- User says "deep research", "thorough investigation", "comprehensive report"
- User explicitly asks for web-grounded synthesis beyond academic databases
- Curated sources have been exhausted and user wants more
NOT-triggers (do NOT use this skill):
- User wants a specific known paper -- use Scholar or Consensus
- User wants papers from their own library -- use Paperpile bib or Readwise
- User wants citation metadata -- use Scholar with
--bibtex - User has not exhausted curated sources yet
- User has not explicitly approved the cost
Commands
Basic (thorough report)
cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts "What is the current state of mandatory climate disclosure regulation worldwide?"
Fast (quick scan)
cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts --fast "ESG disclosure enforcement mechanisms"
Resume polling an existing interaction
cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts --status <interaction-id>
JSON output (raw interaction object)
cd "${CLAUDE_SKILL_DIR}" && bun deep-research.ts --json "corporate governance reform trends"
Models
| Model | Flag | Queries | Time | Cost | Use when |
|---|---|---|---|---|---|
deep-research-max | (default) | ~160 web queries | 5-20 min | ~$3-7 | Thorough reports, literature reviews, multi-faceted questions |
deep-research | --fast | ~80 web queries | 2-10 min | ~$1-3 | Quick scans, narrow questions, time-sensitive requests |
Cost Warning
$1-7 per query. NEVER auto-invoke. Always ask the user first.
This is the most expensive tool in the librarian's toolkit. Before every invocation:
- Confirm the user explicitly requested deep research
- Report which curated sources were already checked and what gaps remain
- State the estimated cost range ($1-3 for fast, $3-7 for thorough)
- Wait for the user's go-ahead
Cost Facts
- Even
--fastcosts $1-3 and takes 2-10 minutes; Scholar, Consensus, Paperpile, NLM, and Readwise are free and return in seconds. Reaching for deep research because it seems "faster than searching multiple sources" is counterproductive on its own terms. - "The curated sources didn't have enough" is only a fact after ALL five (Paperpile, Scholar, Consensus, NLM, Readwise) have been tried. Before that it is an assumption — and $3-7 spent on an assumption without the user's go-ahead is unauthorized spending.
Integration with Librarian Workflow
Paperpile bib --> Scholar --> Consensus --> NLM / Readwise
(local) (free) (free) (free)
|
v
[GAPS IDENTIFIED + USER ASKS]
|
v
Deep Research (THIS SKILL)
$1-7, 2-20 min
|
v
Curate results to NLM
(save report, extract key sources)
After deep research completes:
- Present the report to the user
- Extract key sources cited in the report
- If the user wants to keep specific sources, add them to NLM notebooks
- The report is also saved to
/tmp/deep-research-<id>.mdfor reference
Red Flags
| # | Action | Why Wrong | Do Instead |
|---|---|---|---|
| 1 | Using before checking curated sources | Scholar, Consensus, Paperpile, NLM, Readwise are free and fast | Exhaust all five curated sources first |
| 2 | Using for known paper lookup | Deep Research returns web-grounded reports, not citation metadata | Use scholar lookup --bibtex or scholar cite |
| 3 | Launching without user's explicit request | $1-7 per query; unauthorized spending violates trust | Report gaps, state cost, wait for go-ahead |
| 4 | Calling Interactions API directly (without the script) | Skips retry logic, timeout guard, report file save | Always use bun deep-research.ts |
| 5 | Ignoring the report and re-searching the same topic | Wastes another $1-7 for information you already have | Read the saved report at /tmp/deep-research-<id>.md |
Environment
| Variable | Required | Description |
|---|---|---|
GOOGLE_API_KEY | Yes | Gemini API key with Deep Research access |
Timeout: 65 minutes maximum polling duration (covers the API's 60-min hard limit plus buffer).
Polling interval: 10 seconds between status checks.
Retries: Up to 3 retries on transient errors (503, 429, service_unavailable, rate_limit).
Output
The script prints the research report to stdout as Markdown. It also saves the report to /tmp/deep-research-<interaction-id>.md for later reference.
With --json, it prints the full interaction object as JSON (useful for debugging or extracting structured metadata).
On failure, it prints the error to stderr and exits non-zero.
Limitations
- Beta API -- the Gemini Interactions API is in preview; behavior may change
- No structured output -- reports are free-form Markdown, not JSON-structured citations
- Max 60 minutes -- the API enforces a hard time limit on research sessions
- Cost per query -- $1-7 depending on model; no free tier
- Web-grounded, not academic-specific -- results include news, blogs, and general web sources alongside academic content; always cross-reference claims against Scholar/Consensus for academic rigor
- No incremental results -- the report arrives all at once when polling completes; no streaming
dev — a code change, run through work with a test-first gate
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
The lifecycle is work. Read it and follow it.
This file is a delta: it supplies the domain — the CLARIFY axes, the task-row shape, the lens,
the mechanical checks, the refs, the authority text. It ships no workflow.js and restates none of
work's mechanics.
What makes a run dev rather than plain work is one thing: every implementation task carries a
redCommand (the one exception: redDisposition — see below), so work's probes execute the failing test before the implementer and the passing
test after it, and the JS reads both exit codes. A task whose acceptance cannot be expressed as one
command that fails now and passes later is not ready to plan.
Phase 1 — CLARIFY
work's Phase 1, on these axes. Ask them before any reconnaissance: code says how the system
works, not what the user wants.
- Outcome — what behaviour exists after this run that does not exist now, and what does done look like for one user.
- Exclusions — what must not change: files, interfaces, dependencies, behaviour other callers rely on.
- Automated test framework and the exact command — the runner, and the literal command string a probe will execute. No framework in the repo means the first task is test infrastructure, decided here. A manual-only test proposal is a blocker: resolve an automated approach or leave this workflow. Never waive it silently.
- The user workflow, production protocol and transport the real test must exercise — the action sequence a user performs, the protocol production actually uses, and the observable result. A test over an alternate path proves nothing about the one that ships.
- The first failing test and what RED looks like — which test fails first, and the failure
output that counts. A syntax error, an import error, a broken fixture, a missing dependency or an
unrelated failure elsewhere in the suite is not the RED this workflow accepts. This is
computed, not asserted:
red_probe_gateclassifies a collection/import error ascould-not-runand refuses to dispatch. For a surface that does not exist yet, that means a stub — one that exists and raises, so the failure is an assertion rather than an ImportError — and the stub is named in the plan'sscaffoldPathsso the main thread may author it while the run is armed. - Required runtime evidence — what has to be observed at runtime rather than read in source: logs, HTTP/WS exchanges, rendered output, exit codes, screenshots.
- Review surface — working tree, commit range, or PR.
- Language and its language server — the language this change is written in (observed for an existing repo, decided here for new code), and the LSP that follows from it. Install it before reconnaissance, project-scoped, so recon and every implementer navigate by symbol rather than by grep.
Installing the LSP is two things, and the plugin is only one of them.
claude plugin install <lang>-lsp@claude-plugins-official --scope project
--scope project writes enabledPlugins into .claude/settings.json — a committed file, so it
enables the server for everyone who opens the repo; --scope local writes
.claude/settings.local.json instead. Official servers: pyright (py), typescript (ts/js),
gopls, rust-analyzer, clangd, jdtls, kotlin, csharp, php, ruby, lua, swift,
liquid.
The plugin declares a command, never a binary — pyright-lsp is just
{"command": "pyright-langserver"}. Enabling it while the server is absent registers something
that fails at first use with ENOENT: Executable not found in $PATH.
The binary is global, and that is not a preference. Claude Code resolves command from the
process PATH only — never the plugin root, never ${CLAUDE_PROJECT_DIR}, never a project
node_modules/.bin or pixi env. A project-local language server is unreachable by construction;
install it the way this machine installs system tools (nix here, or the distro package), not with
pixi add. What stays project-scoped is the enablement above and the server's own config —
pyrightconfig.json points a global pyright at the project interpreter, and a global
typescript-language-server still loads the project's own typescript from node_modules.
Registration is separate again: a mid-session install is inert until /reload-plugins. Self-send
it rather than asking the user to type it. Then prove it with one LSP call on a real file in
the repo — documentSymbol is enough. The install command's exit code says the plugin was
enabled, not that a server answers.
work's remaining axes are taken as work states them, with one domain binding: work axis 4
(observable success criteria) is answered with the target project's own test, lint and build
commands, and those strings become mechanicalChecks verbatim.
Then reconnoitre — a scouting Explore or Plan subagent, never a read of the whole tree into this
conversation — and return: the entry point and data flow, the integration boundaries, the existing
test harness and the files a new test extends, and the decisions code cannot answer. Re-ask only what
reconnaissance newly raised.
Before planning, present 2–3 feasible architectures with their boundaries, testing implications and trade-offs, and obtain an explicit choice. A sole viable option still needs its trade-off accepted.
Phase 2 — PLAN
work's Phase 2. The plan opens with frontmatter workflow: dev — required, so a context clear at
approval resumes here and not in work.
Three domain requirements on the table:
redCommandper implementation task — one invocation, no shell operators, failing now for the intended missing behaviour and passing once the task is done. It goes in the plan's Run sizingTest-first:block too, because it costs 2 agents against the fan-out ceiling.redDisposition— the only substitute forredCommand, and plan-lint (dev-red-disposition) refuses any other text:no-behaviour: <reason>(docs, config, rename — nothing to pin) oralready-covered: <test that pins it>. A task carrying one is judged on itsacceptanceCmd/verification, not on a test that went RED.refson every task row and on the lens — required, may be empty.work's spine does not validate it;wc-probeP7 refuses an absent key in THIS file, so a live run assembled from an approved plan is unchecked. Writerefs: []to state "no domain rules" rather than omitting the key.- Narrow
writablePaths— the probe runs a command that loads code the implementer can edit, so a wide writable set lets the implementer reach the thing proving its own RED. scaffoldPathsfor greenfield only, naming files and not directories — the test suite needs no entry (no task writes it, so the guard already allows it); the stub does, because it is the implementer's output and a precondition of its own red gate. A scaffold as wide as a task's writable surface isscaffold-swallows-taskat plan-lint, and it is the guard turned off with extra steps.
Phase 3 — GOAL
work's Phase 3 unchanged.
Phase 4 — the work call
The args go in the plan's <!-- work:dispatch --> arming block, and the dispatch is work's own
work-dispatch.sh — never a hand-written runner line. That script owns the TIER 1 plan-lint
gate, which refuses to dispatch on a major/critical plan finding and fails CLOSED on a verdict it
cannot count; hand-rolling the invocation silently drops it. work owns the wait, the result handling
and the return shape too, and work-result.sh reads the verdict. This run's projectDir is the
session repo, so work's own run directory is already inside it and no --run-dir override applies.
There is no built-in Workflow call — the guard at
~/.claude/hooks/main-thread-guard.sh denies that tool outright.
bash ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/work-dispatch.sh # armed plan; or pass one
bash ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/work-dispatch.sh --provider codex # "run this through codex"
Forward the provider. A provider named in this skill's $ARGUMENTS, however it is spelled —
--provider codex, --dispatch codex, "run this on gpt" — is --provider codex on the line
above. --provider is the only spelling the scripts take, and both reject the others by name.
Omitting it silently runs the user's codex request on claude.
const projectDir = /* the session repo */
const goal = /* one sentence */
const planPath = /* the approved plan's path */
const testCmd = /* the project's test command, from CLARIFY axis 3 */
const lintCmd = /* the project's lint command, or null */
const buildCmd = /* the project's build command, or null */
const taskName = /* the task's name */
const taskWork = /* what to build */
const taskPaths = /* narrow writablePaths — see Phase 2 */
const taskAcceptance = /* the criterion the verifier checks */
const taskRed = /* the exact command from CLARIFY axis 3, scoped to the first failing test */
const args = {
projectDir,
goal,
// The plan's table verbatim. Every implementation task carries redCommand and refs.
tasks: [
{ id: "T1",
name: taskName,
work: taskWork,
writablePaths: taskPaths,
acceptance: taskAcceptance,
redCommand: taskRed,
refs: ["${CLAUDE_PLUGIN_ROOT}/skills/dev/references/tdd.md",
"${CLAUDE_PLUGIN_ROOT}/skills/dev/references/writing-good-tests.md"] },
],
// The TARGET project's own commands, collected at CLARIFY. Nothing generic — a check that does
// not run this repo's suite gates nothing.
// ONE entry point; three lose one silently. A missing --test-cmd, --lint-cmd, --build-cmd or --plan is exit 2 unless --no-lint-cmd / --no-build-cmd / --no-plan declines it.
mechanicalChecks: [
{ name: "dev",
cmd: `bash ${CLAUDE_PLUGIN_ROOT}/skills/dev/scripts/check.sh --project-dir ${projectDir} --plan ${planPath} --test-cmd "${testCmd}"${lintCmd ? ` --lint-cmd "${lintCmd}"` : ' --no-lint-cmd'}${buildCmd ? ` --build-cmd "${buildCmd}"` : ' --no-build-cmd'}` },
],
// check.sh runs every checks/dev/constraints/ script through `workflow-check dev --constraints-only`. work-checks.sh runs ruleChecks as a command (no agent); Jev scores the rules under checks/dev/rules/ (LOOP, MOCK, NET, SHELL, WEAK); a rule with p >= 0.85 lands in rulesThatFailed and blocks, lower ones reach the lens as a ranked checklist.
ruleChecks: { name: 'jev-dev-rules', cmd: `bun ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/rule-check.ts --workflow dev --project-dir ${projectDir} --plan ${planPath}` },
// One lens after verification and mechanical checks; its checklist covers all five dimensions.
lens: {
agentType: "Explore",
refs: ["${CLAUDE_PLUGIN_ROOT}/skills/dev/references/lens-security.md",
"${CLAUDE_PLUGIN_ROOT}/skills/dev/references/lens-performance.md",
"${CLAUDE_PLUGIN_ROOT}/skills/dev/references/writing-good-tests.md"],
prompt: "Judge the change in the working tree against the approved plan and the goal. CHECKLIST, every item in scope on every run: (1) CRITERIA vs ARTIFACTS — for each success criterion in the plan and goal, is there an artifact in the tree that satisfies it? Missing or partial satisfaction is a finding, CRITICAL where the deliverable cannot stand without it. (2) SCOPE intent — path containment is check.sh's scope leg, so do not re-derive it; judge only unrequested features and silently skipped plan items (a plan task with no artifact), CRITICAL where a plan task the goal depends on was skipped. (3) SECURITY of the changed code, against the finding classes in the first ref — read it in full first. A finding names a file and line and states the concrete attack vector: the input an attacker controls, the path it travels, and what it reaches. Pre-existing defects this change did not introduce are out of scope. CRITICAL where the vector reaches attacker-controlled input on a production path. (4) PERFORMANCE — the runtime cost of the changed code, against the finding classes in the second ref, read in full first. A finding names a file and line, sits on a path that runs often enough to matter, and states the cost as Big-O over the input that actually grows or as concrete latency/memory; speculation without a growing input is not a finding. CRITICAL where the growth makes a production path unusable at realistic input size. (5) TEST PINS BEHAVIOUR: for each task, does the test that went RED (for a task with `redDisposition`, its acceptanceCmd/verification instead) assert the behaviour its acceptance names, not an incidental property (exit status only, a log line)? A test that would stay green with the behaviour removed is a MAJOR finding. Severity is MAJOR at minimum for all five. MODE — RED (a task was flagged or a mechanical check failed): diagnose EVERY failure in the digest and route it, naming cause and fix, to the task id whose writablePaths own the file, or to 'plan' when no task can own the fix. MODE — GREEN (everything passed): make one open-ended pass over the whole checklist and report each finding with an ownerTask. In both modes, rule every carried finding open or closed against evidence you actually read; silence leaves it open.",
},
authorityExtra: [
"TDD RULE — no implementation before a genuine RED observation.",
"Write the smallest real test that exercises the production path this task names, run it, and read the output. Accept RED only when it fails for the intended MISSING BEHAVIOUR — never a syntax error, an import error, a broken fixture, an unavailable dependency, or an unrelated failure elsewhere in the suite. If you implemented before observing RED, delete the implementation and restart the task; a test written beside its fix passes whether or not it ever exercised the defect.",
"A test must do what the user does. Exercising an alternate protocol or transport, or inspecting source and logs instead of runtime behaviour, is not evidence for a task whose acceptance names a user-visible result.",
"Your task's redCommand is executed by probes outside your control, before and after you work. A self-reported RED is not evidence and never was.",
"Rules: ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/tdd.md and ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/writing-good-tests.md govern every task — read the latter IN FULL before writing a test. Browser and web work also follows ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/testing-web.md; desktop and native work also follows ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/testing-desktop.md.",
].join("\n"),
verifierAgentType: "Explore",
}
Path containment is mechanical: check.sh's scope leg (work/scripts/scope-check.sh, given --plan) fails a changed path
under no task's writablePaths or scaffoldPaths, naming each. Lens item (2) judges intent only.
verifierAgentType and the lens agentType pin Explore because it has no Edit and no Write: a
judge that structurally cannot modify the tree beats a prompt asking it not to.
implementerAgentType is deliberately unset: the output is code under TDD, and a custom agent body would replace the software-engineering prompt the tasks depend on.
Add ${CLAUDE_PLUGIN_ROOT}/skills/dev/references/testing-web.md or
${CLAUDE_PLUGIN_ROOT}/skills/dev/references/testing-desktop.md to a task's refs when that
task's real test drives a browser or a native app. authorityExtra names them; refs is what makes
an implementer read one in full.
Handle the result per work's Phase 4 — including that a red-not-red, red-unproven or
green-not-green verdict lands in tasksThatFlagged and is fixed as a task, not as a lens.
Phase 5 — HUMAN REVIEW
work's Phase 5 unchanged, on the review surface from CLARIFY axis 7.
Red flags
| Situation | Wrong move | Right move |
|---|---|---|
| Guard denies a pre-dispatch write | --abandon to get moving | read the denial: an uncovered path is already writable, and a covered one that must pre-exist goes in scaffoldPaths. --abandon unguards the rest of the session and silences the Stop nudge with it |
| RED is an ImportError on the module being built | call it red and dispatch | the probe refuses it as could-not-run; write the throwing stub, declare it in scaffoldPaths |
| LSP enabled, binary never installed | trust the install's exit 0 | one LSP documentSymbol on a repo file before recon; ENOENT there is the whole failure |
| No test harness in the repo | proceed and test by hand | test infrastructure is the first task, decided at CLARIFY — absence of tests is never a waiver |
| The failing test needs two steps | redCommand: "build && test" | work throws on shell operators; put the steps in a script and name the script |
| Task's real test would drive a browser or an app | assert on source or logs | the runtime references are refs on that task; a screenshot with no assertion is not evidence |
| Adding a domain review dimension | declare a second lens beside this one | work takes ONE lens; add the dimension to its checklist, or make it a leg of check.sh when an exit code can decide it |
| RED reported by the implementer | accept it | probes execute redCommand on both sides; the verdict is the JS's, never the doer's |
Something work does not obviously do | write a dev/workflow.js | ask which work parameter is missing |
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Contents
- What Dewey Is
- Credential Enforcement
- Download Enforcement
- Access Method Decision Table
- Authentication
- Quick Reference: Featured Datasets
- SafeGraph Global Places Quick Reference
- Additional Resources
What Dewey Is
Dewey Data is an academic data marketplace — one institutional Platform Subscription unlocks a catalog of ~300 datasets from ~40 providers (foot traffic, POI, mobility, consumer transactions, real estate, labor). UVA Library and NYU both hold the institutional subscription; SafeGraph and most providers are free under it.
Dewey is not a SQL warehouse like WRDS. Data is delivered as partitioned Parquet/CSV.gz files downloaded via an API key. You discover datasets, read metadata, sample, filter (by date partition + columns), then download. Think "S3 of presigned Parquet links," not "PostgreSQL."
| WRDS | Dewey | |
|---|---|---|
| Data | Finance/accounting | POI, foot traffic, mobility, consumer, real estate |
| Access | PostgreSQL / SAS on the grid | File download (Parquet/CSV.gz) via API key |
| Query engine | server-side SQL | DuckDB over the files (local or remote presigned URLs) |
| Licensing | per-vendor, negotiated | one platform subscription unlocks the catalog |
| AI access | none | MCP server (api.deweydata.io/mcp) |
Credential Enforcement
IRON LAW: NEVER GUESS, INVENT, OR HARDCODE THE API KEY
The Dewey API key belongs to the **user's** account (`app.deweydata.io` → Connections → Add Connection → API Key). You cannot derive it. Keys are `akv1_` + 35 chars (40 bytes).- ALWAYS look for an existing key first:
DEWEY_API_KEYenv, then~/.config/dewey/apikey(also on rjds at the same path; agenixdewey-api-key.age→DEWEY_API_KEY_FILE). Validate before any pull:GET https://api.deweydata.io/api/v1/external/data/<prj_id>/files?page=1with headerX-API-Key; 200 = valid, 401Invalid API key.= revoked/wrong (Authorization: Beareris the wrong scheme and gives a misleading 401). Ask the user only when no key exists or it 401s. - NEVER write a placeholder like
apikey = "your_api_key"and run it — it will 401 and waste a round trip. Read fromDEWEY_API_KEYenv var or a gitignored file (~/.config/dewey/apikey). - NEVER commit the key, echo it back, print it, or paste it into a script that gets committed.
Guessing or hardcoding the key is NOT HELPFUL — every call 401s, and a committed key is a security incident the user must rotate.
Each product (dataset) has its own product path / project ID (prj_…), obtained from the dataset page: Get Data → (Skip filtering) → Connect to API / Bulk API → API URL. One API key, many product paths. If you don't have the product path, discover it via the MCP server (search_datasets) rather than guessing.
Download Enforcement
IRON LAW: NO BULK DOWNLOAD WITHOUT METADATA + SAMPLE + FILTER FIRST
Before downloading ANY Dewey dataset, you MUST:
- IDENTIFY the product path and what partitions/columns you actually need
- META — call
get_meta(deweydatapy) /get_download_info(MCP) to learn partition columns, date range, file count, total size - SAMPLE — pull 100 rows (
read_sample/ MCPsample_dataset) and INSPECT the schema before committing to a full pull - FILTER — restrict by date partition (
partition_key_after/before) AND columns; for selective pulls use DuckDBCOPY TOover the presigned URLs, never download the whole catalog - DOWNLOAD the filtered subset, then verify row counts / NULLs / date range on disk
This is not negotiable. Skipping the sample-and-filter step is NOT HELPFUL — Dewey datasets are routinely hundreds of GB to multiple TB; an unfiltered pull burns hours of bandwidth and disk for data you'll immediately throw away.
Dewey Facts
- SafeGraph Patterns is multi-TB; "download everything and filter in pandas" fills the disk before the filter ever runs — counterproductive on its own terms. Use DuckDB
COPY TOwith a WHERE clause on the remote parquet to pull only the rows/columns you need. - Column names differ by provider and release (
naics_codevsNAICS_CODE;opened_onmay not exist at all). A full pull against guessed columns is the exact incompetence the sample step exists to prevent —read_sample(nrows=100)BEFORE the full pull. - Most datasets are date-partitioned weekly; "all of it" means every weekly file ever shipped. Set
partition_key_after/beforeto the study window. - Presigned links expire in 24h (
download_files0). For large multi-day pulls usedownload_files1(page-by-page, refreshes links) — a long job ondownload_files0dies mid-pull. - A wrong
prj_product path 404s or returns someone else's data. Get the path from Connect to API or MCPsearch_datasets; hardcoding a guessed path is an unverified claim presented as fact. - Use
deweypy.get_dataset_files, notdeweydatapy.get_meta/get_file_list— the latter'sexternal-api/v3endpoint is dead (returns non-JSON / 500 →JSONDecodeError). Seereferences/deweypy-client.md. - The download service throws transient HTTP 500s on individual presigned URLs, and one bad file aborts a whole-batch DuckDB
COPY read_csv([...]). For filtered pulls: chunk (~20 files), retry per chunk re-minting fresh URLs, fall back to per-file skip; restartable via per-chunk parquet. SetSET http_timeout=120000; SET http_retries=3;. Worked example inreferences/deweypy-client.md. - Some providers gate access behind extra terms (e.g. ConsumerEdge): the web "Get Data" flow shows an "I acknowledge…additional terms" modal you must accept once before the dataset is usable / its
prj_path mints. Don't auto-accept a provider license without the user's OK. - Bulk access is per project, with limits (3 active projects, 5 datasets each; tables of one dataset count once; "New Project" disabled at 3/3). Add the table to a project, then read its API URL there. "Issue New Key" revokes the previous key immediately (a click that looked interrupted still went through): never rotate without the user's OK, and after a rotation update every copy (local file, rjds, agenix). The Bulk API panel renders the LIVE key (readonly "API Key" textbox and the deweypy one-liner), so a
take_snapshotof a get-data page contains it: never print or grep such a snapshot; capture the key withevaluate_script+filePathstraight into the key file. Flow and limits:references/access-options.md. - A "Data Request Form ... requires an institutional Dewey license" modal is not a block (BrightQuery Delaware Stock Filings): the table is added anyway. Cancel it, never submit without the user's OK, and check the project's dataset list.
references/brightquery-delaware-stock-filings.md. - MCP tools load only at session start. After
claude mcp add … dewey-prod, thesearch_datasets/sample_dataset/etc. tools are NOT available in the current session — start a new session to use them.
Red Flags — STOP Immediately If You're About To:
- Call
download_files*without first callingget_meta+read_sample→ STOP. Meta + sample first. - Download a dataset with no
start_date/end_date/ partition filter → STOP. Scope the date range. - Load a whole remote dataset into a DataFrame → STOP. Use DuckDB
COPY TO … (FORMAT PARQUET, PARTITION_BY …)to persist a filtered subset to disk. - Run a pull with
apikey="your_api_key"or any guessed key → STOP. Read env/file and validate; ask the user only if none exists or it 401s. - Write the API key into a script you'll commit → STOP. Env var or gitignored file only.
Access Method Decision Table
| Need | Method | Reference |
|---|---|---|
| Discover/search datasets, check schema, sample — from inside Claude | MCP server (api.deweydata.io/mcp) | references/mcp.md |
| Scripted Python bulk download | deweypy (recommended) or deweydatapy (legacy, product_path API) | references/deweypy-client.md |
| Selective pull — specific columns/rows from huge datasets | DuckDB over presigned URLs (read_parquet($urls) + COPY TO) | references/duckdb.md |
| R workflow | deweyr (download_dewey()) | references/deweypy-client.md |
| One-off, dataset < 2.0 GB | UI CSV download (platform → project) | references/access-options.md |
| Analyze data already on disk | DuckDB / pandas / polars over *.parquet or *.csv.gz | references/access-options.md |
Authentication
Get the key once from app.deweydata.io → Connections → Add Connection → API Key. Store it out of source control:
mkdir -p ~/.config/dewey && echo 'YOUR_KEY' > ~/.config/dewey/apikey && chmod 600 ~/.config/dewey/apikey
# or: export DEWEY_API_KEY=... (add to .envrc, which should be gitignored)
import os, pathlib
apikey = os.environ.get("DEWEY_API_KEY") or pathlib.Path("~/.config/dewey/apikey").expanduser().read_text().strip()
Institutional login (to browse the catalog / create the key) is via UVA NetBadge (use your UVA email) or NYU SSO. The Platform Subscription is what makes SafeGraph etc. free — see references/datasets.md.
Quick Reference: Featured Datasets
| Provider | Dataset(s) | What it is |
|---|---|---|
| SafeGraph | Global Places (POI), Geometry, Spend, Patterns | POI master, building footprints, card spend, foot-traffic visit patterns |
| Advan Research | Monthly/Weekly Patterns, Home Panel | Foot traffic aggregated to place & census-block |
| dataplor | POI | Global POI, strong emerging-markets coverage |
| Veraset | Movement | Device-level mobility (institutional license only) |
| PassBy | Foot Traffic | Per-POI foot-traffic analytics |
| Consumer Edge / PDI | Spend / transactions | Card & product-level purchasing |
| GovFiles | US Business Entity | Secretary of State registries, 98.6M rows incl. dissolved; uneven by state (NV 37K rows, ~2023 formations only; no DE status) |
| Exchange Data International | Global Equity Corporate Actions (WCA/RCAN) | Dividends, splits, mergers, tenders — 2001→, global, w/ CUSIP/SEDOL/ISIN/FIGI bridges |
| LinkUp | Job postings | Labor-market activity, scraped from employer career sites (2007→) |
| ATTOM / Dwellsy / RentHub | Real estate | Property records, rentals |
Full catalog (all ~250 datasets): references/catalog.md — every dataset grouped by category with time coverage, row count, size, and download access (machine-readable: references/catalog.csv). Featured-dataset detail + discovery workflow: references/datasets.md.
SafeGraph Global Places Quick Reference
Core POI schema — columns are UPPERCASE, NAICS_CODE is a string, BRANDS is a JSON-array string (extract with json_extract_string(BRANDS,'$[0].safegraph_brand_name')). Always sample before filtering.
| Column | Meaning |
|---|---|
PLACEKEY | Stable unique POI id (join key across SafeGraph products) |
LOCATION_NAME | POI name |
BRANDS | JSON array: [{"safegraph_brand_name":"…"}] — not plain text |
STREET_ADDRESS,CITY,REGION,POSTAL_CODE,ISO_COUNTRY_CODE | Address (REGION=US state) |
LATITUDE,LONGITUDE | Coordinates |
NAICS_CODE,NAICS_CODE_2022 | 6-digit NAICS (string) |
TOP_CATEGORY,SUB_CATEGORY | Category labels |
OPENED_ON,CLOSED_ON,TRACKING_CLOSED_SINCE | Open/close dates (exist but sparsely populated — NULL for BTMs) |
Resolved empirically: crypto/Bitcoin ATMs do exist as standalone POIs under NAICS_CODE='522320'; all major operators are present. But OPENED_ON/CLOSED_ON are NULL for BTMs in the current release → it's a cross-section, not a time series. Full details, the 7 BTM operators, and the worked example: references/safegraph-places.md and examples/btm_safegraph_pull.py.
Additional Resources
Reference Files
references/access-options.md— all download methods (UI, deweypy, deweydatapy, DuckDB, MCP, R), projects (limits, add-to-project flow, Bulk API/key, CDP tips), 24h link expiry, partitioning, reading data on diskreferences/deweypy-client.md—deweypy(modern CLI +auth/download) anddeweydatapy(get_meta,get_file_list,read_sample,download_files0/1) function reference;deweyrfor Rreferences/duckdb.md— selective remote-Parquet pulls,COPY TO … PARTITION_BYpattern, querying downloaded filesreferences/brightquery-delaware-stock-filings.md— BrightQuery Delaware Stock Filings: 9 tables, measured profile of 4 (rows/cols, schemas, join key, 1990-2024 years, generic class names and no votes, identifier fill, dual-class verdict), Stock Class Get Data disabled, the Data Request Form non-blockreferences/govfiles-business-entity.md— GovFiles US business-entity registry: all 7 table schemas w/ fill rates, Identifiers (documented CIK/LEI bridge is empty in the 2026-10 snapshot; FEIN in 6 states only), the missing officers/parties table, the ungraphable Relationships table (1% counterparty key),FILED_ONsentinel dates, launched Jun 2026 / empty changelog — which sparse fields may backfill and which won't, worked DuckDB joinreferences/linkup-job-postings.md— LinkUp: 12 tables across 2 products (incl. Extracted Salary, Job Descriptions, Structured Fields, Remote Tag — added Jun 2026, NOT in the stalecatalog.csv), join keys (COMPANY_ID/JOB_HASH/BASE_HASH/REQID), salary top-coding at 12k/1M, point-in-time ticker joins, the scrape-log structural-break trap, worked firm-quarter vacancy panelreferences/edi-corporate-actions.md— EDI WCA/RCAN corporate actions: event grain (options=ORs, serials=ANDs), identifier hierarchy + outturn ids, generic label/value slots, future-dated rows, Notes-table coverage gap, worked dividend/CUSIP query +EVENTCDenumerationreferences/mcp.md— Dewey MCP server URL, JSON config, the 9 tools, discovery → schema → sample workflowreferences/datasets.md— featured-dataset catalog, UVA NetBadge / NYU institutional access, discovery workflowreferences/catalog.md+catalog.csv— full enumerated catalog (~250 datasets / 39 partners) by category, with coverage / rows / column count / size / accessreferences/schemas.json— full column schemas for all ~250 datasets (keyed by slug →columns[]with name/type/description; 11,264 columns). Look up a dataset's columns here before pulling, instead of a liveget_dataset_schemacallreferences/linkage.md— cross-dataset join-key map (placekey, ticker, cusip/cik, domain, person id, lat/long, fips, zip…) — which datasets combine and on what spinereferences/safegraph-places.md— Global Places schema, NAICS 522320, BTM operator brands, opened_on/closed_on, the Bitcoin-ATM worked example
Example Files
examples/btm_safegraph_pull.py— acceptance test: filter SafeGraph Global Places to the 7 BTM operator brands + NAICS 522320, verify standalone-POI / open-close coverage, export the US subset to~/projects/batm/
Announce: "I'm using docx-render to convert this document to PDF via doc_render."
DOCX → PDF/PNG rendering
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Office docs → PDF/PNG go through the shared converter ${CLAUDE_PLUGIN_ROOT}/scripts/doc_render.py
(convert()), which picks the best engine and applies the right fixes. Do not
hand-roll soffice/libreoffice or lean on the generic docx skill's own
export — those skip the Word-fidelity path and the x2t kerning/table fixes.
Default to --renderer word (Iron Law)
For any PDF a human will read — a deliverable, an email attachment, a reading-list
file, anything you hand to the user — pass --renderer word. The bare/auto
command is NOT the safe default: auto deliberately EXCLUDES Word (the check is
if avail["word"] and allow_word, and allow_word defaults False) and silently
uses x2t/LibreOffice, which reflow the layout (real case: a 5-page doc shipped
as 7). Word works even from a background/headless job via cmux dispatch (below), so
there is no reason to skip it for a deliverable.
Reserve bare auto for parallel pipeline builds (e.g. many law-review renders
at once) where headless/parallel-safety matters more than line-exact fidelity and
the docs are pipeline-generated (x2t/soffice already grid-faithful there).
On Linux, --renderer word cannot work — it drives Word.app through
AppleEvents. Use --renderer word-remote instead: the same real Word engine
in a Windows guest, driven over SSH. See "Word in a Windows guest" below.
Always verify the engine actually used via the PDF Producer before handing off
(see table below) — auto can fall back, and --renderer word only raises if
Word is truly unavailable.
A legacy .doc goes STRAIGHT to Word — never convert it first
Both Word engines open .doc natively, and WORD_SRC_SUFFIXES admits it:
python3 scripts/doc_render.py IN.doc OUT.pdf --renderer word-remote
Never soffice --convert-to docx a .doc and render the result. That hop
silently corrupts the document, and because the damage is in the intermediate it
survives into a genuine Word render, so the Producer string still says Word and
the output still looks authoritative. Measured on a 23-page expert report:
- every footnote gained a stray superscript
?— soffice writes the real<w:footnoteRef/>and then a literal<w:t>?</w:t>run beside it, where the legacy footnote mark was; - the page count grew by one (23 → 24), so pagination no longer matched what the sender saw.
Rendering the .doc itself reproduced the sender's 23 pages with clean footnote
numbers. If a ? opens every footnote, suspect the conversion, not the sender.
A wrong extension fails silently. Word dispatches on the suffix, so a .doc
staged under a .docx name makes it bail: the guest's scheduled task completes,
writes no PDF, and the only symptom is guest render did not complete. The
transport carries the real suffix through — keep it that way.
Entry point
# CLI (faithful Word engine — DEFAULT for anything a human reads):
python3 ${CLAUDE_SKILL_DIR}/../../scripts/doc_render.py IN.docx OUT.pdf --renderer word
# auto engine (LibreOffice/x2t; NO Word) — ONLY for parallel pipeline builds:
python3 ${CLAUDE_SKILL_DIR}/../../scripts/doc_render.py IN.docx OUT.pdf
import sys; sys.path.insert(0, "<plugin>/scripts")
from doc_render import convert
convert("in.docx", "out.pdf", renderer="word", allow_word=True) # gold standard
convert("in.docx", "out.pdf") # auto (headless)
Agents without the Skill tool (most workflow subagents): run the CLI above
directly — you don't need to invoke this skill, just call doc_render.py.
Which engine, and why it matters
| Engine | Fidelity | Notes |
|---|---|---|
Word (--renderer word) | gold standard | native layout; only engine that keeps an auto-wrapping table as a grid in a hand-authored docx (LibreOffice collapses it to a stacked column). Recomputes Word fields (REF/NOTEREF/PAGEREF/TOC). macOS only — it drives Word.app over AppleEvents. |
word-remote (--renderer word-remote) | gold standard | the same real Word engine, in a QEMU Windows guest over SSH. The Word path on Linux, where --renderer word cannot work at all. Also usable from macOS against a guest on that host. |
| x2t | good | OOXML-native; correct per-section footnote restart; doc_render injects GPOS/kern + EB-Garamond so it matches. |
| LibreOffice | good except | wrong for per-section/page footnote restart; collapses auto-wrapping tables not pre-broken upstream. |
convert() auto-falls-back Word → x2t/soffice. Verify which ran via the PDF
Producer: macOS … Quartz PDFContext = Word; LibreOffice … = LibreOffice.
Garamond documents on macOS need a one-time setup. x2t mis-measures the macOS (Monotype) Garamond italic face badly enough to cram every upright Garamond run.
${CLAUDE_PLUGIN_ROOT}/scripts/setup_garamond_render_override.pywrites a four-face override to~/.config/x2t-render-fonts/garamond/— macOS Garamond for regular/bold, EB Garamond for the slanted faces (--all-ebfor the all-EB variant) — thenrm -rf ~/.cache/x2t-docfontsto re-stage. Full measurements:docs/investigations/2026-06-19_x2t-kerning-patch.md, Part 2.
Word from a background/headless job (the non-obvious part)
A detached Claude job is in a non-console GUI session without Word's TCC grant, so
direct AppleEvents fail with -600. doc_render transparently dispatches the
render into a cmux pane (console session, TCC-granted) and falls back to
x2t/LibreOffice if that's unavailable. Prereqs + full root-cause:
docs/investigations/2026-06-22_word-render-cmux-dispatch.md. Disable with
$DOC_RENDER_NO_CMUX=1.
Driving the Mac's Word from another machine over SSH — doesn't work
The cmux rescue above assumes you are on the Mac. Invoking --renderer word
over SSH from another host (e.g. a Linux box rendering on mbp) fails
differently and has no fallback — cmux dispatch fails too, because there is
no console session on the far end to dispatch into:
doc_render: word renderer failed: Word direct render failed
([Errno 1] Operation not permitted:
~/Library/Containers/com.microsoft.Word/Data/wordrender/<uuid>);
cmux dispatch also failed (…same…)
Note this is a filesystem permission error on Word's app container, not the
AppleEvents -600 of the local case — an SSH session is outside the TCC grant
entirely. launchctl asuser $(id -u) … does not rescue it (Could not switch to audit session: Operation not permitted — needs root).
Fixes, in order of preference:
- Use
word-remote(next section) — a Windows guest is the supported remote path; driving the Mac's Word from off-box is not. - Run the render from a terminal inside the Mac's GUI login session (then the normal local path, incl. cmux dispatch, applies).
- Grant Full Disk Access to
/usr/libexec/sshd-keygen-wrapperin System Settings → Privacy & Security, after which headless SSH renders work.
Do not paper over this by falling back silently — an explicit
--renderer word deliberately raises rather than downgrading.
Word in a Windows guest (word-remote) — the Linux path
--renderer word is macOS-only. word-remote runs the same Word engine in a
Win11 guest and drives it over SSH, so Linux gets gold-standard fidelity:
python3 scripts/doc_render.py IN.docx OUT.pdf --renderer word-remote
Provisioned by the programs.wordRender nix module — word-render and
word-render-install-fonts on PATH, transport at
~/.local/share/word-render/word_render_remote.sh (override with
$WORD_RENDER_REMOTE). Full setup: ~/nix/modules/shared/word-render/README.md.
On Linux the guest is a docker container, not QEMU/qcow2. vm/start-winvm.sh
and ~/.local/share/winvm/*.qcow2 are the macOS path. Linux uses
dockur/windows — the image Omarchy's omarchy-windows-vm drives — as container
omarchy-windows, disk at ~/.windows/data.img, compose at
~/.config/windows/docker-compose.yml. There is no qcow2 to find here.
docker start omarchy-windows # or: omarchy-windows-vm launch -k
until ssh -o ConnectTimeout=5 -o BatchMode=yes word@winvm exit; do sleep 15; done
A cold boot takes several minutes before sshd answers, and the error changes
as it comes up: Connection refused (container down) → Connection reset by peer / timed out during banner exchange (Windows still booting) → success.
Only the first means something is wrong. Wait on the until loop rather than
concluding the guest is broken; docker logs --tail 20 omarchy-windows shows
boot progress.
Selection rules:
- Explicit
--renderer word-remotealways runs it (and raises rather than falling back, like every explicit engine). autopicks it only withallow_word=True, and only when local Word is unavailable — i.e. it is the Linux stand-in forrenderer="word", preferred over the lower-fidelity engines rather than silently downgrading.autonever reaches it in the fallback cascade. Booting/using a VM is not something best-effort should do behind the caller's back.
Availability is a file check on the transport script, not an SSH probe — a
reachability test would cost a round-trip (and can hang on a suspended VM) on
every convert(). A down guest surfaces as a render error naming the fix.
A fresh guest silently renders the wrong fonts. Word substitutes
Cambria/Calibri for any font it can't resolve and still exits 0, so the render
"succeeds" with wrong typography. Run word-render-install-fonts once per
guest, then verify with pdffonts — never trust the exit code. (Stock
lmodern does not work: Word won't render CFF-flavoured OpenType, and it
matches families on name ID 1. The nix module ships a converted set.)
Google Docs exports
A docx exported from Google Docs can carry OOXML package corruption (case-broken
customXML part paths) that makes Word pop a "recover unreadable content" modal
on open — fatal to a headless render. The Word path auto-repairs it via a
preflight (${CLAUDE_PLUGIN_ROOT}/scripts/docx_repair.py); you'll see Word preflight — repaired Google-export package … on stderr. Repair a docx standalone with
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/docx_repair.py in.docx [out.docx].
Related skills
Part of the document skill group (extract → create → repair → build → render → verify):
- law-review-docx — builds a .docx from markdown (template + pandoc), then renders.
- docx (generic) — edits docx content (tracked changes, comments, text).
- docx-repair — repairs a cloud-editor-damaged .docx (package/XML wiring + footnote markup).
- xlsx recalc / pptx-render — spreadsheet recalc / slide inspection.
DOCX Repair (Google Docs / Word Online damage)
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Cloud editors damage a .docx in independent ways. This skill is the front door for all of them; run only the track(s) you need.
| Damage class | Symptom | Fix |
|---|---|---|
| A. Package / OOXML wiring | Word pops "recover unreadable content?" or refuses to open; LibreOffice won't load; phantom blank page | ${CLAUDE_PLUGIN_ROOT}/scripts/docx_repair.py (plugin root) — references/package-repair.md |
| B. Footnote & cross-reference markup | Bios show 1,2,3 not *,†,‡; numbering starts wrong; "supra note N" points to the wrong footnote; missing separator line | the footnote scripts — references/footnote-procedure.md |
| B1. Cross-references aimed at the wrong source | Lin, supra note 130 where note 130 is a different work; Rebuttal Report, supra note 22 where 22 defines the deposition; a short form surviving a renumber that moved its target | check_crossrefs.py <file.docx> — read-only, exit 1 on any problem |
| C. Document content (boxes + headings + cruft) | Visible boxes around freshly-edited text; heading-looking lines not styled as headings; same-style headings rendering differently; blank heading lines; bloated XML full of all-zero rsids, no-op shading, explicit b=0/i=0/u=none, redundant black color & default fonts | fix_footnotes.py's document.xml passes — references/content-cleanup.md |
| D. Presentation hygiene | A footnote renders blue/underlined in the PDF but looks normal in Word; a URL prints one address and navigates to another; heading gaps uneven page to page; tracking params (?utm_source=…) behind a clean-looking link | docx_links.py, docx_spacers.py — references/presentation-hygiene.md |
They are decoupled: package repair fixes the part wiring (never touches content); footnote repair fixes the footnote markup; content cleanup strips Google-Docs leftover content controls and normalizes headings. A file can need any, all, or none. If you don't know which, run the package check first (it's a no-op on a clean package), then the footnote pass (it carries the content cleanup).
Heads-up:
docx-render's Word path already composesdocx_repair.pyas a preflight, so a Google export "just renders" without a manual Track A. Run Track A manually when you need the repaired file itself (to hand back, edit, or footnote-fix), not just a PDF.
Reference index — read the file for the track you are running
All procedural detail lives in references/. Read only what the job needs.
-
references/package-repair.md — Track A. The two concrete Google Docs export defects (case-mismatched
customXML/customXmlOPC part references; leftover<w:evenAndOddHeaders/>), and the CLI + Python API for${CLAUDE_PLUGIN_ROOT}/scripts/docx_repair.py. Read when Word calls the file corrupt, refuses to open it, or shows a phantom blank page — or before handing back a repaired file rather than a PDF. -
references/footnote-procedure.md — Track B, the canonical procedure. The exact four-step order for a Google Docs round-trip (accept changes →
fix_footnotes.py→create_crossrefs.py --baseline→ render with Word), what to verify in the render, the incident-grounded procedure facts (why step 3 must follow step 2, why--baselineis not optional, why LibreOffice lies), the list of symptoms footnote repair applies to, and the quick-start "which script do I want?" routing. Read when running any footnote repair — start here. -
references/footnote-scripts.md — Track B, script detail. Per-script behaviour and every flag for
fix_footnotes.py,create_crossrefs.py(including how--baselineremaps stale numbers) andrefresh_noteref_caches.py(why the naive approaches fail, its requirements and intentional scope), plus the footnote numbering-offset fix (numRestart). Read when you need a specific flag, or the exact semantics of one of the three scripts. -
references/footnotes-reference.md — Track B, OOXML technical reference. (1) Run-level editing gotchas (NBSP, cross-run matching,
xml:space); (2) cloud editor damage patterns — what gets destroyed and why; (3) direct ZIP surgery patterns that bypass Document libraries; (4) the numbering-restart details and the critical rule thatnumRestartgoes insettings.xmlONLY. Read when hand-editing footnote XML or debugging behaviour the scripts do not cover. -
references/content-cleanup.md — Track C. The four content passes
fix_footnotes.pycarries: strippinggoog_rdkcontent controls (the "boxes", default on), heading normalization (--normalize-headings), the OOXML hygiene / de-cruft rules (default on, with the exact strip/keep lists), body-indent normalization (--normalize-body-indent), and applying the template's body styles (--restyle-body), plus the content-cleanup facts. Read when boxes, inconsistent headings, stray indents or XML bloat are the complaint, or before passing any of those flags. -
references/presentation-hygiene.md — Track D.
docx_links.py(tracking-param stripping, SSRN canonicalization, footnote hyperlink unwrapping — and why display text and relationshipTargetmust both be rewritten) anddocx_spacers.py(manual spacer removal, why a text-empty paragraph may not be empty, the title-page collapse hazard), plus why both edit bytes rather than the ElementTree tree. Read when the PDF shows blue/underlined footnotes, a link that navigates elsewhere, or uneven heading gaps.
Related (document skill group)
This skill owns the REPAIR stage — package wiring (Track A), footnote
markup (Track B), and document-content cleanup (Track C) — for a .docx damaged
by a cloud editor. Adjacent stages:
- Build a styled
.docxfrom markdown →law-review-docx(itsbuild_docx.pychains this skill's footnote repair + NOTEREF conversion after the pandoc build). - Render to PDF/PNG →
docx-render/${CLAUDE_PLUGIN_ROOT}/scripts/doc_render.py(Word path composes Track A'sdocx_repair.pyas a preflight automatically). - Check cross-references before believing them.
check_crossrefs.pyis the only pass that READS the target note: it verifies the short form names a source the target actually contains, on top of existence and direction.create_crossrefs.py --dry-runproves only that note N exists and will report "25 targets, 25 valid" over a document whose cites point at the wrong documents — it is a conversion preflight, not an audit. Runcheck_crossrefs.pyon any draft returned by a coauthor, and after every renumber. It name-matches loosely (any capitalised word in the short form appearing in the target), so it catches a cite aimed at the wrong source, never a wrong pincite. - Footnote repair lives only here.
fix_footnotes.pyis the single canonical Google-Docs / Word-Online footnote fixer; there is no second copy. (Bluebook'screate_crossrefs.py+audit_crossref_targets.pyremain a deliberate, actively-used cross-reference fork with their own retargeting strategy — a different concern, not a footnote-fix duplicate.)
See the full document skill group.
DOCX ↔ Typst Bridge
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Typst source is the thing the repo keeps. Word is the thing coauthors edit. This skill moves a document across that boundary in both directions and reconciles what comes back.
The load-bearing fact: typ → docx → typ reaches a fixed point after one pass. So
the pipe's output is itself valid Typst, the canonical form can be committed, and
reconciling a coauthor's returned file collapses from "read two documents side by side"
to git merge-file.
existing.docx ──canonicalize.py --from-docx──> body.typ + media/ (bootstrap, once)
│
main.typ ──typst compile──> PDF │
│ │
│ #include ▼
▼
body.typ ──build.py──> paper.docx ──email──> coauthor edits in Word
▲ │
│ │ sends back
└──── reconcile.py <── merged body.typ <── returned.docx
│
└──comments.py──> comments.json
Scripts
All under ${CLAUDE_SKILL_DIR}/scripts/. Each is self-contained and prints --help.
Run them with uv run --script, not uv run python3. Four of the five declare
lxml in a PEP 723 header, and uv run python3 <path> ignores that header and fails
with ModuleNotFoundError: No module named 'lxml'. --script (or executing the file
directly, since the shebang is uv run) reads the header and provisions the dependency.
| Script | Direction | Does |
|---|---|---|
build.py | typ → docx | Convert with --reference-doc styles and stamp provenance, in one step |
canonicalize.py | docx → typ | Bootstrap an existing Word manuscript (--from-docx); put a body file on its fixed point; --check gates it; --lint guards the body/main split |
reconcile.py | docx → typ | Resolve the ancestor, three-way merge a returned file against the repo source |
comments.py | docx or Drive → JSON | Extract comments with their anchor text, resolved state, and threading |
provenance.py | — | Read/write the stamp directly (build.py already applies it) |
expand_citations.py | typ → typ | Freeze every computed reference — #cite(...) and @label — into the literal body the docx path needs |
make_redline.py | docx × docx → docx | Rebuild a coauthor's untracked edits as real tracked changes, against a baseline (stdlib + LibreOffice; no PEP 723 header) |
The main.typ / body.typ split
main.typ #import / #let / #show / #set, then #include "body.typ" ← typst compiles this
body.typ pure markup: = headings, prose, #emph, #footnote ← pandoc reads this
Both paths see the same prose and neither degrades the other. The split is not stylistic
tidiness — see the first fact row in
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/facts.md.
References — read the one the task needs
| Read | When |
|---|---|
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/conversion.md | Any actual conversion: bootstrapping a Word manuscript into body.typ, building the .docx, reconciling a returned file, rebuilding untracked edits as a redline, extracting comments |
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/citations.md | A manuscript whose numbers must maintain themselves — supra note N, Section IV.B, Id.; the bluebook.typ renderer, bib_to_entries.py, hand-authored short forms, the entries schema, the two-body-file split |
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/facts.md | Before changing the conversion path, or when a build/round trip behaves in a way the commands do not explain — the pandoc and typst defects this skill normalizes around, and why each rule exists |
${CLAUDE_PLUGIN_ROOT}/skills/docx-typst/references/maintenance.md | Changing this skill or its scripts: the test suite, the two tests that pin bugs, and what "verified" requires |
Red Flags — STOP
| Action | Why wrong | Do instead |
|---|---|---|
About to point pandoc at main.typ | Its #show rules destroy heading semantics in the docx | Point it at body.typ |
About to pass --allow-styling to clear a lint error | Ships a headingless Word file | Move the directives into main.typ |
About to hand-edit the returned .docx and call it reconciled | The repo source still diverges; the next build overwrites the edits | Run reconcile.py and merge into the source |
About to resolve <<<<<<< markers by deleting one side wholesale | Discards a coauthor's edit unreviewed | Read both sides; ask the user when the prose choice is theirs |
About to commit a merge without reading .merged.typ.diff | The merge is a claim about someone else's edits, unverified | Read the diff, then commit |
About to send a .docx built from an uncommitted source | Fallback 3 needs a committed blob; the ancestor is unrecoverable | Canonicalize, commit, then build |
About to run --from-docx without --media-dir because the error is in the way | Every figure is dropped and the output still looks complete | Name the sidecar directory; it is one argument |
About to invoke a script with uv run python3 <path> | The PEP 723 header is ignored and the lxml scripts die on import | uv run --script <path> |
About to hand-fix Officers" in a recovered file | The converter did it, not the source; hand-fixes are re-corrupted next pass | Re-recover with current canonicalize.py, which restores ’ |
| About to name a custom citation function in the body file | pandoc dies with Identifier not found and no docx is produced | #show cite: over the built-in #cite(<Key>) |
About to put live #cite in the file reconcile.py merges into | It can never be canonical; the merge churns on every citation | Keep the symbolic form in body-src.typ, generate the literal one |
About to write #cite(<a>); #cite(<b>) in a stacked footnote | typst groups them and the ; vanishes from the PDF | #cite(<a>)#[;] #cite(<b>) |
About to hand-edit the generated cite-data.typ to fix a short form | It is regenerated, and --diff will report your fix as a delta forever | Fix the .bib or the CSL, then regenerate and diff |
About to patch hayagriva to get supra note N | It is statically linked into typst, and note numbers are a layout property a patch cannot reach | Render citations in typst with a #show cite: rule |
About to typst compile a freshly recovered body and conclude the conversion failed | Word's TOC arrives as links to bookmarks that are not labels | Delete the recovered TOC block; #outline() in main.typ replaces it |
Scope
Owns typst → docx, and Bluebook citation rendering in typst — the renderer lives
here rather than in bluebook because it is co-designed with expand_citations.py, which
reads the <bb-out> tag it emits, and that two-file split exists only because of the docx
round trip. bluebook owns the RULES; this skill owns making typst emit them.
law-review-docx and law-econ-docx own markdown → docx; different input format, no
overlap — a markdown manuscript gets Bluebook from pandoc-citeproc and the bundled CSL, and
needs nothing here. docx-repair fixes OOXML damage from a cloud round trip and composes
cleanly before reconcile.py when a returned file is also damaged.
DS rules
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
These are binding on every data task, however you were launched. Each entry is the rule statement from its canonical file, printed at load time from the corpus itself; open that file for the rationale, the examples and the checker that enforces it. When this skill is invoked with arguments, they are your task: do it under these rules.
!c=${CLAUDE_PLUGIN_ROOT}/scripts/load-constraints; [ -x "$c" ] && exec "$c" ds --digest; echo "(rule loader unavailable: NO DS rule was loaded — none of C1-C6, V1-V9, A1-A6, E1-E7 is in your context; read them from the plugin's rules/ before any data work)"
ds — a data project, run through work with a computed data-quality gate
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
DS constraints in force — every rule scoped to this workflow, discovered from the corpus so a new one needs no edit here; read the ones your task touches:
!c=${CLAUDE_PLUGIN_ROOT}/scripts/load-constraints; [ -x "$c" ] && exec "$c" ds; echo "(constraint index unavailable: NO DS rule is listed here — the four aggregates under skills/ds/rules/ are not the whole set)"
The lifecycle is work. Read it and follow it.
This file is a delta: it supplies the domain — the CLARIFY axes, the plan grammar, the lens,
the mechanical checks, the refs, the authority text. It ships no workflow.js and restates none of
work's mechanics.
What makes a run ds rather than plain work is one thing: every declared data output is verified
by a computed runner, not by the agent that produced it. checks/ds/constraints/ds-dq.py reads the approved
plan's ## Data Outputs table and opens each artifact itself; work's JS reads its exit code.
Write surface
Main chat clarifies, plans and dispatches. It does not do the analysis, and it never writes a .py,
.ipynb, .R, .sas, .sql or .qmd file — not by Write/Edit, and not by Bash (python3 -c,
pixi run python, inline pandas/numpy). Analysis runs in a dispatched agent. work's dispatch is
already structural and its judges are pinned to Explore, so this line is a rule on you, not a hook;
reach for the workflow first rather than after a refusal.
Phase 1 — CLARIFY
work's Phase 1, on these axes.
Loading a dataset or proposing a method before the user has stated the outcome anchors the work to whatever data happens to be nearby. Skipping a necessary profile replaces evidence with a convenient guess. Neither shortcut is helpful: it creates an analysis that is fast to start and expensive to correct.
| Axis | Establish with the user |
|---|---|
| Outcome | Question, audience, decision the analysis informs, and deliverables |
| Universe | Unit of observation, entity inclusion rule, sample period, and authoritative source when sources disagree |
| Sources | Available data, access/location, expected refresh/vintage, and known restrictions |
| Constraints | Replication target, required methods, deadline, compute/cost limits, and reproducibility needs |
| Evidence | What makes each planned output credible: a check, an inspectable exhibit, a source comparison, or user judgment |
Ask in one AskUserQuestion call when answers are independent. Ask cascading questions separately:
a source choice that determines available variables must be answered before variable questions.
Three facts decide these axes, and each prevents a defect that is expensive to find later:
- A sample period and an entity universe are research choices, not properties inferred from the first query result. A rate at the wrong grain is a different statistic with a familiar label.
- The universe predicate belongs where scope is decided. Applying it to a lookup or a denominator can turn available values into nulls.
- A criterion that cannot name evidence is a wish. The explicit
TBD (<phase>)convention is legitimate only when profiling is the scheduled evidence-producing phase; never invent coverage.
work's remaining axes are taken as work states them, with two domain bindings: work axis 4
(observable success criteria) is answered with the DQ runner over the plan's ## Data Outputs plus
the target project's own test and lint commands, and those strings become the --test-cmd / --lint-cmd arguments of the single ds mechanical entry (Phase 4); work axis 5 (review surface) is answered by the plan's
## Review Surfaces section. The ds-dq entry is never conditional. CLARIFY records, for each of test and lint, either the
project's command or an explicit "none" — an unanswered one is a refusal (check.sh exits 2), never
a leg that ran nothing. A "none" becomes --no-test-cmd / --no-lint-cmd; work's probe runs a
cmd verbatim, so never leave a placeholder in it.
Then gather planning evidence — a read-only profile, never implementation. For each source that materially affects the plan, collect or dispatch a profile covering: location/access, approximate shape, columns/types, date coverage, and likely row grain/key; nulls, duplicates, type drift, category/distribution anomalies, and likely join risks; raw size and whether source-side filtering, caching, partitioning, or server-side computation is needed; and a small representative sample only when access and project policy permit it. Use direct read-only inspection for one small local source; for independent sources, dispatch read-only profilers in parallel and consolidate. Do not make a profiling agent implement a pipeline, mutate project files, or spawn further agents. Do not pull a full source merely to estimate it. If an answer depends on implementation, state it as a planned evidence task rather than pretending the profile established it.
Four discovery rules run in this same planning step, each governed by its own constraint file:
- Large sources. If a source may be large (roughly 50M rows, 500 MB to ship, or described as
bulk/large/uncertain), read
${CLAUDE_PLUGIN_ROOT}/rules/ds-pull-sizing-decision.mdand follow it. The profile must compare filtered raw and candidate aggregate/server-side paths before planning commits to one. Do not pull a full source merely to estimate it. - External-skill discovery. When the plan will use another skill or data provider, discover its
relevant references and examples before choosing an approach: read
${CLAUDE_PLUGIN_ROOT}/rules/ds-external-skill-decision.mdand follow it. Record the resulting ADOPT, PATCH, or GREENFIELD decision in the plan itself. - Master datasets. For multi-output work that shares a sample, read
${CLAUDE_PLUGIN_ROOT}/rules/ds-master-datasets.md. Plan the minimal canonical analysis dataset(s), their grain and keys, and which planned outputs consume each one. - Parameter transparency. For analytic filters or tunable thresholds, read
${CLAUDE_PLUGIN_ROOT}/rules/ds-parameter-transparency.md; name one configuration location, rationale, and treatment of convenience choices in the plan.
Phase 2 — PLAN
work's Phase 2. The plan opens with frontmatter workflow: ds — required, so a context clear at
approval resumes here and not in work.
Five domain requirements on the plan:
-
The source/access strategy and the profile-derived risks. The plan states how each source is reached, and the data-quality and scale risks the read-only profile actually surfaced. A risk found at CLARIFY that reaches no plan section is a risk nothing acts on.
-
Reproducibility decisions, appropriate to the work: source vintages, seeds, environments and config.
R1(fresh re-run reproducibility) is judged by the review lens's data-quality item, so a plan recording none of these leavesR1with nothing to judge against. -
A
## Data Outputstable, in exactly this grammar, becausechecks/ds/constraints/ds-dq.pyparses it and an output absent from it is one nothing will check:## Data Outputs | Path | Grain | Key Columns | Required Window | |---|---|---|---| | data/processed/panel.parquet | one row per firm-fiscal-year | pk: gvkey, fyear; event: gvkey, datadate | datadate: 2005-01-01..2025-12-31 | | data/processed/industry_xwalk.csv | one row per SIC code | sic | n/a |- Path — the produced artifact, relative to the project root.
.parquet,.csv, or.tsv; one row per artifact the plan promises to produce. - Grain — the declared row grain, in words. This is the claim
DQ3verifies rather than assumes. - Key Columns — the declared primary key
DQ3atests for uniqueness. Writea, bfor a bare primary key, orpk: a, b; event: c, dto also declare the coarser business/event keyDQ3cneeds to catch amendments and restatements. Without anevent:clauseDQ3creportsN/A, and the runner says why. Addconstant: c(one repeated non-null value, e.g.paperin a per-paper file) to exempt that column fromDQ1; an all-null column still fails. - Required Window — the sample period
COVchecks, as[column: ]YYYY-MM-DD..YYYY-MM-DD. Writen/afor an unwindowed output;COVthen reportsN/Awith that as its reason. Naming the column is optional only when exactly one date column exists; an ambiguous window is aFAIL, not a pass.
- Path — the produced artifact, relative to the project root.
-
A
## Review Surfacessection naming the concrete tables, figures, notebook exports, diagnostics, or decisions the user will inspect during human review. -
refson every task row and on the lens — required, may be empty.work's spine does not validate it;wc-probeP7 refuses an absent key in THIS file, so a live run assembled from an approved plan is unchecked. Writerefs: []to state "no domain rules" rather than omitting the key. -
Batch extraction strategy. A task whose work is per-document LLM coding or extraction is a task that LAUNCHES a batch job on pre-cut excerpts, provider picked by
batch-router(with a 1-request synchronous test first). It is never an agent reading the documents. Dual-class share status of a 10-K is already built:${CLAUDE_PLUGIN_ROOT}/skills/wrds/scripts/dual_class/(extractor + gated classifier;${CLAUDE_PLUGIN_ROOT}/skills/wrds/references/dual-class.md).
Phase 3 — GOAL
work's Phase 3 unchanged.
Phase 4 — the work call
The args go in the plan's <!-- work:dispatch --> arming block, and the dispatch is work's own
work-dispatch.sh — never a hand-written runner line. That script owns the TIER 1 plan-lint
gate, which refuses to dispatch on a major/critical plan finding and fails CLOSED on a verdict it
cannot count; hand-rolling the invocation silently drops it. work owns the wait, the result handling
and the return shape too, and work-result.sh reads the verdict. This run's projectDir is the
session repo, so work's own run directory is already inside it and no --run-dir override applies.
There is no built-in Workflow call — the guard at
~/.claude/hooks/main-thread-guard.sh denies that tool outright.
bash ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/work-dispatch.sh # armed plan; or pass one
bash ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/work-dispatch.sh --provider codex # "run this through codex"
Forward the provider. A provider named in this skill's $ARGUMENTS, however it is spelled —
--provider codex, --dispatch codex, "run this on gpt" — is --provider codex on the line
above. --provider is the only spelling the scripts take, and both reject the others by name.
Omitting it silently runs the user's codex request on claude.
const goal = /* one sentence */
const planPath = /* the approved plan's path */
// shq single-quotes a command so quotes and $ reach check.sh intact.
const shq = s => `'${s.replaceAll("'", `'\\''`)}'`
const testCmd = /* the project's test command, or null when CLARIFY recorded "none" */
const lintCmd = /* the project's lint command, or null when CLARIFY recorded "none" */
const args = {
projectDir,
goal,
// The plan's table verbatim. Every task carries refs, empty or not. The four constraint
// aggregates are UNCONDITIONAL — every implementation task carries all four, alongside
// whichever vendored references that task's own work needs.
tasks: [
{ id: "T1",
name: "<data pull or pipeline task>",
work: "<what to build>",
writablePaths: ["<narrow>"],
acceptance: "<the criterion the verifier checks>",
refs: ["${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-conventions.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-analysis-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-engineering-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/references/etl-enforcement.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/references/sql-patterns.md"] },
{ id: "T2",
name: "<task whose acceptance is evidentiary>",
work: "<what to build>",
writablePaths: ["<narrow>"],
acceptance: "<the criterion the verifier checks>",
refs: ["${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-conventions.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-analysis-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-engineering-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/references/verification-patterns.md"] },
],
// The DS gate: the runner reads the approved plan's ## Data Outputs and opens each artifact
// itself. ds-dq is never conditional. tests and lint are the TARGET project's own commands,
// collected at CLARIFY: a command, or an explicit "none" that becomes --no-test-cmd / --no-lint-cmd.
// ONE entry point: a list of N independent commands loses one silently. The project's own test and
// lint commands are ARGUMENTS because this skill cannot know them — omit both a command and its
// opt-out and check.sh exits 2.
mechanicalChecks: [
{ name: "ds",
cmd: `bash ${CLAUDE_PLUGIN_ROOT}/skills/ds/scripts/check.sh --plan ${planPath} --project-dir ${projectDir}${testCmd ? ` --test-cmd ${shq(testCmd)}` : ' --no-test-cmd'}${lintCmd ? ` --lint-cmd ${shq(lintCmd)}` : ' --no-lint-cmd'}` },
],
// check.sh runs every checks/ds/constraints/ script through `workflow-check ds --constraints-only`. work-checks.sh runs ruleChecks as a command (no agent); Jev scores the rules under checks/ds/rules/ (A1, A4, DEN, DQ4, DQ6, E7, M1, R1, UNI; DQ1 waits in checks/ds/rules/uncalibrated/ and ds-dq.py decides it); a rule with p >= 0.85 lands in rulesThatFailed and blocks, lower ones reach the lens as a ranked checklist.
ruleChecks: { name: "jev-rules", cmd: `bun ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/rule-check.ts --workflow ds --project-dir ${projectDir} --plan ${planPath}` },
// One lens after verification and mechanical checks; its checklist covers all five dimensions.
//
// It does NOT pin Explore. Explore is a built-in agent with a PREDEFINED prompt that no preloaded
// skill reaches, and it skips the CLAUDE.md hierarchy — so a constraint-indexed judgement
// dispatched there is graded from memory of constraint ids it was never given. ds-reviewer's body
// is a file this repo controls, and it is read-only by tools allowlist AND by
// tests/agent-contract.test.mjs — the same structural property Explore is pinned for, in an agent
// whose prompt can be told what it is grading. The four aggregates are index tables in skills/ds/rules/ that point at the
// canonical bodies under ${CLAUDE_PLUGIN_ROOT}/rules/ and are not copies of them. They reach the lens as refs,
// with the rule bodies its checklist grades; ds-reviewer has no Skill tool, so refs are its only guaranteed route.
lens: {
agentType: "ds-reviewer",
refs: ["${CLAUDE_PLUGIN_ROOT}/skills/ds/references/ds-checks.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/references/verification-patterns.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-common-conventions.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-analysis-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/ds-engineering-constraints.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-se-matching.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-join-diagnostics.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-p-hacking-prevention.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-sample-selection.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-assumption-over-evidence.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-failure-loudness.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-nondeterminism-sources.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-rerun-equivalence.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-misleading-charts.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-companion-figures.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-sample-coverage.md",
"${CLAUDE_PLUGIN_ROOT}/rules/ds-pull-sizing-decision.md"],
prompt: "Judge the datasets, tables and figures in the working tree against the approved plan and the goal. Read every file in refs in full before judging the items that name one. The binding constraint statements are the rule bodies under rules/ in refs; the four aggregates in refs are the index to them, not the statements. CHECKLIST, every item in scope on every run: (1) CRITERIA vs ARTIFACTS — for each success criterion in the plan and goal, is there an artifact in the tree that satisfies it? Missing or partial satisfaction is a finding, CRITICAL where the deliverable cannot stand without it. (2) SCOPE fidelity — did the changes stay inside the plan's task table and writable paths? Out-of-scope edits, unrequested features and silently skipped plan items are findings, CRITICAL where an edit landed outside every declared writable path. (3) DATA-QUALITY JUDGEMENT, against ds-checks.md. A1, A4, DEN, DQ4, DQ6, E7, M1, R1 and UNI are the jev-rules verdicts in the digest: do not re-grade them, except to rule on a verdict the digest ranks below the block line. A rule in rulesThatFailed is not re-judged either: diagnose it and route it like a mechanical failure. Jev does not score DQ1 or DEL; ds-dq.py decides DQ1. DEL (coverage improved because the base shrank) is yours and MODEL-EVALUATED: report it with the evidence you actually read, never as PASS, which presents a judgement as a computation, and a coverage improvement not attributable to the base shrinking is a finding. Two judgements reach past the wired rule texts and stay yours, stated as broader than any verdict in the digest: whether the sources the outputs combine admit the same entities, which one shared filter predicate does not establish, and whether the vintages, environment and config the plan records would let a fresh re-run reproduce the result, which seeded draws alone do not. Plan criterion compliance is item (1). A judgement you cannot support with evidence you actually read is a finding, never a pass. (4) METHODOLOGY, against verification-patterns.md: a statistic computed at a grain other than the one the plan declared, a universe predicate applied where scope was not meant to be decided, a claim whose evidence is read from code rather than observed from a run (you cannot run anything: report it as unverified), and an output the plan promised that nothing checks. (5) DS-CONSTRAINTS — grade the implemented code and outputs against the rule bodies in refs (the binding statements behind C1-C6, V1-V9, A1-A6, E1-E7); the four aggregates in refs are their index. For a constraint whose body is not in refs, open its file under ${CLAUDE_PLUGIN_ROOT}/rules/ with Read before grading it. Grade every id except the ids item (3) reserves and the ids a line from a checks/ds/constraints/ script in the digest has already decided: those are not re-graded. Grade only the constraints the task actually touches; an engineering constraint applied to a pure analysis task is a wrong finding that costs a round. Report every constraint finding with the file, the line and the quoted code, naming the constraint id, and list every id you considered including those you judged satisfied. NEVER report a constraint judgement as a computation — it is MODEL-EVALUATED. Severity across the whole checklist: `major` at minimum, `critical` where the defect invalidates the output's stated grain, universe or inference, never `minor`, which would leave the gate passing over a real universe defect. MODE — RED (a task was flagged or a mechanical check failed): diagnose EVERY failure in the digest and route it, naming cause and fix, to the task id whose writablePaths own the file, or to 'plan' when no task can own the fix. MODE — GREEN (everything passed): make one open-ended pass over the whole checklist and report each finding with an ownerTask. In both modes, rule every carried finding open or closed against evidence you actually read; silence leaves it open.",
},
authorityExtra: [
"THE DS CONSTRAINT CORPUS IS ONE COMMAND AWAY. Run `${CLAUDE_PLUGIN_ROOT}/scripts/load-constraints ds` for the index — the C/V/A/E ids the ds-rules digest prints plus any further rule scoped to this workflow, each with an openable path — then read the ones your task touches. Do not work from a paraphrase of them in a brief: a paraphrase is a lossy copy of a corpus that moves.",
"IRON LAW OF DS PLANNING — ask before you look; plan only from evidence you actually gathered.",
"IRON LAW OF DS VERIFICATION — no check result without the runner's own output. A mechanical check reported from reading the code, or an N/A justified by a reason the model composed, is the model certifying its own enumeration. Every mechanical result is a line ${CLAUDE_PLUGIN_ROOT}/checks/ds/constraints/ds-dq.py emitted, quoted as emitted.",
"Never report M1, UNI, DEN, DEL or R1 as PASS. The runner lists them as MODEL-EVALUATED; Jev scores M1, UNI, DEN and R1, and DEL is a judgement reported with the evidence read.",
"DQ4 and DQ6 are `N/A` from the runner on a present output (FAIL, like every check, when the output is absent), and that `N/A` is not a third kind of pass — the runner emits a line for them only because ENUM requires one, and an N/A never sets its non-zero exit. Jev's DQ4 and DQ6 verdicts score them against the input → transform → output count chain and the before/after shape. Never read their N/A as `the runner checked this`.",
"An artifact absent from the plan's ## Data Outputs table is one nothing will check and cannot be claimed as verified. Do not verify an output the table never declared.",
"The analysis is done by dispatched agents. Main chat writes no .py, .ipynb, .R, .sas, .sql or .qmd file, by any tool.",
"Standing DS doer authority — every implementation task follows the indexed constraints in the four aggregates under ${CLAUDE_PLUGIN_ROOT}/skills/ds/rules/: ds-common-constraints.md (C1-C6), ds-common-conventions.md (V1-V9 — assumption-over-evidence, deferred verification, statistical validity, P-hacking prevention, sample-selection documentation), ds-analysis-constraints.md (A1-A6 — robustness checks, standard-error specification, visualization integrity, table-figure pairing, chart typography, chart colour) and ds-engineering-constraints.md (E1-E7 — determinism and seeds, schema contracts, join audits with row counts and match rates, idempotency, loud error handling, native document input, network politeness), each indexing the self-contained files under ${CLAUDE_PLUGIN_ROOT}/rules/. The binding statements are the ds-rules digest. If it is not in your context, invoke `Skill(ds-rules)` before writing any code — the aggregates are index tables, not the statements. Open a rule's file under ${CLAUDE_PLUGIN_ROOT}/rules/ when the task turns on its detail. All four are named in every implementation task's refs, and refs are contractual reads, not a reading list: read all four in full before writing code. ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/ds-checks.md stays a separate load.",
"Rules: ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/ds-checks.md defines all thirteen checks; ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/etl-enforcement.md governs pipelines; ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/sql-patterns.md governs data pulls; ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/verification-patterns.md governs evidence; ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/competing-hypothesis.md governs debugging.",
].join("\n"),
implementerAgentType: "ds", // the doer's own prompt replaces Claude Code's software-engineering one, which frames a dataset as a codebase
verifierAgentType: "Explore",
}
implementerAgentType names ds because the default agent carries Claude Code's
software-engineering system prompt, and the deliverable here is a dataset, a table and a figure a
human reads — not a codebase. An agent is justified only by a custom prompt, hooks or preloaded
skills; ds earns it on the first alone — the constraint aggregates reach it as task refs, not as
a preloaded skill.
verifierAgentType pins Explore because it has no Edit and no Write: a judge that structurally
cannot modify the tree beats a prompt asking it not to. The lens buys the same property a different
way — ds-reviewer is read-only by tools allowlist — because Explore's prompt is predefined and no
preloaded skill or CLAUDE.md reaches it; the lens carries the four aggregates (the index) in its own refs and the rule bodies its checklist grades in the same refs. The lens returns schema-validated JSON through StructuredOutput, and the runner writes lens.json. The lens has no Bash, so it cannot re-run anything: a claim that needs a run is reported as unverified, never graded from the code.
Add ${CLAUDE_PLUGIN_ROOT}/skills/ds/references/competing-hypothesis.md to a task's refs when
that task is a diagnosis rather than a build. authorityExtra names it; refs is what makes an
implementer read it in full.
The runner's own contract suite, ${CLAUDE_PLUGIN_ROOT}/skills/ds/scripts/ds_dq_test.py, is the
ds-dq-tests leg of the one mechanicalChecks entry.
Phase 5 — HUMAN REVIEW
work's Phase 5 unchanged, over the plan's ## Review Surfaces. A clean technical verification is
evidence for that conversation, not human acceptance.
Red flags
| Situation | Wrong move | Right move |
|---|---|---|
| Sizing a source | pull the whole thing to see how big it is | the transfer is the failure you are meant to prevent — profile filtered counts and aggregate candidates read-only |
| Profiling a source | treat a head sample as the profile | nulls, grain failures and type drift live outside the head — profile shape, tail/coverage, keys and quality signals |
| Naming a produced artifact | name it in prose | a runner given prose cannot open it, key it, or window it — add its row to ## Data Outputs before approval |
| The DQ result | let the implementer report it | the doer cannot see the assumption it made in both places — the runner is a mechanicalCheck and the JS reads its exit code |
M1/UNI/DEN/DEL/R1 | report them as PASS | that presents a judgement as a computation — Jev's verdicts decide M1, UNI, DEN and R1; DEL is MODEL-EVALUATED with the evidence read |
DQ4/DQ6 reported N/A | read the N/A as the runner having checked them | an N/A on a present output is not a third kind of pass (an absent one is FAIL) — the runner computes neither and an N/A never sets its non-zero exit; Jev's DQ4 and DQ6 verdicts decide them |
| Per-document LLM coding or extraction | fan out agents to read each document | launch a batch-router-picked batch job on pre-cut excerpts (with a 1-request synchronous test first) |
| A judgement that depends on the constraint index | dispatch a built-in agent (Explore, Plan, general-purpose) | their prompts are predefined, no preloaded skill reaches them and they skip the CLAUDE.md hierarchy, so the constraints are graded from memory — dispatch a custom agent whose body you control, like ds-reviewer |
| Handing a doer the constraint aggregates | name the four paths in the task prompt's prose, or copy the aggregates into a skill | prose is discretionary and a copy is a second source of truth tests/constraints-no-duplication.test.ts fails on — put the four canonical paths in the task's refs, which work defines as reads the doer owes in full |
| Project state | write a SPEC.md, STATE.md or LEARNINGS.md | competing state makes progress ambiguous — the approved plan is the authority and work hashes it |
Something work does not obviously do | write a ds/workflow.js | ask which work parameter is missing — mechanicalChecks is what makes the DQ runner the gate |
elide-case — a court opinion cut into a student reading, on the work spine
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Typst rules in force here — read the one that governs what you are writing; they are independent:
!k=notes,prose; command -v typst-rules >/dev/null 2>&1 && exec typst-rules "$k"; echo "(typst corpus unavailable: NO Typst rule is listed here — install the typst plugin, or start a new session so its bin/ reaches PATH)"
The lifecycle is work. Read it and
follow it. This file supplies work's spine with the domain and nothing else — the three CLARIFY
questions, the plan grammar, the task rows, the two scored checks, the one mechanical entry point, the refs
and the authority text. It ships no workflow.js and no .js of any kind, and restates none of
work's mechanics.
The domain rules live in references/ and reach every dispatched agent through refs:
| ref | what it carries |
|---|---|
references/authenticity.md | the two Iron Laws — authentic reporter text, and compile-before-table |
references/retrieval.md | workflows:westlaw as the first route and the .westlaw.docx as the ONLY stored source, the corpus fallback (opinions vs RECAP, the 404, token gating, telling a clean source) and what that fallback's PDF cannot carry, and the two reasons to run on Gemini — a scanned source or a filter-tripping subject |
references/editing-marks.md | one elision mark per addendum, [ ] conventions, never altering subject/tense/sentence boundary, the editors'-note rule (a note is the EXCEPTION, for background), where the mechanical conventions live (the preamble, once), the layout-only widow rule, and the source-mapping marker // elide-source: that check.sh reads |
references/verification.md | what each gate leg decides, and how to read a miss rate |
The Iron Law of Compile-Before-Table. The summary table is written AFTER the compile, from the checker's computed output. Never before, never from an estimate.
Both are stated in full in references/authenticity.md, which is the refs entry every implementer
and the advisory fidelity reviewer must read. Refactoring this file never weakens them.
Phase 1 — CLARIFY: three questions, asked with AskUserQuestion, never self-answered
This interview is the point of the workflow. Ask exactly these three, with AskUserQuestion, and
record every answer verbatim in the plan:
- What is the doctrinal thread, in your own words — and which assigned reading does it cut against? One thread, named by the instructor, plus the case or casebook page it is in tension with. This is what decides every sentence that survives the cut.
- Is this case taught for its HOLDING or for its REASONING? Holding means the disposition and the rule statement survive and the analytical march may be compressed. Reasoning means the analytical steps survive and the disposition may be a sentence.
- What is the page target for each reading? One MIN-MAX per reading, offered with 2-6 as the default option — the skill's established range — since a reading whose length he does not care about takes the default. A lead case and a short blog post reasonably differ, which is why the plan carries one target per row. When the request or the schedule already gives a target, pre-fill the options from it rather than posing the question blind; nothing is inferred silently, and nothing is typed at a command line.
Do not ask anything else. Insertion point and length are not open-ended interview axes. The per-reading page target is question 3 above, set in the interview and enforced from the plan, because a number typed at the command line is a number an agent can invent. Insertion point and class number still come from the request, or off the schedule if the request omits them.
THE INTERVIEW, AND EXACTLY WHAT ENFORCES IT
Never self-answer either question, never state an assumption and continue, never infer the thread
from the Teacher's Manual or from the case itself. work's CLARIFY is conversational and a
dispatched agent cannot call AskUserQuestion — a run started from a farmed-out agent has nobody
to ask, and its only correct move is to stop and report that the interview is unanswered. Elevating
the run to a session that can reach the user is the fix; guessing is not.
A doctrinal target the model states to itself is a silent guess; nothing makes it ask. This workflow exists to remove that path.
The chain, with each link's real strength named. Three things carry the interview, and only one of them is software:
- The answers are recorded in the plan, under
## Doctrinal target, on the three labelled lines Phase 2 specifies. - The instructor reads and approves that plan before the run arms, and the spec hash binds what
he approved. This is the link that makes the answers his. It is a human gate, and it is the only
one there is: an instructor who approves a plan whose
## Doctrinal targetholds answers he never gave defeats the whole chain, and no software in this design closes that. ${CLAUDE_PLUGIN_ROOT}/skills/elide-case/scripts/check.sh --plan <path>refuses a plan whose section is missing or unfilled — absent section, orDoctrinal thread:/Cuts against:missing, empty or a placeholder, orTaught for:missing or saying neither holding nor reasoning. The leg fails closed: passing neither--plannor--no-planis itself a FAIL naming the missing flag, and--no-plan(for fixture and dev runs that check an addendum alone) prints a loud NOT CHECKED line.
What that leg proves, and what it does not. It proves the section exists and every required answer
is non-empty. It is a backstop against an orchestrator that skipped the question — nothing more.
The plan file is written by the orchestrator, so no grep of it can tell the user's answers from the
orchestrator's own, and work's mechanicalChecks run at gate time, after IMPLEMENT, so no
mechanical leg can refuse to proceed past CLARIFY. Do not close the gap with a second
mechanicalChecks entry — P10 refuses one, and the leg belongs inside the single entry point.
Phase 2 — PLAN
work's Phase 2. Domain requirements on the plan:
- A
## Readings In Scopesection, one row per reading:caption | source docs/*.westlaw.docx (a docs/*.txt from an earlier run, or from the fallback route, still resolves) | page target. This section is the source of truth for the page target, and the--planleg enforces it exactly as it enforces the three Doctrinal target lines: an absent section, a row with no page target, or a placeholder one (TBD,<…>,N/A) FAILS the leg naming the offending row. The target is per reading, from interview question 3;check.shmaps each.typcaption to its row and passes that row's own MIN-MAX tocheck-addendum.py. The caption is the identifier the.typand the mechanical entry point key on — the entry point derives its per-reading checks from the captions in the.typ. Each row also generates onecut-<slug>task, which is how a per-reading re-run reacheswork'sonlyTasks. - The first two CLARIFY answers, quoted (the third is the
page targetcolumn above), under## Doctrinal target, on three labelled lines the--planleg parses —Doctrinal thread:,Cuts against:,Taught for:(holding or reasoning). A placeholder (TBD,<…>,N/A) reads as absent and fails the leg. - A
## Non-court readingssection whenever a reading is not court text — one- <caption> — <why>bullet each, and omitted entirely when every reading is an opinion. Declaring a reading non-court text is the INSTRUCTOR's call, recorded in the plan, never a marker the implementer writes into the.typ: a self-exempting marker would let fabricated text switch the verbatim check off on itself, which is the first Iron Law defeated by the file it governs. The quotes leg reads this list from--planand reports those captionsDECLARED-UNCHECKED; a// elide-unchecked:marker in the.typis no longer honoured and its presence is itself a FAIL naming the reading. - Run sizing naming the one mechanical entry point, the two
scoredCheckskeys on theScored checks:line marked ADVISORY, the criteria/scope checklist on theReview lens:line, and the task rows. R readings give R + 2 tasks — oneretrieve, Rcut-<slug>, onecompile-table— and 2R scored items; with one lens and one mechanical check the floor is 2(R + 2) + 1 + 1 + 2R = 4R + 6 agents — 10 for a single reading, 22 for four readings. - The wave assignment, stated:
retrievein wave 1, everycut-<slug>in wave 2,compile-tablein wave 3.work-dispatch.shprints the wave graph it computes; if it does not match those three lines, thedependsOnedges are wrong, not the prose.
Writable paths for a generated run are addenda/, output/addenda/ and docs/ only. notes/,
slides/ and templates/ are excluded, and so is everything else in the course tree.
Phase 3 — GOAL
work's Phase 3, on the PASS phrasing: a run that writes can turn a FAIL into a PASS, and work's
outer fix loop is what does it.
Condition: work-result.sh returned 0 for the plan path, and the tuicr gate returned approved.
/goal work-result.sh has returned 0 for <plansDirectory>/<slug>.md at its current hash, and the
tuicr gate has returned approved, or stop after N turns
Name the plan by path, never by a pinned digest — the FAIL loop amends and re-hashes the plan,
so a named hash self-invalidates. Name work-result.sh's exit code, which is the run's verdict:
overallPass in result.json is not the verdict and must never be read directly (work/SKILL.md's
mechanicalChecks row). Naming the returned verdict is also what makes a gate FAIL reach a stopping
condition at all — including one carried by a lens finding, which otherwise reaches none.
The condition names no legibility or fidelity finding. A clause like "no critical|major finding
survives" would reinstate the non-terminating prose review this design exists to exclude: round
n's fix gives round n+1 new things to object to. Those two run as scoredChecks, which cannot
reach overallPass at all, so their output sits in result.json and is retrievable at Phase 5 by
a human who runs the command there. Nothing carries it to him. Both named verdicts get printed, so
both are judgeable from the transcript.
Phase 4 — the work call
The args go in the plan's <!-- work:dispatch --> block and the dispatch is work's own
work-dispatch.sh — never a hand-written farm.sh --workflow line, which drops the TIER 1
plan-lint gate and both probe gates:
bash ~/.claude/skills/workflows/skills/work/scripts/work-dispatch.sh \
--run-dir /home/eh/.local/state/work "$PLAN" # ABSOLUTE — never the course tree
Forward the provider. A provider named in this skill's $ARGUMENTS, however it is spelled,
becomes --provider codex on that line. Omitting it runs the user's codex request on claude. The
content-filter facts that make this matter are in references/retrieval.md, alongside the other
reason a run belongs on --provider gemini: a source whose text came from OCR.
--run-dir is not optional: work resolves .work/<run-id> against $PWD, which on an elide run is
the course directory — a tree this run writes only addenda/, output/addenda/ and docs/ in.
The args object, complete as it stands.
One task per reading, in three waves. The ## Readings In Scope table generates one cut-<slug>
row each — cut that reading and nothing else — above one shared retrieve row and below one
compile-table row. A second reading therefore adds exactly one task, and adds no lens and no
mechanical entry: the entry point derives its per-reading legs from the .typ itself. This
granularity is not cosmetic. It is what makes a per-reading re-run expressible in work's own
vocabulary — see The FAIL loop below.
The wave shape is forced by work's disjointness rule, not chosen for tidiness. Same-wave tasks
must have pairwise-disjoint writablePaths, checked prefix-aware and refused at arg-validation,
before any agent is dispatched. Every reading needs its source in docs/, so if each reading row
claimed docs they would all land in wave 1 claiming the same directory and a two-reading addendum
would be refused before dispatch. One row owns docs/ instead:
| wave | rows | writable paths | why disjoint |
|---|---|---|---|
| 1 | retrieve (one row, however many readings) | docs | alone in its wave |
| 2 | one cut-<slug> per reading, each dependsOn: ["retrieve"] | addenda/NN-addendum-<slug>.typ | one file each, slugs distinct |
| 3 | compile-table, dependsOn every cut-<slug> | output/addenda, addenda/NN-addendum.typ | alone in its wave |
retrieve writes docs/ and nothing else; the cut rows read it and never write it. A single-reading
run keeps all three rows — collapsing them re-introduces the overlap the moment a second reading is
added.
retrieve stays one serial row (P17 exempted): every Westlaw export shares the user's one signed-in browser, one delivery queue whose top item #0_qitem is clicked, and one Downloads directory, so concurrent retrievals can take each other's file.
const addendum = /* /home/eh/areas/secreg/addenda/NN-addendum.typ with the addendum's number */
const pdf = /* /home/eh/areas/secreg/output/addenda/NN-addendum.pdf, same number */
const planPath = /* the approved plan, /home/eh/areas/secreg/.claude/plans/<slug>.md */
const goal = /* one sentence */
const args = {
projectDir: "/home/eh/areas/secreg", // the COURSE directory — the tree being edited
goal,
// ── The deterministic floor: ONE entry point, whose exit code IS the mechanical verdict. ──
// Seven legs — the plan's interview answers (check-plan.py), typst compile, check-quotes.py per derived caption,
// check-addendum.py --target, the strays leg (canonical widows.py/orphans.py/runts.py
// plus check-stranded-headings.py), check-marks.py (one elision mark per addendum; every
// retained footnote renumbered 1..k and paired), and `bin/workflow-check elide
// --constraints-only`, which runs every constraint under checks/elide/constraints/ (where the six
// checkers live) over the addendum and the plan — none short-circuiting. A second entry here would spread the
// verdict over two commands and lose one silently; P10 refuses it.
// The plan leg FAILS CLOSED: pass `--plan <md>` on a real run or `--no-plan` on a fixture run,
// and passing neither is a FAIL naming the missing flag. `--plan` also carries the plan's
// `## Non-court readings` declarations. It checks that the answers are RECORDED and non-empty;
// that they are the instructor's is carried by his approval of the plan, not by this command.
// `--plan` ALSO supplies each reading's page target, from the plan's `## Readings In Scope`
// rows, so a real run passes NO `--target`. The plan wins if one is passed anyway — the flag is
// ignored with a loud line naming the plan's values. `--target MIN-MAX` / `--no-target` remain
// the fail-closed pair on `--no-plan` fixture and dev runs, where passing neither is a FAIL
// naming the missing flag and the leg must not report length.
// The WIDOWS leg is ON BY DEFAULT, which is its fail-closed form: an absent flag RUNS the
// check, so no invocation can quietly certify page breaks nobody looked at. `--no-widows`
// is the loud waiver for fixture and dev runs; passing it with `--widows` FAILS. Its fix
// vocabulary is LAYOUT-ONLY — spacing or pagination, never the court's words, and when the
// two conflict fidelity wins and the widow stands.
mechanicalChecks: [
{
name: "elide-check",
cmd: `bash ${CLAUDE_PLUGIN_ROOT}/skills/elide-case/scripts/check.sh --addendum ${addendum} --pdf ${pdf} --plan ${planPath}`,
},
],
// Jev scores the calibrated excerpt rules under checks/elide/rules/: an elision mark cutting
// inside a holding or rule sentence, a heading or ordinal sequence that skips with nothing
// marking it, editors' voice set as the court's, stress inside a quotation with no emphasis
// notation, and a reading's note that is only mechanical boilerplate. p >= 0.85 lands in
// rulesThatFailed and blocks; lower ones reach the lens ranked. It scores the assembled
// addendum by --files and names no --project-dir, so it scores that one file WHOLE: the run's
// baseline scoping (--since) is appended only to a cmd that names --project-dir. Earlier
// readings already in the same file are therefore graded too; earlier addenda are other files.
ruleChecks: { name: "jev-elide-rules", cmd: `bun ${CLAUDE_PLUGIN_ROOT}/skills/work/scripts/rule-check.ts --workflow elide --files ${addendum} --plan ${planPath} --project secreg` },
tasks: [
// ── WAVE 1: ONE row, whatever the reading count. It is the only row that writes `docs/`,
// which is what keeps wave 2 pairwise-disjoint. See the wave table above.
{
id: "retrieve",
name: "retrieve every reading's reporter text into docs/",
work: "For EVERY row of the plan's `## Readings In Scope` table, retrieve the opinion FIRST through the `workflows:westlaw` skill, which owns the export procedure and the .westlaw.docx naming convention and yields publisher-keyed text needing no OCR adjudication. THE DOCX IS THE ONLY ARTIFACT: it is the golden copy, its text projection is in memory, and you write no .txt beside it — a stored twin drops the docx's italic runs, so a case name the court italicised would compile in roman. Fall back to ${CLAUDE_PLUGIN_ROOT}/skills/elide-case/scripts/fetch-opinion.sh — saving both docs/<name>.pdf and docs/<name>.txt under the naming convention already in docs/ — only when Westlaw does not cover the case or no Westlaw credential is available; that route yields a court PDF with NO formatting layer, so its italics are unrecoverable and the gate says so on every reading cut from one. On that fallback, a citation 404 means 'not in the opinions corpus', not 'no such case' — take the RECAP docket-entry route. Retrieve and nothing else: write no addenda file, cut no text, and do not edit an excerpt. A reading whose source cannot be retrieved is reported as such — never substituted with a related opinion, a Teacher's Manual summary or a previous excerpt.",
writablePaths: ["docs"],
acceptance: "Every `## Readings In Scope` row has its retrieved source on disk under /home/eh/areas/secreg/docs/ — <name>.westlaw.docx ALONE on the Westlaw route, or <name>.pdf plus <name>.txt on the corpus fallback — and each is the reporter text of that row's caption — `check-source-identity.py` (a leg of check.sh's constraints leg) fails a `.docx` source that does not contain the caption's reporter citation; a Route 2 `.txt` lacking it is reported IDENTITY NOT CHECKED (exit 0), because a slip opinion or RECAP entry carries no reporter citation, so the Phase 5 reader confirms that source by eye. Writing a <name>.westlaw.txt for a source retrieved THIS RUN fails this row — the projection is in memory. A .westlaw.txt already in docs/ from an earlier run is left alone: an addendum whose plan or `// elide-source:` names one still resolves and still passes, and deleting it would break a gate that was green.",
redDisposition: "Retrieval writes no excerpt, so there is no behaviour a redCommand could hold red: the run's mechanical entry point cannot run at all until wave 2 has produced a .typ to check. The per-task Verify channel adjudicates the files this row is responsible for.",
refs: [
"${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/authenticity.md",
"${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/retrieval.md",
],
},
// ── WAVE 2: ONE ROW PER READING. Copy this row for each `## Readings In Scope` line,
// substituting the slug, caption, source basename and page target. Each row cuts its own
// reading, so `tasksThatFlagged` on a FAIL names exactly the readings to re-cut. Each writes
// ONE fragment and never `docs/` — that is what makes the wave disjoint.
{
id: "cut-<slug>",
name: "cut <caption>",
work: "ONE READING, CUT ONLY. Write the excerpt into addenda/NN-addendum-<slug>.typ, from the docs/<source> the `retrieve` task saved (the file the plan's source column names: the .westlaw.docx, or the .txt on the Route-2 fallback) and from nothing else; check-quotes.py projects a docx to text in memory, so read a docx the same way rather than writing one out. THE EXCERPT INHERITS THE COURT'S ITALICS FROM THE SOURCE and must neither invent nor drop them: a case name the projection carries as `_Weinberger_` is set as `_Weinberger_` in the .typ, and emphasis the source does not carry is not added. You do not write docs/; if the source is missing or unreadable, stop and report it rather than retrieving it yourself. Keep only the transaction structure, the analysis of the ONE thread the plan's Doctrinal target section quotes, and the reasoning that puts the case in tension with the assigned reading it names. Holding-taught means the disposition and rule statement survive; reasoning-taught means the analytical steps do. Caption block and judge line follow addenda/01-addendum.typ. EDITORS' NOTE — the default is NO note (instructor ruling). Write one only when this reading needs BACKGROUND the student cannot read the case without: prior history of the same litigation, an earlier related decision, or real-world context. The mechanical conventions — the elision mark, footnotes omitted except as indicated and renumbered, record citations dropped without notation — are stated ONCE in the addendum's preamble and cover every reading; do not repeat them per reading. Elision marks and the disclosure rule are in editing-marks.md — an elided passage may instead be replaced by `#emph[[Summary: ...]]` (<= 40 words, between sentences, no quotation, figure or holding), whose form is in editing-marks.md and which check-marks enforces; record citations, party-brief citations, string cites and internal cross-references may go silently; a dropped substantive footnote, qualifier, alternative holding or dissent may not, and a disclosure specific to THIS reading (e.g. it retains footnote 16 renumbered as 1) goes in this reading's own note. SOURCE MAPPING: check.sh derives each reading's source from its caption and resolves it to the .docx. When that derivation is wrong or ambiguous, declare the file with `// elide-source: <basename.westlaw.docx>`, whose syntax and placement are in editing-marks.md — never point it at a loosely related file to quiet a NOSOURCE, and note that it names a file that must exist, so it cannot exempt anything. You may NOT declare a reading non-court text: `// elide-unchecked:` is no longer honoured and writing one is itself a FAIL, because the agent writing the excerpt cannot be the one who switches the verbatim check off on it. A reading with no reporter source stops and reports to the instructor, who declares it in the plan's `## Non-court readings` section.",
writablePaths: ["addenda/NN-addendum-<slug>.typ"],
dependsOn: ["retrieve"],
// Task-specific: check-quotes.py for THIS caption alone. Not the run's entry point — a task
// graded by the shared command can be reported done because a DIFFERENT task's work made it
// pass, which leaves `work`'s per-task Verify channel nothing task-specific to judge.
acceptance: "python3 ${CLAUDE_PLUGIN_ROOT}/checks/elide/constraints/check-quotes.py /home/eh/areas/secreg/addenda/NN-addendum-<slug>.typ /home/eh/areas/secreg/docs/<source> --caption '<this reading's caption>' --skip-editorial exits 0, having checked a non-zero number of sentences for that caption. <source> is the file the plan's source column names for this reading: the .westlaw.docx, or the .txt on the Route-2 fallback.",
redDisposition: "The behaviour this task must satisfy is already executed by the run's one mechanical entry point, whose check-quotes.py leg is red before the excerpt exists and green after; a redCommand here would be that same command run a second time by a second agent.",
refs: [
"${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/authenticity.md",
"${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/editing-marks.md",
],
},
// ── WAVE 3: alone, so its two writable paths collide with nothing.
{
id: "compile-table",
name: "compile, then write the table from the computed rows",
work: "Assemble the per-reading fragments into addenda/NN-addendum.typ in the plan's Readings In Scope order, compile via teaching:generate-addendum, then run check-addendum.py --target and paste ITS computed rows into the summary table. THE PREAMBLE CARRIES THE MECHANICAL CONVENTIONS FOR THE WHOLE DOCUMENT — one sentence covering the elision mark, footnotes omitted except as indicated and renumbered, and record citations dropped without notation. This row owns that sentence, because it owns the preamble; a per-reading note repeating it is the boilerplate the no-default-note rule removes. Never hand-edit a range to make the checker pass — the checker is reporting the truth about the PDF. On a length or arity miss, fix the offending READING's fragment and recompile; do not rewrite a fragment this row did not break.",
writablePaths: ["output/addenda", "addenda/NN-addendum.typ"],
// Task-specific: check-addendum.py's own arity, table-truth and length legs against the
// compiled PDF. The run's entry point stays the WHOLE mechanical verdict without also being
// this task's only grader.
acceptance: "python3 ${CLAUDE_PLUGIN_ROOT}/checks/elide/constraints/check-addendum.py /home/eh/areas/secreg/addenda/NN-addendum.typ /home/eh/areas/secreg/output/addenda/NN-addendum.pdf --target <the page target the plan's `## Readings In Scope` rows carry; when the rows differ, run the run's entry point instead, which reads each row's own target> exits 0 — summary-table rows equal readings found (arity), every stated page range matches the compiled PDF (table truth), and each reading is inside its target (length).",
dependsOn: ["cut-<slug>" /* , one id per reading row; `retrieve` is reached transitively */],
redDisposition: "Work whose whole acceptance is the run's mechanical entry point; `work` re-runs that command in a shell to adjudicate it, so a redCommand would duplicate the gate rather than gate the task.",
refs: [
"${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/authenticity.md",
"${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/verification.md",
],
},
],
// ── legibility and fidelity are `scoredChecks`, and that is WHY they are advisory. ──
// The `lens` gates by construction: the work skill's workflow.js computes overallPass FROM the
// blocking lens findings that stand, so folding these two into the lens prompt and calling them
// advisory in prose would be a claim about the spine rather than a parameter to it. `scoredChecks` is the mechanism
// that carries the property: overallPass is computed without reading any scored value, there is no
// threshold and no blockBelow, it opens no selector channel, and a dead agent does not flip the
// verdict (work/references/scored-checks.md, S6/S7). That is ~/.claude/CLAUDE.md rule 9 satisfied
// structurally — an open-ended prose review cannot gate a loop, because round n's fix gives round
// n+1 new things to object to.
//
// ONE ITEM PER READING, so a score reads per reading exactly as a task does. Each agent returns
// RAW COUNTS and never sees the weights below, which is what stops a self-graded score.
scoredChecks: [
{
key: "legibility",
items: ["<caption>" /* , one per `## Readings In Scope` row */],
agentType: "Explore",
refs: ["${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/editing-marks.md"],
prompt: "Read this reading's excerpt in addenda/NN-addendum.typ as a student would, with no access to the full opinion. Count ONLY these five defect kinds, and report COUNTS plus the offending sentences verbatim: (1) unparsable — a sentence that does not parse as a sentence; (2) fusedElision — an elision that joins two fragments into something the court did not write; (3) danglingReferent — a pronoun or 'it'/'that' whose referent was cut; (4) unattributedQuote — a block quote whose speaker is not identifiable from the excerpt alone; (5) undefinedTermOfArt — a term of art used before it is introduced. Anything outside those five is not counted here. `itemsChecked` is the number of sentences you actually read; report 0 only if you read none, which marks the item unreliable rather than clean.",
schema: {
type: "object",
additionalProperties: false,
required: ["itemsChecked", "unparsable", "fusedElision", "danglingReferent", "unattributedQuote", "undefinedTermOfArt"],
properties: {
itemsChecked: {type: "integer"},
unparsable: {type: "integer"},
fusedElision: {type: "integer"},
danglingReferent: {type: "integer"},
unattributedQuote: {type: "integer"},
undefinedTermOfArt: {type: "integer"},
offendingSentences: {type: "array", items: {type: "string"}},
},
},
passthrough: ["offendingSentences"],
components: [
{name: "reads", weight: 1.0, base: 10, penalties: {unparsable: 2.0, fusedElision: 2.0, danglingReferent: 1.0, unattributedQuote: 1.0, undefinedTermOfArt: 0.5}},
],
},
{
key: "fidelity",
items: ["<caption>" /* , one per `## Readings In Scope` row */],
agentType: "Explore",
refs: ["${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/authenticity.md"],
prompt: "Verbatim matching cannot see misrepresentation. Judge whether the CUT misrepresents the opinion, reading this reading's excerpt against its full source docs/<source>, the file the plan's source column names. A .westlaw.docx cannot be opened by Read: project it to text with `python3 ${CLAUDE_PLUGIN_ROOT}/skills/westlaw/scripts/westlaw-docx-text.py docs/<source>` (Bash) and read that output; never write it to disk. A .txt (Route-2 fallback) is read directly. Count ONLY these five, and report COUNTS plus the excerpt passage and the source line for each: (1) quotedAsOwnReasoning — a passage the court QUOTES FROM ANOTHER CASE presented as the court's own reasoning (e.g. a block quote the court cites only to reject, rendered as running text); (2) unqualifiedHolding — a holding stated without the qualifier that limits it; (3) omittedAlternativeHolding — an omitted alternative holding that changes what the case stands for; (4) dissentAsMajority — a dissent's language read as the majority's; (5) misstatedSummary — an editorial `#emph[[Summary: ...]]` that misstates, slants or adds to the passage it stands in for (trace it to the source text cut at that point). `itemsChecked` is the number of excerpt passages you traced back to the source; 0 marks the item unreliable rather than clean.",
schema: {
type: "object",
additionalProperties: false,
required: ["itemsChecked", "quotedAsOwnReasoning", "unqualifiedHolding", "omittedAlternativeHolding", "dissentAsMajority", "misstatedSummary"],
properties: {
itemsChecked: {type: "integer"},
quotedAsOwnReasoning: {type: "integer"},
unqualifiedHolding: {type: "integer"},
omittedAlternativeHolding: {type: "integer"},
dissentAsMajority: {type: "integer"},
misstatedSummary: {type: "integer"},
offendingPassages: {type: "array", items: {type: "string"}},
},
},
passthrough: ["offendingPassages"],
components: [
{name: "faithful", weight: 1.0, base: 10, penalties: {quotedAsOwnReasoning: 3.0, unqualifiedHolding: 2.0, omittedAlternativeHolding: 2.0, dissentAsMajority: 3.0, misstatedSummary: 2.0}},
],
},
],
// One lens after verification and mechanical checks; legibility and fidelity remain advisory above.
lens: {
agentType: "Explore",
refs: ["${CLAUDE_PLUGIN_ROOT}/skills/elide-case/references/editing-marks.md"],
prompt: "Judge the delivered artifacts against the approved plan and the goal. CHECKLIST: (1) CRITERIA vs ARTIFACTS — for each success criterion in the plan and goal, is there an artifact that satisfies it? Missing or partial satisfaction is a finding, CRITICAL where the deliverable cannot stand without it. (2) SCOPE FIDELITY — did the changes stay inside the plan's task table and writable paths, limited to addenda/, output/addenda/ and docs/? Out-of-scope edits, unrequested features and silently skipped readings are findings, CRITICAL where an edit landed outside every declared writable path. Severity is MAJOR at minimum. Legibility and fidelity are advisory scoredChecks, not additional checklist items here. Holding-cutting elisions, unmarked heading or ordinal skips, editors' voice in the court's text, un-noted emphasis in quotations and boilerplate notes are the jev-elide-rules verdicts in the digest: do not re-grade them, except to rule on a verdict the digest ranks below the block line. MODE — RED (a task was flagged or a mechanical check failed): diagnose EVERY failure in the digest and route it with cause and fix to its owner task id, or to 'plan' when no task can own the fix. MODE — GREEN (everything passed): make one open-ended pass over the checklist and report each finding with an ownerTask. In both modes, rule every carried finding open or closed against evidence you actually read; silence leaves it open.",
},
authorityExtra: "DOMAIN RULE — every word of the excerpt is authentic reporter text, cut from the .westlaw.docx retrieved THIS RUN into docs/ and projected to text in memory — never from a .txt written beside it, which is not produced. Never from memory, a Teacher's Manual summary, a casebook note, or a previous excerpt. DOMAIN RULE — the summary table is written after the compile, from the checker's computed rows; never hand-edit a range to make the checker pass. WRITE SURFACE — addenda/, output/addenda/ and docs/ only. notes/, slides/ and templates/ are excluded, and notes/07-security.typ may be open in the user's editor. THE TYPST CORPUS IS ONE COMMAND AWAY, AND IS NOT IN YOUR REFS. Run `typst-rules notes,prose` for the index, then read the ones your edit touches; it is on PATH and needs no path or plugin variable. Do not work from a paraphrase of these rules.",
maxAgents: 50,
}
Phase 5 — HUMAN REVIEW
work's Phase 5, on the compiled PDF: the reviewer reads the reading, not the diff. Open the PDF
beside human-review-gate.sh -w.
Where the legibility and fidelity scores actually are
They are in result.json under scores[], and nothing delivers them. work returns each item's
computed score, its raw counts, and — because passthrough declared them — the verbatim offending
text under evidence. Phase 5 is a human step, so no exit code can compel anyone to look. Print them
with one command:
jq -r '.scores[] | "\(.key) \(.item) composite=\(.composite // "null") itemsChecked=\(.itemsChecked // "n/a") \(.reason // "")",
" components: \(.components // {} | tojson)",
" evidence: \(.evidence // {} | tojson)"' \
/home/eh/.local/state/work/<run-id>/result.json
The field is composite, not score — work emits one scores[] entry per (key, item) pair as
{key, item, components, composite, itemsChecked, evidence, reason?}, and there is no cross-item
mean or total to read.
This is an INSTRUCTION, not a gate, and the distinction is the whole design. The scores cannot
fail the run by construction — that is exactly why they are scoredChecks and not the lens —
and therefore a reviewer who does not run that command does not see them. That is the accepted cost
of their not gating, and it is stated here rather than papered over with a sentence claiming they are
"reported".
If that cost is unacceptable on a given run, the remedy is NOT to fold them into the lens prompt
— that reinstates the non-terminating prose gate rule 9 forbids — it is for the instructor to read
the excerpt himself, which is what Phase 5 is for.
When the reviewer does want them in the findings file, write result.json's scores[] there before
opening the PDF, one section per reading, naming the fields:
## Advisory scores — legibility and fidelity (these gate nothing)
### <caption>
legibility composite <n>/10 itemsChecked <n>
unparsable <n> · fusedElision <n> · danglingReferent <n> · unattributedQuote <n> · undefinedTermOfArt <n>
evidence.offendingSentences:
- "<verbatim sentence>"
fidelity composite <n>/10 itemsChecked <n>
quotedAsOwnReasoning <n> · unqualifiedHolding <n> · omittedAlternativeHolding <n> · dissentAsMajority <n> · misstatedSummary <n>
evidence.offendingPassages:
- "<verbatim excerpt passage, and the source line it was traced to>"
Three rules on the rendering, because each is a way the output silently stops meaning anything:
evidence.offendingSentencesandevidence.offendingPassagesare the whole point of the passthrough declarations — the counts say how many, the evidence says which. Render the verbatim text. A section showing counts with no evidence gives the reviewer nothing to act on.- A
nullcompositeis rendered asnullWITH its statedreason— never as0, never as a blank, never omitted.nullmeans the item was unreliable or its agent died;0means it was measured and scored zero. Collapsing them turns "nothing checked this reading" into "this reading is bad", and the reviewer cannot tell. scores: []withscoresRun/scoresReportedasnullis n/a, not clean. Say the phase did not run rather than printing an empty section that reads as nothing found.
The reviewer decides what to do with all of it. A low score re-runs nothing by itself — see The FAIL loop.
The FAIL loop — work's selectors, and where the per-reading granularity comes from
The selector is work's own, all four channels, consumed together: tasksThatFlagged,
mechanicalThatFailed, lensesThatFlagged and planFindings. A scoped re-run is onlyTasks +
priorResults, derived by work-redispatch.sh from the result. There is no caption channel — work scopes by
task id and nothing else, so a re-run instruction phrased as "re-cut the failing caption" names a
dimension work does not have and cannot be executed.
Per-reading re-cutting falls out of one task per reading, not out of any caption machinery.
Because each reading is its own task row, a reading that fails puts its own id in tasksThatFlagged,
and onlyTasks re-runs exactly that reading. A four-reading addendum keeps the three readings the
gate already adjudicated — re-cutting them manufactures new surface for the next round.
mechanicalThatFailed names elide-check, whose leg lines name the failing reading. The lens routes
each failure to its owner task or to plan; task routes narrow the re-run, while planFindings
require amending and re-hashing the plan. Two domain rules on the fix, both in references/verification.md:
- A miss rate above roughly a quarter is the wrong source, not bad sentences. Re-cut the reading
from
docs/<name>.westlaw.docx; never patch sentence by sentence. - Never trust a corruption tell you have not grepped for in both files.
lensesThatFlagged is ["lens"] whenever a blocking finding of the criteria/scope lens still stands,
and [] otherwise. The legibility and fidelity
scores appear in no selector at all: scoredChecks opens no channel, which is the same fact as
their not gating. A low score is
read by the human at Phase 5 and acted on there or not at all; it never re-runs anything by itself.
A null score with a stated reason means the item was unreliable or its agent died — not clean.
The rule for whoever edits this next
Any property this file asserts about work must be traceable to a parameter work reads or to a
command's exit code. Otherwise it is decoration. Three times in this skill's construction a property
was written as a sentence and bound nothing, and a fourth is recorded below: a re-run selector declared to be a "caption", a dimension
work has no channel for; two review dimensions declared ADVISORY while sitting in the gating
review lens, which work computes overallPass from; and an interview refusal declared absolute when no mechanism could carry it — the plan is
written by the orchestrator, so a grep of it cannot identify the answerer, and mechanicalChecks run
after IMPLEMENT, so none of them can stop a run at CLARIFY. The first two now rest on a parameter —
the task-id selector and scoredChecks. The third was scoped down instead: check.sh --plan
proves only that the answers are recorded and non-empty, and the instructor's approval of the plan is
named as the step that carries the rest.
The fourth was the scored checks being "reported to the human". work's Phase 5 is a human step, so no
exit code can compel a human to read a score, and three review rounds said so before the claim came
out. It too was scoped down rather than re-promised: the scores are retrievable at a named jq
command over result.json, a reviewer who does not run it does not see them, and that is the accepted
cost of their not gating.
So: when no parameter and no exit code can carry a property, SCOPE THE CLAIM DOWN to what the mechanism does prove, and name the human step that carries the remainder. An honest smaller claim beats a bigger one nothing enforces. Before writing "this workflow does X", name the parameter or the command that makes X true — and if there is none, say so plainly where the claim would have gone.
Red flags — STOP if you catch yourself
| About to | Do instead |
|---|---|
| Answer either CLARIFY question yourself | Stop and ask, or report that the interview is unanswered. Nothing mechanical can catch a self-answer — the plan is written by the orchestrator, so only the instructor's approval distinguishes his answers from yours |
| Ask about a passage to keep, or about the insertion point | Both were declined; insertion point and class number come from the request or the schedule. The page target IS asked — question 3 |
Type a --target on a real run, or invent a range to fit the excerpt | The target lives in the plan, per reading. With --plan the flag is ignored and said to be ignored; --target is the --no-plan fixture override |
Leave a ## Readings In Scope page target blank or TBD | The --plan leg FAILS naming the row. Ask question 3, or take the 2-6 default |
| Let a legibility or fidelity finding block the run | They are scoredChecks, which work's overallPass never reads — the deterministic gate is the entry point's exit code |
Add legibility or fidelity to the lens prompt because it "reads like a lens" | The lens gates by construction; the advisory property has to be a parameter, not a sentence. Keep them in scoredChecks |
Omit lens or pass {} to turn review off | both select work's generic default. Keep the explicit criteria/scope checklist, with legibility and fidelity only in scoredChecks |
Add a threshold or blockBelow to a scored check | No such knob exists, and adding one chases minors. Read the score; gate on the entry point |
Add a second mechanicalChecks entry | The verdict is one command; P10 refuses a list, and a list drops a check without reporting it |
Write a workflow.js for this skill | work's spine takes a parameter for it. Name the missing parameter and generalize work |
Give every reading row docs in its writable paths | work refuses a wave with overlapping writable paths before dispatch. retrieve owns docs/ in wave 1; the cut rows write one fragment each |
| Write "this workflow does X" with nothing reading X | Name the parameter or the exit code that makes it true, or record that none does — see The rule for whoever edits this next |
| Write that the scores "are reported to the human" | Nothing reports them. They sit in result.json under scores[]; Phase 5 names the jq command that prints them, and a reviewer who does not run it does not see them |
| Re-cut every reading after one FAIL | Re-run onlyTasks on the flagged reading's task id — one task per reading is what makes that possible |
| Scope a re-run "by caption" | work has no caption channel; consume tasksThatFlagged, mechanicalThatFailed, lensesThatFlagged and planFindings |
| Silently skip a reading that has no reporter source | Stop and report it. The instructor declares it in the plan's ## Non-court readings section; an undeclared one FAILS, which is the point |
Write // elide-unchecked: into the .typ to quiet a reading | That marker is self-exempting and is no longer honoured — its presence FAILS. Non-court text is declared in the plan, by the instructor |
Omit both --plan and --no-plan from the entry point's cmd | That is a FAIL naming the missing flag — the leg fails closed. Pass --plan <md> on a real run; --no-plan is the fixture escape and prints a loud NOT CHECKED line |
| Say the workflow "cannot arm" or "does not proceed" without the interview | No mechanism carries that. Say what does: the answers are recorded in the plan, the instructor approves the plan before the run arms, and check.sh --plan refuses a plan whose section is missing or unfilled |
| Write an editors' note on every reading | The default is NO note (instructor ruling). One exists only for BACKGROUND — prior history, an earlier related decision, real-world context — or a disclosure specific to that one reading |
| Repeat the elision-mark and footnote conventions in each reading's note | They are stated ONCE in the preamble and cover the whole document. That repetition is the boilerplate the reversal removed |
| Reword the court's text to close a widow | The fix vocabulary is LAYOUT-ONLY — spacing, or where the page breaks. Rewording defeats the verbatim gate; if the two conflict, fidelity wins and the widow stands |
Pass --no-widows on a real run | It waives the page-break leg and prints a loud NOT CHECKED line. The leg is on by default; leave it on |
Write a .westlaw.txt beside the docx, or cut from one | The docx is the only stored source and its projection is in memory. A stored projection drops the docx's italic runs, so every case name reads in roman |
| Add emphasis the source does not carry, or set a source italic in roman | The excerpt inherits the court's typography. Invented emphasis and dropped emphasis are both misquotation, and neither is caught by matching words alone |
Re-create a .westlaw.txt for a case whose docx is in docs/ | Both gates run green against the docx, so nothing needs one. A .txt naming survives only where the fallback route produced it, and // elide-source: X.westlaw.txt resolves to the .docx rather than failing |
Read a CourtListener/RECAP .txt and assume its typography | That route is a court PDF with no formatting layer — italics are unrecoverable there, not merely unchecked. The gate prints ITALICS NOT CHECKED for it; the route stays, because Westlaw does not carry every case |
| Hand-edit a table range so the checker passes | Fix the source and recompile |
farm-out
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Delegation runs in a separate process on a CLIProxyAPI wrapper, not in this session. This session keeps its own auth, Remote Control, and connectors; the work runs on proxy models.
There is no default runner: each --tasks row is routed by its kind (see
Provider routing below). Name a provider only when the task calls for a specific
model family, or when cross-checking one family against another.
Supersedes the built-in tools
| Instead of | Use |
|---|---|
Agent tool / subagents | scripts/farm.sh (single or fan-out) |
Workflow tool | scripts/farm.sh --workflow <abs script> --args <file> --out <file> |
| in-session agent team | scripts/farm-team.sh |
| persistent / remote agents | the agent-spawn skill (it reaches other machines; this skill does not) |
A description cannot carry this on its own — three rewrites all plateaued near
40% recall, so the routing is enforced by ~/.claude/hooks/main-thread-guard.sh
(PreToolUse on Agent|Task|Workflow|Edit|Write|NotebookEdit). It denies with a
reason that names the runner, which a bare permissions.deny rule cannot do. The
hook exempts FARM_OUT_CHILD=1 (our own runners' children, else farm-team.sh
blocks itself) and named agent types like Explore/Plan/librarian. Add to its
case to exempt another one.
Delegating is a choice the hook cannot make for you — the Agent|Workflow
branch only redirects delegation you already chose. To make a project refuse
main-thread implementation outright, set "farmOutOnly": true in its committed
.claude-workflows.json; every Edit/Write there is then denied unless it
targets that file, .claude/plans/, or .work/. Without it the Edit branch
allows unconditionally whenever no work dispatch is owed.
Two rules that are not optional
1. A returned result is not evidence. A delegated run reports
is_error: false and a confident summary whether or not it did the work.
Observed twice: two runs quoted invented teammate output, with zero tool calls
and zero filesystem trace. Always give the task a checkable artifact and pass
--expect <path>; the runners exit non-zero when an expected artifact is
missing, empty or older than the row (not written this run: <path>), or when the child ended with a background job still running (the row's
failure names it: ended with background job running: <cmd>; a Stop hook in
the child refuses that turn end first). Never relay a delegated summary you did not verify.
2. Every delegated prompt carries the anti-simulation clause. The runners append it automatically. Do not hand-roll a delegation that skips it.
One watcher per SESSION, not one per dispatch
Do NOT arm a Monitor or an until test -s loop to be woken. The watcher mod
(hooks/watch/watcher.ts) reads every farm run, work round and grind loop this session launches from
$TMPDIR/farm-events/<session>/ and wakes YOU — not the user — ONCE per run on DONE and on a run
whose process is gone with no verdict (GONE), naming its report path. farm-status, run as a
statusLine command, prints the running line from the same files (farm: 2 running (ds-rules 43m, …) · work <run> round 2/3 2/5 checks); set it with
{"statusLine":{"type":"command","command":"farm-status","refreshInterval":5}}, or a composing
script that calls it. /farm prints the table: label, state, elapsed, artifact, report. Its timer
restarts with every session start and reload, which the retired farm-runs monitor never did — a
finished run once woke nobody for hours.
Arm a task-specific wait ONLY when a run's finish must trigger a follow-up COMMAND — a wake is
not your next step. Background that step: until test -s <expect>; do sleep 20; done; <next step>.
Never with an unbracketed pgrep -f; the v6.26.3 guard denies a pattern matching its own checker.
A --workflow run prints an hourly CronCreate backstop — it survives --resume/--continue
and a session that was not running (--no-cron opts out; --tasks prints none). Arm it the turn you see it. When
you launch that run DETACHED (setsid nohup … > log) the printout lands in the log, not in front of
you, so create the cron yourself at launch: 7 * * * *, recurring, non-durable, prompt
and? (farm <run-dir name>).
Pick the shape first: sealed worker, or orchestrator
--agent <name> runs the delegation AS one of your agents — its real system prompt,
its preloaded skills, its declared toolset. That last part is the whole decision,
because every persona agent is sealed against DELEGATION: ds is
Read/Grep/Glob/Edit/Write/Bash/Skill with no Agent and no Workflow, so it can
do the work and load a domain skill, but it cannot fan out or dispatch a run.
Skill is deliberately NOT withheld. Withholding Agent/Workflow prevents recursive
fan-out; withholding Skill only blinded the persona to its own domain library — ds
gets the constraint aggregates it is graded against as task refs, while wrds,
crsp-v2, dewey, bmll, marimo and ten more sat unreachable. teaching always
carried Skill; the others not carrying it was drift, not policy.
| Job | Shape |
|---|---|
| one well-scoped piece of work | a one-row --tasks file with "agent": "ds" — the persona's prompt and its narrow toolset are the point |
needs to plan, fan out, dispatch subagents, or run a work workflow end-to-end | same, but omit "agent" — a persona has Skill but no Agent/Workflow, so it cannot fan out or dispatch |
| several independent jobs at once | more rows; each carries its own "agent" |
Row count is orthogonal to the persona question: a fan-out of five can be five ds
workers. "More than one job" does not mean "generic".
An orchestrating child is a full Claude Code session, so it has Agent and Workflow
and can dispatch the persona itself — Agent(subagent_type: "ds") gives the subagent
the same real persona plus the same deliberate restriction. That layering is the design
the work skills already use (skills/ds/SKILL.md sets implementerAgentType: "ds"),
not a workaround.
Agents live in ~/.claude/agents/: ds, writing, writing-econ, writing-legal,
workshop, teaching (workers); ds-reviewer, workshop-reviewer, writing-reviewer
(read-only).
A --workflow run takes no --agent — it picks agents PER LEG, which is the point:
agent(prompt, {agentType: "ds"}) inside the script, or implementerAgentType /
verifierAgentType / lens.agentType in a work args file. One top-level
persona could only apply to every leg, when what you want is ds implementing and
ds-reviewer or Explore judging. Note workflow.js strips the Agent tool from every
leg regardless of agentType, so legs cannot nest further delegation; fan-out is the
workflow's own parallel() / pipeline().
Use
There is one task mode, --tasks, and it always takes a JSON array — one row or
fifty. There is no inline --task: the only caller is a model reading this file, so
an inline prompt saved nobody anything, and a machine-written prompt passed as a shell
argument has to survive quoting (backticks, nested quotes, $) that a JSON file
sidesteps.
S=~/.claude/skills/workflows/skills/farm-out/scripts
# Build the task file with jq, never by hand-quoting a heredoc.
jq -n '[{prompt:"…", expect:"/repo/out.md", label:"count", kind:"script", agent:"ds"}]' > /tmp/t.json
bash $S/farm.sh --tasks /tmp/t.json --cwd /repo --detach
# Rows run in PARALLEL. Per row: prompt (required), workflow (required unless the persona
# implies it), kind (or provider/model), expect (string or array), label, agent.
# Omit "agent" when the row must orchestrate.
# schema (JSON Schema object, inline JSON or a path) -> --json-schema; the CLI-validated value is
# written to the row's ONE expect unless the child wrote it. A persona needs StructuredOutput in tools.
# sandbox: true -> the row's Bash runs in a throwaway bwrap overlay of the tree, its data/ targets and
# $HOME, no network (scripts/sandbox.sh); Edit/Write denied; the agent runs as <agent>-sandboxed.
# workflow script — --out is REQUIRED (the structured return is the result, not the
# summary), and paths resolve against OUR cwd, not --cwd, so pass them absolute.
bash $S/farm.sh --workflow /abs/wf.js --args /abs/args.json --out /abs/result.json --cwd /repo --provider claude
# Long runs: never foreground (a Bash-tool call caps out and kills the run mid-flight).
# Detach, then wait on the artifact:
setsid nohup bash $S/farm.sh --workflow /abs/wf.js --out /abs/result.json --cwd /repo --provider claude \
> /abs/run.log 2>&1 < /dev/null &
# team: named teammates that message each other, one result back
$S/farm-team.sh --prompt-file t.txt --cwd /repo --expect /repo/a.txt --expect /repo/b.txt
In the main thread, run --tasks with --detach and never with run_in_background: it returns at
once with the log path and the watcher's wake on the run's DONE is its one wake; with no watcher
ticking it prints farm: --detach ignored and runs in the foreground.
--provider claude|codex|gemini is optional on --tasks (omit it and every row is
routed), required on --workflow, and defaults to claude on farm-team.sh.
Budget defaults: FARM_TASK_BUDGET=4000000, FARM_RUN_BUDGET=20000000 (FARM_SESSION_BUDGET a deprecated alias),
FARM_TASK_ESTIMATE=750000 per task (capped at its budget). FARM_MAX_TURNS=250
on farm and grind. The pre-launch check refuses a run whose own estimate reaches the run cap; the printed session spend is informational and never enforced.
Read reference.md before changing a runner, debugging a 429, or hand-writing
a proxy call — it holds the verified model-routing and failure-mode details.
Provider routing
Every --tasks row carries a kind — script, judgement, review, bulk or vision. farm.sh hands
each row to ${CLAUDE_PLUGIN_ROOT}/scripts/lib/route.ts, which picks the provider and a pinned model from the committed
${CLAUDE_PLUGIN_ROOT}/scripts/lib/routing.json: the kind's pick, else its first available fallback. A row may name
provider/model itself instead. A row with neither a kind nor a provider/model is
refused — exit 2, no row runs, and it never falls through to a default.
--provider is the legacy whole-run override: that wrapper for every row, route.ts never consulted.
Each row prints farm: ROW <label> <rowId> on stderr and appends one line to
~/.local/state/workflows/farm-outcomes.jsonl (FARM_OUTCOMES overrides). A row whose expect
artifact is missing, whose child ended with no result (GONE), or that left a background job running
(background-orphaned) is labelled wrong automatically.
Once you have verified any other result, label it: farm.sh --verdict <rowId> correct|wrong "<why>".
route.ts --outcomes prints rows, wrong rate and top failing checks per kind x model.
bun route.ts --refresh, run in scripts/lib/ of a checkout of this repo and never the plugin cache,
updates availability (proxy catalog) and prices (OpenRouter) and never touches kinds.
Review its routing.json diff before committing. route.ts --propose reads the refreshed signals
and prints suggested kinds changes with reasons, plus newer same-family models the proxy serves
that beat a candidate at <= its price (candidate C: model <old> -> <new>), writing nothing: you
approve by editing kinds, or that candidate's model and openrouter then --refresh.
Design: docs/DESIGN-routing.md.
Red flags
| Situation | Wrong move | Right move |
|---|---|---|
| Per-document LLM coding/extraction over many files | Use --tasks fan-out to have agents read each document | Use batch-router, which picks the Claude, OpenAI or Gemini batch job |
| Writing style (fonts, hex colours, "X-style look") into a chart brief | The child follows the brief over its rules; farm.sh refuses the row | The subagent's rules own style; the brief says what to show. Only the user's verbatim style words go in a row's userStyle |
A row with no "workflow" | farm.sh refuses the run at launch (exit 2) and nothing spends: a row's output is gated by its workflow's checks, so it must say which | Set "workflow": "<wf>" (any workflow bin/workflow-check <wf> --list resolves: ds, dev, writing, writing/legal, …) or "workflow": "none" out loud. A persona implies one: its name with / for - (writing-legal is writing/legal); a read-only persona implies none |
| Treating a farmed row as done on its own report | The child can ship a join with no audit, a figure in the wrong face, a test that asserts only on a mock or a meta-commentary topic sentence | farm.sh runs bin/workflow-check <wf> after the row on the files it touched: every constraint and Jev rule under checks/<wf>/ and its parents'; findings or an unrunnable check mark the row wrong |
Contents
Fuzzy Name Matching
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Fast many-to-many fuzzy entity matching: char n-gram TF-IDF + sparse top-k cosine similarity (the ING banks recipe). Scales to ~10⁵ × 10⁵ on a laptop in seconds, ~10⁶ × 10⁶ with chunking.
The full recipe — code, threshold guide, gotchas, alternatives considered —
is in references/fuzzy-name-matching.md. Read it before writing match
code; a runnable template is in examples/fuzzy_name_match_sample.py.
Match Enforcement
IRON LAW: NO NAME MATCHING WHILE A SHARED IDENTIFIER EXISTS
Before normalizing anything, ask what identifier both sides already carry. Filings from one regulatory regime usually share a filer key, and it is often sitting in a field nobody read — a path, an accession, an exhibit header.
Example (npx-reconcile): a full session built a name-based
ISS→13F crosswalk reaching 59.45% of the target universe, while the
registrant CIK already parsed from the filing path reached 99.11% with
no name matching at all. Both filings were '40 Act filings from the same
registrant. The linker was not badly built; it was the wrong instrument.
Ask first: are these the same filer under one regime? If yes, find the key. Fuzzy matching is for identifiers that genuinely do not cross.
IRON LAW: NO FUZZY MATCH WITHOUT NORMALIZATION FIRST
Fuzzy matching is the last step of a linkage, never the first:
- NORMALIZE both sides (uppercase, punctuation → space, strip entity
suffixes/titles) — see the
normalize()function in the reference - JOIN exactly on the normalized name, scoped by a secondary key
- MEASURE the exact-join hit rate
- FUZZY-MATCH only the residual rows
- INSPECT a sample of accepted pairs at the chosen threshold
- CLAIM a hit rate only after inspecting matched pairs
Skipping straight to TF-IDF is NOT HELPFUL — every deterministic rule you skip upfront comes back as false positives the user has to find later, in a join they now believe is clean.
Fuzzy Matching Facts
- If the normalized-name exact join yields <70% on ground truth, normalization is leaving money on the table — fix that before reaching for TF-IDF. Fuzzy matching a badly normalized field buys false positives, not coverage.
- IDF weights are corpus-dependent: fitting a fresh vectorizer per scoped subset makes scores incomparable across subsets.
"GABELLI MARIO JOSEPH"↔"GABELLI MARIO J"scores 0.69 in a 2-row toy fit and 0.84 in a 600K-corpus fit — same pair, different context. Fit once onpd.concat([left_all, right_all]), reuse.transform()everywhere. - Scoping is where the signal lives. On the 13,663-row blockholder bridge, the scoped fuzzy pass (≥0.80, per issuer_cik) converted 70% of unmatched rows; the global pass afterward (≥0.90) added 11. Scope by issuer/year/geo and the smaller candidate pool lets you lower the threshold safely.
- TF-IDF on characters is not semantic.
"IBM"↔"International Business Machines"will never match at any threshold — that needs a name dictionary or an embedding model. sp_matmul_topn(A, B, ...)wantsBalready transposed (R.T).sortonly orders the hits within each row — top-k selection always keeps the largest values, sotop_n=1returns the argmax withsort=Falsetoo (verified on 1.2.0: 60/60 rows matched a dense argmax either way). Passsort=Truewhentop_n>1and you care about the order.- Duplicate normalized names on the right side with different ids make
top_n=1pick one of them with no defined rule —sortdoes not disambiguate an exact tie (tested on 1.2.0: it kept the last duplicate in every layout, under bothsortsettings, but nothing documents that). Whichever it is, the id you get is an accident of row order. Deduplicate the right side first.
Red Flags — STOP If About To:
- Run
sp_matmul_topnbefore an exact normalized join has been tried and measured → STOP. You cannot tell false positives from real coverage without the exact-join baseline. - Fit a
TfidfVectorizerinside a per-key loop → STOP. Scores become incomparable across keys; fit once on the full corpus outside the loop. - Accept matches below 0.75, or below 0.90 on an unscoped global pass → STOP. The threshold guide auto-accepts 0.85–0.95 only when scoped; the global pass floor is ≥0.90.
- Report a hit rate without eyeballing matched pairs near the threshold → STOP. Reporting an unverified hit rate is presenting an unverified claim as fact.
- Reach for TF-IDF on <5K × 5K, or on strings under ~3 characters → STOP. RapidFuzz edit distance is better there; n-grams collapse on short strings.
When to Use
Reach for this when linking two datasets whose only shared field is an entity name — bridging identifiers (CIK ↔ permno ↔ gvkey ↔ wficn ↔ EIN ↔ TR personid), deduping filer/fund/insider names, or resolving vendor records that share no key at all.
Don't use it for exact joins (pandas merge), for semantic equivalence, or
below ~5K × 5K rows where RapidFuzz is simpler. The reference's
"Alternatives considered" table covers rapidfuzz, recordlinkage,
dedupe, and edit-distance metrics and when each wins.
The Pipeline
normalize both sides
↓
exact scoped join (issuer/year/geo + norm_name) ← most of your hits
↓
exact global join (unambiguous norm_names only)
↓
fuzzy SCOPED pass — threshold ≈0.80, per-key candidate pools
↓
fuzzy GLOBAL pass — threshold ≥0.90, residuals only, dedup right side first
↓
residual: synthetic ids / leave unmatched — never force a match
Code for each stage, and the threshold table that governs what to accept at
each one, is in references/fuzzy-name-matching.md.
Packages
# pixi.toml
[dependencies]
scikit-learn = "*"
[pypi-dependencies]
sparse_dot_topn = ">=1.2" # ING's fast sparse top-k matmul
conda-forge tops out at sparse_dot_topn 0.3.1, which predates the v1
sp_matmul_topn API this recipe uses (checked against the conda-forge channel). pixi add sparse_dot_topn therefore installs a version without
the function — use pixi add --pypi sparse_dot_topn to get ≥1.2.
Additional Resources
Reference Files
references/fuzzy-name-matching.md— the full recipe: minimal code, threshold guide, normalize-first rule, scoped + global two-pass pattern, seven gotchas, alternatives considered, end-to-end results table
Example Files
examples/fuzzy_name_match_sample.py— runnable template:normalize(),fuzzy_match(),fuzzy_match_scoped(), plus a toy demo linking insider names to reporting-owner CIKs
Related
skills/wrds/examples/blockholders_pipeline/redo_bridge.py— production pipeline this recipe came out of (TRpersonid→ SECrptOwnerCik, 97.4% hit rate)skills/wrds/references/linkage.md— try a real link table first; fuzzy matching is for identifiers that genuinely don't cross vendors
Gemini production batch
## IRON LAW: NEVER use AI Studio for production runsNEVER USE AI STUDIO / THE GEMINI DEVELOPER API FOR PRODUCTION RUNS. Production batch runs use Gemini Enterprise Agent Platform (formerly Vertex AI): genai.Client(vertexai=True, project=..., location=...), Application Default Credentials (ADC), and GCS input/output. Substituting API-key/Files batch is not a shortcut: it sends the production workload to the wrong service.
- Developer API: a 2,111-row grounded batch returned 1,307
code 9 "Precondition check failed"rows; its retry queue stalled over 40 minutes. Tier 1'sgemini-3.8-flashqueue cap was 3M tokens. Repeating that production route recreates the failure, not a cheaper solution. These are Developer limits, not Cloud quotas.
Product name; current SDK docs use enterprise=True / GOOGLE_GENAI_USE_ENTERPRISE=True; vertexai=True remains compatible. Check SDK spelling and precedence before changing pins. aiplatform.googleapis.com and IAM roles/aiplatform.* retain their identifiers. Developer examples retained in references are not for production.
Choose the Cloud tier
Route by row count: ≤10 documents/rows → Cloud Flex PayGo; more → Cloud Batch. This is a judgment default from Flex throughput (~9 rows/hour at peak, measured on Developer Flex). Flex gives the same 50% discount as one direct generate_content call per row: no GCS JSONL, no queue, no import-format failures. A bulk-guard redirect for a few documents lands on Flex, not a Batch job; recipe in Flex.
| Tier | Use | Price / availability | Request path |
|---|---|---|---|
| Cloud Batch (Recommended for >10 rows) | Independent extraction/classification and tested grounded lookups | 50% off real-time; shared capacity; up to 72h queued, then most jobs finish within 24h running | GCS JSONL → client.batches.create; config.dest → GCS |
| Cloud Standard PayGo | Same-model synchronous smoke tests; interactive search | Standard Cloud tariff; capacity/model-dependent | client.models.generate_content with ADC |
| Cloud Flex PayGo (Preview) (Recommended for ≤10 rows) | Few-document extraction; small, synchronous, latency-tolerant lookups | 50% off Standard; higher throttling; global only; timeout up to 30 min | Vertex header X-Vertex-AI-LLM-Shared-Request-Type: flex |
| Cloud Priority PayGo | Latency-sensitive work, not cheap bulk | Higher model-specific tariff; global and supported us/eu multi-regions, not regional endpoints | Same Vertex header, value priority |
Sources: Batch, Flex, Priority, Cloud pricing. Cloud Flex/Priority are documented equivalents, not the Developer Interactions service_tier recipe or its quotas; read tier request patterns.
Cloud Interactions is Preview; generateContent remains fully supported. Production batch uses GenerateContentRequest JSONL, not interactions.create or an Interactions background task.
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Before writing code
READ EXAMPLES BEFORE WRITING ANY CODE. NO EXCEPTIONS.
NO BULK SUBMISSION WITHOUT A SAME-MODEL CLOUD SMOKE TEST AND A 5–10-ROW CLOUD END-TO-END TEST. Skipping these scales bad prompts, lost identifiers and ungrounded answers into a bad dataset.
- Read Cloud batch, the ADC/GCS setup runbook, and
examples/batch_processor.pyorexamples/icon_batch_vision.py. Verify project, API enablement, ADC, IAM and readable input/writable output GCS paths. gcloud user login alone is not ADC. Flex route (≤10 rows): read Flex instead of the GCS runbook. - Fetch current Cloud model cards, locations, batch support, thinking and pricing. New-project extraction defaults:
gemini-3.5-flash-litefor cheap extraction,gemini-3.8-flashfor harder extraction/search. Preserve existing model pins; Flash/Flash-Lite is the measured extraction default, not Pro. See model-selection evidence. - For Gemini 3.x, omit temperature, top_p and top_k. Use exact-model-supported thinking levels: 3.8 Flash supports low/medium/high, not minimal. See Cloud model guidance.
- Send relevant native PDF pages via GCS
fileData.fileUri; preserve layout/scans. For text-native filings, pre-cut relevant sections without truncating target evidence. Historical PDF cost observations are in Files, not a universal tokens/page tariff. - Run one synchronous Cloud request with the exact model/input/schema, then 5–10 rows through the actual Cloud batch/tier. Inspect content, per-row
statuserrors, finish reasons, usage, identifier round-trip and grounding if required. Follow scale-up testing. A ≤10-row Flex job has no separate pilot: its first row is the smoke test.
Production request pattern
from google import genai
client = genai.Client(vertexai=True, project="your-project-id", location="global")
job = client.batches.create(
model="publishers/google/models/gemini-3.5-flash-lite",
src="gs://your-bucket/requests.jsonl",
config={"display_name": "extraction", "dest": "gs://your-bucket/outputs/"},
)
Keep this client alive through submission, polling and retries. dest is a GCS prefix inside config, not a filename or top-level kwarg; after completion use the job's returned destination. Cloud input has request plus the examples' scalar correlation metadata, not Developer Files semantics. Validate with scripts/validate_jsonl.py --backend cloud; see request/output format and schema.
Cloud batch limits
| Limit | Current Cloud Gemini batch |
|---|---|
| Requests/job | 200,000 |
| GCS JSONL input | One file, up to 1 GB |
| Concurrent Gemini jobs / enqueued tokens | No predefined quota; dynamically shared model capacity (not Developer Tier 1 caps) |
| Queue expiry | Up to 72h before starting |
| Running time | Most complete within 24h; incomplete jobs cancelled after 24h running, charged for completed requests |
| Endpoint | Global for base models; supported regional endpoints for residency; global does not satisfy residency |
| Unsupported batch features | Provisioned Throughput, explicit caching, RAG; tuned Gemini 3+ models |
Sources: batch limits, quotas. Embedding batch has separate Cloud limits/schema. BigQuery is a documented alternative, with regional constraints; GCS is this skill's production default.
Grounding and acceptance
- Configure Google Search through the Cloud GenerateContentRequest tool shape and inspect
candidates[].groundingMetadataper row. Cloud grounding and model support are not a promise that every batch+search+schema combination works; the exact Cloud end-to-end sample must prove it. Read measured Cloud gotchas before grounded submission: emptygoogleSearchfails import; M100 grounded 97/100 on 3.8 Flash/global. Cloud batch excludes RAG/File Search. - Measured on Developer API, not a Cloud availability claim: past-tense founder framing grounded 0/138; fresh framing (“Search the web NOW for current pages… report what they say TODAY”) grounded 10/10, 1.9 searches/row. Flex shed load with 503s at peak, about 9 rows/hour. Prefer fresh framing and verify the Cloud sample; do not route bulk to Flex merely because an old prompt failed. More evidence: gotchas.
- Bulk (>10-row) per-document extraction is one Cloud Batch job over pre-cut inputs, never an interactive-agent fan-out. A 297-prospectus coder run consumed all three shared Claude accounts; it belonged in Batch.
- Reconcile every input identifier, duplicate/missing output and row error; preserve IDs on retry. A succeeded job or valid JSON is not correctness evidence. Budget input/output and search separately; a prompt search cap is not an enforced quota. Before a full grounded run, measure about 100 Cloud rows and project query cost from usage, not the requested cap.
- NO GROUNDED RUN OVER ~100 ROWS WITHOUT A USER-APPROVED COST PROJECTION: rows × pilot-measured queries/row × $14/1,000, minus the remaining free allowance; set a budget alert first. Searches were $390 of a $421 bill and every alert arrived after the spend. Formula, SKUs and the Billing → Reports URL: search cost gate.
- Use the harness's background notification mechanism for long monitoring, not a model session repeatedly narrating status. Do not switch backend, model, schema or location to clear an error without retesting.
Red flags — STOP
| About to | Do instead |
|---|---|
Use genai.Client() without vertexai=True for a production batch | STOP. Use explicit Cloud client, project/location and ADC |
| File API upload for a production job | STOP. Use private GCS input and output |
Pass dest= to create, or use a filename as output | STOP. Put the GCS output prefix in config.dest; retrieve this job's actual destination |
| Use a bare model resource after a Cloud batch 404 | STOP. Use publishers/google/models/<id> and verify the exact endpoint |
Infer batch availability from models.list() or downgrade to clear 404 | STOP. Check Cloud support and run the same-model Cloud batch sample |
Put Batch requests into interactions.create | STOP. Batch uses GenerateContentRequest JSONL and client.batches.create |
| Switch platforms or abandon a batch approach | STOP. Cancel every job and runner of the old approach — list its batches, stop the poller/farm — and confirm none are pending. An unstopped AI Studio retry runner billed searches for ~8h after the move to Vertex |
| Accept a succeeded job without row reconciliation | STOP. Inspect status, content, finish reason, identifiers and grounding |
Operations: GCS, CLI, troubleshooting, production patterns. Historical Developer File Search and Files/Interactions/embedding examples are not for production; never use them as production fallbacks.
Google Scholar CLI (scholar)
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Search Google Scholar for academic papers via the scholar command-line tool.
Requires: scholar on PATH (~/.local/bin/scholar → ~/projects/google-scholar-cli/scholar)
Check: command -v scholar || echo "MISSING: scholar CLI not installed"
IRON LAW: No Hallucinated Metadata
NEVER fabricate paper titles, authors, journals, years, or abstracts from memory.
When the user asks about a specific paper or needs citation details:
User mentions a paper
↓
Do you have the exact metadata from a scholar command in this session?
↓
YES → Use that data
NO → Run scholar search/lookup --bibtex FIRST, then report
Metadata Facts
- Training-data paper metadata is unreliable even for famous papers — wrong years, missing co-authors, garbled titles. Presenting remembered metadata is an unverified claim presented as fact; run
scholar lookup "title" --bibtexfirst, then report. --bibtexadds seconds, not minutes, and the BibTeX includes the abstract — reconstructing either from memory is counterproductive on its own terms.
Red Flags
- About to type a paper title from memory → STOP. Run
scholar lookup. - About to list authors without a source → STOP. Run
--bibtex. - Saying "published in" without verification → STOP. Check the BibTeX.
- Writing an abstract from memory → STOP. BibTeX includes the abstract.
Authentication
Before first use, authenticate by extracting cookies from an active Chrome session:
# Chrome must be running with remote debugging enabled
scholar auth --port 9222
Cookies are stored in ~/.google-scholar/cookies (mode 0600).
Core Commands
Scholar Labs Search (AI-Enhanced)
Natural language search using Google Scholar Labs API:
# One-shot search
scholar search "what are the key papers on attention mechanisms"
# JSON output for parsing
scholar search "corporate disclosure and information asymmetry" --json
# With BibTeX citations (includes abstracts)
scholar search "attention is all you need" --bibtex
# Download PDFs for results with full-text links
scholar search "transformer architectures" --download
# Interactive multi-turn mode (follow-up questions)
scholar search --interactive
Traditional Keyword Search
Standard Google Scholar full-text search:
# Keyword search
scholar lookup "machine learning transformers"
# Author search
scholar lookup "author:shleifer disclosure" --json
# With BibTeX
scholar lookup "asset pricing" --bibtex
# With PDF download
scholar lookup "deep learning" --download
Cite (BibTeX by Cluster ID)
Fetch BibTeX directly when you already have a cluster ID from search results:
# Single paper
scholar cite 5Gohgn6QFikJ
# Multiple papers, JSON output
scholar cite 5Gohgn6QFikJ 8409835334886051453 --json
Download (Single PDF)
Note: --download works for open-access PDFs (arXiv, NBER, etc.) but is unreliable for papers behind library link resolvers (institutional access). For paywalled papers, return the URL and let the user download manually.
scholar download "https://arxiv.org/pdf/1706.03762" --output attention.pdf
Quick Reference
| Need | Command |
|---|---|
| Natural language question | scholar search "question" |
| Keyword/author search | scholar lookup "keywords" |
| BibTeX citations | Add --bibtex to search/lookup |
| BibTeX by cluster ID | scholar cite <clusterId> |
| Download PDFs | Add --download to search/lookup |
| Download single PDF | scholar download "url" --output file.pdf |
| JSON output | Add --json to any command |
| Interactive follow-ups | scholar search --interactive |
| Re-authenticate | scholar auth |
Decision Tree: Which Command?
Do you have a cluster ID already?
YES → scholar cite <clusterId>
NO ↓
Do you have a natural language research question?
YES → scholar search "question"
NO ↓
Do you need keyword-exact or author-specific results?
YES → scholar lookup "author:name keyword"
NO ↓
Do you want follow-up refinement?
YES → scholar search --interactive
When to add flags:
Need citation metadata (authors, journal, year, abstract)?
YES → Add --bibtex
Need to download the PDF (open-access only)?
YES → Add --download (unreliable for paywalled papers — return the URL instead)
Need machine-readable output?
YES → Add --json
Verified Paper Information Workflow
When presenting paper information to the user, follow this workflow:
1. Run scholar search/lookup with --bibtex
2. Parse BibTeX fields for authoritative metadata:
- title, author, journal/booktitle, year, abstract
3. Present ONLY the fields returned by BibTeX
4. If BibTeX is missing a field, say "not available" — do NOT fill from memory
BibTeX includes abstracts: The --bibtex flag injects the snippet as an abstract field in the BibTeX entry. Use this instead of generating abstracts.
Output Format
Table output (default): Columns for #, Title, Authors, Year, Cited, Journal, followed by snippets and URLs.
JSON output (--json): Array of ScholarResult objects:
{
"title": "Paper Title",
"authors": "Author A, Author B",
"journal": "Journal Name",
"year": "2024",
"citations": 150,
"snippet": "Abstract excerpt...",
"url": "https://...",
"pdfUrl": "https://... or null",
"clusterId": "12345",
"position": 1
}
BibTeX output (--bibtex): Standard BibTeX entries with abstract field:
@article{key,
title={Paper Title},
author={Author, A and Author, B},
journal={Journal Name},
year={2024},
abstract={Abstract text from Google Scholar snippet...}
}
Journal filtering
scholar lookup --journal "<exact name>" restricts results server-side (repeat
the flag for more than one; they OR together). It is the only way to reach
the student-edited business law reviews — Chicago BLR, Penn JBL, Harvard BLR,
Columbia BLR, NYU JLB, Virginia L&B, Berkeley BLJ, Journal of Corporation Law,
Delaware JCL — none of which Consensus indexes.
scholar lookup "shareholder voting" --journal "Virginia Law & Business Review"
Division of labor with consensus: consensus filters peer-reviewed venues via
--journals-file and covers the T14 general-interest flagships; scholar covers
the business specialties via --journal. Same trusted-journal list feeds both.
--journal is lookup-only. scholar search (Labs) rewrites the query and
silently ignores the filter, so it errors out instead. Journal names go inside
q, which Scholar truncates past ~256 chars — a few journals per search, not
the whole list.
Trusted Journals (shared resource)
When searching Google Scholar, ALWAYS consult the shared trusted-journal list first:
File: ${CLAUDE_PLUGIN_ROOT}/references/trusted-journals.local.md
Shared with the consensus and research skills and the librarian agent — it lives at the plugin root, not under this skill. It is the user's curated list of trusted journals: one exact journal name per line, # comments. That format is load-bearing (consensus search --journals-file reads the same file), so every non-comment line must be a journal name — do not add authors or notes as bare lines. Use it to:
- Prioritize results from known-good journals and authors
- Flag unfamiliar sources - if a result is from an unknown journal, note it
- Suggest related searches - use known authors to refine queries
- Assess quality - weight results higher when they appear in trusted venues
How to Use Domain Knowledge
User asks: "find papers on corporate disclosure"
↓
1. Read `${CLAUDE_PLUGIN_ROOT}/references/trusted-journals.local.md`
2. Run scholar search/lookup
3. Cross-reference results against trusted journals/authors
4. Present results with quality signals:
- ★ = from trusted journal or by trusted author
- Results from unknown sources shown without star
Operational Rules
- No hallucinated metadata — NEVER cite title/author/journal/year/abstract from memory. Use
--bibtexorscholar citeto get verified data. - Scholar is for discovery — Use it to find new papers, not to read them
- Always use
--jsonwhen results will be processed programmatically - Use
--bibtexwhen presenting papers — It provides verified author, journal, year, and abstract fields - Cross-reference the trusted-journal list — Always check results against
${CLAUDE_PLUGIN_ROOT}/references/trusted-journals.local.md - Auth required — If search fails with auth errors, re-run
scholar auth - Rate limits — Google Scholar may rate-limit; space out rapid queries
grind — a loop with no session, whose only memory is one append-only journal
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
A bash while loop that calls a fresh model process per iteration and keeps nothing between them.
Cost inside a chat session is wakes × context, and the context only grows: measured on this
project, one orchestrator spent 1.05 billion cache-read tokens in a day across 1,816
turns on a ~575K-token context, and 208 of those wakes were status reports that needed no judgement
at all. Here nothing accumulates between iterations because nothing survives them.
The loop is scripts/grind.sh. Nothing session-scoped can be load-bearing in it — no CronCreate,
no Stop hook, no Monitor, no session id — so the journal is the whole interface. A human, or a
chat session peeking in, tails it and has no other handle on the run.
Three rules, and each one is a defect this loop already hit
1. The supervisor is a shell command, so no model call is spent while it is red. --gate is a
cheap precondition — a queue check, a file that appears when a round lands. While it exits non-zero
the loop appends a wait, sleeps --sleep, and never invokes the runner. A verdict that takes
ten hours otherwise buys ten hours of model calls whose only finding is that the job is still
queued. Wake on decisions, not on events. Each wait record carries the gate's last output line as
why, so a gate should print the reason it is shut rather than just exiting non-zero. A streak of
consecutive waits is announced through the notification channel while the run continues
(--wait-alert N, default 6, 0 disables), because a gate stuck closed looks identical to
legitimate waiting from outside.
2. The keep decision is --check's exit code, never the agent's opinion. An iteration reports
success whether or not it did anything, and no downstream check catches a confidently wrong answer.
The agent's only write channel is append, and no record it can write makes the loop declare
success.
3. The floor list is read from the journal at the top of every pass, before the prompt is built.
A fresh agent handed the same dead pool re-diagnoses it every iteration, for a week. A key filed as
a floor is injected into every later prompt as an exclusion, which is the only reason an amnesiac
loop converges at all.
done is written only after --check exited 0, in the loop's own process.
A model asked whether the goal is met answers from the pass it just ran, which is the one view that
cannot see the goal. Every other terminal record — stalled, budget, stopped — is likewise a
count or an exit code, never a judgement. Pick a check that names the state the WORK reaches: a
suite passing, a rate under a number, a count at zero. test -f report.md is true while the
objective is unmet, and the loop exits on pass 1.
Start it detached
setsid nohup bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh run \
--journal /abs/run.jsonl \
--prompt-file /abs/PROMPT.md \
--check 'bash /abs/measure.sh --lost-below 120000' \
--gate 'test -f /abs/out/round-ready' \
--sleep 300 --stall-after 5 --max-iters 200 \
>/abs/grind.log 2>&1 </dev/null &
Never foreground it: a Bash tool call caps out and takes the run with it, and a session held open to
watch is the cost this skill exists to remove. --runner defaults to claude-code and --model is
passed through to it; a test or a dry run points --runner at a stub instead. Every ending — done, stalled,
budget, stopped — is announced by default through the launching session's watcher (the farm-events stream);
agent-msg goes only to a --notify-to other session, or when there is no stream or no live watcher
there. A herdr popup too, where herdr is installed. --notify CMD replaces
that and gets GRIND_STATE, GRIND_EXIT and GRIND_JOURNAL; --notify none silences it. A failed
notification never changes the exit code. A loop killed hard reaches no ending, so it also files itself
in the launching session's farm-events stream, whose monitor reports a pid gone with no DONE (--no-events opts out).
If this session will ACT on the result (restart the loop, read the floors, ship the fix), CronCreate
an hourly heartbeat in the same turn as the launch — cron 7 * * * *. The prompt names the session's
standing objective or journal directory, never one batch, so it never goes stale; each tick runs
grind.sh status there. The farm-events monitor is the primary wake; the cron is the backstop that
survives a --resume, which monitors and background tasks do not. NEVER delete the heartbeat when a
loop ends (done, stalled, budget, stopped): the session's mandate (an overnight or autonomous
instruction, or queued next items) outlives any single loop, and a deleted heartbeat leaves the
session with no wake. If the prompt must change, create the new one first. Deleting asks the user
(permissions.ask), so a delete happens only when the user ends the mandate.
Then, from anywhere and at any time:
J=/abs/run.jsonl
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh status --journal "$J" # state, pid, iterations, floors, stall
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh tail --journal "$J" -n 40
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh floors --journal "$J"
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh stop --journal "$J" --why 'grid down for maintenance'
stop is a record, not a signal: the loop honours it at its next boundary, after the pass in flight
finishes. Once the loop answers it with stopped, the same run command resumes the journal,
floors and all. Exit codes are 0 done, 2 refused, 3 stalled, 4 budget exhausted, 5 stopped. The per-iteration watchdog defaults to FARM_MAX_TURNS=250.
That subcommand is refused from inside an iteration, and the refusal names its reason — the loop
marks the environment it invokes the runner in, and an operator's own shell carries no such mark, so
an iteration that reads this file and concludes the work looks impossible cannot end a run the
operator started. The guard closes the documented path and claims nothing beyond it: an iteration
runs bash and can still kill the pid recorded in start, so this prevents a good-faith mistake
rather than guaranteeing the loop cannot be stopped from inside.
Against the work hold — the difference is whether anything stays alive
| the work hold | grind | |
|---|---|---|
| what stays alive | a chat session, held open by a Stop hook and re-woken by a cron heartbeat | nothing — a bash loop outside every session |
| the memory | that session's context, which grows with every turn | one journal; each iteration is a fresh process that starts empty |
| what decides to continue | the hook, running the check when the session tries to stop | the loop, running --check at the top of every pass |
| what it costs | wakes × a context that only grows | one model call per decision, none while the gate is red |
| reach for it when | the work needs that session's own tools, connectors or approvals | the work is a command plus a prompt, and it will take days |
The journal is the only file
There is no pidfile, no state file, no lock and no cache. Every fact is derived by reading the
journal — the iteration counter is the largest i, the floor set is every floor record, the pid
comes from start. Two files that can disagree about one fact are a bug generator.
| kind | written by | means |
|---|---|---|
start | the loop | a run began: pid, check, gate, runner, model, prompt file |
iter / iter_end | the loop | iteration N began; the runner exited with this code |
wait | the loop | the gate was red, so the pass cost nothing |
progress | the agent | the goal moved; this is what --stall-after counts back from |
floor | the agent | this key is dead, and every later prompt is handed it until a reopen names its key |
reopen | the agent or operator | with subject: an exhausted subject is worth another pass; with key: that floor is retracted (a later floor on the key closes it again). Always with a non-empty rerunReason saying what changed |
done | the loop | --check exited 0 — the only record that means success |
stalled / budget | the loop | --stall-after passes in a row with no progress (default 3, 0 disables; an attempt or note is not progress); --max-iters spent |
stop / stopped | the operator / the loop | a stop was requested; the loop honoured it |
That split is enforced, not merely described. The agent may append only progress, floor,
attempt, note and reopen: append refuses every loop-owned kind with exit 2, so nothing an iteration
writes can end a run or report a finish the loop did not compute. A floor is refused unless it
carries a non-empty key, because one accepted without a key is dropped by the reader and the loop
re-diagnoses that family forever.
Any record may also carry a subject — the family the work is about, a CIK or a shard rather than a
free-text key — and that is what groups one iteration's attempt with the earlier ones. Every prompt
carries a GRIND_SUBJECTS block listing each subject's attempts since its last reset, its lifetime
progress count and an excerpt of its newest attempt, so a fresh iteration can see what has already
been tried instead of re-diagnosing it under a new key. A subject with --exhaust-after attempts
(default 3, 0 disables) and no progress since is marked EXHAUSTED and is excluded from the
iteration's choices the way a floor excludes a key. The only way back is a
{"kind":"reopen","subject":"...","rerunReason":"what changed"} record, which resets that subject's
count and is refused without both a subject and a non-empty rerunReason; a progress record on
the subject resets it too. Recording is never refused for exhaustion — an attempt on an exhausted
subject still lands, because the journal must not lie about the work that was done.
A floor is lifted the same way, by key: {"kind":"reopen","key":"<floor key>","rerunReason":"..."}
drops that floor from floors, status and GRIND_FLOORS, ordered by journal position; a floor
filed again afterwards is live again. A note cannot lift a floor — an operator withdrawing one
appends this reopen, or the passes keep honouring it.
Every prompt carries GRIND_JOURNAL, GRIND_SH, GRIND_ITER, GRIND_FLOORS, GRIND_SUBJECTS and
GRIND_NOTES, so an iteration needs nothing from outside itself:
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh append --journal "$GRIND_JOURNAL" \
'{"kind":"progress","key":"2004-noseries","note":"heading index resolved 41K rows"}'
bash ${CLAUDE_SKILL_DIR}/scripts/grind.sh append --journal "$GRIND_JOURNAL" \
'{"kind":"floor","key":"1934-paper-scans","why":"no machine-readable text in the source"}'
Iterations run with FARM_OUT_CHILD=1, so an iteration writes and commits directly; farm out only
work the prompt itself asks to delegate.
An operator steers a running loop with the same channel: append {"kind":"note","key":"...", "note":"..."} and the newest ten notes are printed as GRIND_NOTES in every later prompt, newest
last, outranking the prompt's own ranking. No iteration opens the journal file, so a note that is
not injected is a note nobody reads.
A record must be one JSON object on one line and must fit in a single atomic write, or it is refused — never split, never truncated — because a split record leaves a log nobody can replay. Readers skip lines that do not parse, which is what makes a journal cut short by a crash resumable rather than fatal. Anything larger than a key and a note belongs in a file the record names.
Reading it by hand is the same one pass:
grep -c '"kind":"iter"' "$J" # iterations attempted
grep '"kind":"floor"' "$J" | jq -r '"\(.key)\t\(.why)"' # the set every prompt is handed
grep '"kind":"iter_end"' "$J" | jq -r 'select(.exit != 0) | .i'
Red flags — STOP
| About to | Why wrong | Do instead |
|---|---|---|
| Let an iteration's own report decide the run is finished | the loop would be grading itself, and one confident wrong answer ends a week of work with the goal unmet | only --check exiting 0 writes done; the agent writes records, never verdicts |
Point --check at a milestone — a file existing, a report written, an agent having reported back | it is true while the objective is unmet, so the loop stops on pass 1 | name the state the work reaches: a suite passing, a count under a number |
Write a --check that passes once the inputs are fetched or built | the loop records done with the deliverable missing — 209 exhibit PDFs fetched and none read, 1,404 order PDFs fetched and 0 outcomes coded (board-structuring, twice) | fail the check until the final deliverable exists and is non-empty: coded rows > 0, or every fetched file extracted; print the count so done shows it |
Run without --gate when the verdict takes hours | every pass spends a model call to learn the job is still queued — that is the whole bill | a shell precondition; while it is red the loop records wait and sleeps for free |
| Hand an iteration a key an earlier one closed | it re-diagnoses the same dead family every pass, and the loop never converges | read GRIND_FLOORS before choosing work; file a floor the first time a key dies |
| Append a whole report as one record | over the single-write bound it is refused, and splitting it would leave a log nobody can replay | a key and a one-line note; write the report to a file and name its path |
| Add a pidfile, a progress file or a notes file beside the journal | two files that can disagree about one fact, and the tiebreak rule is the bug | append a record; derive the pid, the counter and the floors from the journal |
| Arm a hold in a session while a grind loop works the same objective | the hold blocks that session's stop while the grind's gate waits for no round in flight: each waits for the other and neither moves | one driver per objective — grind owns a long loop, and no session holds alongside it |
| Foreground the loop from a chat session | the Bash tool call caps out and kills the run mid-flight, and the live session is the cost grind removes | setsid nohup … &, then grind.sh status when you want to know |
kill -9 the loop to end it | the journal then ends on iter, and nothing can tell a kill from a crash | grind.sh stop, honoured at the next boundary |
| About to write a pass prompt that says "do one chunk, commit, exit" | each pass re-reads the investigation doc and the journal, so an hour of analysis became 10+ passes (docket-outcomes reached pass 59, ~20M weighted tokens per 14 passes) | tell the pass to finish the deliverable when it can, committing at checkpoints rather than exiting at them |
About to ask a pass for regression tests, "contracts" or edge-case hardening beyond what --check reads | passes polished instead of finishing, and operator "estimate now" notes took 2-3 passes to land (s220-deterrence) | the pass prompt names the deliverable and the check, nothing else |
Read iter_end with exit 0 as progress | it says the process ran, not that anything moved | progress records, which are what --stall-after counts |
| Launch a loop you will act on without a heartbeat | a --resume restores no monitor and no background task, so a resumed session never hears it end | CronCreate an hourly tick at launch, naming the standing objective; keep it past the ending |
| About to delete the heartbeat cron because a loop/batch ended | the mandate outlives the batch; a loop that lost its only wake this way sat idle for 3 h until the user found it | leave it; the user ends the mandate, and the delete asks them |
| Reach for the work hold to keep this going | the hold keeps a SESSION alive, which is the thing this loop exists to avoid | the hold while a session must live; grind when none should |
Headline Cards for Typst Presentations
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Typst rules in force here — read the one that governs what you are writing; they are independent:
!k=slides; command -v typst-rules >/dev/null 2>&1 && exec typst-rules "$k"; echo "(typst corpus unavailable: NO Typst rule is listed here — install the typst plugin, or start a new session so its bin/ reaches PATH)"
An editorial recreation of a newspaper clipping: light paper stock, the real masthead logo, a top rule carrying a category kicker and dateline, a serif headline with a Last-Week-Tonight yellow highlighter swipe over one phrase, and a short standfirst.
Read templates/theme.typ before writing a card. The implementation is the
authority; if it disagrees with this file, believe it and fix this file.
Architecture
presentation/
├── templates/theme.typ ← headline-card() lives here; read it first
├── data/headlines-NN.json ← card data, NN = the lecture number
├── assets/logos/*.svg ← DARK / full-colour logos (see below)
└── slides/XX-topic/NN.typ ← loops over the JSON, one slide per card
Step 1: the JSON entry
{
"venue": "WSJ",
"date": "June 24, 2026",
"kicker": "Tech",
"headline": "The headline exactly as published",
"highlight": "phrase to swipe in yellow",
"quote": "The article's own standfirst/dek.",
"logo": "../assets/logos/wsj.svg"
}
highlight must be a substring of headline, matching character for
character — the function splits on it, so a straight quote where the headline
has a curly one silently produces no swipe. kicker and quote are optional.
Quote the article, do not summarise it. The standfirst should be the publication's own dek or a vivid line from the piece. "Discusses the impact of proxy advisors" is not a card; "Anyone who gives them money — shame on you" is.
Nothing on the card may be composed. Headline, dek, outlet and date are the
article's own or they do not go on the slide — a card asserts that a publication
printed this. Pull them from the record, and if a field cannot be sourced, omit
it rather than inventing it. Note that Reader's
reader-get-document-details returns null for published_date/source_url
while reader-list-documents --id <id> returns them populated — check both
before concluding a date is unavailable.
Step 2: paper stock is per publication
headline-card picks the stock from venue. FT is famously salmon and the
broadsheets are white; one generic cream for everything reads as wrong to anyone
who knows the paper.
| venue | stock |
|---|---|
| Financial Times, FT | #FFF1E5 salmon |
| WSJ, NY Times, Bloomberg, ABC News | #FFFFFF |
| X, Twitter | #FFFFFF — light-mode surface; the mark is black |
| Bluesky, Mastodon, Threads, LinkedIn | #FFFFFF — same embed layout; accents #01A5FF / #563ACC / black / #0A66C2 |
| anything else | #F7F4EC cream fallback |
Microblog cards are embeds, not clippings
A post takes the same function and the same stock table, but an embed layout, because the masthead layout is what made it read as a clipping of a tweet:
{
"venue": "X",
"name": "Matt Levine",
"handle": "@mattlevine",
"avatar": "../assets/avatars/mattlevine.jpg",
"date": "3:42 PM · Aug 22, 2026",
"headline": "The post text, verbatim."
}
| slot | embed | newspaper |
|---|---|---|
| mark | top right | left, as the masthead |
| identity | avatar + name over @handle, left | venue wordmark |
| date | below the text, over a hairline | top right, with the kicker |
| text | sans regular 21pt | serif bold 28pt |
| border | #cfd9de (X's own) | #ded7c7, radius 0.3em |
name and avatar are optional — with no name the handle carries the slot
alone rather than leaving a bold gap, and with no avatar the row simply starts
at the name. Secondary text is #536471, X's own. Supply logo to use a real X
mark; otherwise a bold sans X stands in.
Bluesky and Mastodon take the same layout. All three microblog embeds have
one shape — avatar + name over handle at the left, mark top right, timestamp
below the text — so this is a venue LIST (X, Twitter, Bluesky,
Mastodon, Threads, LinkedIn), not a branch per platform. Only the mark and its accent differ, and the
handle format is data rather than code: @user (X, Threads),
@user.bsky.social, @user@instance.social. Every microblog ships its real mark inline. post-mark-svg in theme.typ
carries the official path for X, Bluesky, Mastodon, Threads and LinkedIn, each
filled with the platform's own colour, so no deck has to ship a logo asset. The
non-X paths are the simple-icons set (MIT); X's is the official 300×271 mark.
A caller-supplied logo still wins over all of them. The letter fallbacks
(b., m, @, in) now only fire for a venue with no path at all.
X/Twitter is a venue, not a second function. The card suppresses the
newspaper-only chrome for it — top rule, kicker, serif headline face, highlighter
swipe — and takes handle: ("@elonmusk") beside the mark. Everything else, the
stock table included, is shared.
Add a venue to the stocks dictionary in theme.typ rather than passing a
one-off, so the next card for that outlet inherits it. stock: rgb("…")
overrides per card when one genuinely differs.
Legibility does not constrain this choice — ink, dek and dateline land within ~1:1 of each other across every stock — so it is purely identity.
Step 3: the logo
Dark or full-colour, on a light card. Not the -white.svg variant; those are
for dark backgrounds and are invisible here. Use wsj.svg, nyt-dark.svg,
ft-dark.svg, abc-news-dark.svg.
IRON LAW: real vector logos only
If you catch yourself writing an SVG with <text> elements spelling out the
publication name — stop. That is a placeholder, not a logo, and it ships with the
wrong font, weight and spacing.
grep '<text' assets/logos/publication.svg && echo "FAKE — download a real logo"
Download from Wikimedia Commons, the brand's press page, or a logo repo. Wikimedia has SVGs for essentially every major publication; "I can't find one" means search harder.
Sizing needs no per-logo tuning: the function normalises by aspect ratio, so a narrow mark and a wide wordmark carry similar visual weight.
Step 4: render
#{
let cards = json("../../data/headlines-NN.json")
for card in cards {
slide[
#headline-card(
venue: card.venue,
date: card.date,
headline: card.headline,
quote: card.at("quote", default: none),
logo: card.at("logo", default: none),
phrase: card.at("highlight", default: none),
kicker: card.at("kicker", default: none),
stock: card.at("stock", default: auto),
)
]
}
}
One card per slide, and no === heading on it. Grid layouts truncate quotes
and shrink logos below recognition; a heading eats the vertical space the card
fills. Put any section header on the preceding slide.
Step 5: look at it
Compile, get the page from find-slide-page, render it, and read it. What
matters: the logo renders and is dark enough; the swipe covers the intended
phrase; nothing is clipped; no single orphaned word ends the headline or
standfirst.
Fix widows in the JSON text, never in Typst — tighten the wording, or join
the last two words with . Never pad with filler to fix a widow.
Red flags — STOP
| Symptom | Cause | Fix |
|---|---|---|
| Logo invisible | a -white.svg on the light card | use the dark/full-colour variant |
| Logo missing entirely | JSON path taken as relative to slides/ | paths resolve from theme.typ: ../assets/logos/… |
| No yellow swipe | highlight is not an exact substring of headline | match curly vs straight quotes character for character |
| Card doesn't read as an object on white stock | border derived from the stock is near-white | the border is a fixed grey; do not re-derive it from newsprint |
| An FT card looks like every other card | venue missing from the stocks dictionary | add it |
| An X post wears a serif headline, a top rule or a yellow swipe | venue not spelled X/Twitter, so it took the newspaper branch | fix the venue string — never add a second card function |
| Dek is generic | summarised instead of quoted | use the publication's own standfirst |
| A date is invented | it wasn't in the source | omit the field; never fabricate one |
<text> in the logo SVG | placeholder, not a logo | download the real one |
Facts
- The dateline is the smallest text on the card, so it carries its own darker
grey rather than the dek's
mute, which lands ~5.3:1 on newsprint and washes out under projector gamma. - Black on the yellow swipe measures ~13.7:1 — the highlight helps readability rather than hurting it, so spend it on the phrase that carries the point, never decoratively.
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Contents
- When to Use What
- Login Node Enforcement
- Cluster Reference
- Slurm Job Submission
- Array Jobs
- SGE to Slurm Translation
- Environment Variables
- WRDS Data Access
- Monitoring & Debugging
- Resource Billing
When to Use What
Three compute environments, each with a clear role:
| Environment | Use For | Examples |
|---|---|---|
| Local / RJDS | Exploration, prototyping, notebooks | EDA, quick plots, marimo/Jupyter, test on small samples, iterate on code |
| WRDS (SGE) | Data access, SAS ETL, file parsing | SAS jobs against WRDS libraries, SEC filing parsers on /wrds/sec/, scan_covers, ad-hoc SQL |
| UVA HPC (Slurm) | Scale compute | Model estimation (PIN), large polars pipelines, anything needing >10 cores or >1 hour |
The Workflow
1. EXPLORE (local/RJDS) → Prototype code, test on 5-10 items
2. BUILD DATA (WRDS) → SAS ETL or PostgreSQL queries (data lives there)
3. ESTIMATE AT SCALE (HPC) → sbatch when you need 100+ cores
4. ANALYZE RESULTS (local) → Pull results back, notebooks, regressions, tables
Decision Rules
- Does it need WRDS filesystem access? (
/wrds/sec/, SAS libraries) → WRDS - Is it CPU-intensive and embarrassingly parallel? → HPC
- Is it exploratory or iterative? → Local / RJDS
- Is it a quick SQL query? → Either WRDS or HPC (both have PostgreSQL access)
HPC Interactive Partition
The interactive partition (42 nodes, 12h max) is for testing sbatch scripts on one chunk before submitting 176 tasks, not for replacing local dev work:
salloc -p interactive --cpus-per-task=4 --mem=16G --time=1:00:00
# test your script, then exit and sbatch the real job
Why This Split Matters
PIN estimation proved it: WRDS SGE has 10 concurrent slots and took 8+ hours without starting OWR. UVA HPC ran 70+ OWR tasks simultaneously and finished in 30 minutes. But WRDS is still the right place to build the data — the SAS libraries and SEC filings live there.
Login Node Enforcement
IRON LAW: NEVER RUN COMPUTE ON THE LOGIN NODE
The login node is shared infrastructure. Running estimation, bulk processing, or any CPU-intensive work directly via SSH will get the account flagged and the process killed.ALWAYS write a Slurm submission script and submit via sbatch. No exceptions.
ssh uva-hpc 'python3 est.py owr 2020'→ WRONG. Use sbatch.ssh uva-hpc 'nohup ./process &'→ WRONG. Still the login node. Use sbatch.ssh uva-hpc 'for year in 2003..2024; do python3 ...; done'→ WRONG. Use sbatch --array.sbatch run_est.sh owr→ CORRECT.
The login node is for: sbatch, squeue, scancel, sinfo, scp, ls, head, short queries.
Login Node Facts
- Tests go through the scheduler too: write the sbatch script first and test with
--array=1-1. The login-node "quick test" is the run that flags the account — one stock becomes 5,000 when the args change, and you don't know it "only takes 30 seconds" until it runs.
Red Flags — STOP If You're About To
- Write
ssh uva-hpc 'python3 ... > output'→ STOP. Write a submit script. - Write
ssh uva-hpc 'nohup ... &'→ STOP. Use sbatch. - Run a loop over years/permnos interactively → STOP. Use
--array.
Cluster Reference
UVA HPC (Rivanna/Afton)
- SSH:
ssh uva-hpc(configured with ProxyJump through Mac via tailnet) - User:
vwh7mb - Home:
/home/vwh7mb(GPFS, 12PB shared, no per-user quota displayed) - Scratch:
/scratch/vwh7mb/(Weka, 12TB) - Allocation: 10M SUs (service units ≈ weighted CPU-core-hours)
Partitions
| Partition | Nodes | CPUs/Node | RAM/Node | MaxTime | MinNodes | MaxNodes | Use For |
|---|---|---|---|---|---|---|---|
standard | 301 | 40+ | 384GB+ | 7 days | 0 | 1 | Single-node jobs, array tasks |
parallel | 179 | 96 | 768GB | 3 days | 2 | 64 | Multi-node MPI jobs only |
gpu | 44 | 36+ | 257GB+ | 3 days | — | — | GPU workloads |
interactive | 42 | 32+ | 128GB+ | 12 hrs | — | — | Interactive/debugging |
CRITICAL: Partition Selection
**Use `standard` for embarrassingly parallel array jobs** (PIN estimation, file processing, per-year/per-stock tasks).The parallel partition requires MinNodes=2 — it will reject single-node jobs with "Node count specification invalid". It is designed for MPI jobs that span multiple nodes.
Wrong: #SBATCH --partition=parallel for array jobs → submission fails
Right: #SBATCH --partition=standard for array jobs → 301 nodes available
When to Use Each Partition
standard (default choice for most research computing):
- Embarrassingly parallel work: array jobs where each task is independent (PIN estimation, file parsing, per-stock/per-year processing)
- Single-node Python/R with
ProcessPoolExecutor,multiprocessing,mclapply - Any job where tasks don't communicate with each other
- MaxNodes=1, so each array element runs on exactly one node
parallel (multi-node distributed computing):
- MPI jobs where processes on different nodes exchange messages (
mpi4py, OpenMPI, MVAPICH) - Dask distributed or Ray clusters spanning multiple nodes
- Large linear algebra / matrix factorizations that exceed single-node RAM (ScaLAPACK, PETSc)
- Simulations with inter-process communication (CFD, molecular dynamics)
- Key requirement: your code must explicitly coordinate across nodes (MPI, Dask scheduler, etc.) —
ProcessPoolExecutorandmultiprocessingare single-node only - MinNodes=2, 96 CPUs and 768GB RAM per node — use when one node isn't enough
gpu (GPU-accelerated workloads):
- Deep learning training/inference (PyTorch, TensorFlow, JAX)
- GPU-accelerated linear algebra (CuPy, RAPIDS)
- LLM inference or fine-tuning
interactive (debugging and development):
- Testing job scripts before full submission:
salloc -p interactive --cpus-per-task=4 --mem=16G --time=1:00:00 - Debugging segfaults or data loading issues
- 12-hour max — not for production runs
Python/R Environment
- pixi: Install to
$HOME/.pixi/bin/pixiviacurl -fsSL https://pixi.sh/install.sh | bash - Project envs:
$HOME/projects/<name>/.pixi/envs/default/bin/python - Modules (alternative):
module load python— but pixi preferred for reproducibility - NEVER install Jupyter kernels globally on HPC
Slurm Job Submission
Basic Submit Script
#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --partition=standard
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=3:00:00
#SBATCH --output=logs/job-%A_%a.log
mkdir -p logs
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
PYTHON=$HOME/projects/my-project/.pixi/envs/default/bin/python
$PYTHON -u my_script.py --workers ${SLURM_CPUS_PER_TASK:-8}
Submission
sbatch script.sh # submit
sbatch script.sh arg1 arg2 # args passed to script as $1, $2
Note: unlike SGE's qsub run.sh <model>, Slurm passes arguments after the script name directly. Use ${1:?Usage: sbatch script.sh <arg>} to enforce.
Array Jobs
Pattern
#SBATCH --array=1-176 # tasks 1 through 176
#SBATCH --array=1-176%50 # max 50 concurrent tasks
#SBATCH --array=1,5,9,13 # specific tasks only
Year × Chunk Sharding (PIN estimation pattern)
#SBATCH --array=1-176
# 22 years × 8 chunks = 176 tasks
# Decode: year = START_YEAR + (id-1)/NCHUNKS, chunk = (id-1)%NCHUNKS
NCHUNKS=8
START_YEAR=2003
idx=$((SLURM_ARRAY_TASK_ID - 1))
year=$((START_YEAR + idx / NCHUNKS))
chunk=$((idx % NCHUNKS))
Task List Sharding (file processing pattern)
# Equivalent to SGE's sed -n "${SGE_TASK_ID}p" pattern
ITEM=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$TASK_LIST")
Re-running Failed Tasks
# Re-run specific tasks
sbatch --array=5,12,87 script.sh
# Re-run a range
sbatch --array=10-20 script.sh
SGE to Slurm Translation
Directives
| SGE | Slurm | Notes |
|---|---|---|
#$ -N job_name | #SBATCH --job-name=job_name | |
#$ -cwd | (default behavior) | Slurm runs from submit dir by default |
#$ -l m_mem_free=4G | #SBATCH --mem=4G | Per-node memory |
#$ -pe onenode N | #SBATCH --ntasks=1 --cpus-per-task=N | Single-node parallel |
#$ -j y | (default behavior) | Slurm merges stderr into stdout by default |
#$ -o logs/out-$TASK_ID.log | #SBATCH --output=logs/out-%A_%a.log | %A=job, %a=array task |
#$ -t 1-176 | #SBATCH --array=1-176 | |
| (no equivalent) | #SBATCH --partition=standard | Required — no default partition |
| (no equivalent) | #SBATCH --time=3:00:00 | Default 5h, max 7d on standard |
Environment Variables
| SGE | Slurm | Description |
|---|---|---|
$SGE_TASK_ID | $SLURM_ARRAY_TASK_ID | Array task index |
$JOB_ID | $SLURM_JOB_ID | Job ID |
$NSLOTS | $SLURM_CPUS_PER_TASK | Allocated CPUs |
$HOSTNAME | $SLURM_NODELIST | Assigned node(s) |
$SGE_TASK_FIRST | $SLURM_ARRAY_TASK_MIN | First array index |
$SGE_TASK_LAST | $SLURM_ARRAY_TASK_MAX | Last array index |
Commands
| SGE | Slurm | Description |
|---|---|---|
qsub script.sh | sbatch script.sh | Submit job |
qstat -u $USER | squeue -u $USER | List running jobs |
qdel job_id | scancel job_id | Cancel job |
qstat -j job_id | scontrol show job job_id | Job details |
qacct -j job_id | sacct -j job_id | Job accounting |
| (no equivalent) | sinfo -p partition | Partition info |
Conversion Checklist
When converting an SGE script to Slurm:
- Replace
#$directives with#SBATCHequivalents (see table above) - Add
#SBATCH --partition=standard(SGE has no equivalent — partition is implicit) - Add
#SBATCH --time=(SGE defaults to unlimited on WRDS) - Replace
$SGE_TASK_ID→$SLURM_ARRAY_TASK_ID - Replace
$NSLOTS→$SLURM_CPUS_PER_TASK - Replace
$JOB_ID→$SLURM_JOB_ID - Remove
#$ -cwdand#$ -j y(Slurm defaults) - Update log path variables:
$TASK_ID→%a,$JOB_ID→%A - Update data paths from WRDS scratch to HPC scratch
Monitoring & Debugging
Check Job Status
squeue -u $USER # all my jobs
squeue -j 12345678 # specific job
squeue -j 12345678 -t R | wc -l # count running tasks
squeue -j 12345678 -t PD # show pending tasks + reasons
squeue -u $USER --format='%.10i %.9P %.12j %.2t %.10M %.4C %R' # detailed
Common Pending Reasons
| Reason | Meaning |
|---|---|
(Priority) | Lower priority than other queued jobs — will run eventually |
(Resources) | Not enough free nodes/CPUs — waiting for running jobs to finish |
(QOSMaxCpuPerUserLimit) | Hit per-user CPU limit on this QOS |
(AssocMaxJobsLimit) | Hit max concurrent jobs for this account |
Job Accounting (after completion)
sacct -j 12345678 --format=JobID,State,ExitCode,Elapsed,MaxRSS,NCPUS
sacct -j 12345678 -a --format=JobID,State,ExitCode # all array tasks
Log Files
Output goes to --output path. With %A_%a pattern:
logs/est-12345678_1.log— job 12345678, array task 1- Check for errors:
grep -rl 'Error\|Traceback' logs/est-12345678_*.log
Resource Billing
UVA HPC bills in Service Units (SUs), which are weighted CPU-core-hours:
SU = (CPU_cores × 4.6369 + Memory_GB × 0.2842) × hours
Cost Examples (standard partition)
| Config | SU/hour | 176 tasks × 3 hrs |
|---|---|---|
| 1 CPU, 4GB | ~5.8 | ~3,062 |
| 8 CPU, 32G | ~46.2 | ~24,404 |
| 40 CPU, 160G | ~231 | ~121,968 |
With 10M SUs allocated, even aggressive usage (8 CPU × 176 tasks × 3 hrs = ~24K SUs) is negligible (<0.25% of allocation).
Check Balance
allocations # show allocation balance
allocations -a myallocation # specific allocation
WRDS Data Access
WRDS PostgreSQL is accessible from HPC compute nodes. Use polars + connectorx for fast data pipelines that replace SAS entirely.
Connection
- Host:
wrds-pgdata.wharton.upenn.edu:9737 - Credentials:
~/.pgpass(chmod 600) - User:
edwin_hu(UVA account)
Quick Start
from wrds_conn import read_wrds
# WRDS SQL → polars DataFrame in one line
df = read_wrds("SELECT * FROM crsp.msf WHERE date >= '2020-01-01'")
# Write to Parquet for reuse
df.write_parquet("/scratch/vwh7mb/data/crsp_msf.parquet")
wrds_conn.py (see examples/wrds_conn.py) parses .pgpass and builds a connectorx-compatible URI — connectorx doesn't read .pgpass natively.
Pipeline: SQL → polars → Parquet (replaces SAS)
Old: WRDS SAS → .sas7bdat (7GB) → Python HDF5 conversion → .h5 (390MB)
New: WRDS PostgreSQL → polars/connectorx → .parquet
No SAS license needed. Single step. Portable output.
See references/wrds-polars-pipeline.md for full examples (joins, partitioned output, Slurm submission for large queries).
Additional Resources
Reference Files
references/sge-to-slurm.md- SGE → Slurm migration: directive-by-directive translation table for converting WRDS grid jobs to UVA HPCreferences/wrds-polars-pipeline.md- WRDS PostgreSQL → polars → Parquet pipeline (joins, partitioned output, Slurm submission for large queries)
Example Files
Copy these rather than writing a submit script from scratch — both use --partition=standard, which is the choice array jobs actually need:
examples/array_job_filelist.sh- Array job sharded over a file list (one task per chunk of files)examples/array_job_year_chunk.sh- Array job sharded over year × chunk, with multiple workers per task (the PIN-estimation shape)examples/wrds_conn.py- Parses.pgpassinto a connectorx-compatible URI (connectorx doesn't read.pgpassnatively)
Sponsors
Details
- Marketplace
- edwinhu/workflows
- Category
- Development