
medsci-presentation
Academic presentation and PPTX building, PDF/document rendering, environment setup, skill publishing, and offering your changes back to the project.
- GitHub stars
- 387
Install
/plugin install medsci-presentation@medsci-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add Aperivue/medsci-skills.
Contents
Discovered from sourceSkills 62
Academic AIO Skill — Medical AI Paper Visibility for AI Search Engines
Communication Rules
- Surface the checklist in the response. Never apply AIO edits silently.
- When a rule conflicts with journal formatting, defer to the journal and mark the item NA with the reason.
- When introducing a rule, cite its source (TRIPOD+AI, CLAIM, STARD-AI, Wu 2025, Algaba 2024, Aggarwal 2024 GEO) by DOI or arXiv ID from External References. Never invent a citation, DOI, arXiv ID, or reporting-guideline item number; mark anything you cannot verify
[VERIFY].
Section 1 — Title and Abstract Optimization
1.1 Title three-slot rule
Structure: [Task] + [Modality or anatomy] + [Model family or method class]. Include one concrete differentiator (dataset scale, new benchmark, "first …") when defensible. Avoid keyword stuffing (penalized as spam by AI overviews).
- PASS: "Transformer-based segmentation of skull fractures on non-contrast head CT."
- FAIL: "A novel advanced deep-learning AI machine-learning framework for medical image analysis."
1.2 Structured abstract
Use the journal-required structure (Background / Methods / Findings / Interpretation for the Lancet family; Background / Purpose / Materials and Methods / Results / Conclusion for the RSNA family; etc.). If the journal allows unstructured, still structure it internally. Each section stands alone as a semantic chunk of ≤ 3 sentences so that RAG chunk-boundary splits do not break the claim.
1.3 Opening and closing sentences
- First sentence: the problem AND the contribution in one line. Never open with filler ("In recent years, AI has revolutionized …") — it burns the chunk LLM summarizers extract most.
- Last sentence: explicit interpretation ("we show that …", "this implies …"). No hedging-only closes.
1.4 Taxonomy line
Include one sentence that names the field's controlled vocabulary ("diagnostic-accuracy study", "foundation-model evaluation", "LLM-as-judge", "agentic radiology workflow"). Entity linkers in AI indexes use this line.
1.5 Quantified claim
Every abstract must contain at least one numeric primary outcome with a confidence interval (for example, "AUC 0.94 [95 % CI 0.91–0.96]").
1.6 Reporting-guideline anchor
Name the guideline in the abstract or the opening sentence of Methods: "Reported following TRIPOD+AI (Collins 2024) and CLAIM 2024 (Tejani 2024)". Add STARD-AI 2025, DECIDE-AI, or TRIPOD-LLM when applicable. AIO-rule ↔ guideline-item mapping: references/reporting_guideline_mapping.md.
1.7 Keyword, MeSH, and RadLex coverage
Title, abstract, and keywords together should cover the concept's key terms, without repeating title terms in the keywords (92 % of the surveyed papers repeated key terms across title, abstract and keywords; Royal Society 2024, doi:10.1098/rspb.2024.1222). Include:
- Core MeSH terms (verify against the NLM MeSH browser) and RadLex terms where applicable.
- Modality synonyms ("chest radiograph (CXR)", "non-contrast CT (NCCT)").
- Both US and UK spellings when relevant.
Section 2 — Manuscript-Level AIO
2.1 Summary box
Include the journal-specific summary box verbatim when supported; it is the fragment AI search engines most often copy or paraphrase:
- Lancet family: "Research in context" (Evidence before this study / Added value / Implications).
- RSNA Radiology and RYAI: 3 bullets, one claim each — labelled "Key Results" for Radiology and "Key Points" for RYAI in
references/journal_summarybox_templates.yaml. Confirm the label in the current RSNA instructions for authors; the format check accepts either label for Radiology. - npj Digital Medicine: no summary box; Articles carry an unstructured abstract of up to 150 words.
- Nature Medicine: editor's summary (supplied by editorial, but draft one proactively).
Journal-specific templates: references/journal_summarybox_templates.yaml. Never invent a summary-box rule: verify each template against the journal's current instructions for authors before applying it.
Deterministic format check. Validate the drafted box against its journal spec with python3 ${CLAUDE_SKILL_DIR}/scripts/check_summary_box.py --manuscript <file> --journal <stem> --strict (reads references/summary_box_specs.json: Key Points / Key Results top-level bullet count + one-claim-per-bullet, Research-in-context's three sub-blocks each opening a line and carrying text, plain-language word band). It catches the wrong-format / wrong-bullet-count box that a production technical check rejects.
Known limits. The box ends only at a heading or a bold-only label line, so a list under a plain-text label (Abbreviations:) right after the box is counted into it, and an emphasized body phrase that starts with the box label (*Key points* of prior work ...) is taken as the box; read the box by eye when either shape is present. In a Research-in-context box, a bold-only line after the last sub-block label ends the box, so a sub-heading inside Implications (**For clinicians**) cannot be told from the next section; an Implications sub-block left empty that way is reported as an advisory, not a failure.
2.2 Declarative section headings
Headings state a claim, not a generic label: "Model underperforms on rare-finding subset" beats "Subgroup analysis".
2.3 Numeric claim compression
In the Methods and in at least one Results paragraph, compress primary-outcome statistics into one sentence: "On the internal test set (n = 842), the model achieved AUC 0.94 (95 % CI 0.91–0.96), sensitivity 88.2 % (85.1–91.0), specificity 91.4 % (88.7–93.6), at an operating point of 0.37."
2.4 Reproducibility block
Include a labeled block (end of Methods or a standalone Data/Code Availability section) listing data availability and license, code availability with DOI, model weights and checkpoints, prompts and configuration files, random seeds, and compute environment. Flag code described only as "available on reasonable request": scrapers read it as not reproducible and AI tools demote the paper.
2.4a Citing an AI-assisted tool by use-class
Frame an AI-assisted tool by what it did, not by hiding it: verification/QA and analysis uses go in a Software / Code-availability statement (citable); generative drafting or humanizing goes in the journal's AI-use disclosure field, not a citation. A self-citation by the tool's author also requires a COI disclosure and should cite only the functions the work used. Use-class table and rules: ${CLAUDE_SKILL_DIR}/references/ai_tool_citation_framing.md.
2.5 Limitations enumeration
List limitations explicitly and name each one (generalizability, spectrum bias, dataset shift, single-center training, label noise). Flag "clinical grade" or "replaces radiologists" overclaims: LLM trust heuristics demote them and they invite reviewer rejection.
2.6 Standalone figure captions
Each caption re-states the claim, the dataset, and the metric, because captions survive in vector and image-retrieval indexes when the body text is lost.
Section 3 — Preprint, Channel, and Indexing Strategy
3.1 Preprint versus fast-track
- Default: post to medRxiv (clinical), arXiv (methods, cs.CV / eess.IV), or bioRxiv on the day of journal submission. This puts the paper into Semantic Scholar within 24–72 hours and into Perplexity's web index immediately.
- Exception: if the target journal offers a fast-track review cycle (acceptance → online within roughly 30–60 days) AND the authors prefer a single canonical version, the preprint may be skipped; compensate with aggressive post-acceptance SNS seeding and PMC deposit.
- Never skip both preprint and fast-track — a paywall-only paper is invisible to most RAG pipelines.
3.2 Journal preprint policy
Most medical-AI venues allow preprints; a few restrict them or require disclosure. Verify the current policy on Sherpa Romeo or the journal's instructions for authors before posting.
3.3 Indexing time-lag
Read ${CLAUDE_SKILL_DIR}/references/launch_sequencing.md when planning launch timing; it lists how long each index takes to pick a paper up.
3.4 Open-access choice
Prefer gold OA with CC-BY when budget allows; otherwise green OA via preprint plus author-accepted manuscript. Closed-access papers without a preprint lose roughly 30–50 % of AI-tool citations because Elicit, Consensus, and Perplexity Academic cannot extract from paywalled PDFs.
Funder OA-policy decision tree (Plan S, NIH, UKRI, Gates, Wellcome, NRF, MoHW): references/oac_funding_checklist.yaml.
3.5 Post-acceptance channel checklist
- Deposit AAM to PMC or Europe PMC.
- Update ORCID and Google Scholar profile.
- Post to Threads / X / BlueSky with DOI, one-sentence claim, and figure; long-form post on LinkedIn.
- Submit to Papers with Code if the paper reports a benchmark.
- Upload model or dataset to Hugging Face with a model/dataset card.
Section 12 gives the day-by-day order.
Section 4 — Review-Paper Strategy
Read ${CLAUDE_SKILL_DIR}/references/pre_draft_strategy.md when the artifact is a review or taxonomy paper, or the phase is pre-draft.
Section 5 — GitHub, CITATION.cff, Zenodo, Hugging Face
Read ${CLAUDE_SKILL_DIR}/references/repository_and_cards.md when the artifact is a README, CITATION.cff, Zenodo record, or Hugging Face model/dataset card (rules 5.1–5.6).
- Embed JSON-LD
ScholarlyArticle/SoftwareSourceCode/Dataset/Personmarkup in repository pages and author landing pages — templates inreferences/schema_markup_templates/, validated withpython scripts/validate_schema.py path/to/file.jsonld. - Never auto-complete author lists, ORCIDs, or affiliations in
CITATION.cffor Zenodo metadata; surface the empty slots to the user.
Section 6 — Authority and E-E-A-T Signals
- Maintain an author landing page (GitHub Pages, personal domain, or institutional page) listing all papers with DOIs and open-access links.
- Use one consistent affiliation string across papers; inconsistency fragments the author entity in knowledge graphs.
- Keep ORCID complete and linked to Google Scholar. Re-run author disambiguation on Semantic Scholar every 6 months.
- Cross-link related papers by the same group in Discussion sections when defensible; within-group citation raises co-retrieval in RAG.
- Refresh repository and model cards quarterly; quarterly-updated articles outperform single-publish ones in AI-overview retention (Conductor 2026 benchmark).
Section 7 — LLM-Citation Fabrication Defense
Between 50 % and 90 % of LLM answers to medical questions are not fully supported by the sources they cite (Wu et al., Nat Commun 2025, doi:10.1038/s41467-025-58551-6). Defend the paper's identifiers:
- Surface DOI and PMID in copy-friendly text at the top of the paper's landing page and README (for example,
DOI: 10.xxxx/yyyy • PMID: 12345678). - Add a "How to cite" section with BibTeX, APA, Vancouver, and the plain-text line in one place.
- Monitor incorrect citations: a Google Scholar alert for the paper's title variant, plus periodic Perplexity and ChatGPT web queries that record hallucinated bibliographic errors. Report discoverability (Perplexity / Elicit / Consensus retrieval) only from a recorded probe; never estimate it.
- When a reviewer cites an LLM-generated reference, verify the DOI and PMID yourself before accepting it.
Section 9 — Per-Project Application and Pipeline Integration
When invoked, run in this order:
- Read the target artifact (title, abstract, manuscript section, README, or card) and identify its lifecycle phase:
pre-draft/drafting/pre-submission/post-acceptance/post-publication. - Apply Sections 1–5 and 10 relevant to that artifact, filtering each rule by its
applies_to_phasefield inreferences/checklists/AIO_GENERAL.md. Out-of-phase rules become NA rather than FAIL (e.g., do not surface §11.5 multi-disciplinary roster or §12 launch sequencing as FAIL on apre-submissionaudit). Produce a PASS / PARTIAL / FAIL table, one-line reason and concrete fix per item, sorted byexpected_lift(high → medium → low), in the output template ofreferences/checklists/AIO_GENERAL.md. Render viatemplates/aio_audit_checklist.md.j2when programmatic. - Honour
defers_toannotations to avoid duplicate audits. Items annotated with adefers_tofield record only present/absent status here; item-level detail belongs to the linked skill or reference (§1.6 →/check-reporting; §3.4 / §11.3 →references/oac_funding_checklist.yaml). When the manuscript has not had a reporting-guideline audit, invoke/check-reportingfirst; the AIO ↔ guideline-item mapping is inreferences/reporting_guideline_mapping.md. - Apply the Section 6 author-authority audit once per submission cycle, and Sections 11–12 as the
applies_to_phasefilter allows. - Surface Section 7 citation-defense recommendations at
post-acceptancetime. For multi-repo or Hugging-Face-card team audits, runscripts/batch_metadata_audit.py. - Output:
- The phase-filtered checklist (visible).
- A short deferred-item list with one-line status per
defers_torule. - At most 5 concrete edits ranked by
expected_lift(high first, then medium, then low). Alow-lift edit enters the Top 5 only when nohigh/mediumitem remains open.
Integration with other skills
- Run after
/self-reviewand/humanize, so QC-confirmed claims and the final human-readable text anchor the checklist. - Within
/write-paper: apply Section 1 while drafting the title and abstract, Sections 2.5 and 6 (cross-linking) while drafting the Discussion, and the full checklist at QC, after the reporting-guideline check and numerical-claim audit.
Section 10 — Q&A and Entity-Extraction Optimization
10.1 Four-question Q&A block (Discussion or Appendix)
Add a labeled Q&A block, either as the closing subsection of Discussion or as a Supplementary Box:
- What was known before this study? — two-sentence restatement of the prior state.
- What does this study add? — two-sentence statement of the contribution.
- How might this change clinical practice or research? — one-sentence interpretation; avoid overclaim.
- Why does this matter? — one-sentence "so what" framing for non-specialists.
Lancet Digital Health "Research in context" already encodes the first two questions; the block extends them.
10.2 Glossary block with entity IDs
Define each domain-specific acronym inline on first use AND list them in a Glossary subsection at the end of Methods or in the Supplement, with the canonical entity ID where possible: MeSH term ID (clinical concepts), RadLex ID (radiology terms), UMLS CUI (cross-vocabulary mapping), Hugging Face model ID (named models), arXiv ID (cited methods).
10.3 Inline citation anchor text
Avoid bare reference numbers; bind each citation to its specific claim:
- WEAK: "Prior work [12] showed efficacy."
- STRONG: "Smith et al. (DOI: 10.xxxx/yyyy) reported a 12 % accuracy gain on the MIMIC-CXR test set [12]."
When citing one's own prior work, name the cohort or dataset explicitly to enable cross-paper retrieval.
10.4 Explicit challenge statement
Beyond the 2.5 limitations list, include a single-paragraph "Why this is hard" statement near the start of Discussion:
"Building accurate [task] for [modality/anatomy] is constrained by [data scarcity / label noise / dataset shift / regulatory uncertainty / interpretability]. Each of these has been documented [refs], and our results address [subset]."
LLM web-search systems quote challenge statements as authoritative summaries of field state (worked case: references/case_studies/kjr_mllm_2025.md).
Section 11 — First-Mover Timing and Citation-Graph Density
Rules 11.1 (topic-peak detection), 11.2 (editorial-board leverage), and 11.5 (multi-disciplinary author roster) are in ${CLAUDE_SKILL_DIR}/references/pre_draft_strategy.md; read it at pre-draft.
11.3 PMC-auto-deposit journal preference
OA journals that auto-deposit to PubMed Central reach LLM crawlers within 4–6 weeks of publication; non-PMC OA journals can take 3–6 months. When all else is equal, prefer a PMC-auto-deposit journal. Radiology/medical-AI examples (verify per submission; policies change):
- Korean Journal of Radiology (KJR) — auto-deposit confirmed.
- Lancet Digital Health — author-funded green OA, PMC-eligible after embargo.
- Radiology and Radiology: AI — selected articles auto-deposit.
- npj Digital Medicine — auto-deposit (Nature OA).
- JAMIA — author-funded OA route.
- JMIR — auto-deposit (PMC-indexed).
11.4 Citation-graph anchor strategy
Anchor the Discussion in 5–10 high-visibility prior works that LLM training corpora already index well, so the paper is retrieved when users query those works.
- Identify seminal references via Semantic Scholar's "Highly Influential Citations" filter for the topic.
- Cite them with semantic predicates (Section 10.3), not as bare lists.
- Mix recent preprints (currency) with 2018–2022 seminal papers (graph anchoring); a 2024–2025-only citation profile has low LLM retrieval weight because of corpus cutoffs.
Section 12 — Cross-Platform Launch Sequencing
At post-acceptance / post-publication, read ${CLAUDE_SKILL_DIR}/references/launch_sequencing.md for the Day 0 → Month 1 sequence (rules 12.1–12.5).
External References
- GEO: Generative Engine Optimization — Aggarwal et al., KDD 2024, arXiv:2311.09735.
- LLM medical citation support — Wu et al., Nat Commun 2025, doi:10.1038/s41467-025-58551-6.
- LLM citation bias — Algaba et al., 2024, arXiv:2405.15739.
- ExpertQA attribution — Malaviya et al., 2024, arXiv:2309.07852.
- TRIPOD+AI — Collins et al., BMJ 2024. EQUATOR Network.
- CLAIM 2024 — Tejani et al., Radiology: AI 2024, doi:10.1148/ryai.240300.
- STARD-AI — Sounderajah et al., Nat Med 2025, doi:10.1038/s41591-025-03953-8.
- TRIPOD-LLM — Gallifant et al., Nat Med 2024, doi:10.1038/s41591-024-03425-5.
- DECIDE-AI — Vasey et al., Nat Med 2022, doi:10.1038/s41591-022-01772-9.
- Title, abstract, keywords guide — Royal Society Proc B 2024, doi:10.1098/rspb.2024.1222.
- GitHub repository citation advantage — Kang et al., Inf Process Manag 2023, doi:10.1016/j.ipm.2023.103477.
- Semantic Scholar Open Data Platform — Kinney et al., arXiv:2301.10140.
Add Journal Skill
Generate two profiles per journal from its author guidelines: a detailed write-paper profile (~100-150 lines) and a compact find-journal profile (~30 lines). Write profile content in English; the library is shared.
Key Directories
Profiles live in one of two tiers. Pick the target in Phase 0, before any extraction.
- Public (shipped; must meet the verification bar in
${CLAUDE_SKILL_DIR}/../find-journal/POLICY.md):- write-paper:
${CLAUDE_SKILL_DIR}/../write-paper/references/journal_profiles/ - find-journal:
${CLAUDE_SKILL_DIR}/../find-journal/references/journal_profiles/
- write-paper:
- User-local private (per-user, never pushed to git):
- write-paper:
$HOME/.claude/private-journal-profiles/write-paper/ - find-journal:
$HOME/.claude/private-journal-profiles/find-journal/
- write-paper:
POLICY.md also documents promotion from private to public.
Phase 0: Input Collection
- Required: full official journal name; target tier (
publicorprivate). - Strongly encouraged: Author Guidelines URL. If the user gives only a name, ask for this URL before proceeding; it is the primary data source.
- Optional: field focus (used in 3.1), tier estimate (Q1/Q2/Q3), ISSN.
This skill does not judge whether a journal is legitimate. For an unfamiliar publisher, check DOAJ membership (for open-access titles) and indexing before writing a profile, and tell the user.
Public vs. Private target selection
If the user does not state the target tier, ASK before proceeding.
- Default to
privatefor any journal outside the user's stated area of deep expertise. Private profiles need not clear the public bar line by line, but the journal's own Author Guidelines page must still be the primary source: no inference from adjacent journals, no AI policy copied from the publisher family. publiconly when all of these hold:- The user intends to contribute the profile upstream.
- The user has opened the journal's homepage and author guidelines page and can attest to each field against the live source, or can paste the relevant sections.
- AI policy wording is transcribed from the journal's or publisher's own policy page, not inferred from a sibling journal.
- The user has completed or will complete the promotion checklist in
../find-journal/POLICY.mdbefore commit.
If the target is public but the user cannot attest to (2) and (3), switch the target to
private and say that promotion can happen later once the checklist is cleared.
Phase 1: Duplicate Check
Before any extraction:
- Glob all four directories:
${CLAUDE_SKILL_DIR}/../write-paper/references/journal_profiles/*.md ${CLAUDE_SKILL_DIR}/../find-journal/references/journal_profiles/*.md $HOME/.claude/private-journal-profiles/write-paper/*.md $HOME/.claude/private-journal-profiles/find-journal/*.md - Check for an exact filename match on the normalized name (4.1 rules).
- Grep existing profiles for the journal's common abbreviation (e.g., "JCO").
If found, report the paths. If present in only one directory, offer to create only the missing counterpart; if in both, offer to update or abort. Do NOT proceed to Phase 2 without user direction.
Phase 2: Data Extraction
- Path A (WebFetch available): fetch the Author Guidelines URL and parse it. If the page is login-gated, JavaScript-heavy, or sparse, fall back to Path B.
- Path B: ask the user to paste the Author Guidelines content (or key sections). Never fail silently when a fetch fails.
Extraction Checklist
Mark any field that cannot be determined as [TODO: verify at journal site].
For the write-paper profile: full name; abbreviation; publisher; ISSN (print/online); frequency; Impact Factor with its JCR year; OA model; acceptance rate; peer-review type; manuscript types with word, abstract, reference and figure limits; abstract format (structured or not, headings, word limit); required sections for an Original Article; statistical requirements (p-value format, CI, effect sizes); figure specs (DPI, format, color, max count); cover letter requirements.
For the find-journal profile: a 1-2 sentence scope paragraph; 15-20 comma-separated scope keywords; accepted article types; classification (tier Q1/Q2, OA, field).
--- Gate 1: Metadata Confirmation ---
Present the extracted metadata as a structured summary. Ask:
- "Is this information accurate? Any corrections?"
- "What are the common rejection reasons for this journal?" (if not extractable)
- "How would you position this journal relative to similar journals in the field?"
If the user's information conflicts with the source, present the conflict and ask. Do NOT proceed to Phase 3 until the user confirms or corrects.
Phase 3: Profile Generation
3.1 Load Reference Template
Read ONE existing write-paper profile from a similar field as a format reference:
| Field | Template to Load |
|---|---|
| Radiology | Radiology.md or European_Radiology.md |
| General medicine | The_BMJ.md or JAMA.md |
| Medical education | JMIR_Medical_Education.md |
| AI / digital health | npj_Digital_Medicine.md or JMIR.md |
| IR / interventional | CVIR.md or JVIR.md |
| Surgery | Annals_of_Internal_Medicine.md |
| Other | The_BMJ.md (safest general template) |
Also read ONE existing find-journal profile for the compact format.
3.2 Generate Write-Paper Profile
The detailed profile /write-paper loads when drafting for this journal. Read
${CLAUDE_SKILL_DIR}/references/write_paper_profile_template.md when writing it and fill the
literal template.
Follow the canonical 11-section order exactly. /write-paper Phase 7 and /find-journal read
these profiles positionally — a reordered, renamed, or omitted section silently degrades both:
- Journal Identity — full name, abbreviation, publisher, ISSN, frequency, impact factor, OA model, acceptance rate, peer-review type
- Manuscript Types and Word Limits
- Abstract Requirements — word cap and heading structure
- Required Sections (Original Article)
- Statistical Reporting — MANDATORY. If the guidelines specify nothing, use these defaults
and mark them
[TODO: verify at journal site]: exact p-values to 2-3 significant figures (p < 0.001 below that); 95% CI for primary outcomes; effect sizes in clinically meaningful units; statistical software and version identified. - Figures
- Common Rejection Reasons
- Cover Letter
- AI Writing Disclosure Policy — permitted scope, disclosure location, AI-generated-image stance, and the policy URL
- Author Guidelines URL
- Positioning — when to submit here, when not to, and a comparison table against 2-3 similar journals (society, scope emphasis, impact factor range, distinguishing features)
Fill rules. Never guess a value. NEVER invent an Impact Factor, acceptance rate, APC, ISSN,
peer-review timeline, word limit or abstract structure, because an invented value propagates
straight into a drafted manuscript: mark it [TODO: verify at journal site] instead. Use an
approximate range (e.g., "~15-20%") only when the source supports it. Where the journal publishes no
dedicated AI policy, cite the publisher-level policy and say so explicitly.
3.3 Generate Find-Journal Profile
Follow the canonical format exactly:
# {Full Name}
## Identity
- **Abbreviation:** {abbrev}
- **Publisher:** {publisher}
- **ISSN:** {print} / {online}
- **Homepage:** {URL}
- **Author guidelines:** {URL}
## Scope
{1-2 sentence scope description.}
## Scope Keywords
{comma-separated keywords, 15-20 terms}
## Article Types Accepted
- Original Article
- Review Article
- ...
## Classification
- **Tier:** {Q1/Q2}
- **Open Access:** {Full OA / Hybrid / Subscription}
- **Field:** {field}
## Special Notes
{2-3 sentences on positioning, unique aspects, society affiliation.} AI policy: {1-line summary from write-paper profile's AI Writing Disclosure Policy section — e.g., "language editing only, dual disclosure required, AI images banned." or "follows ICMJE — disclose AI use in Methods."}
--- Gate 2: Profile Review ---
Present BOTH complete draft profiles. Ask:
- "Review these profiles. Any corrections needed?"
- "Is the Scope paragraph accurate?"
- "Are the Common Rejection Reasons realistic?"
Do NOT write files until the user approves both profiles.
Phase 4: Write Files and Update Counts
4.1 Determine Filename
Spaces to underscores, special characters removed: "Journal of Clinical Oncology" ->
Journal_of_Clinical_Oncology.md; "JACC" -> JACC.md; "JAMA Network Open" ->
JAMA_Network_Open.md.
4.2 Write Profile Files
Write the write-paper profile and the find-journal profile, each as {filename}.md, into the
directory pair chosen in Phase 0 (Key Directories). For private, create the two private
directories if missing. Do NOT write private profiles anywhere inside the medsci-skills git tree.
4.3 Update Profile Counts
Do not edit SKILL.md, README.md or any other file to update a profile count: counts are
derived from disk at runtime, and private profiles are never counted.
4.4 Confirmation
Report the target tier and the paths written, then:
public: remind the user to commit the two profile files (staged by path) with a message recording which pages were opened and on what date, as POLICY.md requires.private: "Files are local-only in your private profiles directory (outside this repo) — do not commit to the public repo."
Batch Mode
For several journals in one session:
- Collect all journal names and URLs upfront.
- Run Phase 1 (duplicate check) for all journals first.
- Run Phases 2-3 per journal, with both gates.
- Write all files in a single Phase 4 pass.
Statistical Analysis Skill
Generate reproducible code (Python preferred, R when necessary) for medical research analyses, run it, and produce publication-ready tables and figures from its output.
Data Privacy Check
Before reading any data file, check whether it might contain Protected Health Information (PHI):
- If
*_deidentified.*files exist in the working directory, use those. - If only raw CSV/Excel files exist, ask the user whether the data contain patient identifiers
(names, national ID / RRN, contact details); if so, have them de-identify it first with
/deidentify. - Proceed once the user confirms the data are de-identified or contain no PHI.
- NEVER display raw PHI values (names, phone numbers, RRN) in your output, because console
output is copied into transcripts and drafts. If you encounter them, warn the user and suggest
/deidentify.
Reference Files
${CLAUDE_SKILL_DIR}/references/templates/— reusable analysis scripts; read the matching one before generating code.${CLAUDE_SKILL_DIR}/references/analysis_guides/— methodology guides; the Phase 2 table says which to read.${CLAUDE_SKILL_DIR}/references/table-standards/—table-standards.md(universal and AMA rules, footnote order, gtsummary pipeline),journal-profiles/(YAML per journal: radiology, jama, nejm, lancet, european_radiology, ajr),table-types/(one template per table type),tool-comparison.md(R/Python table tools).${CLAUDE_SKILL_DIR}/references/style/figure_style.mplstyle— figure style.
Workflow
Phase 1: Data Assessment
- Read the data file (CSV, Excel, TSV, or other tabular format).
- Report to the user: shape (rows x columns); column names and inferred types (continuous, categorical, ordinal, binary, datetime); missing values per column (count and percentage); first 5 rows; unique value counts for categorical columns.
- Identify the analysis unit: patient, exam, lesion, image, rater, study, etc.
- If a variable name or coding is uncertain, write
[VERIFY: variable_name]and ask the user to confirm it against the data dictionary; never guess a column name or coding.
Phase 2: Analysis Plan
Observational designs (cohort, case-control, cross-sectional, registry, survey): before
planning, look for a variable_operationalization.md from /define-variables (or an equivalent
codebook-backed definition table). If none exists, warn the user and recommend running
/define-variables first, because exposure/outcome/covariate definitions and cutoffs invented
ad hoc from the data dictionary are a common reason reviewers reject observational work. This is
a warning, not a block: proceed on explicit user confirmation and record that the artifact was
not available.
Propose an analysis plan; the user decides the research question and endpoints. Mark any
clinical definition, cutoff, diagnostic criterion or guideline claim you could not confirm
[VERIFY].
-
Detect the analysis type from the table in Analysis-Specific Guidelines, or accept the user's specification.
-
List the specific tests to be performed.
-
Identify primary and secondary endpoints.
-
State the assumptions that will be checked (normality, homogeneity, independence).
-
Note data cleaning needed (recoding, outlier handling, missing-data strategy).
-
Anchor the estimand to the research question.
- Interaction, synergy or effect modification: the primary estimand is the interaction
parameter itself (a likelihood-ratio test of the interaction term, or the interaction
OR/HR on one stated scale) — not a main-effect OR whose CI is then read as "no synergy". A
public-health or biological synergy claim is additive-scale: report RERI, AP or
S, each with a CI (R
interactionR/epiR; Knol & VanderWeele reporting), not only a multiplicative product term, because a non-significant multiplicative interaction is compatible with a large additive one (and vice versa). Joint categories (high/high vs low/low) show joint association, not interaction; "stronger in A than in B" from separate stratum estimates is the difference-in-significance fallacy — report the formal interaction term. - Equivalence or non-inferiority: declare the margin up front (a TOST procedure, or the CI compared against a pre-stated MCID); a non-significant difference is not equivalence without a margin.
- Interaction, synergy or effect modification: the primary estimand is the interaction
parameter itself (a likelihood-ratio test of the interaction term, or the interaction
OR/HR on one stated scale) — not a main-effect OR whose CI is then read as "no synergy". A
public-health or biological synergy claim is additive-scale: report RERI, AP or
S, each with a CI (R
-
Screen every categorical/binary predictor for separation before fitting anything. A predictor that perfectly predicts the outcome has no finite MLE, and the failure is silent:
glmreturns an odds ratio near 0 (or enormous), p ≈ 0.99, and an AUC that ends up in a table. Pathognomonic imaging signs (T2-FLAIR mismatch, the string sign, a halo sign) cause it routinely: 100% specificity means an empty cell by construction.python3 "${CLAUDE_SKILL_DIR}/scripts/check_separation.py" \ --data cohort.csv --outcome idh_mutant --auto --strictCOMPLETE_SEPARATION(an empty cell) andQUASI_SEPARATION(a cell below the sparsity floor) both halt the plan. With two or more predictors the script also tests them jointly (a linear combination can separate the outcome when no single predictor does); that test needs scipy, and without it the report says the joint check was not run. The remedy is a design decision, made in the plan: Firth's penalised likelihood keeps one model, while a two-stage rule — classify the sign-positive cases directly, model only the sign-negative remainder — is usually the clinically meaningful choice for a pathognomonic sign, because a sign-positive patient is already diagnosed.
Present the plan and wait for user approval before executing.
Phase 3: Execute
Generate and run a Python (preferred) or R script. Every number you report MUST come from its executed output, never estimated or hand-typed, because a number no script reproduces cannot be verified.
Script Structure
Start every script with a reproducibility header:
"""
Analysis: {description}
Date: {YYYY-MM-DD}
Random seed: 42
Python: {version}
Key packages: {package==version, ...}
"""
import numpy as np
import pandas as pd
np.random.seed(42)
Execution Rules
- Random seed: always
np.random.seed(42)orset.seed(42). - Figure style: always load the matplotlib style file:
import matplotlib.pyplot as plt style_path = os.path.join(os.environ.get('CLAUDE_SKILL_DIR', '.'), 'references/style/figure_style.mplstyle') if os.path.exists(style_path): plt.style.use(style_path) - Output files: save next to the input data, or in a user-specified output directory.
- Tables as CSV plus a printed markdown version (see Output Conventions); figures as PDF (vector) and PNG (300 DPI).
- Console output: a summary with every number that would appear in the text, formatted for direct copy-paste into the Results section.
Assumption Checking
Choose the summary and test from the design and the distribution, not a preliminary test:
- Shape: QQ plot / histogram and skewness; for Table 1,
|skewness| > 1→ median (IQR) with a rank test, else mean (SD) (table-types/table1_demographics.md). Never gate on a Shapiro-Wilk or KS P value: it flags trivial departures at large n, misses real ones at small n, and test-then-test inflates the type I error (Rochon et al. 2012). - Unequal variances: default to Welch's t / Welch ANOVA, not a Levene gate; Mann-Whitney tests a different hypothesis and does not fix them. State the choice and reason in Methods.
Stratified & Ordinal-Trend Reporting
- Strata disjointness gate (before any ordinal trend test). Before a Cochran-Armitage trend
test (or any analysis that treats tiers as an ordered partition), assert that the strata are
mutually exclusive and exhaustive:
sum(n per stratum) == unique Nandsum(events per stratum) == total events. A trend test on overlapping or non-exhaustive strata is invalid. Print the per-stratum N/event table and the reconciliation (the analysis-side mirror of/self-reviewcheck_cohort_arithmetic.pyPARTITION_OVERLAP). - Secondary stratum HR/OR: report each with (a) its reference contrast (the referent category), (b) the event count in each stratum, and (c) a sparse-stratum caveat when any stratum has few events (< 10 is unstable). "HR 1.55 in lean participants" without the referent and the events is uninterpretable.
- Proportion CI lower-bound clamp: clamp every lower bound to
max(0, lower). A zero-event Wilson/score interval can print a negative or tiny-exponent bound (e.g.3.47e-16) that is a display artifact; report0(or0.0%), and prefer an exact (Clopper-Pearson) interval for zero/near-zero cells.
Output Manifest
After all analyses complete, save _analysis_outputs.md in the output directory, in the format
given in references/analysis_run_workflow.md, so
/make-figures and /write-paper can find the outputs without asking.
For prespecified binary predictions on independent units, use the bundled
scripts/run_analysis.py run workflow in that file: it embeds data/configuration/code/output
hashes, exact counts, metric-specific denominators and the reproduction command in the same
manifest (audit checks recorded versions without rewriting them; compare separates declared
context and numeric equality from byte drift). It does not select thresholds or establish study
validity, privacy clearance or reuse rights. Synthetic example:
python3 ${CLAUDE_SKILL_DIR}/scripts/demo_analysis_run.py --out demo-project.
Phase 3.5: Generated-Code Quality Gate
Before reporting any script as final, lint every emitted .py/.R file:
python3 ${CLAUDE_SKILL_DIR}/scripts/check_generated_code.py {script.py} --strict
# or scan a whole output directory:
python3 ${CLAUDE_SKILL_DIR}/scripts/check_generated_code.py --code-dir {analysis_dir} --strict
Fix every Major before reporting the script: MISSING_SEED (randomness with no seed),
HARDCODED_DATA_LITERAL (hand-typed, table-shaped data instead of read_csv()/read.csv() +
subset), HARDCODED_ABS_PATH (non-portable and a PII risk), and INPLACE_SOURCE_OVERWRITE
(writing to the path read as input — never modify raw data; write derived outputs to a new
path). Fix the flags DEBUG_LEFTOVER and UNUSED_IMPORT when tidying.
Minor, Python only: API_DEFAULT_STUDENT_T (scipy's ttest_ind without equal_var is Student's
t; state Welch or the reason for Student) and API_DEFAULT_PENALIZED_OR (LogisticRegression
with none of penalty, C, l1_ratio, in a file that exponentiates a coef_). R's t.test()
is Welch by default.
Phase 4: Report
After execution, generate manuscript-ready text from the script's output:
- Results paragraph: 3-8 sentences with specific numbers, formatted as:
- Continuous: "mean +/- SD" or "median (IQR)"
- Proportions: "n/N (XX.X%)"
- Test results: "statistic = X.XX, p = 0.XXX"
- Effect sizes: "Cohen's d = X.XX (95% CI: X.XX-X.XX)"
- AUC: "AUC = 0.XXX (95% CI: 0.XXX-0.XXX)"
- Table/figure captions: draft captions referencing table/figure numbers.
- Methods snippet: 2-3 sentences describing the statistical methods, for the Methods
section. Cite only references whose DOI/PMID
/search-litconfirmed; mark any other[UNVERIFIED - NEEDS MANUAL CHECK].
Report statistical results; leave judgments of clinical significance to the user. For designs this skill does not cover (adaptive or Bayesian trials, complex multilevel or causal-mediation models), say so and recommend biostatistician review before the results are reported.
Statistical Reporting Rules (Always Enforced)
- Exact p-values: report exact values (e.g., p = 0.034), not inequalities; below 0.001, report p < 0.001.
- Confidence intervals: always report 95% CIs for primary endpoints.
- Effect sizes: report one alongside every p-value (Cohen's d, eta-squared, odds ratio, risk ratio, etc., as appropriate).
- Multiple comparisons: for 3+ tests on the same dataset, apply Bonferroni or Benjamini-Hochberg correction, state the method, and report both uncorrected and corrected p-values.
- Sample size: state n for each group/analysis.
- Missing data: report how many cases were excluded and why.
- Decimal places: p-values to 3 decimals, proportions to 1 decimal, means/SDs to the precision of the measurement.
- Design/power statistics are code outputs, never hand-computed. Any minimum detectable
effect (MDE), a-priori power, or required sample size that will appear in the
manuscript MUST be printed by the committed script with its method and inputs (n per arm,
alpha, power, allocation ratio, one/two-sided), not computed in a side tool (G*Power, an
online calculator) and pasted in, because a manuscript value no script reproduces cannot be
checked. Use one method family throughout (e.g. the exact noncentral-t via
statsmodelsTTestIndPowerorscipy'snct); do not mix a normal approximation for some values with exact-t for others. Do not report post-hoc (observed) power: it is a function of the P value and adds nothing (Hoenig & Heisey 2001); report the CI of the effect instead. - Estimand & CI output contract. Every primary point estimate — including quantile
estimands (T25, median time-to-event), pooled proportions and subdistribution HRs, not just
ORs/HRs/AUCs — MUST be emitted with its 95% CI, because
/self-reviewtreats a primary metric without one as a defect. In the output CSV carry the interval as explicit columns (estimate, ci_lower, ci_upper) or as one text column inest (lo–hi)form; never a point estimate with no interval beside it. Round ORs/HRs/sHRs to 2 decimals and AUC/C-statistic to 3.
Effect-Size Real-World Translation
Whenever a primary result is a correlation, a standardized coefficient, a regression slope, an OR/HR/RR or a Cohen's d, also report it as a plain-language unit shift a non-statistician can act on — in addition to the effect size (rule 3), not instead of it. This applies to continuous exposure–outcome associations (Spearman's rho, Pearson's r, standardized slopes), to relative measures where the audience needs an absolute-risk feel, and to reader or expert-elicitation studies, clinical-utility framing, abstracts and figure captions.
- Pick an anchored contrast on the exposure, not a 1-unit step. Default: 25th to 75th percentile (IQR), both endpoints in native units.
- Translate to the outcome scale.
- Regression slope b in native units:
delta_outcome = b * (x_p75 - x_p25). Keep the sign: a negative b is a decrease. - Per-SD slope (exposure standardized):
delta_outcome = beta * (x_p75 - x_p25) / SD_x; a fully standardized beta is also multiplied by SD_outcome. - A correlation is not a slope. Pearson r converts to the simple regression slope of the same
data (
b = r * SD_outcome / SD_x), which is the mean change per unit only if the relation is linear; Spearman's rho converts to no change in outcome units. Fit the outcome on the exposure and translate that slope (Theil-Sen if outliers are why Spearman was used); for a monotonic but curved association, report the fitted outcome at x_p25 and at x_p75 instead. - Report as: "going from {x_p25} to {x_p75} {units} is associated with about {delta_outcome} {outcome units} on average", and state the linearity assumption behind it.
- Regression slope b in native units:
- Bound the claim: report the contrast, the assumption and a CI on the coefficient; do not imply causation from a crude or unadjusted estimate.
Output contract (clinical utility is a default, not an optional add-on):
- OR/HR/RR primary outcomes → the relative measure and the absolute risk at a stated baseline, the absolute risk difference, and NNT = 1/ARR (or NNH = 1/ARI), with the baseline risk explicit. A relative-only headline is incomplete.
- Continuous outcomes → the IQR/clinically anchored "Real-world translation" line beneath the effect size.
- Prediction / classification (incl. medical-AI) models → calibration (Brier score,
calibration plot, or calibration slope/intercept) alongside discrimination, because AUC alone
is insufficient; a decision-curve / net-benefit pass at the relevant threshold is standard
output. An incremental claim reports the added-value test on the new term (likelihood
ratio / Wald in the nested model), ΔC-statistic with its CI and Δnet benefit over the
established clinical model, not the new model's AUC alone; NRI only as the categorical,
event/non-event-split version, and IDI with caution. See
references/table-standards/table-types/incremental_value.mdand themake-figuresdecision_curveexemplar (andrender_core_figures.pyfor the rendered curve).
Error Handling
- If a script fails, diagnose the likely cause (missing package, data format mismatch, wrong column name) and present a fix. Do not rerun the same script more than once without modifying it or asking the user.
- If an R package is unavailable, suggest
install.packages()and wait for user confirmation.
Output Conventions
Code, tables, figures and console output are in English.
Tables
Before generating any publication table, load:
${CLAUDE_SKILL_DIR}/references/table-standards/journal-profiles/{journal}.yamlif a target journal is known (it sets footnote markers, P and CI format, title and abbreviation order)${CLAUDE_SKILL_DIR}/references/table-standards/table-types/{type}.mdfor the table type- If no journal is specified, default to AMA style (Radiology profile)
Output formats (always generate all three): a CSV file (downstream use and archival), a
console markdown rendering (user review), and R gtsummary code (Word/LaTeX export). Read
${CLAUDE_SKILL_DIR}/references/table-standards/table-standards.md for the gtsummary pipeline
and the footnote placement order (general note, abbreviations, specific notes, probability
notes).
Validation checklist (run before finalizing any table):
- No vertical lines — horizontal rules only (top, below header, bottom)
- Binary variables show only one level (e.g., Male only)
- Units in headers, not cells; consistent decimal places per column
- Variability measure stated: mean (SD) or median (IQR)
- Statistical test named (footnote or general note); exact P values, never "NS"
- Effect sizes per clinically meaningful unit (per 10 years, not per 1 year)
- Reference category stated for categorical predictors
- Abbreviations defined in footnotes, self-contained per table
Figures
PDF (vector) + PNG (300 DPI), styled with figure_style.mplstyle (Arial, colorblind-safe
palette); width 3.5 inches (single column) or 7.0 inches (double column); axis labels with
units.
Analysis-Specific Guidelines
Before generating code for any row, read the files listed for it (paths relative to
${CLAUDE_SKILL_DIR}); they carry the method rules this file does not repeat.
| Analysis type | Read before generating code | Template |
|---|---|---|
| Table 1 (demographics) | references/table-standards/table-types/table1_demographics.md | references/templates/table1_demographics.py |
| Diagnostic accuracy (Se, Sp, PPV, NPV, AUC) | references/analysis_guides/diagnostic_accuracy.md | references/templates/diagnostic_accuracy.py |
| Added value beyond a baseline model (ΔAUC, NRI/IDI, net benefit) | references/table-standards/table-types/incremental_value.md | — |
| Reader study (MRMC) | references/table-standards/table-types/reader_study.md | — |
| Prediction model (outputs a risk used for a decision) | references/analysis_guides/calibration.md | — |
| Inter-rater agreement (kappa, ICC, Bland–Altman) | references/analysis_guides/agreement_reliability.md, references/table-standards/table-types/agreement.md | references/templates/agreement_analysis.py |
| Meta-analysis (pairwise, single-arm proportion, DTA) | references/analysis_guides/meta_analysis.md | references/templates/dta_meta_analysis.R (DTA) |
| Network meta-analysis (≥3 interventions) | references/analysis_guides/network_meta_analysis.md | — |
| Health economic evaluation (CEA, CUA, budget impact) | references/analysis_guides/health_economic_evaluation.md | — |
| Survey/Likert (ordinal rating scales) | the Survey/Likert section below | references/templates/likert_summary.py |
| Survival, competing risks, interval-censored events | references/analysis_guides/survival.md, references/table-standards/table-types/survival_results.md | references/templates/survival_analysis.py (--cluster <id> for nested units) |
| Group comparison, correlation | references/analysis_guides/test_selection.md | — |
| Logistic or linear regression | references/analysis_guides/regression.md | references/templates/regression.py (regression_type = "logistic" or "linear") |
| Propensity score (PSM, IPTW, SIPTW, overlap weighting) | references/analysis_guides/propensity_score.md | references/templates/propensity_score.py |
| Survey-weighted (KNHANES, NHANES, KCHS) | references/analysis_guides/survey_weighted.md | references/templates/survey_weighted_analysis.py |
| NHIS claims-based studies (ICD-10 definitions) | references/analysis_guides/nhis_icd10_mapping.md | — |
| Repeated measures (LMM, GEE, RM ANOVA) | references/analysis_guides/repeated_measures.md; for missing data see references/analysis_guides/missing_data.md (an LMM already uses every observed outcome under MAR, so missing outcomes alone do not call for MICE) | references/templates/repeated_measures.py |
| Mediation | references/analysis_guides/mediation.md | — |
| Multiple testing, many-exposure scans (ExWAS/EWAS) | references/analysis_guides/multiplicity.md | — |
| Mendelian randomization | references/analysis_guides/mendelian_randomization.md | — |
| Polygenic risk score | references/analysis_guides/polygenic_risk_score.md | — |
| Burden of disease, decomposition, forecasting | references/analysis_guides/burden_decomposition_forecasting.md | — |
Comparing models: use a paired DeLong test for two separate scores on the same patients. For a nested "baseline + new term" model fitted and evaluated on the same data, DeLong is not the added-value test — test the term (likelihood ratio) and report ΔAUC with its CI.
The rules below have no separate guide.
Survey/Likert
- Descriptives per item: median, IQR, frequency distribution. Group comparisons: Mann-Whitney or Kruskal-Wallis (ordinal data). Visualization: diverging stacked bar chart.
- Internal consistency: Cronbach's alpha with item-total correlations.
- Reverse-coding guard (run before reliability): recode every negatively worded item
(min+max) - xbefore computing the scale total or Cronbach's alpha. An un-recoded reverse item produces a negative item-rest correlation and can make alpha negative — a coding bug, not evidence of a multidimensional construct; do not defend it as one.likert_summary.pyprints per-item item-rest correlations, flags negative ones as reverse-code suspects, warns on a negative alpha, and applies the recode with--reverse-items E3 .... To screen at cleaning time, runpython3 "${CLAUDE_SKILL_DIR}/../clean-data/scripts/check_reverse_coding.py".
Covariate Pitfalls: Structural Zeros & Dose/Duration Variables
Applies to any multivariable adjustment (logistic / linear / Cox / propensity score / survey- weighted) with a dose/duration variable anchored to a categorical exposure (pack-years under smoking status, grams/week under alcohol use, cessation duration under former smoker):
- Structural-zero guard (do not impute): a never-smoker's
pack_yearsis a structural zero — known to be 0 by definition of the category, not missing. Imputing it (MICE/MNAR) fabricates a dose for unexposed subjects and corrupts the exposure contrast. Before imputing a dose/duration column, set the implied zero explicitly (IF status == 'never' THEN dose = 0) and impute only the genuinely missing residual among the exposed./clean-dataflagsneverrows with a NULL dose (scripts/check_structural_zero.py). - Complete-case collapse (adjust for status, not dose): a dose variable in a complete-case model drops the unexposed stratum wholesale when its zeros are stored as NULL, collapsing n (commonly 40–60%) and distorting subgroup estimates. Adjust for the categorical status (never/former/current); reserve the continuous dose for an exposed-only secondary analysis. Report n before and after model fitting and confirm the denominator did not silently collapse.
Covariate Selection: Over-adjustment in a Cross-Sectional Outcome Model
Applies to any cross-sectional / single-visit outcome regression (temporal order is not observed). Select covariates causally, not statistically:
- Do not adjust for a consequence or mediator of the outcome — that is over-adjustment / collider bias and removes part of the effect under study. Signature case: with eGFR as the outcome, serum uric acid is renally excreted (a lower eGFR raises urate), so it is an outcome-consequence, not a confounder; blood pressure and HbA1c are often similarly downstream. Classify each candidate covariate against a DAG as confounder / mediator / outcome-consequence / collider, and keep only confounders in the primary model.
- "It differs in Table 1" is not a confounder-selection criterion. Imbalance justifies considering a variable; a mediator or outcome-consequence stays out however imbalanced it is.
- Report the suspect-covariate sensitivity + VIF. Make a parsimonious, history-/design-based model primary; report the fuller model as a sensitivity analysis that drops (or adds) the suspect covariate and show whether the headline estimate moves. Print VIF and the n actually fitted. If dropping the covariate materially changes the estimate, carry that to the abstract and conclusion.
- Compare adjusted vs unadjusted on the SAME frame. When extended adjustment adds covariates with missingness, the analytic n shrinks (e.g. 84 → 49 events), and comparing that estimate with the full-frame unadjusted one confounds adjustment with who was dropped. Refit the unadjusted model on the reduced complete-case frame and report unadjusted and adjusted on that frame beside the full-frame estimate; never describe "adjustment changed the estimate" from a comparison across different frames (or use multiple imputation so all models share one frame).
/architecture-zoo was renamed to /model-selection in v6.0; this redirect is removed in v7.
Run /model-selection now, passing the same arguments the user gave /architecture-zoo.
Start at /model-selection Phases 1–3 (choose a paper-grounded architecture family and write the decision note).
/author-strategy — PubMed Author Strategy Analysis
Work from PubMed metadata and the title/abstract text already fetched — nothing else. Do not
retrieve full text, follow external links, or resolve preprints. Signals that need citations,
citation half-life, venue-impact tier, repository/preprint links, or corresponding-author role are
unavailable and surface as [VERIFY] — never inferred.
Prerequisites
- Python 3.10+ with
biopython,pandas,matplotlib,seaborn, andpyyaml(PyYAML is required by the archetype classifier and the rubric renderer) - Scripts:
${CLAUDE_SKILL_DIR}/fetch_pubmed.py,${CLAUDE_SKILL_DIR}/analyze_patterns.py,${CLAUDE_SKILL_DIR}/pubmed_parse.py(stdlib parser),${CLAUDE_SKILL_DIR}/classify_archetypes.py,${CLAUDE_SKILL_DIR}/render_archetype_doc.py - Rubric:
${CLAUDE_SKILL_DIR}/references/trajectory_archetypes.yaml(canonical) and${CLAUDE_SKILL_DIR}/references/trajectory_archetypes.md(generated)
Workflow
Step 1: Gather Input
Ask the user for:
- Author name (PubMed format, e.g., "Kim DK" or "Lee KS")
- Last name for position classification (auto-detected if ambiguous)
- Output directory (default:
~/.local/cache/author-strategy/{AuthorName}/) - Email for NCBI E-utilities (passed as
--email)
Step 2: Fetch PubMed Data
python "${CLAUDE_SKILL_DIR}/fetch_pubmed.py" "{Author Name}" \
--last-name "{LastName}" \
--output "{output_dir}/data/{name}_publications.csv" \
--email "{user_email}"
Review the console summary (total count, study type distribution, author position). If count is 0, suggest alternative name formats (e.g., "Kim DK" vs "Kim D" vs the full first name) rather than generating data.
Step 3: Generate Visualizations and Report
python "${CLAUDE_SKILL_DIR}/analyze_patterns.py" "{output_dir}/data/{name}_publications.csv" \
--output-dir "{output_dir}/report/" \
--author-name "{Author Name}"
This produces 7 PNG charts (01-07) and analysis_report.md with the strategy breakdown.
Study types come from the keyword rules in pubmed_parse.py, first match in this order: GBD,
SR/MA, NHIS/Claims, Cross-national, National survey, Biobank, AI/ML, Clinical trial, Case report,
Letter/Commentary; anything else is "Other". Present the script's labels as they are — never
reclassify a paper by guess.
Step 4: Interpret and Present
Read analysis_report.md and present to the user; every count and rate comes from that
report or the fetched CSV, never from memory:
- Executive summary: total publications, growth trajectory, most frequent journals (venue-impact tier is
unavailable— never present a "high-tier" rate) - Primary strategy: what study type dominates and why
- Author position analysis: first/last positional rate vs middle (positional heuristic only — not leadership or corresponding-author metadata, which are unavailable here)
- Topic clusters: research focus areas
- ROI quadrant: which study types combine volume with first/last positional rate (chart 07; no venue-tier axis)
- Replication opportunities: which patterns are replicable with Claude Code + public databases
State the classifier's limits when they matter: it is tuned for Korean epidemiology and public health researchers and may undercount specialized study types in other fields, and NHIS studies that lack its keywords fall into "Other".
Known limits: a supplied --orcid or --initials that contradicts every same-surname author on
a paper yields match_basis orcid-conflict / initials-conflict and position unknown (a
namesake is never attributed). The journal_tier CSV column always reads unavailable [VERIFY].
The A3 reporting-quality term list no longer contains claim (it matched "claims database"), so
papers naming only the CLAIM checklist do not count toward dual_mode_corpus.
Step 5: Optional — MA Gap Identification
If the user asks "what MA topics are feasible with this professor?":
- Cross-reference topic clusters with the user's existing MA plans
- Identify gaps where the professor has domain expertise but no MA published
- Output a prioritized list of MA proposals
Optional: Trajectory-Archetype Classification
An opt-in path that classifies the trajectory into abstract career archetypes (A1–A6 + a
composite) as an explainable, multi-label, confidence-scored heuristic — not an objective
verdict, using the canonical rubric references/trajectory_archetypes.yaml.
Step 6: Disambiguation Gate (required before classification)
A surname alone never resolves an author. Pass disambiguators so the target author is uniquely attributed:
python "${CLAUDE_SKILL_DIR}/fetch_pubmed.py" "{Author Name}" \
--initials "{Initials}" --orcid "{ORCID}" \
--affiliation "{Institution}" --year-from "{YYYY}" --year-to "{YYYY}" \
--output "{output_dir}/data/{name}_publications.csv" --email "{user_email}"
This writes the CSV, a candidates.json of affiliation/year candidate clusters, and a
corpus_manifest.json with review_status: pending. Present the candidate clusters to
the user for review. The user decides include/exclude. Only after the user has reviewed
the clusters do you finalize and approve the corpus (the --approve flag is a human gate
— never set it without explicit user review/approval):
python "${CLAUDE_SKILL_DIR}/fetch_pubmed.py" "{Author Name}" \
--initials "{Initials}" --affiliation "{Institution}" \
--include-pmids "{included.txt}" --exclude-pmids "{excluded.txt}" --approve \
--output "{output_dir}/data/{name}_publications.csv" --email "{user_email}"
The manifest is cryptographically bound to the CSV (csv_sha256 + pmid_set_hash); the
classifier refuses to run on an unapproved or mismatched corpus.
Step 7: Run the Classifier and Present
python "${CLAUDE_SKILL_DIR}/classify_archetypes.py" \
"{output_dir}/data/{name}_publications.csv" \
--manifest "{output_dir}/data/corpus_manifest.json" \
--rubric "${CLAUDE_SKILL_DIR}/references/trajectory_archetypes.yaml" \
--output-dir "{output_dir}/report/"
Read archetype_report.md and present it to the user, stating up front that the labels
are explainable heuristics, not objective classifications. For each surfaced archetype,
show the score, confidence band, and the author's own evidence PMIDs. Honor the [VERIFY]
markers (h-index/citation/venue-tier are unavailable) and the A5 participation flag. List
the insufficient evidence archetypes too — below the minimum sample or with conflicting
signals, never force a label.
To retune the rubric, edit only the YAML and regenerate the narrative doc:
python "${CLAUDE_SKILL_DIR}/render_archetype_doc.py" # regenerate the .md
python "${CLAUDE_SKILL_DIR}/render_archetype_doc.py" --check # CI/test sync gate
Output Structure
{output_dir}/
data/
{name}_publications.csv
candidates.json # disambiguation candidate clusters (Step 6)
corpus_manifest.json # review_status + csv_sha256 + pmid_set_hash (Step 6)
report/
analysis_report.md
01_yearly_stacked.png
02_study_type_pie.png
03_author_position.png
04_journal_heatmap.png
05_topic_distribution.png
06_growth_curve.png
07_strategy_roi.png
archetype_report.md # trajectory-archetype classification (Step 7)
archetype_results.json # machine-readable labels + scores + evidence
Batch Cohort Analysis Skill
Generate one analysis script per exposure/outcome combination from a single validated methodology template; only the variables change between scripts.
Inputs
- Database path(s): CSV/SAS data files (KNHANES, NHANES, NHIS, or any cleaned cohort)
- Methodology template: One of:
- Path to a validated R/Python analysis script (from /replicate-study or /cross-national)
- A paper type template name:
nhis_cohort,cross_national,survey_weighted - A source paper to extract methodology from (falls back to /replicate-study Phase 1)
- Combination spec: A list of exposure/outcome pairs, provided as:
- Inline list:
exposures: [depression, obesity, smoking]; outcomes: [diabetes, hypertension, CVD] - CSV file with columns:
exposure,outcome, (optional)subgroup_vars "all"keyword: generates all pairwise combinations from the lists
- Inline list:
Optional Inputs
- Covariate set: Fixed covariate list for all analyses (default: use template's set)
- Subgroup variables: Variables to stratify by (default: sex, age group)
- Output format:
code_only(just scripts) |execute(run + collect results) |full(code + results + summary) - Cross-national mode:
cross_national: truegenerates paired scripts for both countries per combination (see Cross-National Batch Mode)
Example:
/batch-cohort
DB Korea: /path/to/knhanes/HN18.csv
DB US: /path/to/nhanes/
Template: cross_national
Exposures: [depression, obesity, smoking]
Outcomes: [diabetes, hypertension, metabolic_syndrome]
cross_national: true
Mode: execute
Workflow
Phase 1: Template Validation
- Read the methodology template (R script or paper type reference).
- Identify the slot variables — parts that change per combination:
EXPOSURE_VAR(raw variable name in the database),EXPOSURE_LABEL(label for tables/figures),EXPOSURE_CODING(derivation of the binary/categorical exposure), andOUTCOME_VAR,OUTCOME_LABEL,OUTCOME_CODINGlikewise. - Confirm the survey design: weighted analysis is mandatory for KNHANES/NHANES and is inherited from the template; NHIS and other claims databases have no sampling weights (standard regression).
- Verify the template runs successfully on at least one combination before batch generation.
- Output: template summary with identified slots → user approval.
Phase 2: Variable Specification
For each exposure and outcome in the combination spec:
- Look up the variable in the database: KNHANES — the name exists in the CSV header; NHANES —
which table contains it (codebook.csv if available); NHIS — claims code or variable name.
For ICD-10 claims algorithms read
${CLAUDE_SKILL_DIR}/../analyze-stats/references/analysis_guides/nhis_icd10_mapping.md; for survey variable coding,survey_weighted.mdin the same folder. If a mapping is uncertain, write[VERIFY: variable_name]and ask the user to confirm it against the data dictionary; never guess a variable name, column name or coding. - Define coding: binary as a threshold or category mapping (e.g.,
HE_glu >= 126 → diabetes = 1); categorical as level definitions (e.g.,smoking: current/former/never). Outcome definitions MUST include physician diagnosis, because lab-only definitions systematically overestimate exposure→outcome associations: Diabetes = FPG≥126 OR HbA1c≥6.5 OR physician-diagnosed (KNHANES: DE1_dg=1, NHANES: DIQ010="Yes"); Hypertension = SBP≥140 OR DBP≥90 OR physician-diagnosed (KNHANES: DI1_dg=1, NHANES: BPQ020="Yes"). - Set covariates: the full 8-covariate set (age, sex, education, income, smoking, alcohol, obesity, CVD) is the default unless explicitly justified — minimal models (age+sex+BMI only) leave residual confounding, which can bias effects in either direction. Remove self-adjustment: if the exposure is (or derives from) a covariate, drop that covariate (exposure = BMI → drop obesity; exposure = education/income → drop the same variable); if outcome = MetS, consider dropping obesity. Document every removal in the matrix Notes.
- Output: combination matrix (
combination_matrix.csv) with all variable specifications.
| # | Exposure | Exposure Coding | Outcome | Outcome Coding | Covariates (adjusted) | Notes |
|---|----------|-----------------|---------|----------------|----------------------|-------|
| 1 | Depression (PHQ≥10) | BP_PHQ sum ≥10 | Diabetes | HE_glu≥126|HbA1c≥6.5|DE1_dg=1 | age,sex,edu,income,smoking,alcohol,obesity,CVD | — |
| 2 | Obesity (BMI≥25) | HE_obe ≥4 | Diabetes | same | age,sex,edu,income,smoking,alcohol,depression,CVD | obesity removed from covariates |
Phase 3: Batch Code Generation
Never modify the core methodology across combinations — only swap exposure/outcome/covariates. For each combination in the matrix:
- Clone the template script.
- Replace slot variables with the combination-specific values.
- Apply the combination's adjusted covariate set from the matrix.
- Set output paths: each combination gets its own results subdirectory.
- Generate a master runner script (
run_all.Rorrun_all.sh) that executes all N scripts sequentially (or in parallel viafuture/parallel), captures errors per script without stopping the batch, and logs execution time per analysis. - Generated-code quality gate: one reproducibility slip (a missing seed, an absolute path, a
hand-typed data literal) replicates across the whole batch, so lint the generated scripts with
the
/analyze-statscode-quality gate (python3 ${CLAUDE_SKILL_DIR}/../analyze-stats/scripts/check_generated_code.py --code-dir {batch_dir} --strict) and clear every Major (MISSING_SEED,HARDCODED_DATA_LITERAL,HARDCODED_ABS_PATH,INPLACE_SOURCE_OVERWRITE) before batch execution.
Phase 4: Batch Execution (if execute or full mode)
- Event count check: before running, verify ≥10 outcome events per model parameter (count the exposure and every dummy/spline term; use min(events, non-events)), in the full sample and in each subgroup that is modelled. Flag underpowered combinations.
- Run the master script.
- Collect results from each combination's output directory.
- Log which combinations failed and why (common: convergence issues, too few events, empty
subgroups) in
failed_runs.csv, and suggest fixes.
Phase 5: Summary Matrix
Aggregate all results into a single summary. Every number comes from a combination's executed results files; never fill a cell by hand.
Main Results Matrix (summary_matrix.csv):
| Exposure | Outcome | N | Events | Model 1 OR (95% CI) | Model 2 OR (95% CI) | Model 3 OR (95% CI) | p-value | Significant |
|---|
Subgroup Summary (subgroup_matrix.csv): Same format, stratified by subgroup variables.
Heatmap (optional): Visual matrix of effect sizes × significance, exposure on Y-axis, outcome on X-axis.
- Multiple comparisons: whenever more than one combination is tested, include a Bonferroni- (or Holm-) corrected significance column in the summary matrix, with m = the number of tests in the reported family (combinations × models × subgroups), state m, and add a note on exploratory vs confirmatory framing.
- No p-hacking framing: the summary matrix is for hypothesis generation, not confirmation.
State this explicitly in README and any manuscript output. Any citation there needs a
/search-lit-confirmed DOI/PMID; otherwise mark it[UNVERIFIED - NEEDS MANUAL CHECK].
Output Files
{working_dir}/batch_{timestamp}/
├── README.md — Batch run summary (N combinations, template used, date)
├── combination_matrix.csv — All exposure/outcome specs with coding
├── template/
│ └── base_template.R — The validated template (frozen copy)
├── scripts/
│ ├── 01_depression_diabetes.R
│ ├── 02_obesity_diabetes.R
│ ├── ...
│ └── run_all.R — Master execution script
├── results/
│ ├── 01_depression_diabetes/
│ │ ├── table1.csv
│ │ ├── main_results.csv
│ │ └── subgroup_results.csv
│ └── ...
├── summary/
│ ├── summary_matrix.csv — Main results across all combinations
│ ├── subgroup_matrix.csv — Subgroup results across all combinations
│ ├── failed_runs.csv — Combinations that failed + error messages
│ └── heatmap.png — Optional effect size × significance visual
└── logs/
└── batch_execution.log — Timing + error log
Reproducibility: freeze the template version (template/) and include a SHA256 hash of the
data file in README.
Cross-National Batch Mode
When cross_national: true:
- Generate paired scripts for each combination (Korea + US), using /cross-national's dual-survey-design approach
- Summary matrix includes both countries side-by-side, with a ratio-of-ORs column (95% CI; P) computed
as in
/cross-nationalPhase 4, not a direction-agreement tick
Calc-Sample-Size Skill
Decision Tree
Walk the user through this tree one question at a time; do not assume answers. Reader studies, segmentation and model-comparison designs sit outside the tree: see Tests 14–17.
What is your primary outcome?
|
+-- Binary (yes/no, positive/negative)
| |
| +-- Paired data (same subjects, two methods)?
| | +-- YES --> [5] McNemar test
| | +-- NO --> How many groups?
| | +-- 2 groups, superiority --> [4] Two-proportion comparison (chi-square)
| | +-- 2 groups, non-inferiority --> [10] Non-inferiority / equivalence
| | +-- Multivariable model --> single-predictor hypothesis test? --> [9] Logistic regression
| | --> clinical prediction / AI model for use?
| | +-- developing the model --> [12] Prediction-model development (Riley)
| | +-- externally validating --> [13] External-validation (Riley)
| |
+-- Continuous (measurement, score)
| |
| +-- How many groups?
| +-- 2 groups --> [6] Independent t-test
| +-- 3+ groups --> [8] One-way ANOVA
|
+-- Time-to-event (survival, recurrence)
| |
| +-- Two groups, unadjusted --> [7] Log-rank test
| +-- Multivariable / adjusted HR --> [7] Log-rank (Schoenfeld) + [11] Cox EPV
|
+-- Agreement (inter-rater, reproducibility)
| |
| +-- Continuous measurements --> [2] ICC
| +-- Categorical ratings --> [3] Kappa
|
+-- Diagnostic accuracy (Se, Sp, AUC precision)
|
+--> [1] Diagnostic accuracy (precision-based)
Tests 1–11
Once the test is chosen, read ${CLAUDE_SKILL_DIR}/references/formulas.md § Test N — the
parameter table with defaults, the effect-size interpretation, the formula, R/Python code and the
methodological reference.
| # | Test | Use when |
|---|---|---|
| 1 | Diagnostic accuracy — Se/Sp precision | desired 95% CI half-width for sensitivity or specificity |
| 2 | ICC agreement (Walter 1998 test; Bonett 2002 CI width) | inter-/intra-rater agreement on continuous measurements (tumor size, angle) |
| 3 | Kappa agreement (Donner & Eliasziw 1992; needs the trait prevalence) | agreement on categorical ratings (BI-RADS category, lesion present/absent) |
| 4 | Two-proportion comparison (chi-square) | two independent groups (AI vs conventional detection rate) |
| 5 | McNemar (paired proportions) | paired binary outcomes (two readers on the same cases, before/after) |
| 6 | Independent t-test | means in two independent groups (lesion size, malignant vs benign) |
| 7 | Survival / log-rank (Schoenfeld events, then patients) | time-to-event between two groups |
| 8 | One-way ANOVA | means across 3+ independent groups |
| 9 | Logistic regression (Peduzzi EPV + Hsieh 1998, continuous or binary predictor) | multivariable binary outcome, single-predictor hypothesis test |
| 10 | Non-inferiority / equivalence | new method not worse than standard by more than a pre-specified margin, or equivalent within it |
| 11 | Cox regression EPV | multivariable Cox model — enough events for stable estimates |
- Test 9: Peduzzi EPV ≥ 10 is a minimum baseline for a single-predictor hypothesis test only. For a clinical prediction / medical-AI model intended for use, EPV-10 is outdated and reviewer-vulnerable — use Test 12 (development) / Test 13 (validation). Always report both Peduzzi and Hsieh and recommend the larger N.
- Test 10: NI alpha is one-sided (typically 0.025); orient the difference so > 0 favours the new
method. Equivalence is TOST, powered jointly. The margin must be clinically justified (see
formulas.md§ Margin selection). - Test 11: EPV ensures model stability, not power for a specific HR. If an HR is available, also run Test 7 (Schoenfeld) and recommend the larger N.
Tests 12–17 (specialised designs)
Each has its own reference file (parameters, method, reporting); read it once the test is chosen.
- Test 12 — Prediction-model development (Riley). Developing a clinical prediction /
classification model (including a medical-AI model evaluated as one) for use. EPV-10 does not
apply: N is the largest satisfying all Riley criteria — three for a binary or time-to-event
outcome, four for a continuous one (R
pmsampsize). - Test 13 — External validation (Riley). Validating an existing prediction/AI model: size for the
CI width of the C-statistic, calibration slope, O:E and (if claimed) net benefit (R
pmvalsampsize); ≥ 100 events and ≥ 100 non-events is only a floor. For Tests 12–13 read${CLAUDE_SKILL_DIR}/references/prediction_model_sample_size.md. - Test 14 — MRMC reader study (Obuchowski–Rockette). Readers with vs without the AI, or AI
non-inferior to readers. Test 1 under-sizes it because readers are a random effect: size readers
J× cases from pilot/literature variance components withRJafroc/MRMCaov/iMRMC(do not hand-roll the OR algebra) and report theJ × Npower grid. Read${CLAUDE_SKILL_DIR}/references/mrmc_reader_study_sample_size.md. - Test 15 — Segmentation-metric precision (Dice / HD95 / NSD). The outcome is a per-case score,
not a proportion:
n ≈ (1.96·SD/δ)²from the pilot SD of per-case Dice, sized on the worst structure; CI by patient-level bootstrap (BCa). This is precision; a comparison is Test 16. Read${CLAUDE_SKILL_DIR}/references/segmentation_metric_sample_size.md. - Test 16 — Between-model comparison. Model A beats B, C, …: power the paired per-case
difference,
n = ((z₁₋α/₂ + z₁₋β)·SD_Δ/Δ)²(a CI sized to just exclude zero has ~50% power), not each model's precision; for > 2 models pre-specify one primary contrast or pay the family-wise correction; a ranking claim needs multiple seeds. Read${CLAUDE_SKILL_DIR}/references/multi_model_comparison_sample_size.md. - Test 17 — Segmentation usability. Clinicians can use it (acceptability rate, catastrophic-failure
bound, edit time): acceptability is a proportion sized per structure class;
DE ≈ 1 + (m−1)ρholds only when each case has its own readers — the same readers on every case (crossed) add a reader term more cases cannot shrink; bounding failures at ≤ 1% needs ~300 clean cases (rule of three). Read${CLAUDE_SKILL_DIR}/references/segmentation_acceptability_sample_size.md.
Out of Scope
Do not compute adaptive trials (group-sequential, sample size re-estimation), cluster-randomized trials (design effect, ICC-based inflation), Bayesian sample size determination, crossover designs, or multi-endpoint correction (mention Bonferroni if asked, but do not compute corrected sample sizes). Say the design is beyond this skill and point to G*Power (free, https://www.psychologie.hhu.de/gpower), PASS, or a biostatistician.
Workflow
Phase 1: Understand the Study
- Ask the user to describe the study briefly (design, primary outcome, groups).
- Walk the decision tree to a test, and confirm it with the user before proceeding.
Phase 2: Collect Parameters
- Present the selected test's parameter table.
- For each parameter without a user-provided value, explain it and offer the default.
- Estimate effect sizes from prior literature (ask for the references) or pilot data; use Cohen's conventions only as a last resort, noting that convention-based estimates are less precise.
Phase 2b: Retrospective Studies
When the dataset already exists, formal power analysis is often impractical. Offer:
- Fixed extract: read
${CLAUDE_SKILL_DIR}/references/observational_cohort.mdand report event budget / confidence-interval precision instead of forcing a prospective recruitment-style power calculation. - Experience-based justification (acceptable for IRB and many journals):
- Institution volume:
total exams in period × prevalence × (1 − exclusion rate) = expected N. Ask for the annual exam volume for the modality, study period, prevalence and exclusion rate. This gives a realistic upper bound for N. - Prior studies: report the N of 3–5 comparable published studies and cite them; the user's N should be in the same range or larger.
- IRB templates for both:
${CLAUDE_SKILL_DIR}/references/justification_examples.md§ Retrospective.
- Institution volume:
Use a formal calculation (Phase 3) even for a retrospective study when a subset is enrolled prospectively, the primary analysis tests a hypothesis (not just estimation), the journal's Instructions for Authors require a power analysis, or the IRB requires it.
Phase 3: Calculate and Report
- Generate R code (primary) and Python code (alternative) from the reference formula.
- Run the R code via Bash; the reported N is the number it prints, not a hand calculation.
- Present the result in the Output Format below. Cite methodological sources only from
formulas.mdor the test's reference file; any other reference needs a DOI/PMID confirmed via/search-lit, otherwise mark it[UNVERIFIED - NEEDS MANUAL CHECK]. Mark an effect size, clinical definition or threshold you could not confirm[VERIFY]. - In a project, save the IRB text as
protocol/sample_size_justification.mdand the scripts asprotocol/sample_size_calc.R/.py:/write-protocoland/write-paperembed that text verbatim, so the numbers are never retyped.
Phase 4: Sensitivity Analysis (Optional)
If a parameter is uncertain or the effect-size estimate is vague, flag it and offer a table of N across plausible values (e.g., varying effect size, or power from 0.80 to 0.90).
Output Format
Always structure the final output as follows:
## Sample Size Calculation Report
### Study Design
[1-2 sentence summary of the design and test selected]
### Parameters
| Parameter | Value | Source |
|-----------|-------|--------|
| ... | ... | user / literature / convention |
### Result
- **Required sample size**: N = [value]
- **With [X]% attrition adjustment**: N_adj = [value]
### R Code (Reproducible)
```r
# [complete, self-contained R script]
# Dependencies: [list packages]
# Run: Rscript sample_size_calc.R
Python Code (Alternative)
# [complete, self-contained Python script]
# Dependencies: [list packages]
# Run: python sample_size_calc.py
IRB Justification Text
A sample of [N] participants is required to detect [effect description] with [power]% power at a [one/two]-sided significance level of [alpha], assuming [key assumptions]. Accounting for an estimated [X]% attrition rate, we plan to enroll [N_adj] participants. This calculation is based on [formula/method reference].
Effect Size Interpretation
[Cohen's benchmark classification + clinical meaning in the context of this study]
The IRB text must state N; name the test and its formula source; give every assumed parameter
(effect size, alpha, power); state the attrition adjustment and final enrollment target; cite the
methodological reference (e.g., "Schoenfeld, 1981"); and use formal, third-person language. Read
`${CLAUDE_SKILL_DIR}/references/justification_examples.md` for per-design exemplars when writing it.
Check-Reporting Skill
Reference Files
Checklists are vendored under ${CLAUDE_SKILL_DIR}/references/checklists/, one file per
instrument. Each file's header gives its version, source citation and licence, and says when its
item text is an own-words summary rather than the published wording.
STROBE.md-- STROBESTROBE_MR.md-- STROBE-MR 2021RECORD.md-- RECORD 2015 (RECORD-PE for drug studies)REMARK.md-- REMARKTARGET.md-- TARGET 2025GATHER.md-- GATHER 2016CHEERS_2022.md-- CHEERS 2022CROSS.md-- CROSS 2021 + CHERRIES (internet surveys)SRQR.md-- SRQR 2014 (all qualitative approaches)COREQ.md-- COREQ 2007 (interviews and focus groups)STARD.md-- STARD 2015STARD_AI.md-- STARD-AI 2025TRIPOD.md-- TRIPOD 2015TRIPOD_AI.md-- TRIPOD+AI 2024TRIPOD_LLM.md-- TRIPOD-LLM 2025PGS_RS.md-- PGS-RS / PRS-RS 2021CONSORT.md-- CONSORT 2025CONSORT_AI.md-- CONSORT-AI 2020SPIRIT.md-- SPIRIT 2025SPIRIT_AI.md-- SPIRIT-AI 2020CLAIM_2024.md-- CLAIM 2024DECIDE_AI.md-- DECIDE-AI 2022MI_CLEAR_LLM.md-- MI-CLEAR-LLMCLEAR.md-- CLEAR (radiomics)ARRIVE_2.md-- ARRIVE 2.0CARE.md-- CARE 2013SQUIRE_2.md-- SQUIRE 2.0GRRAS.md-- GRRASPRISMA_2020.md-- PRISMA 2020PRISMA_2020_Abstracts.md-- PRISMA 2020 for Abstracts, 12 items. A separate instrument, not a subset of the 27-item checklist (main item 2 defers to it): score it with its own denominator.PRISMA_DTA.md-- PRISMA-DTAPRISMA_P.md-- PRISMA-PPRISMA_ScR.md-- PRISMA-ScRMOOSE.md-- MOOSESWiM.md-- SWiMAMSTAR2.md-- AMSTAR 2QUADAS3.md-- QUADAS-3QUADAS2.md-- QUADAS-2QUADAS_C.md-- QUADAS-CRoB2.md-- RoB 2ROBINS_I.md-- ROBINS-IROBINS_E.md-- ROBINS-EROBIS.md-- ROBISROB_ME.md-- ROB-MERoB_NMA.md-- RoB NMAPROBAST.md-- PROBASTPROBAST_AI.md-- PROBAST+AINOS.md-- NOSCOSMIN_RoB.md-- COSMIN RoB
Workflow
Step 0: Existing-checklist staleness pre-check
If a checklist already exists for this project (qc/reporting_checklist.json or a prior .md
report), verify it targets the current manuscript before reusing it — one generated against an
older version carries stale section/line references and a stale version label:
python3 "${CLAUDE_SKILL_DIR}/scripts/check_checklist_version.py" \
--checklist qc/reporting_checklist.json --manuscript manuscript_v8.md
A non-zero exit means the checklist is stale (older target_version, changed source_sha256,
different target_manuscript) or pre-dates the version contract: regenerate it against the current
manuscript (Steps 1–5) rather than reusing it. Every report you generate carries the
target_manuscript / target_version / source_sha256 fields (Part A header + Part D JSON) so
this check works next round.
Step 1: Select Guideline
Use the guideline the user names; otherwise auto-detect it from the study type. If the study type is ambiguous, ask the user to confirm before selecting a guideline.
Auto-detection mapping:
| Study Type | Primary Guideline | AI Extension |
|---|---|---|
| Observational study | STROBE | -- |
| Mendelian randomization study | STROBE-MR (base STROBE + MR extension) | -- |
| Health economic evaluation (cost-effectiveness / cost-utility / cost-benefit / budget-impact) | CHEERS 2022 | -- |
| Observational study using routinely-collected data (claims / EHR / registry / health-checkup DB) | RECORD (base STROBE + RECORD extension; RECORD-PE for drug studies) | -- |
| Survey / questionnaire study (KAP, physician/patient, cross-sectional, e-survey) | CROSS (+ CHERRIES for internet surveys) | -- |
| Scoping review (maps breadth/nature of evidence, clarifies concepts, identifies gaps — not a focused effectiveness/accuracy question) | PRISMA-ScR (base PRISMA + scoping-review extension) | -- |
| Qualitative study (interviews, focus groups, ethnography, grounded theory, phenomenology, document analysis) | SRQR (all qualitative approaches); COREQ (interviews/focus groups specifically) | -- |
| Randomized controlled trial | CONSORT 2025 | CONSORT-AI |
| Diagnostic accuracy study | STARD 2015 | STARD-AI |
| Prediction model (development/validation) | TRIPOD | TRIPOD+AI |
| Polygenic (risk) score prediction study | PGS-RS (with TRIPOD / TRIPOD+AI) | -- |
| Prognostic tumor-marker / biomarker study (single or multiple markers; e.g., ctDNA / molecular residual disease) | REMARK (pair with STROBE for the observational-design items; TRIPOD / TRIPOD+AI if a prognostic model is developed) | -- |
| Causal / comparative-effectiveness question emulated on observational data (treatment vs treatment, screening vs none, drug A vs B on registry / EHR / claims data) | TARGET (pair with the /design-study target-trial-emulation module for design; RECORD / STROBE for the routinely-collected-data items) | -- |
| Health-estimate / burden-of-disease modeling study (GBD or GBD-satellite, comparative-risk / population-attributable-fraction, cause-of-death or prevalence/incidence estimation, with or without forecasts) | GATHER (pair with /analyze-stats burden-decomposition-forecasting guide for the analytic layer) | -- |
| Systematic review / meta-analysis | PRISMA 2020 | PRISMA 2020 for Abstracts (run on the abstract, scored separately) |
| DTA systematic review / meta-analysis | PRISMA-DTA | PRISMA 2020 for Abstracts (run on the abstract, scored separately) |
| Meta-analysis of observational studies | MOOSE | PRISMA 2020 (use both) |
| Risk of bias (DTA studies) | QUADAS-3 (current recommended version) | QUADAS-2 only when appraising or reproducing a review that used it |
| Risk of bias (RCTs) | RoB 2 | -- |
| Risk of bias (non-randomised intervention studies) | ROBINS-I | -- |
| Risk of bias (non-randomised exposure studies) | ROBINS-E | -- |
| Risk of bias (comparative DTA studies) | QUADAS-C | QUADAS-3 (use both; apply the E&E's adaptation — see Using QUADAS-C with QUADAS-3 in QUADAS3.md) |
| Risk of bias (prediction models) | PROBAST | PROBAST+AI |
| Risk of bias (systematic reviews) | ROBIS | AMSTAR 2 |
| Risk of bias (missing evidence in MA) | ROB-ME | -- |
| Risk of bias (network meta-analysis) | RoB NMA | -- |
| Risk of bias (measurement properties) | COSMIN RoB | -- |
| Quality assessment (observational) | NOS | -- |
| Case report | CARE | -- |
| Study protocol | SPIRIT 2025 | SPIRIT-AI |
| Animal study | ARRIVE 2.0 | -- |
| AI/ML study in clinical imaging | CLAIM 2024 | -- |
| Study using a large language model (develop/fine-tune/prompt/evaluate an LLM) | TRIPOD-LLM | MI-CLEAR-LLM (use alongside when LLM accuracy is an outcome) |
| Early-stage / live clinical evaluation of an AI decision-support system (human factors, workflow, safety) | DECIDE-AI | -- |
| LLM accuracy evaluation in healthcare | MI-CLEAR-LLM | STARD-AI or CLAIM 2024 (use alongside) |
| Reliability / agreement study | GRRAS | -- |
| SR protocol | PRISMA-P | -- |
| Synthesis without meta-analysis | SWiM | PRISMA 2020 (use both) |
| Quality of systematic reviews | AMSTAR 2 | ROBIS |
| Radiomics study | CLEAR | CLAIM 2024 (if deep learning component) |
| Educational / QI study | SQUIRE 2.0 | -- |
| Generative AI images ARE the study object (realism / real-vs-synthetic reader study / model-vs-model quality) | (no single guideline -- assemble) | see decision aid below |
QUADAS-3 has two protocol-stage phases, and this skill usually runs too late for them. Phase 1 (state the synthesis question) and phase 2 (define the ideal test accuracy trial each judgement is made against) are review-level and belong in the protocol, with the review-specific guidance for answering each signalling question. Reaching them for the first time during manuscript QC means writing the comparator after seeing the results. If they are missing, say so as a limitation rather than reconstructing them, and route the protocol work to
/meta-analysisPhase 1. Phases 3–6 are what a QC pass can genuinely run.
Rules:
- A guideline the user names is the one you score. The rules below choose the guideline only when
the user names none. If they call for a different or additional instrument (STARD-AI for an AI index
test, TRIPOD+AI for an ML prediction model, an AI extension), score the named guideline and put one
line at the top of the report saying which instrument fits better and the command to rerun, e.g.
/check-reporting STARD-AI. Do not switch or add instruments on your own: the user maps the report to the checklist form the journal asked for. - If the study involves AI/ML, apply the AI version of the guideline. CONSORT-AI and SPIRIT-AI are scored alongside their base guideline; TRIPOD+AI, STARD-AI and TRIPOD-LLM replace theirs (below).
- Exception — TRIPOD: TRIPOD+AI 2024 (Collins et al., BMJ 2024) is a complete rewrite, not an addendum to TRIPOD 2015 (Moons et al., Ann Intern Med 2015). For non-AI prediction models, use TRIPOD 2015 only. For AI/ML prediction models, use TRIPOD+AI 2024 only. Do NOT apply both simultaneously.
- STARD-AI (Sounderajah et al., Nat Med 2025) extends STARD 2015 with 14 new and 4 modified items (40 total) and incorporates all STARD 2015 items. For AI diagnostic accuracy studies use STARD-AI only — do NOT apply STARD 2015 and STARD-AI simultaneously.
- TRIPOD-LLM (Gallifant et al., Nat Med 2025) is the reporting guideline for studies that develop, fine-tune, prompt, or evaluate a large language model for a clinical/biomedical task. It extends the TRIPOD family (TRIPOD 2015 → TRIPOD+AI 2024 → TRIPOD-LLM 2025); name the base instrument and the extension and cite each. It is modular — task-specific items (Annotation, Prompting, Summarization, Instruction-tuning) are N/A when that component is absent. Use TRIPOD-LLM for LLM studies in place of TRIPOD+AI; pair with MI-CLEAR-LLM when LLM accuracy is an evaluated outcome. The vendored checklist is an educational summary (own-words paraphrase of item intent); complete the official instrument for a submission checklist.
- MI-CLEAR-LLM is a supplementary checklist (8 item categories in the 2025 update; the 2024 original had 6), not a standalone reporting guideline. Always pair it with the study's primary guideline (e.g., STARD-AI for AI diagnostic accuracy, CLAIM for imaging AI). Apply it whenever the study evaluates LLM accuracy as an outcome — do NOT apply it merely because the manuscript was written with LLM assistance. Its scope is LLM accuracy studies (including VLMs interpreting images); it does not apply at study level when a generative model produces the images under study (next bullet).
- Generative-AI images as the study object (a generative model synthesizes images and the study evaluates their realism, controllability, real-vs-synthetic distinguishability, or model-vs-model quality) has no single dominant checklist. Assemble: CLAIM 2024 (imaging-AI umbrella; model-development items N/A when commercial models are used as-is) + FUTURE-AI traceability + MI-CLEAR-LLM transparency items only (prompt/model/version/params/runs — for generation provenance, not study-level compliance) on the generator side; STARD-AI (for real-vs-synthetic detection) + GRRAS (reader reliability) + MRMC reporting on the evaluation side. Map applicable items and cite base + extension; never claim wholesale compliance. Full decision aid:
${CLAUDE_SKILL_DIR}/references/genai_image_study_object_decision_aid.md. - If multiple guidelines apply (e.g., a diagnostic accuracy study that is also an AI study), check against all relevant guidelines and merge into one report.
Step 2: Load Checklist
-
Run the fail-fast guard first for every guideline you intend to apply:
python "${CLAUDE_SKILL_DIR}/scripts/check_checklist_exists.py" --guideline "STARD-AI"- Exit 0 → the vendored checklist exists; read it from
${CLAUDE_SKILL_DIR}/references/checklists/and proceed. - Exit 1 (
MISSING_CHECKLIST_CONTRACT_VIOLATION) → the guideline is routed but no checklist file is vendored. Do not construct items from memory. Halt, report the violation to the user, and stop unless they explicitly opt in (next bullet). - Exit 2 (
UNKNOWN_GUIDELINE) → the name is not recognised; confirm the correct guideline with the user.
- Exit 0 → the vendored checklist exists; read it from
-
No silent fallback. A from-memory checklist is permitted only when the user explicitly accepts it — re-run the guard with
--allow-from-memory(exit 0 + a NON-AUTHORITATIVE warning). The report MUST then carry a prominent banner that the assessment was constructed from model knowledge and is not backed by a vendored checklist, andsubmission_safemust not be asserted on its basis.
Step 3: Scan Manuscript
Read the whole manuscript before assessing any item — including tables, figures and their captions, supplementary material, and the reference list (registration numbers and protocol references often sit there).
Step 4: Assess Each Item
Items most often missing in medical manuscripts — look for these first, whichever guideline applies: registration number and registration/amendment date consistency (run Step 4c), sample-size justification, missing-data handling, blinding, funding and conflicts of interest, ethics approval with committee name and approval number, and a data availability statement; for AI studies, the training/validation/test split, model architecture and hyperparameters, failure-mode analysis, fairness/bias assessment, and commercial interests with data/code availability.
For every checklist item, determine:
| Status | Criteria |
|---|---|
| PRESENT | The item is fully addressed with sufficient detail. |
| PARTIAL | The item is mentioned or partially addressed but lacks required detail. |
| MISSING | The item is not found anywhere in the manuscript. |
| N/A | The item does not apply to this particular study (justify why). |
For each item, record:
- Status: PRESENT / PARTIAL / MISSING / N/A
- Location: Section name and paragraph or approximate position (e.g., "Methods, paragraph 3")
- Notes: What was found (if PRESENT/PARTIAL) or what should be added (if MISSING)
Be strict. PARTIAL means the item is mentioned but lacks specificity; a vague reference does not count as PRESENT — the detail level must match what the guideline expects. "We used appropriate statistical tests" = PARTIAL (which tests?); "We used the Mann-Whitney U test for continuous variables and Fisher's exact test for categorical variables" = PRESENT. If an item is genuinely unclear in its applicability, mark it N/A with justification.
Two gaps that are easy to pass:
- Power-aware framing of a null result (STROBE 16a / 18 / 20) — for an observational study whose headline is a non-significant association, a flat "X was not associated with Y" overreads the data when the analysis is not powered to exclude a clinically meaningful effect. Mark item 18/20 PARTIAL unless the manuscript states the precision as an exclusion (e.g., "the 95% CI excluded an eGFR difference larger than ~1.7") or reports a minimum detectable effect — "no effect" vs "could not exclude an effect of size X" are different claims, and a negative conclusion needs the latter.
- Confounder-selection rationale, not "adjust for everything that differs" (STROBE 16a explicitly asks which confounders were adjusted for and why) — flag a kitchen-sink adjustment set chosen because variables differ in Table 1. The Methods must give a causal rationale (DAG / prior literature) and must not adjust for a mediator or consequence of the outcome (over-adjustment, e.g. serum uric acid in an eGFR model); both an unjustified inclusion and an unjustified omission are item-16a gaps.
What is appraised is the source paper's reporting — never your convenience in using it. This holds for every instrument here, reporting checklists and risk-of-bias / quality tools alike, and is easiest to lose in a systematic review, where you read each paper in order to extract from it. An item asking "are the results clearly reported?" is not asking "were they reported in the unit my pool needs". If a downgrade's stated reason turns on a denominator, an analysis unit, a subgroup you needed and they did not report separately, or a format you could not parse, it is an extraction note, not a scoring reason: record it in a separate extraction-note column and restore the score. An extraction limitation often belongs in your limitations paragraph; folded into the score it makes the appraisal unreproducible, because another assessor with a different pool would score the same paper differently.
Step 4b: Section Boundary Check
In addition to checklist items, verify that:
- Results section contains only factual findings: no interpretation, no "why" explanations, no prior literature comparisons, no evaluative adjectives without numbers.
- Discussion section does not introduce new data not presented in Results.
- Flag any boundary violation as a separate finding in Part C Action Items with the label
[BOUNDARY].
Step 4c: Registration / Protocol Timing Consistency Check
Applies to: systematic reviews, meta-analyses, and intervention studies with prospective registration (PRISMA 2020, PRISMA-DTA, PRISMA-P, MOOSE, CONSORT, SPIRIT). The registration identifier is a single checklist item and can pass Step 4 while the manuscript is inconsistent about when registration and amendments happened relative to the analysis.
Read ${CLAUDE_SKILL_DIR}/references/step4c_registration_timing.md (item-by-item procedure,
JSON schema, flagging edge cases) and run its five checks: (1) registration identifier present in
Methods, Abstract, and cover letter; (2) initial registration date precedes — or is explicitly
disclosed as post-dating — the extraction milestone; (3) amendment dates appear in Methods, the
described change is visible in Methods, analysis was re-run if the amendment post-dates the lock,
and no amendment post-dates submission; (4) Methods agree with the registry record (PROSPERO PDF,
ClinicalTrials.gov export) — a silent discrepancy is a finding; (5) a retrospective-registration
disclosure paragraph when evidence suggests post-extraction filing.
Flagging: any failure is logged in Part C Action Items with label [REGISTRATION-TIMING].
fixable_by_ai: false when reconciliation requires an external amendment filing; true only when
the fix is a Methods-text insertion of a date already disclosed elsewhere. Part D JSON includes a
registration_timing object (registry, id, initial_registration_date, amendments[],
timing_consistency, findings[]).
Registration-ID format gate: a PROSPERO ID is CRD42 + 9 digits = 14 characters
(^CRD42\d{9}$, e.g. CRD42024500001). Run grep -oE 'CRD42[0-9]+' manuscript.md and
assert each match is 14 characters long; a 15-character ID (a stray inserted digit) is a
transcription error logged as [REGISTRATION-TIMING] (fixable_by_ai: false — verify against
the live PROSPERO record, do not guess the correct digit).
Step 4d: PRISMA Figure 1 Arithmetic & Cross-Reference Audit
Applies to: systematic reviews and meta-analyses using PRISMA 2020 / PRISMA-DTA / PRISMA-P. Triggers when Item 16a (flow diagram) is PRESENT — the diagram can pass Step 4 while its numbers do not add up or disagree with the text.
-
Choose the Figure 1 source, in this order: (a)
analysis/figures/Figure1_PRISMA.mdmarkdown manifest, (b) caption text inmanuscript.md, (c) PPTX text run if a.pptxexists, (d) manual entry from PNG/SVG. -
Run the audit. It checks the five subtractions (screened = identified − duplicates; sought-for-retrieval = screened − excluded at screening; retrieved = sought − not retrieved; assessed for eligibility = sought − not retrieved; included = assessed for eligibility − excluded with reasons) and that the body-text PRISMA numbers match the Figure 1 boxes 1:1, and writes
qc/prisma_figure_audit.json:python3 ${CLAUDE_SKILL_DIR}/scripts/check_prisma_figure.py \ --md <manuscript.md> --figure <Figure 1 source: .md manifest / caption / text export> \ --out qc/prisma_figure_audit.jsonExit
1= an arithmetic or cross-reference MISMATCH; exit2= missing/unparsable input. When the Figure 1 numbers exist only in a PNG/SVG, transcribe them by hand and run the same checks manually as${CLAUDE_SKILL_DIR}/references/step4d_prisma_figure_audit.mdspecifies (regex set, JSON schema, and edge cases: duplicates across databases, the citation-searching strand, dual-reviewer screening). -
By hand (the script does not do this): the reasons for exclusion in Methods and the Figure legend must agree on counts and category names, and when identification is split across sources (databases, registers, other methods) the per-source counts must sum to the
identifiedtotal — the script reads only the firstidentifiedcount. -
If
analysis/figures/_figure_manifest.md(from/make-figures) exists, verify that the row whoseType = prisma(orType = prisma-dta) points at the same file used as the audit source, and that itsCriticfield isyesorpartial(notno). A missing row, mismatched path, orCritic = nologs[MANIFEST-XREF](advisory); the arithmetic check still runs.
Flagging: any MISMATCH or arithmetic failure logs a Part C Action Item with label
[PRISMA-FIGURE], fixable_by_ai: false (the author must reconcile the numbers).
PRISMA Cascade Arithmetic Auto-Verify
When PRISMA 2020 or PRISMA-DTA is selected and round-by-round screening TSV artifacts are
available, recompute the screening cascade from the raw decisions. Off-by-one errors in the prose
cascade are a high-frequency reviewer red flag (e.g., 151 + 108 + 39 + 1 + 1 + 4 = 304
followed by a prose summary "305" four lines later).
python "${CLAUDE_SKILL_DIR}/scripts/prisma_cascade_check.py" \
--round1 2_Screening/round1.tsv \
--round2 2_Screening/round2.tsv \
--round3 2_Screening/round3_adjudication.tsv \
--manuscript manuscript.md \
--out qc/prisma_cascade.json --strict
The script counts INCLUDE / EXCLUDE / MAYBE decisions per round, computes the cascade, and
reports per-stage drift where the manuscript's stage counts disagree. Treat any
manuscript_drift entry as a P0 blocker — fix the prose to match the computed cascade and re-run.
A --manuscript path that is not a file exits 2. When the manuscript is read but none of the
script's stage phrases is found in it, manuscript_check is "unverifiable",
submission_safe is false, an UNVERIFIED line is printed, and --strict exits 1: the
drift check did not run, so compare the prose stage counts to stage_counts by hand.
stages_compared lists the stages that were actually checked.
Step 4e: Reporting-Framework Naming Audit
Applies to: any manuscript that invokes an AI/extension reporting framework
(PROBAST+AI, STARD-AI, TRIPOD+AI, TRIPOD-LLM, CONSORT-AI, SPIRIT-AI, PRISMA-DTA, QUADAS-C).
A base reporting tool and its extension are distinct instruments with separate citations, and
Step 1 does not police how the framework is named in prose. The recurring failures: invoking an
extension without ever naming or citing the base instrument it extends; mixing +AI and -AI
hyphenation for one family within a single document; coining item labels like "12-AI"; and waving
at "recent guidance" instead of naming the framework.
python3 "${CLAUDE_SKILL_DIR}/scripts/check_framework_naming.py" \
--manuscript manuscript.md --out qc/framework_naming.json --strict
Verdicts: BASE_MISSING (extension used, base instrument never named standalone) is a
Major and logs [FRAMEWORK-NAMING] in Part C with fixable_by_ai: true (insert the base
name + its citation). HYPHEN_MIX, CITE_MISSING, SELF_COINED_LABEL, and VAGUE_GUIDANCE
are Minor (fixable_by_ai: true). Part D JSON includes a framework_naming object mirroring
the script's claims[].
Step 4f: Critical-item floor cross-check
Applies to: every guideline assessment for which the floor defines a row (load and
check only those; do not invent a floor for an unlisted guideline). After the item-by-item
table, read ${CLAUDE_SKILL_DIR}/references/critical_item_floor.md and check the small set of
non-waivable items for this study type — and, for AI/ML and radiomics manuscripts, the
methodological-quality / risk-of-bias instrument (PROBAST+AI, METRICS/RQS, APPRAISE-AI) and
concerns it lists. A MISSING critical item is surfaced as a Critical gap and becomes the
report's headline regardless of the overall percentage — a high percentage with a missing
critical item (undefined reference standard, no leakage-controlled partition, calibration absent
for a prediction model, an unreconciled flow diagram) is not "broadly acceptable."
Step 5: Generate Report
This report is an internal working audit — it carries auto-fix annotations, a
machine-readable JSON block (compliance_pct, fixable_by_ai, …), and Action Items. It is
NOT the official reporting checklist a journal expects (that is the blank guideline form with
Item | Recommendation | Reported in page/section, which the authors fill in). Never submit
this report as the submission checklist. So that the file is self-identifying and cannot be
reused by filename into a later submission package, the report MUST begin with this banner as
its very first line:
<!-- INTERNAL AUDIT — NOT FOR SUBMISSION. This is the /check-reporting working
report, not the official journal checklist. Do not upload to a submission portal. -->
Write the checklist content and report in English (matching the guideline originals), whatever
language you use with the user. Save it as qc/reporting_checklist.md and the Part D JSON as
qc/reporting_checklist.json.
Read ${CLAUDE_SKILL_DIR}/references/report_templates.md when you write the report — it holds the
literal templates for the four parts:
- Part A — Summary. Header (manuscript file, version token, guideline, date), the
PRESENT/PARTIAL/MISSING/N-A count table, and overall compliance. The headline is the critical
items (Step 4f), not the percentage: report
{present}/{total}and name every missing critical item with the section it belongs in. - Part B — Item-by-item checklist. One row per item:
# | Section | Item | Status | Location | Notes. - Part C — Action items (MISSING and PARTIAL only), ordered by: items most journals enforce strictly (ethics approval, registration, sample size) → items in Methods (easiest to fix) → everything else.
- Part D — Machine-readable JSON, appended as a fenced block. MUST be present under
--jsonor when called from/write-paperPhase 7, which parses it.
JSON field contract (the part other skills depend on — get these right):
compliance_pct—present / (total_items - na) * 100, one decimal;nullwhen every item is N/A (total_items == na).action_items— MISSING and PARTIAL only; PRESENT and N/A are excluded.fixable_by_ai—truewhen the fix inserts or expands text using information already in the manuscript or inferable from it;falsewhen it needs external facts the author alone holds (registration number, IRB approval number, protocol details, sample-size rationale and target).suggested_fix— concrete draft text, insertable as written. A fix that still contains a bracketed placeholder ([N],[rationale]) is not insertable and isfixable_by_ai: false.source_sha256— first 12 hex chars of the SHA-256 of the manuscript bytes, so a stale report cannot be silently attributed to a newer manuscript.
In notes and suggested fixes, cite a reference only with a /search-lit-confirmed DOI or PMID and
mark any other [UNVERIFIED - NEEDS MANUAL CHECK]; never invent clinical definitions, diagnostic
criteria, or guideline recommendations — flag anything uncertain [VERIFY] and ask the user.
When the user asks for the filled checklist a journal requires, build it from the Submission
checklist export template in ${CLAUDE_SKILL_DIR}/references/report_templates.md.
Skill Interactions
| When | Call | Purpose |
|---|---|---|
| During manuscript writing | /write-paper Phase 7 | Final compliance check |
| Need to add Methods text | /write-paper Phase 3 | Draft missing Methods content |
| Need statistical details | /analyze-stats | Generate missing statistical reporting |
| Need flow diagram | /make-figures | Generate CONSORT/STARD/PRISMA diagram |
Gates
| Gate | Severity | Trigger | Action on fail |
|---|---|---|---|
| Mandatory items present | ENFORCED at submission | < 100% of guideline-mandatory items marked PRESENT | Auto-fix MISSING items where text exists; otherwise route to /write-paper Phase 7 for re-draft |
| Step 4d PRISMA Figure 1 arithmetic & cross-reference audit (PRISMA / PRISMA-DTA only) | ENFORCED for SR/MA | flow numbers don't sum (e.g., screened ≠ included + excluded), or in-text counts mismatch flow diagram | HALT; reconcile against extraction artifacts |
| Optional items (e.g., supplementary AI declarations) | ADVISORY | < 80% of optional items present | warn; user accepts |
| Cross-reporting-guideline routing (study type → guideline) | ENFORCED | study type undeclared or guideline missing | Ask user; do not silently default |
Data Profiling and Cleaning Skill
Profile, flag, and generate cleaning code for a clinical dataset in three stages, each ending at a user-approval gate. You generate code and reports; you do NOT auto-clean data. Every cleaning decision needs the researcher's explicit confirmation, because clinical cleaning calls need domain knowledge.
PHI first. If the dataset contains PHI or PII, run /deidentify before proceeding, and use
*_deidentified.* files when they exist in the working directory. Otherwise work from the data
dictionary / codebook alone, or in a local-only environment with no network access. The skill writes
code that runs on the data; it does not need to see the raw rows to write it.
Three-Stage Workflow
Stage 1: Profiling
Input: CSV/Excel file path OR data dictionary/codebook
- Adapt
${CLAUDE_SKILL_DIR}/references/profiling_template.py(pandas) to the dataset. It reports: variable and row counts, data types; missing count and percentage per variable; unique counts for categorical variables; min/max/mean/median/SD for numeric variables; histograms and bar charts. - If the user provides a codebook, cross-reference variable names, expected types, and expected
ranges. Never guess a column name or coding: when a mapping is uncertain, write
[VERIFY: variable_name]and ask the user to confirm it against the data dictionary. - Present the summary (see Output Format). Every count and statistic comes from the executed script's output.
Gate: The user reviews the profile. Ask whether to proceed to Stage 2 (Flagging) and whether any variables should be excluded or focused on.
Stage 2: Flagging
Read ${CLAUDE_SKILL_DIR}/references/cleaning_patterns.md for missing-data mechanisms, outlier
decision rules, duplicate detection, date handling, and clinical pitfalls (inequality-prefixed lab
values, mixed units, sentinel values). Flag issues in these categories:
- Missing values: variables with >5% missing; pattern analysis (MCAR/MAR/MNAR heuristic).
- Statistical outliers: IQR method (Q1 - 1.5IQR, Q3 + 1.5IQR) and Z-score (|z| > 3).
- Duplicates: exact row duplicates AND near-duplicates (same patient ID, different dates).
- Type mismatches: numeric stored as string, dates in inconsistent formats.
- Implausible values: flag against the codebook's valid range or, when the codebook is silent,
the domain-default hard physiologic bounds in
references/implausible_value_rules.md§1. An implausible value is a likely data-entry/unit/sentinel error (correct or set missing); a statistical outlier (#2) is biologically possible (keep + sensitivity analysis). Check units before calling a bound violation an error. Never auto-fix. 5b. Cross-field inconsistencies: logical contradictions perreferences/implausible_value_rules.md§2 — temporal ordering (birth ≤ event ≤ death, admission ≤ discharge), derived-vs-source (recomputed BMI/age; subset ≤ superset; total = sum of parts), sex-/state-specific fields, and min ≤ max / diastolic < systolic pairs. Name the rule that fired; a hard contradiction is High severity. - Category inconsistencies: typos and variants in categorical values ("Male", "male", "M", "MALE").
- Categorical-implied zeros: when a category defines a natural zero for a dose/duration variable
(
smoking_status == 'never'impliespack_years == 0;alcohol_use == 'never'impliesgrams_per_week == 0), flag records that store the implied zero as NULL. This is a contradiction, not a missing-data pattern: complete-case models silently drop those never-smokers and MICE imputes them a non-zero dose, corrupting the exposure contrast. Suggested action: "Set dose = 0 where category == reference level; impute only the residual missingness among the exposed." Detected byscripts/check_structural_zero.pygiven the category↔dose mapping; pairs with/analyze-stats"Covariate Pitfalls: Structural Zeros & Dose/Duration Variables". - Reverse-coded scale items: in a multi-item Likert scale that mixes positively and negatively
worded items, every reverse item must be recoded
(min+max) - xbefore the scale total or Cronbach's alpha is computed. An un-recoded reverse item correlates negatively with the rest and collapses alpha, often to a negative value. A negative alpha is a reverse-coding bug, not "multidimensional structure". Suggested action: "Recode reverse-worded items, then recompute reliability." Detected byscripts/check_reverse_coding.py(negative item-rest correlation and negative raw alpha, given the scale item columns); the recode itself is applied downstream by/analyze-statslikert_summary.py --reverse-items.
Present the flag report as a table:
| Variable | Issue Type | Count | Severity | Suggested Action |
|---|---|---|---|---|
| age | Outlier (IQR) | 3 | Medium | Review: values 150, 200, -5 |
| pack_years | Categorical-implied zero | 12421 | High | Set 0 where smoking_status=='never' (structural zero, not missing) |
Severity levels:
- High: likely data errors that will affect analysis (type mismatches, impossible values)
- Medium: potential issues that need expert review (statistical outliers, moderate missingness)
- Low: minor inconsistencies that are easy to fix (category labels, trailing whitespace)
Gate: The user marks each row (A) Approve the suggested action, (R) Reject / keep as-is, or (M) Modify the action. Only approved actions generate cleaning code.
Stage 3: Code Generation
For ONLY user-approved actions, generate Python (or R if requested) code:
- Missing value handling: listwise deletion, mean/median imputation, or MICE setup (code only, the user runs it and picks the variables and method)
- Outlier handling: winsorization, removal, or keep-and-flag
- Duplicate removal: exact dedup with logging
- Type conversion: standardize dates, numeric parsing
- Category harmonization: mapping table for inconsistent labels
All generated code MUST include:
- Before/after row counts printed to console
- Logging of every modification to a cleaning log DataFrame
- Reproducibility:
np.random.seed(42)andrandom.seed(42)where applicable - Output: cleaned CSV (a new file; the input file is never overwritten) +
cleaning_log.csv - Clear comments explaining each cleaning step
End the generated script with this notice:
"This code implements ONLY the cleaning rules you approved. Review the cleaning_log.csv output to verify all changes before proceeding to analysis."
Out of scope: free-text extraction from clinical notes, and image data or DICOM metadata. After
cleaning, hand off to /analyze-stats. Take any citation from /search-lit, never from memory.
Output Format
Structure all reports using this template:
## Data Profiling Report
### Dataset Overview
- Rows: [N]
- Columns: [N]
- File size: [size]
- Date range: [if applicable]
### Variable Summary
| Variable | Type | Missing N (%) | Unique | Min | Max | Mean | SD |
|----------|------|---------------|--------|-----|-----|------|-----|
| ... | ... | ... | ... | ... | ... | ... | ... |
### Flags
| Variable | Issue | Count | Severity | Suggested Action |
|----------|-------|-------|----------|-----------------|
| ... | ... | ... | ... | ... |
### Cleaning Code
[Python/R script -- only for approved actions]
### Cleaning Log
[What was changed, how many rows affected, before/after counts]
Contribute
The user is a clinician who has probably never opened a pull request. Do the git work for them: never make them type a git command, and never use the words "rebase", "upstream", or "HEAD" in anything they read. Their local edits sit next to real patients and real manuscripts, so the safety steps below, not the git mechanics, are why this is a skill.
The one rule that cannot bend
Nothing leaves the machine until the author has seen every line that would leave it and said yes. Until something has been sent, say explicitly, every time, that nothing has been sent.
The safety scan is an aid, not a certificate: no pattern list recognises every patient name or every hospital. Say so out loud, because a user who believes the scanner is complete stops reading the diff, and that is exactly when the leak happens.
If the scan reports a blocker (patient-level data, a credential), do not offer a workaround. The line gets deleted. A contribution never needs patient data to make its point.
Phase 0: What did they change?
python3 "${CLAUDE_SKILL_DIR}/scripts/find_local_changes.py" --target claude --json \
--out qc/local_changes.json
This compares the installed skills against the hashes of what was shipped. It reads only.
- Nothing changed → say so plainly and stop; that is normal, not a failure. Offer the feedback path (Phase 4) instead.
- Something changed → show it, grouped by skill:
- A new file (a journal profile, a citation style, a reporting exemplar) — usually the most valuable kind of contribution, because it is domain knowledge nobody in the project has. Say that.
- An edited file — ask what was wrong for their specialty; the answer is the pull-request description. Never guess at the reason for an edit.
- A deleted file — usually not a contribution. Ask before including it.
A change that only makes sense in one hospital (internal rules, a template with the department's letterhead) is a local adaptation, not a contribution: say so, leave it out, and tell them keeping it is fine.
Phase 1: The safety scan (blocking)
python3 "${CLAUDE_SKILL_DIR}/scripts/check_contribution_safety.py" \
--changes qc/local_changes.json --out qc/safety.json
This gate fails closed: finding anything at all is a non-zero exit, unlike every other detector in the repository — deliberately, because a tool that returns success while printing a hospital name will eventually be trusted to have said nothing.
Verdicts: PHI_SUSPECTED, SECRET (blockers — the line is deleted, not argued with),
IDENTITY, INSTITUTION, APPROVAL_ID, MANUSCRIPT_ID, LOCAL_PATH. Fix each finding with
them, following the do line the scan prints for it (a name → a role, a hospital → a generic
descriptor, an IRB or manuscript ID → removed, a home path → ~). A manuscript ID matters because
a paper under review is confidential and its ID identifies it.
A file that is not UTF-8 text (an image, a binary) cannot be scanned or read, so it is listed
as NOT SCANNED and the scan fails until it is taken out of the contribution. qc/safety.json
records the sha256 of every file it scanned; the submit step refuses unless the files it is about
to send are exactly those, unchanged. After any edit, re-run the scan — an older report, or one
made with --text on a different file, clears nothing.
Then print the full text of every file that would be sent and ask them to read it. Not a summary — the text. This is the step that actually protects them.
Phase 2: Does it meet the project's own bar?
Run the repository's validator against the change if the repo is available locally; otherwise check by eye:
- A journal profile cites the journal's public author guidelines: word limits, abstract structure and AI policy are quoted from them or left out. No impact factors or numbers from memory.
- A citation style is the official CSL from the Zotero repository.
- A skill edit still says what it does, and the change is general — it would help someone at another hospital. If not, it is a local adaptation; keep it local.
If it does not pass, say what to change, in their words. Do not send a contribution that will be rejected — that is a worse experience than not contributing at all.
Phase 3: Send it
python3 "${CLAUDE_SKILL_DIR}/scripts/submit_contribution.py" \
--changes qc/local_changes.json --safety qc/safety.json \
--title "Add a journal profile for <journal>" --dry-run
Always --dry-run first, and show them the plan. Then, on their explicit yes, run it again
without --dry-run. The script takes the highest rung available:
- GitHub tool installed and signed in → it forks, copies the files, and opens the pull request. They type nothing.
- Installed but not signed in → it prints the single command (
gh auth login) and stops. Walk them through it: GitHub.com → HTTPS → log in through the browser. - Not installed → it writes the change to a file and opens a pre-filled issue in their browser. Do not make installing a developer tool a condition of helping.
Tell them what happens next: a maintainer reads it, and if something needs changing they can reply in plain language on that page. They will not be asked to do any git work.
Phase 4: Feedback that is not a code change
A false positive or a failed step is often the most useful thing a clinician has: it is the only evidence of how a detector behaves on a real manuscript rather than a synthetic fixture. Collect:
- A detector fired wrongly: which detector (the
detectorfield in itsqc/*.json), the verdict, and the smallest possible snippet that reproduces it — rewritten with fake numbers and names if the real one cannot be shown; the shape of the sentence is what matters. - A step failed: the command, the error, the host (Claude Code / Codex / Cursor / Copilot), the operating system.
- Something was wrong for your specialty: what the toolkit assumed, and what is actually true.
Send it as an issue, after the same safety scan — a repro from a real manuscript is exactly where PHI hides:
python3 "${CLAUDE_SKILL_DIR}/scripts/check_contribution_safety.py" --text qc/feedback.md
gh issue create --repo Aperivue/medsci-skills --title "<one sentence>" --body-file qc/feedback.md --web
--web opens it in the browser pre-filled: they read it and press Submit. Nothing is sent by us.
Phase 5: Keep their change when they update
Tell them the installer already takes a permanent backup of any skill they modified before
it overwrites it (under ~/.medsci-skills/backups/), so updating never destroys their work —
many people avoid updating because they assume it would. If the change is worth keeping,
contributing it means the next update will contain it.
Settings — nobody is nagged
python3 "${CLAUDE_SKILL_DIR}/scripts/contribution_prefs.py" --status
python3 "${CLAUDE_SKILL_DIR}/scripts/contribution_prefs.py" --on # remind me
python3 "${CLAUDE_SKILL_DIR}/scripts/contribution_prefs.py" --off # never mention it (default)
- Reminders are opt-in and off by default. The installer mentions the option once, at the end of a first install, and never again. Opted in, a reminder appears only when they actually changed something, and at most once a month.
- The setting governs reminders only.
/contributeworks whenever it is run. - The setting cannot weaken safety: there is no key for that, and
check_contribution_safety.pyreads no configuration. Opting in never means "send without asking".
Saying thanks
Many clinicians who find this useful have never starred the repository because nobody told them what a star is for. Once the toolkit has actually done something for them, tell them — the script explains what a star is and makes it one command:
python3 "${CLAUDE_SKILL_DIR}/scripts/star_repo.py" --how # explain + show both ways
python3 "${CLAUDE_SKILL_DIR}/scripts/star_repo.py" --now # do it from here (GitHub CLI)
- Never ask twice — being shown once is recorded, and doing nothing is an answer.
- Never dress this up: do not imply the toolkit is worse off without their star, and do not ask before the toolkit has done something for them. Asking for thanks in advance is begging.
Scope
Touches only the installed skills — never the user's manuscripts, data, or any other file. Contributions arrive as a fork's pull request or an issue; never push to the main repository.
Known limits
- The patient-identifier patterns cover a fixed set of labels and formats. Among what they do not recognise: Korean-language labels for a registration or patient number, a resident registration number written without its hyphen, an alphanumeric patient ID, "hospital number", and a date of birth written with a month name or introduced by "born". Widening them means keyword matching over prose, which has produced false positives before, so it has not been done. Reading every line (Phase 1) is what covers these.
Cross-National Comparison Study Skill
Inputs
- Research question: exposure → outcome association to compare across countries
- Korean data path: KNHANES CSV file
- US data path: NHANES CSV directory (multiple tables to merge)
- Harmonization table (optional): CSV mapping variables across surveys
- Default:
/replicate-study'sreferences/harmonization_knhanes_nhanes.csv
- Default:
Reference Files
/write-paper'sreferences/paper_types/cross_national.md— writing template/analyze-stats'sreferences/analysis_guides/survey_weighted.md— survey-weighted analysis guidereferences/chns_coding.md— CHNS files, merge keys, coding and warnings; read it when the design includes China (3-country design)references/additional_variables.md— asthma, sleep, physical activity, diet, treatment and non-HDL-cholesterol coding, plus composite-score (LE8) warnings; read it when the study uses any of them
Workflow
Phase 1: Study Definition
- Confirm research question: Exposure → Outcome
- Define variable coding for both countries:
- Exposure: PHQ-9, BMI category, smoking, etc.
- Outcome: diabetes, hypertension, mortality, etc.
- Covariates: age, sex, education, income, smoking, alcohol, obesity, CVD
- Check harmonization table for variable availability
- Output: study protocol summary for user approval
Phase 2: Data Preparation
Never guess a variable name, dataset column name, or variable coding. If a mapping is uncertain,
output [VERIFY: variable_name] and ask the user to confirm it against the data dictionary.
KNHANES (single CSV):
-
Load CSV and keep every row — the age ≥20 (or per-protocol) restriction is applied to the design in step 3
-
Derive variables using KNHANES coding:
Variable Raw Var Coding Smoking BS3_1 1,2=Current; 3=Former; 8=Never Alcohol BD1_11 2-6=Frequent (current drinker); 1=Occasional (past-year abstainer); 8=Never Obesity HE_obe 1-3=Normal; 4-6=Obesity (BMI≥25, Asian cutoff) Depression BP_PHQ_1~9 Sum ≥10 = depression Diabetes HE_glu, HE_HbA1c, DE1_dg FPG≥126 or HbA1c≥6.5 or DE1_dg=1 CVD DI4_dg, DI5_dg, DI6_dg Any = 1 → CVD yes Education edu 1-3=Non-college; 4=College Income incm Quartile (1=lowest … 4=highest): 1-3=Bottom 75%; 4=Top quartile -
Set survey design on the full file, then restrict to the analytic domain: des <- svydesign(id=~psu, strata=~kstrata, weights=~wt_itvex, nest=TRUE, data=df); des_ad <- subset(des, age >= 20). Never filter rows before svydesign() — dropping them changes the standard errors (see
survey_weighted.md, subpopulation analysis)
NHANES (multiple CSVs):
-
Load and merge tables by SEQN (DEMO_J, DPQ_J, GHB_J, GLU_J, BMX_J, SMQ_J, ALQ_J, DIQ_J, MCQ_J, BPQ_J, BPXO_J)
-
Derive variables using NHANES coding. CRITICAL: NHANES data downloaded via R
nhanesApackage uses TEXT LABELS, not numeric codes.Variable Raw Var Coding (text labels) Sex RIAGENDR "Male" / "Female" (NOT 1/2) Smoking SMQ020 + SMQ040 100 cigs (SMQ020 "Yes" / "No") + now smoke (SMQ040 "Every day" / "Some days" / "Not at all") Alcohol ALQ121 + ALQ111 Frequent (current drinker): any ALQ121 frequency except "Never in the last year"; Occasional (past-year abstainer): "Never in the last year"; Never (lifetime non-drinker): ALQ111 == "No" (ALQ121 will be NA) Obesity BMXBMI (BMX_J, kg/m²) ≥30 (WHO cutoff, NOT Asian) PHQ-9 DPQ010~DPQ090 "Not at all"→0, "Several days"→1, "More than half the days"→2, "Nearly every day"→3; sum ≥10 = depression Diabetes LBXGLU (GLU_J, fasting subsample, mg/dL), LBXGH (GHB_J, %), DIQ010 LBXGLU≥126 | LBXGH≥6.5 | DIQ010=="Yes" (DIQ010: "Yes" / "No" / "Borderline"). CRITICAL: fasting glucose is GLU_J LBXGLU, not BIOPRO_J LBXSGL — CDC says the serum LBXSGL should not be used to determine undiagnosed diabetes; an analysis that uses LBXGLU needs the fasting-subsample weight (step 3). LBXGLU is NA for non-fasters, and in R NA | TRUEis TRUE butNA | FALSEis NA, so this composite goes missing only for non-fasters who would be non-diabetic; dropping those NAs under WTMEC2YR inflates prevalence. Either analyse the FPG composite only in the fasting domain with the fasting weight, or use the non-fasting variantLBXGH>=6.5 | DIQ010=="Yes"on WTMEC2YR, and say which invariable_mapping.csvCVD MCQ160B/C/D/E MCQ160B=="Yes" (CHF) | MCQ160C=="Yes" (CHD) | MCQ160D=="Yes" (angina) | MCQ160E=="Yes" (MI); labels "Yes" / "No" / "Don't know" HTN BPXOSY2+3, BPXODI2+3, BPQ020 mean(BPXOSY2, BPXOSY3)≥140 | mean(BPXODI2, BPXODI3)≥90 | BPQ020=="Yes" (BPXOSY3 is the 3rd reading, not an average; the mean of the 2nd and 3rd matches KNHANES) Education DMDEDUC2 5 text levels -
Set survey design on the full file, then
subset()the design object to the analytic domain (e.g. RIDAGEYR >= 20): svydesign(id=~SDMVPSU, strata=~SDMVSTRA, weights=~WTMEC2YR, nest=TRUE). The weight follows the files: the single-cycle_Jtables above take WTMEC2YR; WTMECPRP goes only with the pre-pandemicP_files (P_DEMO, P_BMX, ...). A variable from the fasting subsample (GLU_J LBXGLU, TRIGLY_J LBXTR/LBDLDL) takes the fasting weight instead: WTSAF2YR (_J) or WTSAFPRP (P_); a variable that combines a fasting-subsample component with full-sample ones (diabetes above) is defined only in that fasting domain, so it cannot enter a WTMEC2YR model. Pooling cycles follows the NCHS rules: divide each cycle's weight by the number of cycles pooled (1999–2002 has its own 4-year weights), and to combine 2015–2016 with 2017–March 2020 use 2/5.2 × WTMEC2YR and 3.2/5.2 × WTMECPRP.
CHNS (3-country design): read references/chns_coding.md before preparing China data.
Phase 3: Parallel Analysis
For EACH country independently:
- Table 1: Baseline characteristics by exposure (weighted counts + percentages)
- Main analysis: Sequential logistic regression models
- Model 1 (unadjusted)
- Model 2 (age + sex)
- Model 3 (fully adjusted: + education, income, smoking, alcohol, obesity, CVD)
- Subgroup analyses: By sex, age group, education, income, alcohol, smoking, CVD, obesity.
Whether the association differs between subgroups is tested with an exposure × subgroup
interaction term fitted on the full design (
svyglmon the whole sample), not by comparing the subgroups' P values. - Dose-response (if applicable): RCS with 3 knots
Phase 4: Cross-National Comparison Table
Generate a side-by-side comparison:
| Analysis | Korea wOR (95% CI) | US wOR (95% CI) | Ratio of wORs (95% CI); P |
|---|---|---|---|
| Overall (fully adjusted) | ... | ... | ... |
| Male | ... | ... | ... |
| Female | ... | ... | ... |
| ... | ... | ... | ... |
Compare the countries with the ratio of their odds ratios, not with whether the directions agree: two estimates in the same direction can differ, and opposite directions can be compatible. With log odds ratios b₁, b₂ and their design-based standard errors SE₁, SE₂ from the two independent surveys, the ratio is exp(b₁ − b₂) with 95% CI exp(b₁ − b₂ ± 1.96·√(SE₁² + SE₂²)) and z = (b₁ − b₂)/√(SE₁² + SE₂²) (Altman & Bland, BMJ 2003;326:219).
Every number comes from executed code output (analysis_korea.R, analysis_us.R) — never an
invented p-value, effect size, confidence interval, or sample size.
Phase 5: Output Files
{working_dir}/
├── cross_national_report.md — Study summary + comparison tables
├── variable_mapping.csv — Variable mapping with match status
├── analysis_korea.R — KNHANES analysis (self-contained)
├── analysis_us.R — NHANES analysis (self-contained)
├── results/
│ ├── table1_korea.csv
│ ├── table1_us.csv
│ ├── main_results_comparison.csv
│ └── subgroup_comparison.csv
└── manuscript_draft/ — Optional: Methods + Results draft
├── methods_draft.md
└── results_draft.md
Take every citation in the report or draft from /search-lit (confirmed DOI/PMID); mark any other
[UNVERIFIED - NEEDS MANUAL CHECK], and never generate references from memory.
Critical Rules
- NEVER pool data across countries. Each country analyzed with its own survey design.
- Country-specific BMI cutoffs: Korea ≥25 (Asian), US ≥30 (WHO).
- Country-specific income: KNHANES quartile, NHANES PIR → harmonize to binary.
- Weighted analysis mandatory: Both KNHANES and NHANES are complex surveys. CHNS has no
survey weights — analyse it unweighted (see
references/chns_coding.md). - Document all harmonization decisions: What matches, what needed recoding, what differs.
- Same analytic approach: Identical model specifications for both countries for fair comparison.
Define-Variables Skill
Map each exposure, outcome, covariate, and eligibility variable to a canonical guideline/consensus
definition, cross-check it against prior operationalizations in comparable cohorts, then map it to
the available DB variables. Call after /design-study (and /search-lit), before
/write-protocol.
Inputs
- Research question (one sentence)
- Candidate variables — exposure, outcome, key covariates, eligibility filters
- Data dictionary path (xlsx / csv / markdown) OR explicit list of available DB columns
- Cohort type (e.g., health-screening, NHANES-like, claims, registry) — informs which prior-art cohort to compare against
Missing inputs → ask once, then proceed.
4-Tier Pipeline (DB codebook + token-efficient literature)
Tier 0 — DB codebook lookup (mandatory for DB-backed observational studies)
Trigger: project has a project.yaml::db.dictionary_path field pointing to a machine-readable codebook (xlsx/csv/markdown), OR the user supplied a dictionary path in inputs. If neither, skip to Tier 1.
For every candidate DB variable — before touching literature — open the dictionary and record, verbatim, the sheet name, row number, and code→meaning mapping. This prevents the most common observational-study error: assuming a column code (status == 0, grade == 4) means what it intuitively reads like, when the codebook says otherwise.
Per variable:
- Locate the variable in the dictionary by exact column name.
- Copy verbatim: the sheet title, row number, and full code→meaning mapping (or unit/range statement for continuous vars).
- Paste into the
Dict. sheet & row+Dict. verbatimcolumns of the operationalization table. - If the variable is not found, OR the codebook is silent on a specific code value, file a question to the DB owner / data steward. Do NOT infer from cross-tabs, do NOT guess, do NOT proceed with that variable until a verbatim answer exists.
Empirical checks (value distributions, cross-tabs with related columns) are useful for sanity testing after the verbatim codebook meaning is recorded — never as a substitute for it.
Recommend committing a DICTIONARY_FIRST_POLICY.md at the project root (or shared-config path) with the canonical dictionary path and the escalation contact.
Exit gate: before Tier 1, cross-check every row's Dict. sheet & row and Dict. verbatim against the source dictionary; no DB-backed row may be left blank.
Tier 1 — Canonical index lookup (no API calls)
Look the variable up in references/common_definitions.md (hepatology, metabolic/endocrine, renal, pulmonary, cardiovascular, oncology/imaging incidentalomas, alcohol exposure). On a hit, record the guideline, year, canonical cutoff, and BibTeX key. Done — no /search-lit call.
Tier 2 — Targeted /search-lit (focused queries only)
For variables NOT in Tier 1, OR when subgroup justification is needed (Asian-specific cutoff, pediatric, young-adult, pregnancy, etc.), call /search-lit with one query per variable — never a general sweep, which buries the signal. Query pattern:
"{construct} definition {cohort type} {subgroup qualifier}"
e.g., "obstructive sleep apnea prevalence Korean health screening cohort"
Stop searching a variable once the first 1-2 papers converge on the same definition. If more than five variables need Tier 2, list them and confirm with the user before running the rest.
Tier 3 — Verification
Every definition, cutoff, and era anchor must come from a verified source — a clinical guideline, a peer-reviewed paper with DOI, or an established registry data dictionary. Never take a phenotype threshold from the model's prior or a reference from memory. Before finalizing, run /verify-refs on the accumulated BibTeX to confirm every citation exists in PubMed/CrossRef. A choice with no canonical source is flagged Ad-hoc: yes, justified in 1-2 sentences, and confirmed by the user before it propagates into /write-protocol or /analyze-stats.
Output Template
Write {project_root}/variable_operationalization.md (or the path the user specifies) from templates/variable_operationalization.md. Required structure:
-
Header: research question, cohort type, date, author
-
Operationalization table — one row per variable:
| Variable | Role | Dict. sheet & row | Dict. verbatim | Canonical source | Definition | Cutoff | DB vars | Implementation | Ad-hoc? |
Role: exposure / outcome / covariate / eligibilityDict. sheet & row: e.g.5-1.복부초음파 r12— mandatory if a DB dictionary existsDict. verbatim: full code→meaning string copied from the dictionary — mandatory under the same conditionCanonical source: BibTeX key (e.g.,@rinella2023_aasld_masld), so downstream skills can re-verifyDefinition: one line, verbatim from the guideline where possibleCutoff: numeric + unitsDB vars: exact dictionary column names usedImplementation: SQL/pandas-style pseudocode (e.g.,bmi>=25 & (b_tg>=150 | b_hdl<40))Ad-hoc?: yes/no. If yes, justification below the table
-
Ad-hoc justifications — for each yes row
-
Mapping gaps — variables in the protocol with no DB equivalent; list proxy / omit / request decisions
-
References — BibTeX block
Out of scope: statistical analysis → /analyze-stats; manuscript drafting → /write-paper; data cleaning / missingness → /clean-data; sample size → /calc-sample-size.
Failure Modes to Avoid
- Column-first framing — starting from what columns exist, then picking a definition that matches. Always flip: definition first, then map. Tier 0 still applies once a column is picked: quote its codebook entry verbatim before using its values.
- Cutoff drift — using a different cutoff than the cited guideline without justification (e.g., BMI≥23 cited as WHO Asian while text says ≥25).
- Mixing eras — 2020 MAFLD criteria with 2023 MASLD criteria in the same analysis. Pick one and note why.
- Dose/duration structural-missingness — operationalizing a dose/duration covariate (pack-years, cessation-years, alcohol grams/week) anchored to a categorical exposure (smoking status, alcohol use) without specifying what the reference level (never-smoker, never-drinker) does to the dose. A never-smoker's pack-years is a structural zero, not a missing value; conflating the two collapses the analytic sample under complete-case modeling and lets MICE fabricate a non-zero dose for the unexposed. Operationalize it explicitly — add a row with
Role = covariateandImplementation = "IF status == 'never' THEN dose = 0 ELSE measured_value"— and adjust on the categorical status variable, reserving the continuous dose for an exposed-only secondary analysis./clean-data(categorical-implied-zero flag) and/analyze-stats("Covariate Pitfalls") enforce this downstream.
De-identification Skill
You are guiding a medical researcher through data de-identification. The actual de-identification is performed by a standalone Python script that runs WITHOUT any LLM. Your role is to explain, guide, and verify — not to see or process raw PHI data.
Critical Safety Rules
- NEVER ask the user to paste, show, or upload raw data containing PHI. The script processes data locally. You never need to see patient-level data.
- NEVER read or display the mapping file contents. It contains original PHI values.
- You may read the scan and reviewed reports (column classifications; no cell values) and the audit log (keyed hashes; no original values). Read the de-identified output only after the researcher has run the review and confirmed it; it has the identifiers they chose to anonymize removed, and nothing more (see "What the tool does not do").
- Always communicate in the user's preferred language about the process, but use English for technical terms (PHI, HIPAA, Safe Harbor, etc.).
- The researcher runs the script in their own terminal, not through you (no
!prefix, no Bash tool call): the review prints sample values of every column.
Reference Files
${CLAUDE_SKILL_DIR}/references/hipaa_18_identifiers.md— HIPAA Safe Harbor checklist${CLAUDE_SKILL_DIR}/references/korean_phi_patterns.md— Korean-specific regex patterns${CLAUDE_SKILL_DIR}/references/date_shift_guide.md— Date shifting best practices
Read relevant references before advising the researcher.
Prerequisites
- Python 3.9+
openpyxl(for .xlsx files):pip install openpyxl- Supported formats: CSV, TSV, Excel (.xlsx)
Five-Phase Workflow
Phase 1: Assessment
Ask the researcher:
- What file format is the data? (CSV, Excel, etc.)
- What PHI do you expect in the data? (names, dates, IDs, etc.)
- Does your IRB require specific de-identification documentation?
- Do you need to re-identify later? (affects mapping file choice)
Based on answers, recommend the appropriate command:
- Full pipeline (most common):
python3 deidentify.py full <file> --locale <code> - Step-by-step (cautious):
python3 deidentify.py scan <file> --locale <code>first
Available locale codes: kr (Korea), us (USA), jp (Japan), cn (China), de (Germany),
uk (United Kingdom), fr (France), ca (Canada), au (Australia), in (India), it (Italy), es (Spain).
If --locale is omitted, the script shows an interactive country selection menu.
Users can provide a custom locale file via --locale-file custom.json.
Phase 2: Script Execution
Guide the researcher to run the script. The script is located at:
${CLAUDE_SKILL_DIR}/deidentify.py
Full pipeline (recommended for most users):
python3 ${CLAUDE_SKILL_DIR}/deidentify.py full data.xlsx \
--locale kr \
--output-dir ./deidentified/
Step-by-step (for careful review):
# Step 1: Scan
python3 ${CLAUDE_SKILL_DIR}/deidentify.py scan data.xlsx --locale kr --output-dir ./deidentified/
# Step 2: Review (interactive)
python3 ${CLAUDE_SKILL_DIR}/deidentify.py review ./deidentified/scan_report.json
# Step 3: Apply (refuses a report that was not reviewed, has a column without a decision,
# or was made from data that has changed since: edit the file, then scan and review again)
python3 ${CLAUDE_SKILL_DIR}/deidentify.py apply ./deidentified/reviewed_report.json
Options:
--locale CODE: Country locale for PHI patterns (kr, us, jp, cn, de, uk, fr, ca, au, in, it, es)--locale-file PATH: Custom locale JSON file (copylocales/_template.jsonto create one)--auto-accept-safe: Keep SAFE columns without showing them. Not recommended: SAFE means no pattern matched, not that the column holds no identifiers (a name typed into a short comment matches no pattern), and this option means nobody looks at those columns--hash-mapping: Store unkeyed SHA-256 hashes instead of original names/IDs in the mapping file. Dates and numeric IDs can be recovered from such hashes by trying every candidate, so mapping.json stays restricted either way--mapping-in PATH(apply/full): Reuse a previous run's mapping.json, so values already de-identified keep the same pseudonym, ID, and date offset. Without it, every run numbers from 0001: merging outputs from separate runs silently gives two patients the same pseudonym (and one patient different ones). A--hash-mappingfile cannot be reused this way, since its keys are hashes, not original values. An imported mapping is validated before use: all three sections must be present, replacement values must be P/ID-plus-digits, unique within their section, and different from the original they replace, duplicate JSON keys are rejected, and date offsets must be within 1..365 days (never 0). Row-keyed offsets from a run with no patient key are not carried across runs, and a batch with nothing to anonymize keeps the prior mapping instead of erasing it. With or without it, a new pseudonym or ID skips every original value in the anonymized columns, the patient key column and the prior mapping, so a source ID already shaped likeID0002is never written out unchanged. An earlier batch cannot see values that arrive later: if source IDs share the P/ID-plus-digits form, an earlier batch's pseudonym can still equal a later batch's original value--output-dir: Where to save de-identified file, mapping, and audit log-v/--verbose: Enable debug logging
Phase 3: Interactive Review Guidance
The script's terminal review has three passes:
- Pass 1 — Column Classification: Each column is shown as PHI / REVIEW_NEEDED / SAFE, with sample values. The researcher confirms or overrides each classification.
- Pass 2 — Undecided Items: Columns that weren't resolved in Pass 1 get a second look with more sample values displayed.
- Pass 3 — Final Summary: A table of all planned actions. The researcher can edit individual decisions before confirming.
- Patient key (only when dates will be shifted): the researcher picks the column that
identifies the patient, or
rowif every row is a different patient. Each patient gets their own offset; without a key the tool does not shift dates.
Coach the researcher. Deliver these prompts in the researcher's preferred language:
- "Columns classified as PHI are anonymized by default. Press 'k' to keep the original value."
- "REVIEW_NEEDED are columns the script could not vouch for: free text, a PHI word inside a longer column name, ID-like numbers, a rare address. Read the sample values and type 'a' or 'k' — Enter alone is not accepted for these."
- "SAFE means no pattern matched, not that the column is free of identifiers. Read its sample values too, and press 'r' if a column holds names or anything else identifying."
- For free-text columns, 'a' replaces each whole text with
[REDACTED]: the script cannot find a name inside a sentence, so it does not try to keep the rest of the text.
Phase 4: Verify and Document
After the script completes, help the researcher verify:
-
Read the audit log (no original values;
before_hashis an HMAC-SHA256 under a per-run key that is kept only in mapping.json):cat ./deidentified/audit_log.csv | head -20Verify the number of changes, affected columns, and PHI types.
-
Ask the researcher to spot-check the de-identified file first, in their own terminal: pseudonyms (P0001, etc.), shifted dates and [REDACTED] markers where expected, and no names in the columns they kept. Read it yourself only after they confirm.
-
Check that sensitive columns are actually removed: Verify no original names, phone numbers, or RRN values remain.
-
Mapping file security:
- Remind the researcher: "mapping.json contains original patient identifiers — treat it as restricted."
- Recommend storing it separately from the de-identified data
- File permissions are automatically set to 0600 (owner-only)
Phase 5: Documentation
Generate a de-identification methods paragraph for the manuscript or IRB:
Template:
Direct identifiers were removed from the dataset prior to analysis using a rule-based de-identification tool (deidentify.py, medsci-skills) with the [COUNTRY] locale pattern pack. The tool scanned column names and cell values using regex patterns for country-specific identifiers (e.g., national ID numbers, phone numbers), email addresses, dates, and addresses. Each column classification was reviewed by the researcher in an interactive terminal session. Names were replaced with pseudonyms (P0001, P0002, ...), dates were shifted by a random per-patient offset (1-365 days, either direction) preserving relative temporal intervals, and direct identifiers (phone numbers, email addresses, national ID numbers) were suppressed. A total of [N] cells across [M] columns were de-identified. The de-identification mapping file was stored separately under restricted access (file permissions 0600).
Customize based on the actual audit log statistics. Do not call the dataset "de-identified under HIPAA Safe Harbor" (or anonymised under another law) unless the gaps listed in "What the tool does not do" were closed as well, or an expert determination covers them.
Cross-Skill Integration
- deidentify sits BEFORE
clean-datain the research pipeline - After de-identification, hand off to
/clean-datafor data quality profiling /analyze-statscan safely process the de-identified output/write-paperMethods section should reference the de-identification process/write-protocolcan use the HIPAA/PIPA reference files for protocol documentation
Output Files
| File | Contains PHI? | Safe for Claude? | Purpose |
|---|---|---|---|
*_deidentified.xlsx/csv | Only what the researcher kept, and what the tool cannot detect (see below) | After the researcher confirms it | Data for analysis |
mapping.json | YES | No | Original ↔ pseudonym mapping, per-patient date offsets, audit hash key |
audit_log.csv | No original values (keyed hashes) | Yes | What was changed and where |
scan_report.json | No cell values | Yes | Column classification results |
reviewed_report.json | No cell values | Yes | Researcher-reviewed classifications and patient key column |
What the tool does not do
The tool removes or replaces the identifiers the researcher marked for anonymization. It does not by itself make a dataset HIPAA Safe Harbor de-identified, or anonymous under other laws:
- Ages over 89 are kept as they are. Safe Harbor requires ages 90 and over to be grouped (e.g. "90+"); do it before sharing.
- Shifted dates keep a day and a month. Safe Harbor removes every date element except the
year; date shifting is a different method (typically justified by an expert determination
or used within a limited data set). Dates the script cannot parse become
[DATE_SHIFTED]. - Names in text are not found. Free-text columns are REVIEW_NEEDED and, if anonymized, replaced as a whole. A name in a short comment column can still be classified SAFE: the review shows every column's sample values so the researcher can catch it.
- Rare combinations are not assessed. Small cells, rare diagnoses, or quasi-identifiers (age + sex + ZIP + admission month) can identify a patient; no k-anonymity check is run.
- Geography is only as fine as the patterns. Addresses and postcodes are detected per locale; truncating ZIP codes to three digits (Safe Harbor) is not done.
- Known limits — bare 5-digit US ZIP codes: a ZIP column is caught by its name (
zip,zipcode,zip_code) or by the ZIP+4 form (94110-1234). A bare 5-digit value cannot be told apart from other 5-digit codes (e.g. procedure codes), so a ZIP column under another name can be classified SAFE: check the sample values in review. - Known limits — dates: these spellings are still classified SAFE: month-first with a
dash (
Mar-15-2024), year-first with a month word (2024-Mar-15), month and year only (March 2024), non-English month words (15 mars 2024,15. März 2024), and two-digit-year day/month dates outside theuslocale (15/03/24underuk). Underus, anyM/D/YY-shaped value (e.g.1/2/10) is treated as a date, so a column of such three-part numbers is flagged PHI/date. - Not detected: URLs, IP addresses, account, licence, vehicle and device numbers (fax numbers only where they look like the locale's phone numbers), and identifiers written in column names or file names. Only the first sheet of an Excel file is processed (the other sheets are not written to the output).
Scope and Limitations
Supported (v1):
- Structured tabular data: CSV, TSV, Excel (.xlsx)
- 12 country locales with country-specific PHI patterns:
- Korea (kr): RRN (주민번호), phone, email, address, Hangul names, dates
- USA (us): SSN, US phone, US address, zip codes
- Japan (jp): マイナンバー, Japanese phone, 都道府県 address, Kanji names
- China (cn): 身份证号, Chinese phone, 省市区 address, Chinese names
- Germany (de): Steuer-ID, German phone, Straße address
- UK (uk): NHS Number, NI Number, UK phone, postcodes
- France (fr): NIR/INSEE, French phone, Rue address
- Canada (ca): SIN, Canadian phone, postal codes
- Australia (au): TFN, Medicare number, AU phone
- India (in): Aadhaar, PAN, Indian phone, pin codes
- Italy (it): Codice Fiscale, Italian phone, Via address
- Spain (es): DNI/NIE, Spanish phone, Calle address
- Universal patterns (all locales): email, ISO dates, high-cardinality numeric IDs (MRN)
- English column names recognized across all locales
- Custom locale support via
--locale-filewith template - Pseudonymization, date shifting, ID replacement, suppression
NOT supported (planned for v2):
- DICOM image metadata (PS3.15 Annex E) — requires pydicom
- Clinical free-text NER (clinical notes, radiology reports)
- Automated k-anonymity / l-diversity assessment
- SPSS (.sav), SAS (.sas7bdat), or other statistical formats
Design-AI-Benchmarking Skill
Standard Output
## AI-Benchmark Design Review
Evaluation question: ...
Arms / systems compared: ...
Reference (human-expert panel): ...
Unit of rating: (item / case / output)
### Rubric (decoupled dimensions)
- dimension -> construct -> anchors (1..k)
### Calibration probes (blinded, randomized)
- positive-control / known-bad / instability / mechanism-contradiction
### Reviewer panel
- n reviewers, metadata captured, per-reviewer randomized order
### Reliability plan
- IRR target on the anchor set of real items (ICC form + 95% CI) + control-item hit rate (a competence check, reported separately)
### Judge strategy
- human-as-judge / LLM-as-judge / both + adjudication rule
### Validity risks
1. ...
### Minimal fixes
- ...
### Decision
- Ready to collect / Needs rubric revision / Needs arm or judge redesign
Reviewer ratings, reference labels, probe outcomes and agreement statistics come only from collected
rating records. Never invent them: a reported ICC, kappa or score with no underlying rating record is
the failure this skill exists to prevent. Cite a reference only with a /search-lit-confirmed DOI or
PMID; mark any other [UNVERIFIED - NEEDS MANUAL CHECK]. Flag an unconfirmed clinical definition,
diagnostic criterion or guideline recommendation [VERIFY] and ask the user.
Workflow
Phase 1: Define the evaluation question and arms
Pin down, in writing:
- the exact claim the benchmark must support (e.g., "system A's outputs are perceptually indistinguishable from expert outputs", not "system A is deployment-ready")
- every arm/system and what each receives as input (same items, same information access, same output format), so no arm has a hidden advantage
- the human-expert reference: who they are, and whether they set ground truth, form a comparison arm, or both
- the unit of rating (item, case, output) and how many units each reviewer sees
Gate: Present the reconstructed evaluation question, arms, and reference to the user and confirm before designing the rubric. A wrong reconstruction misdirects the entire benchmark.
Phase 2: Design a decoupled multi-dimensional rubric
- Decouple the axes. Each rated dimension measures one construct. Keep "is the output valid/correct" separate from "is it novel", "is it feasible/measurable", "does it add value over current tools", and "would it change action". A candidate can be high-validity yet low-added-value ("real but redundant"); a single blended score hides this.
- Anchor every scale point with a short verbal descriptor; pilot the anchors with at least one reviewer before locking.
- Pre-specify discriminant validity: hypothesize which dimensions should correlate vs be orthogonal, then report the full inter-dimension correlation matrix.
- Start from
${CLAUDE_SKILL_DIR}/references/elicitation_rubric_template.md(dimensions, anchors, probe flavors).
Phase 3: Insert and randomize calibration probes
Plant a few deliberate control items, blinded and randomized across raters (record who received which
via a probe_arm flag), to anchor the scale, measure rater drift/fatigue, and audit the rubric and
pipeline. Four flavors:
- Positive control / "too-good" item — near-tautological; tests whether raters equate "largest effect" with "best", and whether the construct-independence gate (Phase 7) works.
- Known-bad negative control — an engineered defect (fabricated reference, missing key statistic).
- Instability item — an estimate that reverses or fails to replicate on a holdout.
- Mechanism-contradiction item — an empirical direction that opposes the proposed mechanism.
Probes are planted or adjudicated, never fabricated to fit a hypothesis.
Phase 4: Construct the reviewer panel
- Recruit reviewers spanning the intended expertise gradient; pre-specify any stratification.
- Capture reviewer metadata (years of experience, prior AI-evaluation experience, subspecialty).
- Randomize item order per reviewer (not one global seed), record the order, and plan to analyze order and fatigue effects.
- Require each item to be judged standalone; cross-item references in free text signal non-independent rating.
Gate: Present the panel composition, stratification, and randomization plan for user review before recruitment is finalized.
Phase 5: Set inter-rater reliability targets
- Pre-specify the reliability statistic and its form — for several raters, an ICC with the model stated (e.g. ICC(2,1): two-way random, absolute agreement, single rater) or Krippendorff's α (ordinal); weighted (Cohen's) kappa is a two-rater statistic — and a justified target. Size the number of items for the CI width you need: ICC precision is driven mainly by the item count (Koo & Li, J Chiropr Med 2016).
- Planted control items are a rater-competence check, not reliability evidence. Report the hit rate on known-good vs known-bad controls. Do not present an ICC computed on the controls as evidence of rubric or scale validity: ICC scales with the spread between items, controls are extreme by construction, so their ICC is high whatever the raters' reliability on real items (same five raters, same noise: ICC(2,1) 0.89 on four planted controls, 0.16 on 40 real items), and it rests on a handful of items.
- Report reliability on the representative anchor set of real items — ICC with its 95% CI and stated form — with an agreement measure alongside (exact agreement, SEM), since reliability depends on how heterogeneous the rated items are (de Vet et al., J Clin Epidemiol 2006). A low ICC on real items with a good control hit rate points to little true spread among the items or a rubric that does not separate them, not to careless raters.
- Plan the minimum ratings-per-item for a stable agreement estimate (the math goes to
/analyze-stats).
Phase 5b: Reader allocation under burden constraints (anchor-and-rotate)
When the item pool exceeds what one reader can rate in a session, do not make every reader rate
every item (that caps the pool at the per-reader limit). Use anchor-and-rotate (an anchor set
plus rotating incomplete blocks — not a balanced incomplete block design, since reader pairs are
not co-rated equally often): all readers rate a shared anchor set of real items (which carries
the inter-rater reliability; the planted controls carry only the competence check) plus a
rotating unique block each. The binding
constraint is usually the number of available expert readers, so solve the reverse problem (largest
pool for R readers) to size the must-rate set. Pre-specify anchor membership, raters-per-item, and the
rotation seed before rating. Read ${CLAUDE_SKILL_DIR}/references/anchor_rotate_reader_allocation.md
for the formulas, trade-offs, and a stdlib implementation.
Phase 6: Choose the judge strategy and adjudication
- Decide human-as-judge, LLM-as-judge, or both. An LLM judge is one more arm whose ratings must be validated against the human panel on the anchor set of real items, not only on the extreme control items.
- Pre-specify the adjudication rule for disagreement (majority, a third senior reviewer, consensus discussion) and who adjudicates.
- Blind judges to arm identity wherever feasible; record any unavoidable unblinding.
Phase 7: Construct-independence and leakage guards
- Exclude any predictor or input that is a definitional component of the outcome, and flag near-tautological composites built from the outcome's defining components — they produce an inflated, near-circular result and belong as labeled probes, not discoveries.
- Verify no arm sees post-decision or outcome-derived information the others do not.
- Confirm the reference labels were not derived from the same model output being evaluated.
Phase 8: Lock a structured export schema
Write the machine-readable rating record as a JSON schema, starting from
${CLAUDE_SKILL_DIR}/references/benchmark_export_schema.json: per-item ratings on every rubric
dimension, free-text justifications, follow-up flags, the probe_arm flag, reviewer id and metadata,
item order, and timing.
Gate: Present the final rubric, probe set, panel plan, judge strategy, and export schema together; collect explicit user approval before any rating begins, because changes after collection starts compromise the comparison.
Handoff Rules
- route to
/analyze-statsfor ICC (stated form) / Krippendorff's α / kappa (two raters) / DeLong, agreement sample size, and effect-size real-world translation of the benchmark results - route to
/check-reportingfor STARD-AI, CLAIM, or TRIPOD+AI item-level reporting once the design is locked - route to
/design-studywhen the broader study around the benchmark (cohort logic, analysis unit, comparator) also needs review - route to
/peer-reviewor/self-reviewonly after ratings exist and a manuscript is being assessed
Design-Study Skill
Core Review Questions
Always inspect:
- The exact research question.
- The analysis unit: patient, lesion, exam, study, phase, report.
- The index date or decision point.
- How inclusion and exclusion criteria are applied.
- Any information leakage.
- The reference standard or endpoint definition.
- A clinically meaningful comparator.
- The validation strategy.
- The uncertainty reporting required.
- The best-fitting reporting guideline.
- Whether exposure/outcome/covariate definitions are literature-grounded or invented ad hoc from the data dictionary. If ad hoc, defer to
/define-variablesbefore drafting Methods.
Standard Output
## Study Design Review
Question: ...
Study type: ...
Analysis unit: ...
Index date / prediction timepoint: ...
### Strengths
- ...
### Major validity risks
1. ...
2. ...
### Minimal fixes
- ...
### Reporting fit
- Recommended guideline: ...
### Decision
- Ready for analysis / Needs redesign / Drafting can proceed with limitations
Cite a reference only with a /search-lit-confirmed DOI or PMID; mark any other [UNVERIFIED - NEEDS MANUAL CHECK]. Never invent a clinical definition, diagnostic criterion, or guideline recommendation: flag an unconfirmed one [VERIFY] and ask the user.
Workflow
Phase 1: Reconstruct the study
Extract from the protocol, draft, slides, tables, or notes: clinical problem, intended use case, population, inputs, outputs, outcome definition, and timing of variable availability.
Gate: Present the reconstructed study summary (question, analysis unit, intended use) to the user and confirm it before proceeding — a wrong reconstruction misdirects the entire validity review.
Phase 2: Check structural validity
Read a reference only when its condition holds:
| File | Read it when |
|---|---|
references/dag_adjustment.md | confounding control needs an explicit adjustment set |
references/target_trial_emulation.md | the design emulates a target trial |
references/venue_accept_recipe.md | a clinical DL / AI-validation study must decide which venue tier the achievable design can be accepted at, and the one design move that reaches the tier above (the bridge into /find-journal); skip when there is no publication-tier decision |
references/combine_models_ablation_design.md | the model is built by combining / adapting / fine-tuning existing models (nnU-Net, TotalSegmentator, SAM/MedSAM, a pretrained backbone) and the comparator must be designed as an ablation; skip for a model trained de novo |
references/multi_model_comparison_design.md | the contribution is comparing several models / architectures head-to-head and the comparison must be fair; skip for a single-model study (one model's ablation → the row above; AI-vs-human → /design-ai-benchmarking) |
references/segmentation_failure_characterization_design.md | the claim is that a segmentation model is clinically usable, not that it scores well; skip when the endpoint is benchmark accuracy (metric choice → /model-assessment; abstention / risk–coverage → /model-assessment) |
A. Analysis unit
Look for mismatches: a patient-level claim from lesion-level analysis; an exam-level split with patient overlap; phase-level samples treated as independent.
B. Leakage
Look for:
- postoperative features used for preoperative prediction
- normalization or thresholding performed before the data split
- repeated exams across train/test
- reader annotations derived from outcome information
- input-text contamination for NLP/LLM extraction tasks: if the input includes clinical history, indication, impression, prior diagnosis, or referral text, confirm it does not name or strongly imply the target label. If it does, the task is information retrieval under label leakage, not phenotype inference: redesign the input mask, report a sensitivity analysis excluding those fields, or reframe.
- construct dependence (a predictor that is a definitional component of the outcome): (i) an input that computes the outcome (fasting insulin and glucose when the outcome is HOMA-IR), or (ii) a near-tautological composite of the outcome's defining components (inflated, near-circular association). Test: "could this predictor be derived, in whole or part, from the outcome's definition or the same measurement?" If yes, exclude it, or keep it only as a labeled calibration probe.
F. Time origin & survivorship (incident / transition models)
For any time-to-event or incident/transition design, check before drafting:
- Time origin per model. Each incident model starts its at-risk clock at the correct origin. Watch for immortal-time bias (a span in which the event cannot occur, misattributed to one group) and left-truncation / delayed entry.
- Mediator-ascertainment-window survivorship. A "progressor" / transition label conditional on surviving to a later ascertainment (a second scan, a follow-up visit) is survivorship-biased; plan a landmark time or an explicit intermediate-state (multistate / illness-death) model.
- Primary-analysis-set selection. If the primary is not the full cohort, pre-specify why, from the missingness mechanism relative to the outcome: complete-case regression is unbiased when missingness is independent of the outcome given the covariates (even with a covariate MNAR) and generally biased when it depends on the outcome, including under MAR, where multiple imputation is the usual primary (Hughes et al., Int J Epidemiol 2019; logistic-regression odds ratios are the exception when missingness depends on the outcome alone). Never pick it for significance.
- A design that cannot yet answer these should say so — but at review time a Methods/Limitations admission that it was "not formally assessed" is escalated to MAJOR by the survival probe (S1).
C. Reference standard
Check who established ground truth, when, whether blinding was possible, and whether only a subset had gold-standard verification. Then:
- Construct ↔ nominal-definition match. Restate each construct's nominal definition and confirm every included case satisfies it. An "incidentaloma" defined as an indeterminate finding must not include frank malignancy reads; a label that overshoots its definition inflates the cohort and breaks the κ.
- Per-flag reference-standard concordance. Report concordance per flag category, not only overall; a construct where most flags (e.g., ~86%) miss the reference standard measures something else.
- Manuscript definition ↔
variable_operationalization.md. Methods definitions must match the/define-variablesoperationalization table verbatim, and a blinded re-classification form must quote the analytic protocol's definition verbatim. A paraphrase or "common-sense extension" in the form is the documented cause of a low κ that is a definition mismatch, not real disagreement.
D. Validation
Classify: apparent only, internal split, cross-validation, temporal, external, or multi-center external.
E. Reader / expert-elicitation studies
When the study elicits expert ratings — a reader study, an annotation panel, an AI-output evaluation —
read references/reader_elicitation_design.md before data collection. The acceptance ceiling of a
perceptual / reader AI study is fixed at design time: no quality of execution lifts a ceiling baked
into the comparator, the estimand, or the reader cohort. For an AI-system-versus-human-expert
benchmark, route to /design-ai-benchmarking (arm definition, LLM-as-judge versus human-as-judge
adjudication, structured export schema).
Phase 3: Clinical framing
Ask whether the comparator and endpoint support the stated claim:
- Is the model better than current practice, or just another model?
- Is the endpoint clinically meaningful?
- Does performance translate to action?
- incremental value: if the model/marker is framed as adding value beyond / on top of /
incremental to an existing tool (a clinical score, a routine test, a baseline model), pre-specify
the baseline comparator built from the in-routine-use predictors and how added value will be
shown: test it with the likelihood-ratio (or Wald) test of the new term in the development data;
report ΔC-index / ΔAUC with a paired CI as the size of the gain, not as a second test (a DeLong
test of nested models fitted and evaluated on the same data is invalid; DeLong is for two models
frozen before a held-out evaluation); prefer the change in decision-curve net benefit at a
pre-specified threshold; NRI only as a pre-specified categorical NRI reported separately for
events and non-events, never the continuous NRI (analyze-stats
incremental_value.md; Kerr et al., Epidemiology 2014). A standalone discrimination number does not support a "beyond X" claim, and the nested-model comparison cannot be added post hoc without the baseline model. - fine-tuning contribution baseline: if an NLP/LLM study claims that fine-tuning, LoRA, prompt
engineering, or a multi-agent wrapper improves extraction/classification, pre-specify a
same-backbone zero-shot or few-shot comparator on the identical input, output schema, and test
split; a weaker or unrelated baseline cannot show the adaptation adds value. For imaging:
- a model built by combining / adapting / fine-tuning existing models → the full ablation ladder
(un-adapted base, best single component, direct-train vs transfer), per
references/combine_models_ablation_design.md; - a head-to-head comparison of several models → comparison fairness: one frozen
split/preprocessing through every model, a strong fairly-tuned baseline, a matched (or disclosed)
compute budget, and a paired delta test, per
references/multi_model_comparison_design.md; - a claim that a segmentation model is clinically usable → a pre-specified failure taxonomy, an
acceptability endpoint with a named judge and adjudication rule, the tail beside the mean, and edit
effort paired against manual-from-scratch, per
references/segmentation_failure_characterization_design.md. A mean DSC cannot be converted into a usability claim after the fact.
- a model built by combining / adapting / fine-tuning existing models → the full ablation ladder
(un-adapted base, best single component, direct-train vs transfer), per
- endpoint↔conclusion scope: decide up front what kind of conclusion the design can support. A
cross-sectional / single-visit / prevalence design cannot support a prognostic or surveillance claim
(rescreen interval, disease progression) — that needs longitudinal follow-up. A binary surrogate
endpoint (present/absent, >0, dichotomized) is risk stratification, not a patient-care directive
(defer/withhold/initiate therapy). At review time
/self-review§D +check_scope_coherence.pyflagCROSS_SECTIONAL_PROGNOSTIC/SURROGATE_CARE_DIRECTIVEagainst the conclusion.
Phase 4: Reporting fit
Recommend one primary guideline — TRIPOD+AI, CLAIM, STARD, STROBE (TARGET for a
target-trial emulation), PRISMA, CARE, or ARRIVE — plus journal-specific additions if needed.
Frequent Failure Modes
Diagnostic AI
No clinically relevant comparator; exam-level instead of patient-level split; unclear reference standard; AUROC-only reporting without threshold metrics.
Prognostic modeling
Unclear time zero; immortal time bias; feature timing mismatch; no calibration.
Retrospective cohort / screening database
- Time zero misalignment: cohort entry ≠ follow-up start → immortal time bias.
- Interval-censored outcomes treated as exact → underestimated event times.
- Unacknowledged healthy-volunteer bias → inflated external-validity claims.
- Surveillance bias from unequal follow-up frequency between groups.
- Map each threat explicitly to one of the three bias classes (Hernán/Robins): selection bias (who enters), information bias (how measured), confounding (what else differs).
- Comparative / causal question → emulate a target trial. For a treatment-vs-treatment,
screening-vs-no-screening, or drug-A-vs-drug-B question on routinely-collected data, specify the
seven target-trial components (eligibility, strategies, assignment, time zero, outcome, causal
contrast, analysis plan) before extraction. This prevents immortal-time, prevalent-user, and
confounding-by-indication bias and turns an association into a defensible causal contrast.
New-user + active-comparator design, grace-period clone-censor-weight, and negative controls are in
references/target_trial_emulation.md. - Confounding completeness: pre-specify the adjustment set from a DAG (not a Table-1 p < 0.05
rule), and plan an extended-adjustment sensitivity model to show whether any measured covariate that
turns out imbalanced by exposure but lies outside the adjustment set leaves the primary estimate
robust. Pre-screen the proposed covariates with
scripts/adjustment_set_helper.py(flags mediator / collider / descendant adjustment and omitted confounders, and proposes a candidate backdoor set), then derive the minimal sufficient set with dagitty — seereferences/dag_adjustment.md. At review time/self-reviewPhase 2.5e and the O-probes inobservational_confounding.md(O1–O18) check this against Table 1 — including O7 over-adjustment, O10 overlapping-subset-gradient discipline, O11 design-based weighting for complex-survey data, O12 data-driven-threshold mining, O13 (a cross-sectional mediation claim cannot order X→M→Y), and O14 (a synergy/joint-effect claim needs the additive interaction scale — RERI/AP/S — not a multiplicative-only test).
Multimodal LLM / report generation
No clear rubric for clinical correctness; benchmark labels derived from noisy reports without adjudication; unsupported claims about safety or workflow benefit; input text containing the target label or diagnosis being predicted; no same-backbone zero-shot/few-shot baseline for a fine-tuning or prompt-engineering claim.
Imaging meta-analysis
Overlapping cohorts; paired modalities analyzed as independent; heterogeneity metrics missing; zero-cell handling unspecified.
Minimal-Fix Principle
Recommend the smallest feasible repair first: clarify the claim, narrow the target population, add a limitation statement, add a clinically relevant baseline, re-run one key sensitivity analysis, or redefine the endpoint more explicitly. Escalate to redesign only when the central claim is not defensible otherwise.
Handoff Rules
- route to
analyze-statswhen the design is basically sound but analysis details need refinement - route to
check-reportingafter the design is locked - route to
self-reviewwhen the user wants a pre-submission quality check on their own manuscript - route back to
write-paperonly after the main validity risks are documented
This skill does not compute statistics, draft full manuscript prose, resolve raw data-engineering issues, or replace a full peer review when journal-facing tone is required.
/explainability was renamed to /model-assessment in v6.0; this redirect is removed in v7.
Run /model-assessment now, passing the same arguments the user gave /explainability.
Start at /model-assessment Part D, Phases 12–13 (saliency sanity checks, localisation, attribution framing).
Fill-ICMJE-COI Skill
The official ICMJE coi_disclosure.docx keeps every field inside Word Content Controls (SDTs),
which python-docx cell.text silently ignores. The script therefore does literal-string
replacement in word/document.xml, which only works when the target strings already exist — hence
the shipped pre-filled synthetic seed.
Core Principles (Do Not Violate)
- Never author SDT XML from scratch. Only replace existing strings in an already-populated seed, because programmatic Content Controls are fragile and Word-version-dependent. If an error seems to require altering the seed structure, stop and escalate to the user.
- Never use or commit a seed that contains a real author's data. The shipped
templates/icmje_coi_seed_synthetic.docxis PII-scrubbed (synthetic name, title, date; metadataICMJE/Anonymous); real-person seeds leak PII through bothdocument.xmlanddocProps. Keep custom seeds with real names in private per-project directories, and before promoting any seed checkunzip -p seed.docx docProps/core.xmlfor real names. - Never modify the 13 disclosure items or the certification checkbox. The script replaces only Date, Name and Title; the ☒ None entries come from the seed unchanged. Never say the script "handled the disclosures": it cloned the seed, and no author-specific disclosure reasoning happened. An author with a real disclosure edits their own form in Word.
- The seed must be pre-filled (all-None ☒ + text). The script does not work on a blank ICMJE template.
Skip this skill when the target journal uses its own declaration form (Elsevier Declaration of Interest, a BMJ ICMJE derivative, etc.) rather than the canonical ICMJE form — check the author guidelines first.
Execution
Phase 1 — Intake
Ask the user (or extract from the conversation):
- Manuscript title (exact, as on the title page)
- Submission date (e.g., "April 20, 2026")
- Author list — ordered, one name per slot:
[(1, "Author One"), (2, "Author Two"), ...], taken verbatim from the manuscript title page or the user's list - Output directory — typically
submission/{journal}/icmje_forms/
Gate 1 — user approval. Present the intake back before generating anything. Name which authors will get all-None forms and remind the user that anyone with a real disclosure must fill their own form in Word instead.
Phase 2 — Generate
python3 ${CLAUDE_SKILL_DIR}/scripts/fill_icmje_coi.py \
--seed ${CLAUDE_SKILL_DIR}/templates/icmje_coi_seed_synthetic.docx \
--seed-name "Placeholder Author" \
--seed-title "Placeholder Manuscript Title" \
--seed-date "January 1, 2000" \
--new-title "{exact manuscript title}" \
--new-date "{submission date}" \
--out-dir {out_dir} \
--authors '[[1,"Author One"],[2,"Author Two"],...]'
It writes one ICMJE_COI_{NN}_{Name}.docx per author and exits nonzero if any seed string is not
found. It exits 2, before writing anything, if any --seed-* or --new-* value or author name is
empty: an empty seed string would match nothing and leave the seed's value in every form.
Phase 3 — Verify
For each generated docx confirm: ☒ count = 14 (13 disclosure items + final certification);
"None" disclosures = 13; the correct name after "Your Name:" and title after "Manuscript Title:";
no seed strings left. SEED must be the exact --seed-name, --seed-title and --seed-date
values used in Phase 2 (the shipped placeholders below, or a custom seed's own strings); TITLE
must be the --new-title value.
python3 - {out_dir}/*.docx <<'PY'
import sys, zipfile
SEED = ("Placeholder Author", "Placeholder Manuscript Title", "January 1, 2000") # = --seed-* values
TITLE = "{exact manuscript title}" # = --new-title
assert all(s.strip() for s in SEED) and TITLE.strip(), "empty SEED/TITLE value"
for f in sys.argv[1:]:
xml = zipfile.ZipFile(f).read("word/document.xml").decode("utf-8")
assert xml.count("☒") == 14, f"{f}: bad ☒ count"
assert xml.count(">None<") + xml.count(">None ") == 13, f"{f}: bad None count"
for ph in SEED:
assert ph not in xml, f"{f}: seed leak {ph!r}"
t = TITLE.replace("&", "&").replace("<", "<").replace(">", ">")
assert t in xml, f"{f}: title missing"
print("✓", f)
PY
This snippet does not locate the name inside the "Your Name:" field; confirm the name by opening each form.
Gate 2 — user review. Present the verification results before handing off the files.
Phase 4 — Circulation Guidance
Give the user this copy to send with each personalized form, written in the co-authors' language:
Please review the attached ICMJE COI form.
- If the contents are correct, sign and reply with a PDF.
- If a change is needed, edit/check the relevant item, sign, and reply.
- If there are no changes at all, reply "no changes" and return the signed PDF separately.
All authors can be emailed in one draft batch (e.g., gws gmail draft). Gate 3 — the user
approves the batch before anything is sent.
Custom Seeds and Seed Provenance
Read ${CLAUDE_SKILL_DIR}/references/seeds.md when the user wants a custom seed (different default wording, a common
grant pre-filled in items 2/3) or asks how the shipped seed was made.
Fill-Protocol Skill
Core Principles (Do Not Violate)
- Open the existing template — never create from scratch. Use
Document(template_path), notDocument(): rebuilding loses the header logo, custom margins, footer placeholders and page numbering. Replace cell/paragraph text in place; never a singlecell.text = "value"assignment, which erases run-level styles (bold, color, eastAsia font). - Convert .doc → .docx via LibreOffice headless before any editing.
pandoc -f docis not supported;textutilcorrupts table structure (merged cells dropped). - Match cells by left-label text, not row/column coordinates such as
table.cell(2, 1). Templates evolve and coordinate matching breaks silently. - Apply
cantSplitto every filled row so a row never breaks across pages. - For CJK languages, set the
eastAsiafont attribute, not justrun.font.name, or Hangul/Kanji/Hanzi render in fallback fonts. - Validate every fill operation: resolve every
[MISS]line and everyWARN:line ([TABLE-MISS],[SECTION-MISS],[RAW-NEWLINE]) before the filled form is submitted.
Dependencies
Python packages docxtpl python-docx pyyaml are always required. LibreOffice is needed only for a
legacy .doc template (~700 MB on macOS). The bundled setup.sh detects what is missing:
bash ${CLAUDE_SKILL_DIR}/setup.sh check # report what's installed (read-only)
bash ${CLAUDE_SKILL_DIR}/setup.sh install # install missing pieces (asks before each)
When invoking this skill on behalf of a user:
- Skip LibreOffice entirely if the template is already
.docx. - Before calling
doc_to_docx.py, runsetup.sh check. If LibreOffice is missing, ask the user before installing, because the cask is ~700 MB. - Never pass
--yestosetup.sh installunless the user has explicitly authorized unattended installation in this session, because it installs without prompting. - If the user declines, ask them to convert the
.docmanually (Word/LibreOffice/Pages → Save As → .docx) and re-run with the converted file.
Workflow
Step 1 — Convert legacy .doc to .docx (if needed)
python3 ${CLAUDE_SKILL_DIR}/scripts/doc_to_docx.py path/to/template.doc path/to/template.docx
Step 2 — Inspect the template structure
python3 ${CLAUDE_SKILL_DIR}/scripts/inspect_template.py path/to/template.docx
It lists every table, every cell (coordinates and content preview) and every top-level paragraph. Take the exact labels for the YAML from this output.
Step 3 — Author a content YAML
Start from ${CLAUDE_SKILL_DIR}/examples/example_irb_template.yaml. Three fill modes; all keys are
optional.
protections:
korean_font: "맑은 고딕" # CJK font (set to "Noto Sans CJK KR", "SimSun",
# "MS Mincho", etc. for other locales)
cant_split: true # Apply <w:cantSplit/> to every filled row
# Readability options (defaults shown)
blank_between_paragraphs: true # Enter between \n\n chunks
blank_around_section_header: true # Enter above/below filled sections
blank_around_all_section_headers: false # opt-in; also touches untouched sections
normalize_page_breaks: true # empty page-break paragraphs -> pageBreakBefore
# Mode 1 — table key/value (left-label cell → right value cell)
table_kv:
"Study Title": "Multi-center prospective validation of ..."
"Principal Investigator": "Last, First (Department)"
"연구 목적": "본 연구는 ..."
# Mode 2 — section replacement (find numbered header, replace until next header)
section_replace:
"1. Background":
"Hepatocellular carcinoma is the third leading cause of ..."
"4. 연구 배경 및 이론적 근거":
"..."
# Mode 3 — single paragraph in-place text replacement
paragraph_replace:
"Title:":
"Title: Multi-center prospective validation of ..."
Read ${CLAUDE_SKILL_DIR}/references/best_practices.md (Readability knobs) before changing any
readability option from its default.
Content rules: put a reference in only with a /search-lit-confirmed DOI or PMID, otherwise mark
it [UNVERIFIED - NEEDS MANUAL CHECK]; mark any unconfirmed clinical definition, diagnostic
criterion or guideline claim [VERIFY] and ask the user.
Step 4 — Run the fill
python3 ${CLAUDE_SKILL_DIR}/scripts/fill_form.py \
--template path/to/template.docx \
--content content.yaml \
--output path/to/filled.docx
The CLI prints [OK] / [MISS] for every fill operation, a summary, then any WARN: lines.
Investigate every [MISS] and WARN: before submitting (Principle 6).
Every value under table_kv, section_replace and paragraph_replace is written exactly as
typed (012345, 12:30, 1.10, No stay text). An empty value or a list/mapping value stops the
run with exit 2 and names the label; write '' to blank a field on purpose. A section_replace
on a header with no later numbered header replaces everything to the end of the document, and its
[OK] line says so with the number of paragraphs replaced. If a signature/date block or anything
else follows the last section, bound it with section_end (header -> regex matched against the
paragraph that starts the kept block):
section_end:
"18. References": "^Investigator signature"
Step 5 — Visual verification
soffice --headless --convert-to pdf path/to/filled.docx
Without LibreOffice (a .docx template skips it), export the PDF from Word instead. Open the PDF and confirm: page count is sensible, no table row was split across pages, no font fell
back to Times New Roman, all required fields are populated.
Read ${CLAUDE_SKILL_DIR}/references/best_practices.md when a label or section header does not
match, the template has no numbered section headers (section_end), a cell
needs multi-line content, or cells are merged.
Known Limitations
- HWP / HWPX input is not handled directly — convert it to
.docxfirst. - Merged cells: filling a label cell that participates in a vertical merge may overwrite the merged region's content. Test on a copy first.
- Embedded form fields (Word content controls): not supported; plain paragraph and table cell content only.
- Right-to-left scripts (Arabic, Hebrew): untested.
- Last section with no later header: replaced through the end of the document unless a
section_endpattern is given; the filler cannot tell the section body from a trailing block.
Find-Cohort-Gap Skill
Output directory: user-specified (default: current working directory).
Phase 0: Cohort Intake
The cohort does not have to be one this skill has heard of. Route on what the user actually has.
| The user has… | Do this |
|---|---|
| A named public cohort (NHIS, UK Biobank, KNHANES, …) | Fill the profile from published documentation. Cite the source for every field. |
| A codebook / data dictionary / CSV export of their own registry or EMR extract | Run the input adapter below. This is the common case — an institutional registry or single-centre export that no public documentation describes. |
| A review, guideline, or preprint defining the clinical domain | Attach it as domain context (--context), as a file or a URL. |
Input adapter (local codebook / documents)
python3 "${CLAUDE_SKILL_DIR}/scripts/build_cohort_profile.py" \
--codebook data_dictionary.csv \
--context narrative_review.pdf --context https://example.org/guideline \
--cohort-name "Institutional CT registry" --out-dir .
Formats: .csv / .tsv / .json / .md / .txt (stdlib), .xlsx (needs openpyxl),
.pdf (needs pdftotext). A .csv is auto-detected as a codebook (rows are
variables) or a data export (the header row is the variable list). Writes
cohort_profile.md + cohort_profile.json (+ context_extract.md).
Do not read the codebook yourself and summarise it. A paraphrased, merged, or invented
variable poisons every downstream claim — the intersection matrix, the feasibility gate, and
eventually the manuscript's Methods. The adapter enumerates variables verbatim with provenance
(file:row). Read cohort_profile.md; do not re-derive it.
The adapter infers, and shows its work for, the variable cluster map, serial /
repeated-measure groups (evidence for P1 Longitudinal Advantage), and endpoint
candidates (evidence for P2 Endpoint Upgrade). A variable matching no cluster keyword is
left unclassified — review those, since the lexicon is not exhaustive.
What the adapter cannot know — ASK, never guess
A codebook does not state any of the following; each is emitted as [UNKNOWN - ask the user]:
- Sample size (N at baseline, N with follow-up)
- Time span (enrollment period, follow-up duration, measurement intervals)
- Known limitations (healthy volunteer bias, attrition, missing-data patterns)
- Existing publications from this cohort (to avoid duplicating them)
- IRB status and data-access route
Collect these from the user before Phase 2, because a guessed N flows into the Phase 5 feasibility gate and makes it pass or fail for a reason unrelated to the cohort. Also confirm the setting (institution type, country, population type) and any special strengths the variable names cannot reveal — registry linkage, biobank availability, a distinctive population.
Gate: Present the cohort profile summary, including the [UNKNOWN] list and the
unclassified variables. Confirm before proceeding.
Phase 1: PI/CA Profiling
Profile the intended PI or corresponding author to find topic-expertise alignment. If no PI is specified, skip this phase and use variable clusters alone in Phase 2.
- Search PubMed for the PI's recent publications (last 5 years) with
/search-lit's E-utilities script:bash "${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh" search "AuthorLastName AuthorFirstInitial[Author]" 30Extract top keyword clusters from titles/abstracts. - Identify specialty signals: academic society positions (president, board member, editor), subspecialty focus areas, preferred journal tiers.
- Build a PI keyword map: 5-10 keyword clusters ranked by publication frequency.
Output: PI profile card (name, affiliation, top keywords, society roles, preferred journals).
Phase 2: Intersection Matrix
Cross cohort variable clusters with PI expertise to generate candidate topics.
Method
Create a matrix: rows = DB variable clusters, columns = PI keyword clusters. Score each cell 0-3:
- 3: PI has published in this exact intersection (direct match)
- 2: PI's subspecialty covers this area (strong relevance)
- 1: Tangential connection (possible but needs framing)
- 0: No connection
Candidate Generation
- Extract all cells scoring 2-3 as primary candidates.
- For cells scoring 1, apply the A-B substitution test: "Has someone published [this analysis] with [a different exposure/outcome] in a similar cohort?" If yes, substituting the PI's specialty variable creates a viable candidate.
- Generate 20-40 candidate topic statements in PICO format — P population from the cohort, E exposure/predictor variable(s), C comparison group, O outcome (preferably hard endpoint).
Discipline Alignment Filter
Before saturation scanning, identify the intended first author's department/specialty. The primary exposure variable must belong to that discipline (e.g., radiology first author → imaging variable as the primary exposure). Kill candidates where the primary exposure is outside the first author's discipline — a strong PI match alone is insufficient if the first author cannot claim ownership of the core variable.
Gate: Present the intersection matrix and top 20 candidates (post-discipline filter). User selects 8-12 for saturation scanning.
Phase 3: Literature Saturation Scan
For each selected candidate, determine how saturated the literature is.
Search Strategy
For each candidate:
- Build a PubMed query:
(exposure terms) AND (outcome terms) AND (cohort OR longitudinal OR prospective) - Execute search via
/search-litE-utilities. - Count total results, then ask the Critical Filter question — "Has anyone published this with serial/repeated measurements?" — and check whether a meta-analysis exists. Grade:
| Papers | Longitudinal papers | MA exists? | Grade | Interpretation |
|---|---|---|---|---|
| 0-2 | 0 | No | Blue Ocean | First report possible. Verify the topic has audience interest. |
| 3-10 | 0 | No | Green Field | Optimal zone — established interest, longitudinal gap wide open. |
| 11-30 | 0 | No | Green Field (upgraded from Yellow) | As above. |
| 1-30 | 1+ | No | Yellow | Viable only with very specific angle (unique population, novel endpoint). |
| >30 | Any | No | Yellow (borderline Red) | As above. |
| Any | Any | Yes, outdated (>5 yr) or limited scope | Yellow | As above. |
| Any | Any | Yes, recent | Red | Avoid unless doing NMA or using truly unique data. |
"So What" Test
For each candidate, articulate 2-3 potential clinical implications of the findings. If you cannot state why a clinician or policymaker would care about the result, the topic fails regardless of gap score.
Output: Saturation table with grade, paper count, longitudinal gap status, and "So What" statement for each candidate.
Gate: Present saturation results. User selects 3-5 finalists for deep scoring.
Phase 4: 6-Pattern Scoring + Comparison Table
Apply the 6-Pattern framework to each finalist. Score each pattern 0 or 1.
6 Patterns (Universal)
Read the detailed rubric at ${CLAUDE_SKILL_DIR}/references/pattern_scoring_rubric.md and score
each finalist on P1 Longitudinal Advantage, P2 Endpoint Upgrade, P3 Cohort Uniqueness,
P4 PI-Topic Alignment (skip if no PI specified), P5 Comparison Table Gaps (3+ features
unique to THIS STUDY in the table below), and P6 Complementary Design.
Comparison Table Construction
For each finalist, build a table comparing the top 3-5 existing papers against THIS STUDY,
one row per feature (design, N, serial data, hard endpoint, population, ethnicity, subgroup
analysis, …) — the rubric's P5 lists the construction steps and differentiator categories.
Cite existing papers only with a /search-lit-confirmed DOI or PMID; otherwise mark the
reference [UNVERIFIED - NEEDS MANUAL CHECK].
Score Interpretation
| Total Score | Recommendation |
|---|---|
| 5-6 | Top-tier journal target (Lancet sub, JACC, J Hepatol level) |
| 3-4 | Specialty journal target (solid publication) |
| 0-2 | Restructure or kill — find a stronger angle before proceeding |
Gate: Present scoring results and comparison tables. User approves final ranking.
Phase 5: Feasibility Gate
For each scored finalist, verify practical feasibility.
Checks
- Sample size adequacy:
- Cox and logistic regression: minimum 10 events per predictor variable (EPV rule)
- For large cohorts (N>100K) with many outcome events: warn that small effects become statistically significant (power depends on the number of events, not N — check the event count first), so focus on effect size thresholds (e.g., HR >1.2 or <0.8 for clinical relevance)
- Consider negative control strategy (EPCV) for very large samples
- Missing data: key exposure variable <20% missing acceptable; key outcome <5% missing; if serial data, assess the attrition pattern (MCAR/MAR/MNAR)
- Follow-up adequacy: the outcome must have plausible latency within available follow-up — cancer outcomes minimum 5 years, CVD events minimum 3 years, mortality minimum 5 years
- Operational definition:
- Can the exposure be defined from available variables?
- For claims data: ICD codes alone = 40-60% accuracy. Require combination strategy (diagnosis + prescription + visit frequency + special codes)
- Cross-check expected prevalence against known epidemiological data
- Flag any clinical definition, diagnostic criterion, or guideline claim you cannot
source with
[VERIFY]and ask the user.
- IRB/ethics: is the data already IRB-approved for this type of analysis? Any additional approvals needed for data linkage?
- Disease Novelty Bonus (informational, not Go/No-Go): idiopathic etiology or debated mechanism → higher journal interest; established mechanism → needs stronger methodological novelty
Decision
- Go: All checks pass.
- Conditional Go: Minor issues solvable (e.g., missing data manageable with imputation).
- No-Go: Fatal flaw (insufficient events, no valid endpoint, key variable unavailable).
Output: Feasibility report for each finalist with Go/Conditional/No-Go status.
Phase 6: Output — Ranked Proposals + One-Pagers
Ranked Summary Table
| Rank | Topic (PICO) | Saturation | 6-Pattern Score | Feasibility | Target Journal | Timeline |
|------|--------------|------------|-----------------|-------------|----------------|----------|
| 1 | ... | Green (0 longitudinal) | 5/6 | Go | JACC | 6 months |
| 2 | ... | Blue (0 papers) | 3/6 | Conditional | Radiology | 8 months |
One-Pager for Each Finalist
Fill every section of the template at ${CLAUDE_SKILL_DIR}/references/onepager_template.md;
give the target journal with its rationale (PI alignment, scope match, gap fit). Save one-pagers
as markdown files: {output_dir}/gap_proposal_{rank}_{short_topic}.md
Downstream: output feeds into /design-study → /write-paper.
Find Journal Skill
Phase 1: Input Collection
Required Inputs
- Abstract text or key findings summary
- Study type: original research, meta-analysis, case report, technical note, review, letter, AI validation, diagnostic accuracy, etc.
Optional Inputs
- Preferred tier: Q1 / Q1-Q2 / any (default: any)
- OA preference: Full OA / Hybrid OK / No preference (default: no preference)
- Field focus: radiology, medical AI, clinical specialty, methodology, education, general medicine
- Journals to exclude: journals that have already rejected this manuscript
If only an abstract is given, infer the study type from it. When called from /write-paper or
another skill, take the abstract and study type from the calling context and skip these questions.
Phase 2: Theme Extraction
From the abstract or key findings, extract: disease/condition; modality/technique; methodology (e.g., retrospective cohort, diagnostic accuracy, systematic review, RCT); population; and innovation type (new algorithm, clinical validation, workflow improvement, educational tool).
Phase 2.5: Acceptance-Readiness & Design-Ceiling Pre-flight
Editors screen importance/novelty and the design ceiling before scope fit (lack of novelty/importance is the top desk-rejection cause). This phase estimates that first filter, so the skill does not recommend a venue whose bar the design cannot clear. It is advisory, and its flags are not auto-fixable: the author decides.
2.5.1 Run the deterministic pre-flight (preferred)
If a manuscript or abstract file is available, run:
python3 ${CLAUDE_SKILL_DIR}/scripts/assess_acceptance_readiness.py <manuscript_or_abstract.md>
# add --json for a machine-readable report
It returns flags in four categories — DESIGN_CEILING, UNFIXABLE_DEFECT,
IMPORTANCE_RISK, CLAIM_MISMATCH — and a ceiling verdict
(NO LISTED SIGNAL MATCHED - NOT A DESIGN CLEARANCE … / IMPORTANCE-FRAMING REVIEW … /
SPECIALTY / TOLERANT-VENUE OR DESIGN FIX … / HIGH-IMPACT VENUE UNLIKELY …).
The script reads UTF-8 markdown or plain text only. A .docx, .pdf, .doc or other non-text
file exits 2 with no report ("No scan was run"): convert it (e.g. pandoc -t markdown) and
rerun, or fall back to 2.5.2. It skips the References/Bibliography section.
Known limits. The scan matches listed phrases, line by line. It does not understand negation or context ("Unlike prior single-center studies…" still flags single-center), and it misses paraphrases ("one institution", "has not been externally validated", the same patients in training and test sets). So an empty result is not a clearance: apply the §3 taxonomy with judgement (as in 2.5.2) before treating feasibility as uncapped, and adjudicate each flag before it demotes a venue.
2.5.2 If only pasted text is available
Read ${CLAUDE_SKILL_DIR}/references/acceptance_signals_schema.md §3 (taxonomy and verdict
bands), apply the same taxonomy to the pasted text with judgement, and assign the same ceiling
verdict.
2.5.3 Carry the verdict forward
Record the ceiling verdict and its top flags for Phase 3.2 Axis 2 and for the Phase 4 Acceptance-Readiness Summary and Cascade plan. Do not block the recommendation on a ceiling: route to a venue the design can clear, or recommend a design change or presubmission inquiry.
Phase 3: Profile Loading and Matching (2-Pass)
3.1 Pass 1: Load Compact Profiles
Read both tiers and merge them into one profile set:
# Public (shipped with the skill)
${CLAUDE_SKILL_DIR}/references/journal_profiles/*.md
# User-local private (optional, may be empty or absent)
$HOME/.claude/private-journal-profiles/find-journal/*.md
On a filename collision the private copy wins (user override). If either directory is missing or empty, continue with the other and say which tier was unavailable. Count the profiles actually loaded after the merge and state the total in the output — never hard-code the count.
From each profile parse Scope, Scope Keywords, Article Types Accepted, Classification (Tier, OA,
Field), Special Notes (includes a 1-line AI policy summary), and the optional Acceptance
Signals block (format: ${CLAUDE_SKILL_DIR}/references/acceptance_signals_schema.md §1).
Do NOT read write-paper profiles in this pass — they are 4-5x larger and carry formatting detail irrelevant to matching.
3.2 Two-Axis Scoring (scope fit × acceptance feasibility)
Score each journal on two independent axes: scope fit ("does this journal cover my topic?") and acceptance feasibility ("can this manuscript's design and importance clear this journal's bar?"). Keep them separate — a perfect scope match can still desk-reject the design.
Axis 1 — Scope fit. Compute a composite scope-fit score:
| Factor | Weight | Description |
|---|---|---|
| Scope alignment | 40% | How well the manuscript's themes match the journal's scope and keywords |
| Study type fit | 25% | Whether the journal accepts this article type and values this methodology |
| Tier match | 20% | Alignment with user's preferred tier (if specified) |
| OA match | 10% | Alignment with user's OA preference (if specified) |
| Special fit | 5% | Bonus for unique alignment with journal's Special Notes |
Axis 2 — Acceptance feasibility. Weigh the Phase 2.5 ceiling verdict against each journal's Acceptance Signals (selectivity band, desk-reject triggers, design expectations); for a profile without that block, use its Special Notes plus the Phase 2.5 taxonomy. Assign High / Medium / Low:
- Low / ceiling-mismatch when the journal's bar is one the manuscript's ceiling cannot clear
(e.g., a
highly-selectivevenue that desk-rejects single-center surrogate-endpoint designs, and the manuscript is exactly that). - High when no ceiling signal collides with the journal's stated bar.
Report a band with reasons, NEVER an acceptance probability, because no acceptance-rate data source exists and a number would be false precision (schema §4).
3.3 Filtering
Before scoring, exclude journals on the user's exclusion list and journals that do not accept the manuscript's study type (e.g., a case report to a journal that only takes original research). If no journal survives, relax the filters — OA constraint first, then tier — and re-score.
3.4 Ranking
Rank primarily by the Axis-1 scope-fit score, then apply Axis 2:
- Demote (or, if the mismatch is severe, drop below a better-feasibility peer) any journal whose acceptance feasibility is Low / ceiling-mismatch.
- Never silently demote — always carry the reason so Phase 4 can surface it.
- Select the top 5 by the feasibility-adjusted order. If a strong scope match is demoted for feasibility, still mention it in the comparison note with the mismatch spelled out (the author may choose to fix the design rather than change venue).
3.5 Pass 2: Enrich Top-5
For each top-5 journal, look for a detailed write-paper profile in both tiers (private wins on collision):
# Public
${CLAUDE_SKILL_DIR}/../write-paper/references/journal_profiles/{journal_filename}
# User-local private
$HOME/.claude/private-journal-profiles/write-paper/{journal_filename}
If found, take from it: manuscript types and word limits, abstract format, statistical reporting requirements, the full 5-field AI Writing Disclosure Policy, common rejection reasons, and Acceptance Signals (these sharpen the Axis-2 call and the cascade plan). If none is found or the directory is not accessible, use the compact profile only.
Phase 4: Output
Every journal fact you print (URLs, article types, OA model, AI policy) comes from the loaded profile or from the journal's own website; never invent journal metadata, an impact factor, an APC or a submission policy.
For each of the top 5 recommended journals, present:
### Rank [N]: [Journal Name] ([Tier])
**Scope fit:** [2-3 sentences explaining why this manuscript matches this journal's scope.
Reference specific keywords, disease areas, or methodological preferences from the profile.]
**Article types accepted:** [relevant types from profile]
**Open Access:** [Full OA / Hybrid / Subscription]
**Acceptance feasibility:** [High / Medium / Low] — [1 line: how the manuscript's
Phase 2.5 ceiling meets this journal's bar; spell out any mismatch, e.g., "scope fit
High, but this highly-selective venue desk-rejects single-center surrogate-endpoint
designs — add external validation or target a selective/accessible venue"]
**Homepage:** [URL]
**Author guidelines:** [URL]
**AI disclosure:** [Required / Recommended / Not specified] — [brief summary of permitted scope and disclosure location, if available in profile]
After all 5, add a 2-3 sentence comparison note on the key tradeoffs between the top choices (e.g., scope breadth vs. specialty depth, tier vs. acceptance feasibility). Then add the two blocks below, then the Mandatory Disclaimer.
Acceptance-Readiness Summary
---
### Acceptance-Readiness Summary
**Ceiling verdict:** [from Phase 2.5]
**Top flags:** [the 2-4 most consequential design-ceiling / unfixable / importance flags, each with its 1-line reason]
**Implication for venue tier:** [1-2 sentences — e.g., "An unfixable single-center +
surrogate-endpoint ceiling makes flagship venues unlikely without external validation;
the recommendations above are routed to venues this design can clear."]
_Advisory only — a risk band, not an acceptance prediction; flags are not auto-fixable._
Cascade plan
---
### Cascade plan (primary → reject-fallback)
1. **Primary:** [top recommendation] — [why first]
2. **If rejected:** [fallback 1] — [same-publisher transfer if applicable, else one tier
down / different scope angle]. An editor's transfer offer within the same publisher
is usually worth taking: the manuscript skips a fresh desk screen.
3. **Then:** [fallback 2]
[If the Phase 2.5 risk is IMPORTANCE rather than design, recommend a **presubmission
inquiry** to the primary target before full submission — it lets the editor quick-assess
contribution/fit and saves a desk-reject cycle.]
[If a confidential corresponding-author editor-bar overlay exists in
`$HOME/.claude/private-journal-profiles/find-journal/`, its notes have already been
merged into the per-journal feasibility calls above.]
Mandatory Disclaimer
Always append this disclaimer at the bottom of every recommendation output:
---
**Important Disclaimer**
Impact Factor, APC fees, acceptance rates, and turnaround times change frequently
and are subject to copyright restrictions. Please verify current values directly
at each journal's homepage before making your submission decision.
Recommended verification sources:
- Journal Citation Reports (JCR) via institutional access: for Impact Factor
- Journal homepage -> Author Guidelines: for current APC and formatting requirements
- Clarivate Master Journal List: for indexing status
Special Modes
Post-Rejection Mode
When the manuscript was rejected by a specific journal:
- Exclude the rejecting journal.
- Distinguish the rejection type — it changes the advice:
- Desk-reject (no peer review) — usually an importance/novelty or design-ceiling
verdict, not a fixable-revision signal. Re-run Phase 2.5; if a ceiling or importance flag
is present, the next same-tier venue will likely desk-reject again. Route one or two
tiers down (or to a tolerant/
accessiblevenue), or advise the design/importance fix first, and consider a presubmission inquiry. - Post-peer-review reject — prefer a same-publisher transfer (Springer Nature Transfer Desk / Elsevier Article Transfer Service / Wiley / Nature Portfolio) that carries the referee reports; otherwise same tier or one down, with the fatal flaw addressed.
- Desk-reject (no peer review) — usually an importance/novelty or design-ceiling
verdict, not a fixable-revision signal. Re-run Phase 2.5; if a ceiling or importance flag
is present, the next same-tier venue will likely desk-reject again. Route one or two
tiers down (or to a tolerant/
- If rejected from Q1 other than by a ceiling/importance desk-reject, recommend a mix of Q1 (different scope angle) and strong Q2.
- In each scope-fit explanation, note how the recommendation differs from the rejected journal's focus.
- Suggest scope adjustments — or, when Phase 2.5 found a ceiling, design changes — that might improve feasibility for the new target.
Case Report Mode
When the study type is case report, prioritize journals known for valuing educational or rare cases. If fewer than 5 profiled journals accept case reports, say so and suggest the user consider case-report-specific journals outside the profile set.
Submission Directory Scaffolding
When the user selects a target journal, create submission/{journal_short}/ (lowercase with
underscores, e.g., radiology_ai, european_radiology, ajr) and report the path so
/write-paper Phase 8+ and /peer-review know where to write:
submission/
└── {journal_short}/ # e.g., radiology_ai/
├── cover_letter.md # Generated by /write-paper Phase 8+
├── checklist.md # Journal-specific submission checklist
└── peer_review.md # Generated by /peer-review (journal scope-aware)
Fulltext Retrieval Skill
Batch download open-access full-text PDFs from a DOI list using legitimate OA APIs only.
Paywalled articles fail by design and are listed in manual_needed.txt for institutional access
or ILL; never work around a paywall or publisher access control.
Pipeline
DOI → arXiv (10.48550/arXiv.* DOIs) → Unpaywall → PMC (Europe PMC / OA FTP / web) → OpenAlex → Crossref → landing page
Each DOI goes through these sources in order until a valid PDF (≥10 KB, %PDF- header) is found.
arXiv DOIs (10.48550/arXiv.2401.01234, version suffixes, old-style hep-th/9901001, or a bare
arXiv: id) resolve directly to the arXiv PDF first.
Run
Requires Python 3.10+ (stdlib only) and a contact email, which Unpaywall's Terms of Service require. The script paces its requests (0.3–0.5 s delays) for the APIs' rate limits.
python "${CLAUDE_SKILL_DIR}/fetch_oa.py" dois.txt --output pdfs/ --email your@email.com
# Verbose mode for debugging (per-DOI source trace)
python "${CLAUDE_SKILL_DIR}/fetch_oa.py" dois.txt -o pdfs/ -e your@email.com --verbose
Input formats:
- Plain text — one DOI per line.
- TSV / CSV with header, or a Markdown pipe table — must contain a
DOIcolumn; optionalPMID,Title, andFirstAuthor(surname or full name) columns.
A PMID makes the PMC lookup more reliable (PMID → PMCID conversion). Supply Title where
available: a DOI-only worklist can download a PDF but cannot establish title agreement.
FirstAuthor is optional additional evidence.
Output
- PDFs saved as
{DOI_safe}.pdf(slashes replaced with underscores). pdfs/retrieval_report.json— structured per-DOI report (below); override with--report PATH.<output>/manual_needed.txt— DOIs that could not be retrieved via OA; when a PMCID was resolved, the line also carries it and the PubMed Central article URL to open in a browser.- Summary with arXiv/OA/PMC/fail/skip counts.
Retrieval report (--report)
Every run writes the report (default <output>/retrieval_report.json), schema 2:
{
"schema_version": 2,
"generated_by": "fetch_oa.py",
"counts": {"total": 4, "retrieved": 3, "not_retrieved": 1, "title_mismatch": 1,
"source_identity": {"consistent": 1, "conflict": 1, "unresolved": 1, "unavailable": 1}},
"items": [
{"doi": "10.1000/synthetic.example", "pmid": "", "title": "Example title",
"first_author": "", "status": "oa", "source": "unpaywall",
"file": "10.1000_synthetic.example.pdf", "size_bytes": 482113, "page_count": 9,
"file_sha256": "<SHA-256 of the downloaded file>", "title_match": "match",
"source_identity": {"status": "consistent", "reason": "title_and_identifier_agree",
"text_scope": "first_page_front_matter", "title_match": "match",
"doi_match": "match", "observed_identifiers": ["10.1000/synthetic.example"],
"first_author_match": "unavailable"}}
]
}
status (arxiv | oa | pmc | skip | fail), source, and counts.retrieved describe the
resolver result, including existing files (skip). They do not count identity-verified
papers. No PDF is automatically deleted or rejected. page_count comes from Poppler's
pdfinfo (null without it) and is recorded, not judged — a 3-page "article" or a 4-page "book"
is worth opening.
source_identity.status | Meaning / action |
|---|---|
consistent | Complete normalized title and a compatible DOI/arXiv identifier occur in the bounded first-page front matter, with no supplement / preface / table-of-contents heading and no retraction / erratum / correction / corrigendum / expression-of-concern heading there; an optional supplied author must also match. Evidence agrees, but this is not independent source verification or claim validation. |
conflict | Both the title and observed identifier differ. Inspect the PDF and requested record. |
unresolved | Evidence is incomplete or ambiguous: title-only, DOI-only, missing author, multiple identifiers, a matching title with another DOI/version, or a supplement / preface / table-of-contents file that names the work without being it (supplement_or_front_matter). A retraction notice, erratum, correction, corrigendum or expression of concern whose heading line sits in that area is likewise unresolved (correction_or_retraction_notice). Inspect before using as evidence. |
unavailable | No usable extracted text, Poppler unavailable, no output PDF, or the PDF changed during assessment. No current identity assessment was possible. |
Evidence is limited to the first page before a recognized abstract/body/reference heading (at
most 40 lines / 4,000 characters), so a title cited in the body or references does not count. A
title_match of match needs the complete normalized title on up to six consecutive lines;
partial overlap is unavailable and low overlap an advisory mismatch. Cover sheets, unusual
reading order, short or changed titles and DOI footers outside that area can stay unresolved.
PDF metadata and the filename alone are not identity evidence. Explicit arXiv versions must
agree; a preprint/published-version DOI difference needs review, not automatic rejection.
Downstream reports must preserve source_identity and file_sha256, keep unresolved items
visible, and check the hash still identifies the file being used. Older reports without identity
evidence remain unassessed; do not infer identity from retrieved or title_match=match.
Full-text conversion does not resolve an identity warning.
Attach PDFs into Zotero ("Find Available PDF")
For paywalled-but-licensed papers the OA resolvers miss, references/find_available_pdf.js is a
user-run snippet for Zotero's Tools → Developer → Run JavaScript (no-code equivalent:
right-click → "Find Available PDF"). It triggers Zotero's own addAvailablePDF /
addAvailablePDFs, so it reuses the user's OpenURL resolver / institutional proxy config;
no credentials, proxy hosts, or institutional identifiers are hard-coded or leave the Zotero
client. It is user-initiated and depends on the live Zotero session, so record its results
manually — they are not reproducible CI evidence. /lit-sync Phase 2.7 orchestrates both routes
and reconciles them in a report.
PDF → Markdown Conversion (Optional)
Convert downloaded PDFs to Markdown when the same papers will be read repeatedly (data extraction from k≥5 studies, a meta-analysis pipeline); for one-pass screening, read the PDF directly.
# Install (one-time)
pip install pymupdf4llm
# Convert all PDFs in a directory (.md files land alongside the .pdf files)
python "${CLAUDE_SKILL_DIR}/pdf_to_md.py" pdfs/ -v
# Custom output directory
python "${CLAUDE_SKILL_DIR}/pdf_to_md.py" pdfs/ -o markdown/
# First 10 pages only (useful for long supplements)
python "${CLAUDE_SKILL_DIR}/pdf_to_md.py" pdfs/ --pages 0-9
# Overwrite existing conversions
python "${CLAUDE_SKILL_DIR}/pdf_to_md.py" pdfs/ --force
Images are skipped, so figures survive only as caption text. Scanned-only PDFs (no text layer)
convert poorly. pdf_to_md.py needs pymupdf4llm
(AGPL-3.0), an optional dependency; fetch_oa.py stays stdlib-only.
Generate Codebook Skill
Turn a raw tabular dataset into a structured, citable data dictionary (codebook). This is the
generator side of the dictionary-first workflow: it produces the artifact that /define-variables
and dictionary-first QC later consume. Distributions, types, and missingness are observable and the
bundled script profiles them; the meaning of a coded value (fatty_liver_grade = 0) is not
observable from the data and lives only in the authoritative data dictionary. You generate code and
review output — you do not invent the meaning of coded values.
Deterministic Script
Run the bundled profiler rather than describing columns from memory:
python "${CLAUDE_SKILL_DIR}/scripts/generate_codebook.py" data.csv --out-dir .
Supports .csv/.tsv/.xlsx/.parquet/.dta/.sas7bdat. Flags: --max-levels N
(categorical cutoff, default 20), --json-only, --md-only. The script is
pandas-only, runs locally, and never sends data anywhere.
Workflow
Step 1: Profile (deterministic)
Run generate_codebook.py on the dataset. It writes codebook.json (machine-
readable) and codebook.md (review table), reporting per variable: role
(id / continuous / categorical / binary / date / text), dtype, missingness,
unique count, level frequencies or quantile summary, and a needs_dictionary flag.
Step 2: Review with the researcher (gate)
Read ${CLAUDE_SKILL_DIR}/references/codebook_schema.md (the codebook.json schema, the
role-inference heuristics, the needs_dictionary rule) before interpreting the output. Present
codebook.md and walk the user through it. Gate: role inference is a heuristic, so the user
confirms the inferred roles (e.g., an integer-coded scale mis-read as continuous, or an id column).
Do not proceed to definition work until the user approves the role assignments.
Step 3: Resolve [NEEDS DICTIONARY] items (gate)
For every variable flagged needs_dictionary: true, the level codes are
uninterpretable without the authoritative source. Gate: ask the user to
supply the meaning of each code from the real data dictionary (file/sheet/row),
or to confirm none exists. Fill label, units, and per-level meanings into the
codebook only from that source, citing file > sheet > row — never from inference. If the user
cannot supply it, leave the [NEEDS DICTIONARY] marker in place; do not erase it.
Example: in a cohort file, sex (levels 1/2) and fatty_liver_grade (0..4) are flagged because
their levels are bare codes; smoking_status (never/former/current) is not. Never write
sex: 1 = male because "that is the usual coding" — if the dictionary is unavailable, the flag stays.
Step 4: Hand off
The completed codebook.json becomes the input dictionary for /define-variables
(operationalization) and the citation source for dictionary-first QC. Gate:
confirm with the user that no needs_dictionary flags remain unresolved before
the codebook is treated as authoritative for downstream analysis.
Run /deidentify on the raw data before a codebook is shared externally. Cleaning or transforming
data is /clean-data.
Known limits
- Codes longer than 3 characters that are not numbers (
C50.9,S001) are not recognised as codes, so their column is not flaggedneeds_dictionary. - An integer-coded column with more distinct values than
--max-levelsis classedcontinuousand is not flagged. The Step 2 role review is where such columns are caught. - The same holds for a
textcolumn holding number strings: any number value (45,88) is read as a measurement, so integer codes exported as strings with more than--max-levelsvalues are not flagged.
Output Format
codebook.json (schema in references) and codebook.md (review table with a
"Columns requiring dictionary lookup" section). Summarize the counts
(rows, columns, needs_dictionary_count) in chat; do not paste the full JSON.
Grant-Builder Skill
Write proposal prose in the language the target call requires. Produce whichever parts the request needs: concept summary, Significance, Innovation, Approach, specific aims, work packages, milestone table, role split by institution, evaluation framework, reviewer-risk memo.
Korean Government Grant Mode
When the user requests a Korean industry-academia grant (산학과제) or research plan (연구계획서) — e.g., MOHW, MOTIE, MSS or regional industry-academia programs — apply the adaptations below. Korean program terms are preserved in parentheses because they are the literal form used on the funding agency's template.
Document Structure (three-attachment format)
| Attachment | Contents |
|---|---|
| 1 (첨부1, 기본정보) | project title, participating institutions, investigator CVs, publication / patent record |
| 2 (첨부2, 매칭확인서) | per-institution cost-share confirmation, typically finalized after a kickoff meeting between the institutions |
| 3 (첨부3, 연구계획서) | the 10-page research plan — structure below |
Attachment 3 Standard Structure
1. Significance & Aims (약 2p)
- clinical problem with quantitative framing
- domestic + international trends (3–5 year literature / guideline window)
- differentiation of the proposed work
2. Research Content & Methods (약 4p)
- staged roadmap (Phase 1 – N with time ranges)
- pipeline schematic (mandatory when an AI pipeline is in scope)
- per-subproject institution and personnel assignment
3. Team Capability (약 1p)
- expertise + representative record (SCI papers, patents) per investigator
- cross-institution synergy (hospital = data / clinical; university = algorithm)
4. Expected Outcomes & Utilization (약 2p)
- quantitative targets: SCI papers, patents
- qualitative targets: clinical impact, standardization contribution
- linkage to follow-on larger grants (positioning as a seed)
5. Budget Plan (약 1p)
- RA salaries, computing equipment, consumables, academic activities, indirect costs
Small-Scale Grants (< KRW 30 million)
- Write for a non-specialist reviewer; assume the evaluator is not in your subfield.
- Emphasize feasibility over technical novelty.
- Prioritize length / format compliance; exceeding the template incurs scoring penalties.
- Include preliminary data or pilot results whenever available.
- Keep quantitative targets conservative — undershooting a committed target is punished more than overdelivering on a modest one.
Workflow
Phase 1: Decode the funding call
Extract funding body, call theme, eligibility constraints, deliverable expectations, timeline and evaluation criteria. If no call text is available, infer a generic academic-medical AI proposal structure and label the assumptions.
Phase 2: Frame the problem
Define the clinical pain point, the current workflow limitation, why existing AI or standard care is insufficient, and who benefits if the project succeeds.
Gate: Present the problem framing (clinical pain point, gap, proposed solution) to the user. Confirm before building proposal sections — a misframed problem produces an unfundable proposal.
Phase 3: Build the proposal spine
Always articulate: problem, gap, proposed solution, why this team can execute it, measurable outputs.
Phase 4: Convert to proposal sections
- Significance must answer why this matters clinically, why now, and why the proposed solution is worth funding.
- Innovation: what is genuinely different, why the integration is new, why the novelty is useful and not just technical.
- Approach: dataset and participating sites, model or workflow components, validation plan, benchmark/comparator, failure analysis, risk mitigation.
Route to search-lit to support significance and prior-art positioning; cite a reference only with
a /search-lit-confirmed DOI or PMID, otherwise mark it [UNVERIFIED - NEEDS MANUAL CHECK]. Mark
any unconfirmed clinical definition, diagnostic criterion or guideline recommendation [VERIFY] and
ask the user. Route to design-study if the evaluation framework is weak, and to write-paper only
when the proposal requires publication-style narrative sections.
Phase 5: Execution plan
Generate milestones by quarter or year, institution-level responsibilities, dependencies and handoffs, and required infrastructure. Do not fabricate budget details, and do not promise datasets, partners or infrastructure the user has not evidenced.
Default Structure
## Proposal Summary
Title: ...
Goal: ...
Clinical problem: ...
### Significance
...
### Innovation
...
### Approach
Aim 1. ...
Aim 2. ...
Aim 3. ...
### Milestones
- ...
### Consortium roles
- ...
### Major risks and mitigations
- ...
Before Finalizing
Check, and flag any failure to the user:
- Is the clinical need explicit and credible?
- Is the novelty more than "we will use AI", with a clinical consequence?
- Are the aims linked to measurable outputs with a concrete benchmark or success criterion?
- Is the validation plan convincing, including external validation or a deployment path?
- Is the multi-site structure realistic, with each institution in a distinct, functionally integrated role?
- Are compute, annotation, and regulatory needs acknowledged?
- Are there too many aims for the timeline?
- Does it read as a funded program rather than a paper?
Humanize Skill
This skill only removes AI patterns; it does not perform general copy-editing, evaluate scientific quality, check journal formatting, or translate.
Read ${CLAUDE_SKILL_DIR}/references/ai_patterns.md at the start of every session, before
scanning: the definitions, watch words, examples, detection greps, per-pattern fixes and the
section-by-section priorities for all 27 patterns live only there.
Workflow
Phase 1: Scan
Scan the section(s) the user provides for all 27 patterns. For response-to-reviewers letters and
cover letters, prioritise Patterns 22-24. For a full manuscript, follow the per-section priorities
in ai_patterns.md (Section-Specific Application Guide). For each pattern found, record its number
and name, the count, the exact passage, and its location (paragraph number or line range).
Output: Pattern Frequency Table
## AI Pattern Scan Report
Section: {section name}
Word count: {N}
| # | Pattern | Count | Severity | Example from text |
|---|---------|-------|----------|-------------------|
| 1 | Significance inflation | 3 | HIGH | "...pivotal role in diagnostic imaging..." |
| ... | ... | ... | ... | ... |
Patterns not detected: 2, 4, 9, 14, 15
Total AI pattern instances: {N}
AI pattern density: {N per 1000 words}
Phase 2: Report
- Severity per pattern: HIGH (>3 occurrences), MEDIUM (1-3), LOW (0, clean).
- Density: total instances across all 27 patterns per 1000 words. Target: < 2.0.
Gate: Present the report and ask the user which patterns to fix. Default: fix all HIGH and MEDIUM.
Phase 3: Fix
Rewrite flagged passages with each pattern's fix from ai_patterns.md, under these rules:
- Preserve technical accuracy. Every number, statistic, p-value, confidence interval, and clinical fact must remain identical.
- Preserve citations. Never add, remove, or relocate a citation.
- Keep the formal academic register of an experienced radiologist writing for peers in a top-tier journal. Never make the text casual or conversational.
- Keep domain-specific terminology intact. "Convolutional neural network," "apparent diffusion coefficient," "Fleiss' kappa" stay as-is.
- Never introduce new claims, remove existing ones, or change a sentence's meaning — rephrase, never reinterpret. If a passage cannot be fixed without changing its meaning, leave it and flag it for the user.
- Use active voice where natural: "We analyzed" rather than "Analysis was performed."
- Vary sentence structure. Mix short declarative sentences (8-12 words) with longer ones
(25-35 words). A de-AI pass tends to flatten rhythm — it shortens the long sentences and pads
the short ones toward a comfortable middle, which is itself a tell.
scripts/check_sentence_variety.pyverifies this rule in Phase 4. - Thin out antithesis and cleft constructions (Pattern 27, the M2 heuristic). For each "X
rather than Y", "not X but Y" or "X, not Y", apply the negative-form test: delete the negative
half and rewrite the clause in the positive. If a fact disappears, the contrast was functional —
keep it; if nothing disappears, it was decoration — cut it. Judge by the manuscript's overall
rate, not instance by instance, and keep two or three for emphasis. Rewrite clefts ("What … is
…", "It is … that …") in plain subject-verb order ("What matters is X" → "X matters").
scripts/check_rhetorical_density.py(in/self-review) measures this in Phase 4.
Output: Present the rewritten text with changes highlighted using diff format or tracked changes.
Phase 4: Verify
Keep the pre-rewrite text. Before editing in place, copy the original somewhere the fidelity
check can read it (cp manuscript.md /tmp/pre_humanize.md). Without it Phase 4 can only re-scan
for patterns — it cannot tell whether the rewrite preserved numbers and citations.
Run both deterministic checks, then re-scan the rewritten text using the same 27 patterns.
python3 "${CLAUDE_SKILL_DIR}/scripts/check_rewrite_fidelity.py" \
--before /tmp/pre_humanize.md --after manuscript.md \
--out qc/rewrite_fidelity.json --strict
python3 "${CLAUDE_SKILL_DIR}/scripts/check_sentence_variety.py" \
--manuscript manuscript.md --out qc/sentence_variety.json
NUMBER_DRIFT, NUMBER_REASSIGNED, CITATION_DROP or CITATION_MOVED means the rewrite broke an
invariant — revert that passage, redo it, and flag it for the user. EDIT_FOOTPRINT_HIGH is advisory: Patterns 6 and 18 replace whole
paragraphs by design, so a correct pass over an inflated draft can exceed 60% of words changed.
Read the diff and confirm the author's argument survived rather than assuming the percentage is a
defect.
Known limits: the fidelity gate does not check an added or removed negation, a number written in words, a changed unit, or a direction word next to a non-percentage. A clean exit does not clear these; read the diff for them.
Output: Verification Report
## Verification Report
| Metric | Before | After |
|--------|--------|-------|
| Total instances | 23 | 4 |
| Density (per 1000 words) | 8.2 | 1.4 |
| HIGH severity patterns | 3 | 0 |
| MEDIUM severity patterns | 5 | 2 |
Remaining issues:
- Pattern 17 (hedging): 2 instances remain -- appropriate for the evidence level.
Verdict: PASS (density < 2.0)
If the density remains above 2.0, run another fix-verify cycle (max 3 rounds). When called by another skill, return the verification report so the calling skill can check the pass/fail status.
The 27 Detection Patterns
All 27 are defined in references/ai_patterns.md. Two carry rules to apply exactly as written:
| # | Pattern | What to look for | Fix |
|---|---|---|---|
| 13 | Em dash overuse | More than 2 em dashes per 1000 words (the /self-review classical-style gate fails a manuscript above 25 prose em-dashes) | Use parentheses or restructure. After converting — X — appositives to (X), run the paren-span safety scan (python3 "${CLAUDE_SKILL_DIR}/../self-review/scripts/check_paren_spans.py"): a bulk conversion can pair two unrelated dashes across a sentence boundary and wrap a whole sentence (or an ordinal "Sixth, …" limitation) inside one parenthesis — paren-balanced but broken, so a balance check misses it. Operate per-sentence; never match across . |
| 21 | AI Disclosure boilerplate (body) | "## Artificial Intelligence Disclosure", "Generative AI was not used to create..." in manuscript body | Put it where the target journal asks (the journal profile's Disclosure location): Methods or Acknowledgments for some journals, cover letter, title page or submission form only for others. Do not delete a disclosure the journal requires in the body |
Patterns 22-24 apply only to response-to-reviewers letters and editor cover letters, not
manuscript bodies. They are defined once in ai_patterns.md (Response-Letter Patterns); for
authoring guidance, see the revise skill's references/r2r_voice.md.
Gates
| Gate | Severity | Trigger | Action on fail |
|---|---|---|---|
| AI-pattern density target | ADVISORY | density > 2.0 patterns / 1000 words after sweep | warn; surface remaining flagged passages for manual review |
| Pattern 13 — paren-span corruption after em-dash conversion | ENFORCED | after a — X — → (X) sweep | run python3 "${CLAUDE_SKILL_DIR}/../self-review/scripts/check_paren_spans.py" --strict; PAREN_SPAN_ORDINAL / PAREN_SPAN_SENTENCE means a conversion wrapped a sentence/ordinal inside parens — fix before finalizing |
Pattern 19 — § symbol | ENFORCED (senior MA reviewer prep) | grep -c "§" manuscript.md > 0 | auto-strip; verify post-rewrite count == 0 |
Pattern 20 — (see Methods §X) self-reference | ENFORCED | match found | rewrite to direct section name reference |
| Pattern 21 — AI disclosure in the wrong place | ENFORCED | an AI-use disclosure in the body of a journal that wants it elsewhere, or repeated in several places | move it to where the target journal asks (journal profile); never reword a required disclosure to hide the tool |
| Pattern 26 — aphorism density | ENFORCED | negative-definition rate AND short-declarative share both over threshold | run python3 "${CLAUDE_SKILL_DIR}/../self-review/scripts/check_aphorism_density.py" --manuscript manuscript.md; APHORISM_DENSITY (Minor) means the prose is a run of epigrams with the explanatory sentences compressed out — absorb most of them into the neighbouring sentence and write the explanation back, keeping two or three for emphasis; do NOT simply delete them, which shortens the prose further |
| Pattern 27 — antithesis / cleft density | ENFORCED | "rather than" / "not X but Y" / "X, not Y" or "What … is …" / "It is … that …" over a per-1000 threshold AND raw-count floor | run python3 "${CLAUDE_SKILL_DIR}/../self-review/scripts/check_rhetorical_density.py" --manuscript manuscript.md; ANTITHESIS_DENSITY / CLEFT_DENSITY (both Minor) — apply Fix rule 8 (the M2 test). A lone functional "rather than" or "instead of" never fires |
| Pattern 25 — inline-emphasis over-use | ENFORCED | italic-emphasis density over threshold after allowlist | run python3 "${CLAUDE_SKILL_DIR}/../self-review/scripts/check_emphasis_density.py" --manuscript manuscript.md; EMPHASIS_OVERUSE (Minor) means strip inline italics (keep only stat symbols / Latin / gene-species); whole-clause italics are the strongest tell |
| Patterns 22-24 — R2R editing-mechanism / draft line-number / tooling leak | TRIAGE (response letters); § = 0 hard | detection greps in ai_patterns.md R2R section surface candidates | review each hit (analysis narration, quoted additions, revised-manuscript page/line are NOT tells); rewrite confirmed tells to substantive prose |
| Citation preservation invariant | ENFORCED | a citation item (each key of a multi-key or locator Pandoc citation, or a numeric marker) removed or changed, or moved out of the sentence of the word it was attached to while that word kept its place | scripts/check_rewrite_fidelity.py --before <pre> --after <post> --strict → CITATION_DROP / CITATION_MOVED (Major); revert that single rewrite and flag for the user |
| Numerical preservation invariant | ENFORCED | a numeric token's count changed (sign, inequality sign and %-direction are part of the token; writing a sign out in words also fires), or values traded places while the words around them stayed | same script → NUMBER_DRIFT / NUMBER_REASSIGNED (Major); revert and flag. Known limits: a negation, a number written in words, a unit, or a direction word next to a non-percentage is not checked — read the diff for these |
| Rewrite footprint | ADVISORY | fraction of word tokens changed exceeds --warn-pct (default 70) | EDIT_FOOTPRINT_HIGH (Minor) — never blocks; read the diff (Phase 4) |
| Fix rule 7 — sentence-length uniformity | ADVISORY | prose has no short (≤12 words) or no long (≥25 words) sentences | scripts/check_sentence_variety.py --manuscript <file> → SENTENCE_UNIFORM (Minor); break up or combine sentences until both bands exist. Silent below 15 sentences |
Imaging-Data Skill
The dataset decides more of a study than the architecture does, and it decides it first. Phases 1–3
establish what the data is and what it will not support, while that is still cheap; Phases 4–7 design
and audit the preparation pipeline so it is leakage-safe before /model-scaffold builds the repo.
Describe-and-audit only: never modify, resample, reorient, split or write image data, never run
preprocessing on real patient data, and wire MONAI / TorchIO transforms by reference rather than
writing a new normalisation or resampling implementation.
Elsewhere: tabular/clinical variables → /generate-codebook, /clean-data; auditing the split
table, held-out metrics, calibration, subgroup results → /model-assessment; choosing an
architecture → /model-selection; building the repo → /model-scaffold.
Workflow
Phase 1 — Profile every case
python3 ${CLAUDE_SKILL_DIR}/scripts/profile_imaging_dataset.py \
--split train:imagesTr:labelsTr \
--split test:imagesTs \
--dataset "MSD Task09 Spleen" \
--declared-labels 0=background,1=spleen \
--target-label 1 \
--plan resample=true,reorient=false,loss=dice_ce,metrics=dice+hd95 \
--out eda/profile.json
One record per case: grid, spacing, orientation, intensity percentiles, the label values actually
present, foreground fraction, and target volume in mL. A --split given no label directory is
recorded as unlabelled — itself a finding. Requires nibabel + numpy; the gate does not.
Every profile figure comes from opening the files — never from a dataset's README, a similar dataset,
or memory. A README can be wrong about its own label indices; the labels cannot.
--target-label on a multi-structure atlas. Foreground defaults to every non-zero index — the
whole annotated anatomy. Measured on the AMOS22 CT cases, that pools to 3.2 % instead of the spleen's 0.20 %, so the
pooled figure sits above the 1 % imbalance threshold while the target sits far below it and the
imbalance verdicts go quiet exactly where the risk is. Naming the target also makes LABEL_EMPTY mean
this case has no spleen. Pass --target-label all for a genuinely multi-class study; leave it out
on a multi-structure atlas and the gate raises TARGET_LABEL_UNDECLARED.
Phase 2 — Gate the profile against the declared plan
python3 ${CLAUDE_SKILL_DIR}/scripts/check_dataset_profile.py --profile eda/profile.json \
--out qc/dataset_profile.json --strict
Stdlib-only, so the audit re-runs anywhere the JSON travels. Never report a profile "pass" without running it.
| Verdict | Severity | Fires when |
|---|---|---|
LABEL_SHAPE_MISMATCH | Major | label grid ≠ image grid |
LABEL_EMPTY | Major | a labelled case has zero foreground |
LABEL_VALUE_UNEXPECTED | Major | label values outside the declared set |
TEST_SET_UNLABELLED | Major | a split named test/held-out/external/eval carries no labels |
ACCURACY_UNDER_IMBALANCE | Major | accuracy is planned while the target is a sliver of the volume |
LABEL_MISSING | Minor | a case in a labelled split has no label file |
SPACING_HETEROGENEOUS | Minor | spacing spans ≥ ratio on an axis and no resampling is declared |
ORIENTATION_MIXED | Minor | >1 orientation code and no reorientation declared |
INTENSITY_SCALE_INCONSISTENT | Minor | some cases on the HU scale, others not |
EXTREME_IMBALANCE | Minor | median foreground below the threshold with no Dice-family loss |
TARGET_LABEL_UNDECLARED | Minor | >1 structure declared, no target named, so foreground pools them all |
The gate flags an undeclared decision, not variability: 5× spacing spread and two orientation
codes pass once resampling and reorientation are declared (the clean challenge fixture proves this).
--spacing-ratio (default 2.0) and --imbalance-frac (default 0.01) are screening defaults, not
published cut-points — never present them as such; the values applied are printed in the output
and belong in the Methods. A split the profile shows unlabelled is never a held-out test set, however
the directory is named.
Phase 3 — Turn the profile into research decisions
Write these decision notes into the study record, so /design-study, the preparation phases below,
and /write-paper inherit them instead of re-deriving them:
- Resampling target — from the spacing distribution, not a tutorial default (carried into Phase 5).
- Loss and metric family — from the foreground fraction. Segmentation reports Dice and a
boundary metric per structure (
/model-assessment); accuracy is not on the list. - Pre-specified subgroups — from the clinical spread the profile shows (target volume, slice thickness, modality). Pre-specifying them here is what separates a subgroup finding from a post-hoc one.
- Where the held-out set comes from — especially when the shipped "test" directory is unlabelled.
- What the cohort cannot support — n, single-source acquisition, absent subgroups: the seed of the Limitations paragraph, written before results can bias it.
Phase 4 — Inventory the preparation steps and fix fit scope
Collect the modality, the data manifest (one row per image/slice with a patient_id), the resample
spacing, the intensity transform (fixed HU window vs a fitted z-score / min-max / histogram match),
and the augmentation plan. Read ${CLAUDE_SKILL_DIR}/references/preprocessing_guide.md for the
modality-aware normalisation, physiology-preserving vs -breaking augmentation, and MONAI / TorchIO wiring.
- Fit dataset-level normalisation on the training split only — never all/full/test.
- Run any data-fitted transform after the split; before it there is no train/test distinction.
- Prefer per-image (per-sample) normalisation where clinically appropriate — leakage-free even before the split.
- Keep augmentation train-only; augmenting val/test folds undisclosed test-time augmentation into the metric.
- Split at the patient level, then map slices to their patient's split.
Phase 5 — Emit the preprocessing manifest
Write preprocessing_manifest.json, which /model-scaffold consumes and the gate checks. Every value
comes from the real data manifest and the declared pipeline — never invented patient IDs or split
assignments.
{
"split_seed": 42,
"transforms": [
{"name": "hu_window", "type": "clip", "fit_scope": "none", "stage": "before_split"},
{"name": "train_zscore", "type": "standardize", "fit_scope": "train", "stage": "after_split"},
{"name": "flip_rotate", "type": "augmentation", "stage": "after_split", "applies_to": ["train"]}
],
"split_assignment": [
{"patient_id": "P001", "unit_id": "P001_s1", "split": "train"}
]
}
fit_scope: train (OK) · all/full/dataset/test (leak) · sample/per_image/none/fixed
(not data-fitted). stage: before_split / after_split. The fields must describe what the code
actually does — never tag a dataset-fitted transform per-sample to clear the gate; that hides the leak.
A declared dataset-level fit_scope is judged whatever the type is, so a library class name
(HistogramStandardization, NormalizeIntensityd) fit on all is a leak like standardize would be.
Declare the fit scope of resampling too. A target spacing chosen in advance is fit_scope: fixed
and never leaks. A target derived from the cohort does: nnU-Net sets its target spacing from a
percentile of the dataset fingerprint, so a resample fitted over every case carries held-out geometry
into the training grid exactly as an intensity statistic would. The fingerprint's scope decides which
you have, not the word "resample".
Phase 6 — Gate the manifest
python3 ${CLAUDE_SKILL_DIR}/scripts/check_preprocessing_leakage.py --manifest preprocessing_manifest.json \
--out qc/preprocessing_leakage.json --strict
Verdicts: PREPROCESS_BEFORE_SPLIT, NORMALIZATION_LEAKAGE, PATIENT_CROSS_SPLIT (Major);
AUGMENTATION_ON_EVAL, UNSPECIFIED_FIT_SCOPE, MISSING_SEED (Minor), reproduced by set arithmetic
and rule on the manifest. A green gate is the precondition for handing the manifest to
/model-scaffold; its split_assignment is the same patient-level split /model-assessment later
re-verifies. Never report a pass without running it.
Phase 7 — Before inference on a new cohort: check the normaliser's domain
Phase 6 asks whether a transform was fit on the right scope. Before running a trained model on a cohort it was not trained on, ask whether that cohort sits in the intensity domain the trained normaliser assumes:
python3 ${CLAUDE_SKILL_DIR}/scripts/check_normalizer_domain.py \
--profile eda/<cohort>_profile.json \
--contract work/nnUNet_results/.../plans.json \
--splits external_mri --out qc/normalizer_domain.json --strict
Its challenge card holds a cohort in the contract's own domain that must come back clean, an arbitrary-unit cohort that must raise a Major, and an unreadable contract that must refuse rather than pass.
Outputs and hand-off
eda/profile.json(Phase 1),qc/dataset_profile.json(Phase 2), and the decision notes (Phase 3).preprocessing_manifest.jsonwith the augmentation-appropriateness and normalisation fit-scope notes (Phases 4–5),qc/preprocessing_leakage.json(Phase 6),qc/normalizer_domain.json(Phase 7).
The manifest feeds /model-scaffold, which also reads the qc/ reports: keep them in qc/ beside
the manifest (or ../qc/), or pass them with --imaging-qc. An unresolved Major there refuses the
scaffold until it is fixed and re-gated or acknowledged with a stated reason (--ack-qc); Minor and
Flag claims are carried into the repo's IMAGING_QC.md, and a missing report is recorded as not
assessed. Re-run a gate after fixing its finding — a stale report still blocks. The manifest documents the CLAIM 2024 / TRIPOD+AI data-preprocessing items
for /check-reporting; /self-review's model_development probe looks for exactly this pipeline in a
finished manuscript. Regression: bash ${CLAUDE_SKILL_DIR}/scripts/check_dataset_profile_challenge/verify.sh,
bash ${CLAUDE_SKILL_DIR}/scripts/check_preprocessing_leakage_challenge/verify.sh,
bash ${CLAUDE_SKILL_DIR}/scripts/check_normalizer_domain_challenge/verify.sh,
bash ${CLAUDE_SKILL_DIR}/tests/test_dataset_profile.sh,
bash ${CLAUDE_SKILL_DIR}/tests/test_preprocessing_leakage.sh.
Intake-Project Skill
Canonical Manuscript Folder Structure
/manage-project init scaffolds new projects; its script is the source of truth for that tree, including the qc/ folder the QC skills write to. Use the layout below to reorganize an existing project that was never scaffolded, mapping each artifact into one of these slots or into a folder the init scaffold or another skill already writes to.
{project_root}/
├── HANDOFF.md # session handoff entry point
├── README.md # project overview
├── data/ # raw data (NEVER edit; read-only)
├── analysis/ # reproducible scripts (00_* → 04_*)
├── output/ # analysis outputs: CSVs, PNGs, intermediates
├── qc/ # QC reports the skills write (self-review, checklists, audits)
├── irb/ # IRB/ethics docs
├── proposal/ # original protocol / approved proposal
├── reviews/ # external correspondence
├── manuscript/ # SOURCE manuscript + drafting
│ ├── manuscript_v{N}.{md,docx,pdf} # current canonical working version (top level)
│ ├── build_unified_docx.py # or pandoc wrapper
│ ├── archive/ # ALL prior versions v1 .. v{N-1}
│ ├── reviews/ # review notes (QC reports go in qc/ at the root)
│ ├── figures/ # figure scripts + rendered PNG/PDF
│ └── tables/ # table scripts + rendered docx
└── submission/ # per-journal packages
└── {journal-slug}/ # e.g., chest/, kjr/
├── CHECKLIST.md
├── cover_letter.{md,docx,pdf}
├── title_page.docx # separated for double-anonymized
├── manuscript_anonymized.{docx,pdf}
├── supplement.{docx,pdf}
├── strobe_checklist.md # or PRISMA / CONSORT
├── circulation_email.md
└── figures/ # submission-ready DPI copies
Rules
manuscript/= source;submission/{journal}/= derived artifacts. Regenerate submission files frommanuscript/manuscript_v{N}.md; never edit anonymized/title-page directly.- One canonical working version at
manuscript/manuscript_v{N}.{md,docx,pdf}. Older versions move tomanuscript/archive/immediately on version bump. - No loose files at project root. Only
HANDOFF.md,README.md, the project memory files (Phase 3), the contract files/manage-project initwrites (SSOT.yamlorproject.yaml,project_state.json,artifact_manifest.json), and folder entries. - QC artifacts (self-review, reporting checklists, cross-reference and reference audits) live in
qc/at the project root, where the skills that produce them write them, not at manuscript top level. - On rejection/retarget:
cp -r submission/{old} submission/{new}, then rewrite cover letter and reformat. - Double-anonymized journals (Chest, AJRCCM): title page and anonymized manuscript MUST be separate files under
submission/{journal}/. - Keep existing project labels and file names in the language the workspace already uses.
When to apply
- At project intake: scaffold through
/manage-project init— unless the user only wants a quick assessment. - At first submission prep: create
submission/{journal}/and populate. - Mid-project cleanup: when
manuscript/has >3 versioned files or QC docs at top level, reorganize. - Before session handoff: reorganize if structure is drifting.
Workflow
Phase 1: Discover context
- Read top-level folder names and key files.
- Detect manuscript-like files, tables, figures, protocols, and analysis outputs.
- Extract:
- project title or working title
- study question
- dataset or cohort hints
- collaborators or institutions
- venue/journal hints
Phase 2: Classify project and stage
Determine:
- project type:
original | review | meta-analysis | case report | technical note | grant | peer review | challenge | career-doc - primary domain:
radiology | medical AI | multimodal LLM | intervention | survival/prognostic | diagnostic accuracy | workflow - target output:
paper | abstract | grant | review | rebuttal | CV - likely target journal or venue — only if the files name one
- one current stage:
idea | data assembly | analysis planning | analysis in progress | drafting | revision | submission prep | archived/unclear
If the folder mixes several studies, say so rather than collapsing them into one.
Gate: Present the classification (project type, stage, target output) to the user. Confirm before creating any files — misclassification leads to wrong scaffold and wrong skill routing.
Phase 3: Surface missing inputs
Check for blocking dependencies and common gaps:
- no explicit study question
- no target journal
- no analysis plan
- no variable dictionary
- no claims-to-results map
- no review log for revised manuscripts
If missing, propose or create lightweight anchor files — PROJECT.md, STATUS.md, CLAIMS.md,
DATA_DICTIONARY.md, ANALYSIS_PLAN.md, REVIEW_LOG.md — only those the project type justifies.
Read ${CLAUDE_SKILL_DIR}/references/memory_templates.md when creating PROJECT.md or STATUS.md.
Phase 4: Produce normalized summary
Output this structure:
## Project Intake Summary
Project: ...
Type: ...
Current stage: ...
Likely target: ...
### What exists
- ...
### What is missing
- ...
### Risks / ambiguities
- ...
### Recommended next actions (3-5, in dependency order)
1. ...
2. ...
3. ...
Handoff Rules
After intake:
- route to
search-litif the literature basis is weak - route to
design-studyif the research question exists but design logic is unclear - route to
manage-projectif the folder should be scaffolded - route to
write-paperonly after the project phase is clearlydrafting
Literature Sync: Zotero + Obsidian Pipeline
Takes the .bib output of /search-lit (or any user-specified .bib file), synchronizes the
references into the Zotero library and Obsidian literature notes, and extracts cross-cutting
concept notes once enough literature notes accumulate.
Prerequisites
- Project owner only. Collaborators consume the committed
manuscript/_src/refs.bibsnapshot read-only. If the current user is a collaborator (no Zotero access perSSOT.yamlreference_manager.required_for), abort with instructions to flag[@NEW:topic]placeholders in the manuscript and notify the owner. - Zotero desktop 7.x + Better BibTeX plugin, with Better BibTeX "Keep updated" auto-export
configured to
<project>/manuscript/_src/refs.bib. - Zotero MCP server (if not connected, skip Phase 2; auto-export still refreshes once Zotero is reopened).
- Obsidian CLI or direct file writing to the vault; vault path from the user's environment
(e.g.,
$OBSIDIAN_VAULT).
Artifact Contract
/lit-sync is the sole writer of manuscript/_src/refs.bib (via the Better BibTeX auto-export
trigger), references/zotero_collection.json, and references/fulltext_retrieval.json (Phase 2.7).
NEVER write refs.bib directly — not even to compensate for a missing Zotero connection —
because only Better BibTeX's export keeps it in step with the library; if auto-export is broken, fix
the Zotero setup. Direct hand edits to refs.bib are drift — revert on sight.
Phase 1: Parse BibTeX
Parse the user-specified .bib file, or the references/library.bib just produced by
/search-lit, with regex. Extract per entry: citekey, doi, pmid, title, authors (first + last
minimum), journal, year, and volume/number/pages if present. Log any parse failures and skip those
entries. All bibliographic data in later phases comes from these entries or from API responses.
A parsed citekey is not yet a library key: /search-lit candidate keys (Kim_2024_Validation) are
provisional, and Better BibTeX mints the real key when Phase 2 adds the item. Never compose a key;
Step 3.2 §Citekey provenance says where the note's key comes from.
Phase 2: Zotero Sync
If the Zotero MCP is not connected, skip this phase (still write the Step 2.3 file with
status: "skipped") and proceed to Phase 3.
Step 2.1: Determine project collection
Identify the project from the current working directory or an explicit user override. Reuse a
recorded collection key; otherwise check existing Zotero collections for the project and, if none
exists, create one with zotero_create_collection. Record the collection key, and report it to the
user when a new collection is created.
Step 2.2: Dedupe + add
For each entry:
- Use
zotero_search_itemsto search by DOI or title — if already present, skip. This search-first step is what prevents duplicates;zotero_add_by_doidoes not dedupe by itself (it fetches CrossRef and creates the item), so never skip the search. - Otherwise call
zotero_add_by_doi(when a DOI is available) orzotero_add_by_url(the PubMed URL, when only a PMID is available). An entry with neither DOI nor PMID is not added: count it as failed and ask the user to add it manually.zotero_add_by_doiaccepts anattach_modeargument that governs the OA child-PDF attach attempt at add time (the installed server treatslinked_urlas "bookmark the PDF URL"; other values download/import). Exact accepted values are server-version-specific — verify against the connected server. Do not usezotero_add_from_fileto attach a PDF to an item added here: it has no parent-item argument and would create a duplicate parent item.
- Use
zotero_manage_collectionsto place the item in the project collection.
Step 2.3: Result report
Zotero Sync:
Added: 8 papers (new)
Skipped: 3 papers (already in library)
Failed: 1 paper (no DOI/PMID)
Collection: RFA-Meta (TZQEP4NH)
Always write references/zotero_collection.json in the project workspace:
{
"schema_version": 1,
"status": "synced",
"collection": "RFA-Meta",
"collection_key": "TZQEP4NH",
"added": 8,
"skipped": 3,
"failed": 1
}
If Zotero is unavailable, write the same file with status: "skipped" and a
human-readable reason.
Phase 2.5: refs.bib snapshot refresh
Better BibTeX "Keep updated" auto-export normally refreshes manuscript/_src/refs.bib within
seconds of a Zotero change. This phase verifies the snapshot actually updated before downstream
skills consume it.
Step 2.5.1: Resolve path
Read SSOT.yaml → truth.refs_bib. Default: manuscript/_src/refs.bib. If absent (legacy project), fall back to manuscript/_src/refs.bib and emit a WARN recommending SSOT migration.
Step 2.5.1b: Precondition assertion (early-exit, do NOT poll)
Before entering the 10s polling loop in Step 2.5.2, verify both preconditions. If either fails, abort Phase 2.5 with setup instructions instead of waiting for a timeout that will never resolve.
-
Better BibTeX is answering. Probe the running plugin, not a file on disk:
curl -s -m 5 -o /dev/null -w "%{http_code}" \ http://127.0.0.1:23119/better-bibtex/json-rpc # expect 200A non-200 means Zotero is closed or BBT has not finished starting. Retry once after Zotero's window is up; BBT registers its endpoint a few seconds after the app does.
⚠️ Do not gate on
~/Zotero/better-bibtex/read-only.json. Current BBT releases keep auto-export registrations in their own store, so that file is routinely[]on a healthy install; treating it as "not configured" skips this phase on working setups, which is how a stalerefs.biband an invented citekey reach a manuscript.On failure print:
Phase 2.5 skipped: Better BibTeX did not answer on
127.0.0.1:23119(HTTP<code>). Open Zotero, wait for it to finish loading, then re-run/lit-sync. -
Target refs.bib exists. The resolved
truth.refs_bibpath from Step 2.5.1 must exist on disk (even empty is OK — BBT will overwrite). On failure print:Phase 2.5 skipped: target snapshot
<path>not found. Configure BBT auto-export with "On Change" to the SSOT path, then re-run.
In either early-exit, set refs_bib_refreshed: false + reason: "precondition:<which>" in the
Step 2.5.3 JSON, tell the user, and return control to the caller. Nothing downstream enforces this
flag, so telling the user is what stands between a stale refs.bib and a manuscript.
Step 2.5.2: Verify refresh
After Phase 2 adds items:
- Capture
stat -f "%m" manuscript/_src/refs.bibbefore Zotero writes. - Wait up to 10s (Better BibTeX debounce). Poll mtime.
- If mtime unchanged after 10s:
- Prompt user to check Zotero is running and BBT export is "Keep updated".
- If BBT auto-export path is wrong, print the expected path (
<project>/manuscript/_src/refs.bib). - As last resort, offer manual export:
File → Export Library → Better BibTeX → target path.
- Once mtime advances, grep for the newly added citekeys. All must be present; if any is missing, report as failure (do NOT fabricate entries).
Step 2.5.3: Record in zotero_collection.json
Append to the JSON written in Step 2.3:
{
"refs_bib_path": "manuscript/_src/refs.bib",
"refs_bib_mtime": "2026-04-24T14:32:11Z",
"refs_bib_refreshed": true,
"citekeys_verified": ["smithDeepLearningRadiology2024", "..."]
}
If refresh failed, set refs_bib_refreshed: false and include reason.
Phase 2.7: Fulltext Retrieval (opt-in, owner-only)
Run only when the user asks for full text (e.g. "download the PDFs", "fetch full
text", or a worklist supplied with that intent). Default /lit-sync stays metadata-only
and network-light — do not auto-run this phase. Runs after items are in Zotero (Phase 2)
and the snapshot is verified (Phase 2.5), before Obsidian notes (Phase 3).
Offer both routes below and reconcile them in one report. Retrieve full text only through them: never automate authenticated browser sessions, never bypass paywalls or access controls, and never hard-code institutional proxies, credentials, or hosts into this skill. Route what neither reaches to institutional access, interlibrary loan, or author contact.
Route A — disk OA PDFs (for downstream skills)
Delegate to the /fulltext-retrieval engine (do not re-implement the OA cascade or
import its code; invoke it by path):
ENGINE="${CLAUDE_SKILL_DIR}/../fulltext-retrieval/fetch_oa.py"
python3 "$ENGINE" <worklist> -o pdfs/ -e <contact-email> --report pdfs/retrieval_report.json
<worklist> is the DOI/PMID(/Title) list — the Phase-1 .bib DOIs, a worklist supplied in
Standalone Modes, or the project collection's DOIs. Output: pdfs/*.pdf for /meta-analysis and
pdf_to_md.py, plus pdfs/retrieval_report.json (schema 2: retrieval status/source,
source_identity, and file_sha256). Keep the distinction between having a file and assessing its
identity.
Route B — in-library PDFs (Zotero-native, higher yield, proxy-aware)
Emit ${CLAUDE_SKILL_DIR}/../fulltext-retrieval/references/find_available_pdf.js
for the user to paste into Zotero (Tools → Developer → Run JavaScript) with the project
collection selected. It triggers Zotero's own addAvailablePDF/addAvailablePDFs, which
reuse the user's OpenURL resolver / institutional proxy — so it typically retrieves more
than OA-only, while no credentials or institutional identifiers enter this skill. The
no-code equivalent is right-click → "Find Available PDF". Record its {attached, missing} summary
from the printed JSON.
Report
Merge Route A's pdfs/retrieval_report.json (and the user-reported Route B summary) into
references/fulltext_retrieval.json, and append a short fulltext counts block to
references/zotero_collection.json. Read ${CLAUDE_SKILL_DIR}/references/fulltext_report.md
before writing them — it has the schema and the identity-evidence rules (a retrieved file is not
a verified paper; a consistent status is advisory; unassessed items go to
identity_review_needed).
Phase 3: Obsidian Literature Notes
Detect the vault's existing layout before creating notes. If the vault already uses a folder
structure (including a Korean one), honor it — never silently rename a user's folders. For a new
or unclear vault, default to the English folders Literature/ and Concepts/ with the English
templates (Step 3.2, Step 4.3). For a Korean-structured vault, or a user who prefers Korean notes, use the Korean
folder layout and Korean-heading templates in
${CLAUDE_SKILL_DIR}/references/locale/ko/note_templates.md (and the vault's own hub-note names).
Step 3.1: Check existing literature notes
# Default English layout; substitute the vault's existing folder if one is present
# (e.g. "02 연구/문헌/" for a Korean-structured vault — see references/locale/ko/note_templates.md).
ls "$VAULT/Literature/" | grep -v "📊" | wc -l
Step 3.2: Create literature notes
For each .bib entry, create Literature/{citekey}.md (or the vault's existing literature folder).
Skip if the file already exists — never overwrite, because the user may have added highlights
or personal notes.
Citekey provenance — the note filename is a claim about the library
A note's filename and citekey: field assert that Zotero has an entry with that key. [@key] in a
manuscript, [[key]] between notes, and the Zotero Integration plugin's {{citekey}}.md all depend
on it, and a key that resolves to nothing still looks correct. So the key is read, never
composed:
- Take it from the Better BibTeX-exported
refs.bibentry, or ask Better BibTeX (item.searchover json-rpc — see${CLAUDE_SKILL_DIR}/references/bbt_lookup.md). - If the paper is not in Zotero, add it first (
zotero_add_by_doi) and let BBT mint the key. Phase 2 owns that step for a reason: a note written ahead of its library entry has no key to be right about. - If it cannot be added (no DOI, offline), write the note with
citekey: ""and the tag_needs-citekey. An empty field is recoverable; an invented one is not, because nothing downstream can tell it apart from a real key.
Verify before finishing:
python3 "${CLAUDE_SKILL_DIR}/scripts/check_citekey_provenance.py" --vault "$VAULT" --bib "$REFS_BIB"
INVENTED means the note's key is absent but its DOI resolves to a real key; fix it here.
MISMATCH means the key is real but the note's DOI is the library's DOI for another key, so it
cites another paper; rename to the suggested key (known limit: a note DOI spelled differently from
the library's, e.g. quoted or braced, never raises it). UNPARSED means a literature note's
frontmatter never closes, so its key was not read. With --strict these three exit 1, and a scan
that finds no literature note exits 2. Match notes to papers by DOI, never by key (AMBIGUOUS, UNUSABLE, NO_IDENTIFIER: see the script's
--help), and search the full library (--live) before importing anything it reports missing.
Template
---
notetype: literature
citekey: "{citekey}"
title: "{title}"
authors: "{authors}"
journal: "{journal}"
year: {year}
doi: "{doi}"
pmid: "{pmid}"
created: "{today}"
tags:
- type/literature
- _unread
---
# {title}
## Bibliographic info
- **Authors**: {authors}
- **Journal**: {journal}{volume_issue_pages}
- **Year**: {year}
- **DOI**: [{doi}](https://doi.org/{doi})
{pmid_line}
## Key points (in my own words)
## My thoughts
## Related notes
- [[Research Hub]]
- [[Papers & Reviews]]
-
-
Rules:
notetype: literature— compatible with the Zotero Integration template._unreadtag — change to_readlater after the user reads the PDF in Zotero and adds highlights.- Leave
## Key pointsand## My thoughtsblank — the user fills these in personally. ## Related notescontains 2 hub links + 2 empty slots (reserved for later concept-note linking).- If a PMID is available, add a PubMed link.
Step 3.3: Result report
Obsidian Literature Notes:
Created: 8 notes (new)
Skipped: 3 notes (already exist)
Location: Literature/
Total in vault: 12 literature notes
Phase 4: Concept Extraction (conditional)
Trigger condition
Run this phase only when there are ≥10 literature notes in the vault. If fewer exist, print a status message like "N literature notes — concept extraction unlocks at ≥10" and stop.
Step 4.1: Cross-cutting concept scan
Read all files under Literature/*.md (or the vault's existing literature folder), extract
keywords from each paper's title, journal, and tags and major concepts from the .bib entry titles,
and identify concepts that co-occur across ≥3 literature notes.
Step 4.2: Filtering (5 exclusion rules)
Exclude from concept candidates: model names (GPT-4, Claude, etc.), dataset names (MedQA, ImageNet, etc.), journal names, institution names, and generic technique names (too unspecific). Whatever remains becomes a concept-note candidate.
Step 4.3: Draft concept note
Create Concepts/{concept name}.md (or the vault's existing concept-note folder) from the
template in ${CLAUDE_SKILL_DIR}/references/concept_note_template.md (Korean-structured vault:
the concept template in references/locale/ko/note_templates.md).
Key rules:
- Never auto-fill
## Definition— keep the> TODOmarker; the 2nd-layer note only becomes meaningful once the user writes the definition in their own words. statusalways starts at🌱Seedling.- At least 4 wikilinks under
## Related notes(vault convention).
Step 4.4: Propose to the user
Concept-note candidates (≥3 papers cross-referenced):
1. {Concept A} (4 papers)
2. {Concept B} (3 papers)
3. {Concept C} (5 papers)
Create? (all / selected / skip)
Auto-draft, but create only after user confirmation.
Standalone Modes
This skill can run without a fresh .bib file.
Concept extraction only
On an explicit concept-extraction request, scan existing Literature/*.md (or the vault's existing
literature folder) and run only Phase 4.
References tidy
On a "tidy this project's references" request, locate .bib files inside the workspace and run
Phase 1–3.
Zotero sync only
On a "sync Zotero" request, diff the Zotero collection against the .bib file and add whatever is
missing.
PMID-list ingestion (no .bib)
When the user supplies a list of PMIDs (e.g., from a HANDOFF or a colleague), resolve PMIDs to DOIs via PubMed esummary first, then enter Phase 2 with the DOIs:
PMIDS="12345,67890,..."
curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=${PMIDS}&retmode=json" \
| jq -r '.result | to_entries[] | select(.key != "uids") | "\(.value.uid)\t\(.value.elocationid)\t\(.value.title)"'
For items already in the library (found by the Step 2.2 search), use zotero_manage_collections to
attach them to the project collection without re-adding — re-adding by URL/PubMed-URL would
bypass the search dedup and create duplicates. Record both added and existing items in
references/zotero_collection.json. If a PMID has no DOI in PubMed (older or non-indexed papers),
fall back to zotero_add_by_url with the PubMed URL and mark the entry no_doi: true.
Worklist ingestion (DOI/PMID/Title; no .bib)
When the user supplies a worklist file (a .tsv/.csv/.md table with a DOI column,
optional PMID/Title, or a plain DOI-per-line list — e.g. an SR include set), enter
Phase 2 directly from it: resolve any PMID-only rows to DOIs (esummary above), then run the
search-first dedupe + add loop. The same worklist file feeds Phase 2.7 Route A
(fetch_oa.py reads .tsv/.csv/.md/plain natively), so no reformatting is needed.
MA Scout Skill
This skill handles the pre-protocol phase of a meta-analysis — from idea to ranked topic list.
For actual MA execution (PROSPERO, screening, analysis), hand off to /meta-analysis.
Mode Selection
| Signal | Mode |
|---|---|
| Professor name or profile URL provided | A: Professor-first |
| Clinical question, keyword, trend, or "find me a topic" | B: Topic-first |
| Both supplied (e.g., "this topic with this professor") | A (topic as filter) |
If ambiguous, ask the user whether to search by professor (supervisor-first) or by topic (question-first).
Inputs
Mode A: Professor-first
- Professor name (native-language + English); profile URL (ScholarWorks, SKKU Faculty, Google Scholar, ORCID); PubMed author link (preferably with cauthor_id for disambiguation); known specialty; affiliation history (e.g., "Hospital A → Hospital B → retired")
- Minimum required: name + at least one profile URL or PubMed link
Mode B: Topic-first
- Clinical question or keyword; radiology subspecialty scope; MA type preference (DTA, prognostic, intervention — optional); desired role (solo first author / co-first / supervisor-matched)
- Minimum required: clinical question or keyword
Workflow
Mode A (Professor-first): Phase 0 → 1 → 2 → 3 → 4 → 5 Mode B (Topic-first): T-Phase 0 → T-1 → T-2 → T-3 → T-4 → T-5 Phase 2 (MA Gap Analysis) and Phase 4 (README template) are shared between both modes.
Query PubMed with /search-lit's E-utilities scripts, not WebFetch — they are faster and return
structured JSON/XML: ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh and
parse_pubmed.py beside it. Rate limit: 350 ms between calls (100 ms with NCBI_API_KEY).
MODE A: PROFESSOR-FIRST WORKFLOW
Phase 0: Disambiguation & Context Confirmation
Resolve the author's identity BEFORE any PubMed search:
- Resolve the full English name first. If a cauthor_id is provided, fetch that PMID page to
get the full name + affiliation. NEVER start with an initials-only search, because common
Korean/Asian initials (e.g., "Lee KS") return 300+ papers with massive contamination. The first
search must be
"[Full Name]"[Author]. - Confirm the affiliation chain with the user. Ask whether
{detected affiliation}matches the professor's history (professors move institutions — do not assume), and ask the user's relationship to the professor so topic proposals can be tuned. Skip only if the user already gave an explicit affiliation history. - Profile URL fallback chain:
- 1st: PubMed full-name search (always works)
- 2nd: Google Scholar profile (WebSearch
"[Full Name]" radiology scholar) - 3rd: ResearchGate profile (WebSearch
"[Full Name]" researchgate radiology) - 4th: ScholarWorks / SKKU / university faculty page (if URL provided)
- Last: Scopus/ScienceDirect — returns 403 or a login redirect; never rely on it
Phase 1: Profile Exploration (E-utilities API)
Goal: Identify the professor's 5-6 distinct research pillars.
Step 1 — Total publication count + PMID list:
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh search \
'"[Full Name]"[Author]' 200 \
| python3 ${CLAUDE_SKILL_DIR}/../search-lit/references/parse_pubmed.py esearch
If the parser exits non-zero (an error body, or no count), the count is unknown — re-run the
search; never record it as 0 papers or 0 MAs.
Step 2 — Fetch metadata for MeSH-based clustering (parallel):
# Get PMIDs from Step 1, then fetch summaries
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh fetch_json \
"PMID1,PMID2,..." \
| python3 ${CLAUDE_SKILL_DIR}/../search-lit/references/parse_pubmed.py esummary
Step 3 — Topic-specific counts (launch 4-5 searches in parallel Bash calls):
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh search \
'"[Full Name]"[Author] AND "keyword1"' 5
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh search \
'"[Full Name]"[Author] AND "keyword2"' 5
# ... repeat for each suspected pillar keyword
Step 4 — MeSH term extraction for automatic pillar clustering:
# Fetch full XML for top-cited papers to extract MeSH headings
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh fetch \
"PMID1,PMID2,...,PMID20" \
| python3 -c "
import sys, xml.etree.ElementTree as ET
from collections import Counter
root = ET.fromstring(sys.stdin.read())
mesh_counts = Counter()
for article in root.findall('.//PubmedArticle'):
for mh in article.findall('.//MeshHeading/DescriptorName'):
mesh_counts[mh.text] += 1
for term, count in mesh_counts.most_common(30):
print(f'{count:3d} {term}')
"
→ Top MeSH terms reveal natural research pillars (e.g., "Colonography, Computed Tomographic" = CTC pillar).
Step 5 — Google Scholar profile (parallel with PubMed calls): WebSearch
"[Full Name]" radiology scholar google for h-index and citation data; WebFetch any other profile
URL the user provided (skip Scopus).
Output: Pillar Summary Table. Publication counts and pillar assignments come from E-utilities output and the h-index from the Scholar profile — never estimate them.
| Pillar | Domain | Representative keywords | MeSH terms | Est. # papers |
|---|---|---|---|---|
| 1 | ... | ... | ... | ~N+ |
Phase 2: MA Gap Analysis (Multi-Source)
Goal: For each pillar, determine if a viable MA topic exists using PubMed + Consensus + Scholar Gateway + bioRxiv + PROSPERO.
Run pillars in parallel: up to 4 subagents, each covering 1-2 pillars and running 2a–2g, each reporting raw k, realistic k and every source checked. If PubMed returns 0 or Consensus/Scholar Gateway is unavailable, state that limitation rather than guessing.
2a. PubMed E-utilities — Existing MAs + Primary studies
# Existing MAs (structured count)
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh search \
'[pillar keywords] AND ("meta-analysis"[pt] OR "systematic review"[pt])' 50
# Primary studies with extractable outcomes
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh search \
'[pillar keywords] AND ("sensitivity" OR "specificity" OR "accuracy" OR "prognosis" OR "outcome")' 50
2b. Consensus MCP — Semantic MA gap detection
Use mcp__claude_ai_Consensus__search to find existing SRs/MAs that PubMed keyword search might miss:
query: "systematic review OR meta-analysis [pillar topic] [imaging modality]"
Consensus returns citation-ranked results — check if any highly-cited MA already covers the proposed scope. Limit: max 3 Consensus calls per Phase 2 batch, in total across all agents (rate limit). If rate-limited, wait 30 s and retry once.
2c. Scholar Gateway — Semantic similarity search
Use mcp__claude_ai_Scholar_Gateway__semanticSearch to find MAs under different terminology
(e.g., "pooled analysis" instead of "meta-analysis"), scope-overlapping MAs that use different
keywords, and methodological reviews that partially cover the topic.
2d. bioRxiv/medRxiv — In-press competition detection
Use mcp__claude_ai_bioRxiv__search_preprints to catch MAs posted as preprints but not yet in
PubMed, SR/MA protocols shared as preprints, and very recent primary studies that could change
feasibility.
query: "[pillar keywords] meta-analysis OR systematic review"
server: "medrxiv" (clinical topics; "biorxiv" for preclinical)
2e. Assessment matrix
| Factor | Criteria |
|---|---|
| MA gap | 0 existing = best, 1-3 = check scope overlap, >5 = saturated |
| Primary k | ≥8 for DTA, ≥6 for prognostic (minimum), ≥15 ideal |
| Recency | Last MA >5 years old = update opportunity |
| Competition | Check the last two years for very recent MAs that block entry |
2f. PROSPERO competition check (MANDATORY)
- Search PROSPERO via WebSearch
site:crd.york.ac.uk/prospero [topic keywords]; also try WebFetchhttps://www.crd.york.ac.uk/prospero/#searchadvanced - Look for registered-but-unpublished protocols that could block entry; if a PROSPERO match is found → flag as 🚫 competition risk in ranking
- Record the exact query, the search date and one status:
searched (match found),searched (no match), ornot checked (unavailable/failed); a fetch that returns the site shell or an error page instead of a result list isnot checked. Onlysearched (no match)clears the PROSPERO gate.
2g. Realistic k estimation
- Raw PubMed hit count is NOT the real k — most studies lack 2x2 data or HR, so raw counts overestimate by 3-7x
- Apply conservative discount: k_realistic ≈ raw_count × 0.15–0.30 for DTA topics
- Flag if k_realistic < 8 (DTA) or < 6 (prognostic) as ⚠️ feasibility risk
- Report both raw and realistic estimates, e.g.,
estimated k: ~130 (raw) → ~20–40 (extractable DTA data)
2h. Niche subtopic discovery (if pillar appears saturated, >5 prior MAs)
Try these angles, and use Consensus to check whether the niche angle has already been covered:
- "First MA" rule: the professor's most unique/niche subtopic where MA = 0
- AI/radiomics overlay: classical imaging topic + AI approach
- Treatment response: diagnosis MAs are often saturated; treatment monitoring is often open
- Modality comparison: head-to-head (e.g., CEUS vs MRI) is often underserved
- Guideline gap: professor-authored guidelines → MA supporting/updating them
- Population niche: specific subpopulation, disease subtype, or regional population (e.g., parasitic diseases, TB)
- Temporal update: last MA >5 years old + significant new primary studies since
Phase 3: Topic Ranking
Goal: Rank all viable topics by composite score. Score each candidate on 5 criteria (★1-5):
| Criteria | Weight | Description |
|---|---|---|
| Professor fit | Highest | Core area of the professor's career, publication count, distinctive contribution |
| MA gap | High | No prior MA > ≥5 yr since last MA > recent MA exists |
| Feasibility (k) | High | Number of includable studies and extractability of 2×2 or HR data |
| Clinical impact | Medium | Whether the topic directly informs clinical decision-making |
| Execution ease | Medium | Completable from literature alone; difficulty of managing heterogeneity |
Output: Ranked Topic Table
| Rank | Topic | Professor's Pillar | Prior MA | Estimated k (raw→realistic) | PROSPERO competition | Verdict |
|---|---|---|---|---|---|---|
| 1 | ... | ... | 0 | ~98 → 15–30 | None | ✅ Best fit |
Phase 4: Folder & README Scaffolding
Goal: Create project folders and README for each viable topic.
- Folder location:
{working_dir}/ma-scout/{initials}_{professor_name}/{NN}_{topic_slug}/- Professor folder:
{initials}_{name}(e.g.,KDK_Kim,LKS_Lee) - NN: sequential number within professor (01, 02, ...); topic_slug: English, underscore-separated
- Check existing folders with
lsbefore creating
- Professor folder:
- README.md (PROSPERO-ready): copy the template block from
${CLAUDE_SKILL_DIR}/references/project_readme_template.mdinto{topic_folder}/README.mdand fill it (PICO/PIRD frame, preliminary search, target journal table, backward-planned timeline). Write the research question, PICO/PIRD and README content in English, medical terms always in English, unless the user asks for the Korean PI-facing variant that reference names.
Phase 5: Output Summary
- Save the ranked topic table and README files to the working directory.
- Summarize: total topics scanned, viable topics found, recommended next steps (see Handoff).
MODE B: TOPIC-FIRST WORKFLOW
T-Phase 0: Topic Clarification & Scope
Goal: Refine the user's clinical question into a searchable, PROSPERO-registrable scope. When
the user asks for topic suggestions without a specific idea, read
${CLAUDE_SKILL_DIR}/references/topic_discovery_heuristics.md first to generate candidate questions.
- Parse the input — disease/condition (e.g., "hepatocellular carcinoma"); imaging modality or intervention (e.g., "dual-energy CT", "AI CAD"); outcome type: DTA (Se/Sp), prognostic (HR/OR), intervention (RR/MD), dosimetry; population specifics (e.g., "screening setting", "cirrhotic patients").
- Expand to neighboring angles — propose 3-5 variations, e.g.:
user input: "AI for lung nodule malignancy prediction" → variant 1: AI vs radiologist for lung nodule malignancy prediction (DTA) → variant 2: Radiomics for lung nodule malignancy (DTA) → variant 3: Deep learning for incidental pulmonary nodule management (prognostic) - User selects 1-3 angles to investigate further.
T-Phase 1: Landscape Scan (Multi-Source)
Goal: For each selected angle, rapidly assess the MA landscape. Run all angles in parallel. For each angle:
1a. PubMed — Existing MA count
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh search \
'[topic keywords] AND ("meta-analysis"[pt] OR "systematic review"[pt])' 50
1b. PubMed — Primary study pool
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh search \
'[topic keywords] AND ("sensitivity" OR "specificity" OR "hazard" OR "outcome")' 100
1c. Consensus MCP — Semantic MA discovery
query: "systematic review [topic] [modality]"
Check for MAs using different terminology.
1d. bioRxiv/medRxiv — Preprint competition
query: "[topic] meta-analysis"
server: "medrxiv"
1e. PROSPERO — Registered protocols
WebSearch: site:crd.york.ac.uk/prospero [topic keywords]
Output: Landscape Summary Table
| Variant | Existing MAs | Primary k (raw) | k (realistic) | PROSPERO | Preprint MA | Verdict |
|---|---|---|---|---|---|---|
| 1 | 3 | 120 | 18-36 | 1 | 0 | ⚠️ Competitive |
| 2 | 0 | 85 | 13-25 | 0 | 0 | ✅ Optimal |
T-Phase 2: Feasibility Deep-Dive
Goal: For viable angles (MA ≤ 2, no PROSPERO conflict), run the same Phase 2 (MA Gap Analysis) as Mode A — steps 2a through 2h. With no "Professor fit" to evaluate, focus on:
- Gap certainty — are existing MAs truly non-overlapping with proposed scope?
- k quality — are primary studies heterogeneous enough to warrant MA, or too uniform?
- User's domain fit — does this align with user's radiology AI / imaging expertise?
T-Phase 3: Topic Ranking (Topic-first weights)
| Criteria | Weight | Description |
|---|---|---|
| MA gap | Highest | No existing MA > update opportunity > saturated |
| Feasibility (k) | Highest | k_realistic ≥ 8 (DTA) or ≥ 6 (prognostic) |
| User domain fit | High | Does it match the user's area of expertise? |
| Clinical impact | Medium | Potential to change guidelines; directly tied to clinical decisions |
| Co-author availability | Medium | Access to a domain expert (existing relationship or easy to reach) |
| Execution ease | Medium | Can be done solo vs requires expert interpretation |
Output: Ranked Topic Table
| Rank | Topic | Existing MAs | Est. k | PROSPERO | Co-author needed | Overall |
|---|---|---|---|---|---|---|
| 1 | ... | 0 | 25 | None | Optional | ✅ Optimal |
T-Phase 4: Co-Author Matching (Optional)
Goal: If the user wants a senior co-author, find candidates.
Strategy 1 — Existing network: check memory files and existing professor folders in the working directory for professors whose pillar naturally covers this topic (best match).
Strategy 2 — PubMed reverse search:
# Find prolific authors in this specific topic
bash ${CLAUDE_SKILL_DIR}/../search-lit/references/pubmed_eutils.sh search \
'[topic keywords] AND ("{user_country}"[Affiliation])' 100
Then E-utilities efetch → author frequency; the top 5 most-published authors in this niche are potential co-authors. Cross-check Google Scholar for h-index and recent activity.
Strategy 3 — Self-led (no senior co-author): viable when the user has 2+ published MAs and the topic is methodologically straightforward. A 2nd reviewer (junior colleague or peer) is still needed — flag this in the README. Corresponding author = user.
Output: Co-author recommendation table or a "solo-viable" judgment.
T-Phase 5: Folder & README Scaffolding (Topic-first)
- Folder location:
{working_dir}/ma-scout/TOPIC/{NN}_{Topic_Abbreviation}/(e.g.,01_AI_Lung_Nodule_DTA/). Topic-first projects use theTOPIC/prefix, not professor initials; if a co-author is matched later, the folder can move under the professor folder. - README.md: the Phase 4 template with the Solo-Mode Adaptations in
${CLAUDE_SKILL_DIR}/references/project_readme_template.md(Lead/Domain rows, Team Expertise, self-led timeline). - Summary: same as Mode A Phase 5 — save ranked results and recommend next steps.
Quality Gates
Before finalizing a topic as viable:
- (A) Author identity confirmed — full name resolved via E-utilities efetch, no initials-only contamination
- (A) Affiliation confirmed with user (or from reliable source)
- (A) Professor's publication record demonstrates clear authority in this area
- (B) Clinical question refined to PICO/PIRD (not just a keyword)
- (A) Confirmed MA = 0 or last MA >5 years (via PubMed E-utilities, not assumption)
- Cross-validated via PubMed plus every connected semantic or preprint source (Consensus, Scholar Gateway, bioRxiv/medRxiv); a source that was unavailable is named and recorded as not checked, never as "no competing work found" — no hidden MAs with different terminology, no preprint MA in progress
- Confirmed k_realistic ≥ 8 (DTA) or ≥ 6 (prognostic), where k_realistic = raw count × 0.15–0.30 (§2g: keep 15–30% of raw hits, i.e. a 70–85% discount — not a 15–30% discount)
- PROSPERO searched — the search ran and returned results (query + date recorded, §2f), and no registered competing protocol was found. A PROSPERO search that failed or was unavailable is "not checked", never "none found": this item stays open
- No competing MA from the last two years in press or preprint
- Research question is specific enough for PROSPERO registration
- (B) User's domain expertise sufficient for clinical interpretation (or co-author identified); 2nd reviewer identified or plan to recruit
- (B) If self-led: user has ≥ 2 published MAs (otherwise, recommend co-author)
- README contains: complete PICO/PIRD, PubMed search strategy, Embase draft, target journal with IF, timeline
Handoff
- To
/meta-analysis: when a topic is approved and ready for the PROSPERO protocol (README has PICO + search strategy) - To
/manage-project: when the project folder needs full scaffolding or the user wants results saved to project management - To
/search-lit: when a deeper preliminary search is needed before committing - To
/analyze-stats: when feasibility requires power/sample-size calculation for the estimated k
Phase 6: Pre-Proposal Pipeline (Post-Scout)
After MA Scout identifies viable topics, prepare a "ready-to-propose" package before contacting the professor. Topics are independent: run up to 4 agents per wave, each doing search → fetch → triage → write files.
- Search Execution — E-utilities with broadened synonyms (retmax=200)
- Primary search:
[topic] AND [outcome keywords] - Existing MA search:
[topic] AND ("meta-analysis"[pt] OR "systematic review"[pt])
- Primary search:
- Metadata Collection —
fetch_json→esummary(batch 40-50 PMIDs) - Title-Based Triage — classify as INCLUDE / MAYBE / EXCLUDE
- Check for existing MAs within the results — the initial scout may have missed them
- Separate sub-approaches that contaminate the pool (e.g., bronchoscopic vs percutaneous)
- Flag the professor's own papers (authority evidence) and retracted papers
- PRISMA Flow Draft — Identification → Screening → Eligibility → Included (estimated)
- Gap Re-assessment — update the MA count and re-position if needed: MA=0 → "first MA" | MA=1 (>5yr) → "update MA" | MA≥3 (recent) → skip/niche
- Output Files:
candidates.md— full triage table + PRISMA flow + gap findingREADME.md— updated Preliminary Search section with actual numbers
Professor Contact Package — the pre-proposal gives the professor: candidate count + gap evidence (e.g., "MA = 0, 35 studies to include"), a clear role description (e.g., "independent screening review + discussion only"), and the urgency of PROSPERO pre-registration to secure the topic.
Make-Figures Skill
Data Privacy Check
Before reading any data file, check whether it might contain Protected Health Information (PHI):
- If
*_deidentified.*files exist in the working directory, use those preferentially. - If only raw CSV/Excel files exist (no
*_deidentified.*counterpart), ask the user, in their language, whether the data contains patient identifiers (names, national ID / RRN, contact details, etc.) and, if so, to de-identify it first with the/deidentifyskill. - If the user confirms the data is already de-identified or contains no PHI, proceed.
Reference Files
- Figure specifications:
${CLAUDE_SKILL_DIR}/references/figure_specs.md— read it before generating any figure (journal dimensions, DPI, file formats, palettes, font sizes, panel layouts, caption format). - Figure style:
${CLAUDE_SKILL_DIR}/../analyze-stats/references/style/figure_style.mplstyle
Journal AI-Image Policies (CRITICAL — check BEFORE generation)
| Journal family | Policy on AI-generated images | Disclosure required |
|---|---|---|
| JACC family (incl. JACC: Asia, JACC Imaging, JACC EP, JACC BTS) | Prohibited without prior Editor-in-Chief permission (JACC pathway, PMC10167500) | Cover-letter pre-submission inquiry + ICMJE-style declaration |
| NEJM | AI image generation prohibited | N/A |
| Radiology / Radiology AI | Allowed with disclosure | Manuscript disclosure block |
| Springer Nature (Nature Portfolio, BMC, Springer journals) | Allowed only when the visual is derived from independently verifiable data, source material, methods or code; AI visuals without verifiable inputs are "opaque" and not permitted. Charts from datasets and author-reviewed code-generated figures are the policy's own allowed examples (AI in manuscript preparation) | Declare model and purpose; disclose non-generative image edits in the caption |
| Elsevier journals (publisher policy) | Explanatory images (flow charts, schematics) allowed. Data visualizations only when directly derived from the underlying data by reproducible methods. Primary research images, including radiology scans and patient images, must not be created or altered with AI. AI images produced as part of the research methods are permitted. Graphical abstracts: dedicated illustration tools, not general-purpose generative AI (Generative AI policies for journals) | Caption + AI disclosure statement (explanatory); Methods (data visualizations, research-method use) |
| Lancet family | Disclosure required, generation discouraged | Manuscript disclosure |
| Default (target unknown) | Treat as prohibited until confirmed | N/A |
Publisher pages set the floor; a journal's own guide can be stricter (JACC is an Elsevier journal). Check the target journal's guide as well as its publisher's.
Hard rule: For JACC, NEJM, or any "unknown" target journal, never use Gemini / DALL-E / Midjourney / Stable Diffusion / Nano Banana to create images that will appear in figures, Central Illustrations, or graphical abstracts. AI text-editing of the manuscript prose remains acceptable subject to standard disclosure.
Default workflow when AI images are not allowed
- SMART Servier Medical Art — https://smart.servier.com/, CC BY 4.0, free, 3,000+ vector medical icons (anatomy, organs, ethnicity-specific human figures, drugs, devices). Commercial / journal use allowed. Required attribution (1 line in figure legend OR methods):
Anatomical icons modified from SMART Servier Medical Art (CC BY 4.0).
- NIAID BioArt (https://bioart.niaid.nih.gov) — public domain (US Govt), microbiology / immunology / lab-tech focus.
- BioRender (https://www.biorender.com) — institutional license usually required; use the exported "Publication-ready" PNG/TIFF and cite per BioRender publication policy.
- For "diseased" variants not directly available (e.g., calcified vessel from a clean vessel): reuse the healthy asset and overlay disease markers via matplotlib
scatter/Circle/PathPatch. Keeps the entire pipeline non-AI and reproducible.
Even when AI images are allowed, experienced reviewers recognise AI-generated illustrations (small decorative icons that add no information, overly uniform layouts, generic clip-art style). For high-impact submissions, prefer Servier / BioArt / BioRender + matplotlib overlays over AI.
Asset directory convention
manuscript/figures/_assets_servier/ # CC BY 4.0 source PNGs
manuscript/figures/_assets_servier/CITATION.md # source URL + download date per asset
manuscript/figures/_assets_data/ # data-driven raster (R / matplotlib heat maps, KM, etc.)
manuscript/figures/_legacy/ # archived prior versions
Composition scripts should load only from _assets_servier/ and _assets_data/. If a script imports from _assets_ai/, treat it as a policy violation for JACC/NEJM/unknown targets.
AI Image Generation (Optional)
AI illustration is a supplementary option: every figure and visual abstract can be completed without an API key. If GEMINI_API_KEY is set, generate_image.py can generate illustrations (procedural schematics, anatomical illustrations), subject to the policy table and hard rule above:
python ${CLAUDE_SKILL_DIR}/scripts/generate_image.py \
"Clean medical illustration of a CT-guided lung biopsy procedure, \
flat vector style, white background, no text" \
--output output.png --aspect 16:9
If GEMINI_API_KEY is not set, use ${CLAUDE_SKILL_DIR}/references/medical_illustration_sources.md.
Visual Abstract / Graphical Abstract
European Radiology made graphical abstracts mandatory for all Original Articles from first revision
(Jan 2025); other journals encourage or accept them (status list in template_guide.md). Check the
target journal profile (write-paper/references/journal_profiles/) for its visual abstract
requirements before starting.
Workflow
- Check journal template. Look for an official PPTX template in
${CLAUDE_SKILL_DIR}/references/visual_abstract_templates/{journal}.pptx. If no journal-specific template exists, usemedsci_default.pptx. - Extract content from the manuscript:
- Title: Full article title
- Hypothesis/Question: Derived from Key Point 1 or study objective (max 1 sentence)
- Methodology: Brief flowchart or ≤3 bullets, <6 words each
- Visual element: Study's own figure (ROC curve, flow diagram, representative image)
- Badges: Patient cohort (N=...) | Modality/organ | Single/Multi-center
- Main finding: Derived from Key Point 3 (<20 words)
- Citation: Journal (year) Authors; DOI
- Select visual element (priority order):
- Study's own figures (ROC, flow diagram, representative image) — always preferred
- Free illustration from Servier Medical Art or NIAID BioArt
(see
${CLAUDE_SKILL_DIR}/references/medical_illustration_sources.md) - Manual drawing in PPT/Keynote/Figma
- AI generation via
generate_image.py --style medical(only if GEMINI_API_KEY set)
- Generate using the script:
python ${CLAUDE_SKILL_DIR}/scripts/generate_visual_abstract.py \ --template medsci_default \ --title "Article Title" \ --hypothesis "Research question" \ --methods "Method 1|Method 2|Method 3" \ --finding "Main finding statement" \ --citation "Eur Radiol (2026) Author A et al; DOI:..." \ --visual figures/fig1_roc_curve.png \ --badges "N=450|CT chest|Multi-center" \ --output figures/visual_abstract.pptx - Review with user. Open the PPTX to verify layout and content. Iterate.
- Export. PPTX is the primary deliverable. For PNG: open in PowerPoint/Keynote → export,
or use LibreOffice CLI (
soffice --headless --convert-to png).
Design: one page, landscape (16:9) or per the journal template; three sections (study question → key method → main result); the study's actual figures rather than generic graphics; minimal text; no decorative clip-art.
Available Templates
| Template | File | Use When |
|---|---|---|
| MedSci Default | medsci_default.pptx | Any journal without an official template |
| JACC Central Illustration | jacc_central_illustration.pptx | JACC family journals (use --type central-illustration) |
Using a journal's own template. We do not redistribute journals' templates (e.g., European
Radiology's EURA-GA-Jan2025.pptx): download it and pass its absolute path:
python ${CLAUDE_SKILL_DIR}/scripts/generate_visual_abstract.py \
--template /absolute/path/to/EURA-GA-Jan2025.pptx ...
--template takes an absolute path to any .pptx. The script locates the fields by their text
content rather than by shape name, so a journal's own template works unmodified. If the path does
not exist it falls back to medsci_default.pptx.
To add a new journal template: see ${CLAUDE_SKILL_DIR}/references/visual_abstract_templates/template_guide.md.
Central Illustration vs Visual Abstract
JACC family journals (JACC, JACC: Asia, JACC: Cardiovascular Imaging, JACC: Heart Failure, JACC: CardioOncology, JACC: Clinical Electrophysiology, JACC: Basic to Translational Science) require a Central Illustration (CI) with every Original Article. A CI is not a Visual Abstract: it conveys one key finding and contains no methods. Before making one, read ${CLAUDE_SKILL_DIR}/references/jacc_central_illustration_principles.md (CI vs VA table, the five Fuster-Mann rules, layout and validation thresholds).
CI mode invocation
python ${CLAUDE_SKILL_DIR}/scripts/generate_visual_abstract.py \
--type central-illustration \
--visual figures/central_illustration_v2.png \
--citation "FirstAuthor Last et al. Journal Name 2026; vol(issue):pages." \
--output submission/jacc_asia/central_illustration.pptx \
--ci-zones 3 --ci-label-words 22 --ci-numerical-points 2 \
--ci-raw-text "warranty drops to 3 years in age 45+ with cardiometabolic burden; MASLD HR 1.77"
CI mode validates before rendering and rejects (exit 2) if any of: zones > 3, label words > 30, numerical points > 4, or methodology terms (cohort flow / inclusion criteria / exclusion criteria / study design / enrollment / randomized / sample size / CONSORT / PRISMA / STARD) appear in --ci-raw-text. Override individual rules with --ci-allow {zones|words|numerical|methods} only when you have a defensible reason. Submit only the content figure + citation: JACC editorial applies the red border and blue "CENTRAL ILLUSTRATION:" header after acceptance.
Workflow
Any reference in a caption, legend or illustration needs a DOI or PMID confirmed via /search-lit;
mark one you cannot confirm [UNVERIFIED - NEEDS MANUAL CHECK]. Mark an unconfirmed clinical
definition, criterion or guideline claim [VERIFY] and ask — the submission gates block on both.
Step 1: Specify
Before specifying figure type, read ${CLAUDE_SKILL_DIR}/references/design_principles.md —
identify (1) the one-sentence key message, (2) audience and reading-time budget, and
(3) whether a figure is the right vehicle (vs a small table or in-line text). Skip only when the
figure is mandated by a reporting guideline (e.g., PRISMA / CONSORT flow), and even then apply the
cognitive-load checklist.
For reporting-guideline figures, also load
${CLAUDE_SKILL_DIR}/references/reporting_guideline_figure_map.md — which guideline mandates which
figures and whether this skill ships an official template (✅), generic flow only (⚠️), or needs
manual production (❌). Critical for AI-extension guidelines (CONSORT-AI, STARD-AI, TRIPOD+AI,
CLAIM 2024, DECIDE-AI).
For medical AI / engineering pipeline figures (DICOM workflow, annotation pipeline, federated
learning topology, model architecture), also load
${CLAUDE_SKILL_DIR}/references/pipeline_concepts_medical_ai.md — canonical layouts, required
annotations, and tool selection per type.
Optional flags:
--study-type <type>: One of:diagnostic-accuracy,ai-validation,meta-analysis,dta-meta-analysis,observational-cohort,rct,case-report. When set, auto-generate the full figure set from the Study-Type Figure Sets table below without prompting for individual figure types.--data-dir <path>: Directory containing analysis outputs (CSVs,_analysis_outputs.md). Default: current working directory.
Ask the user for (infer what the context already gives, and confirm before proceeding):
- Figure type — skipped when
--study-typeis provided - Data source (file path, DataFrame, or manual values)
- Target journal (for dimension/font requirements)
- Panel layout (single panel, multi-panel, or let you decide)
- Any special requests (annotations, highlights, reference lines)
- Study type (if not passed via
--study-type): determines the required figure set
Step 2: Configure
- Load the figure style file (skeleton below).
- Look up journal-specific dimensions in
${CLAUDE_SKILL_DIR}/references/figure_specs.md. - Set the colorblind-safe palette (Wong palette by default).
- Set font sizes per element type (title, axis label, tick label, legend, annotation) from
figure_specs.md.
Step 3: Generate
Use Python (matplotlib/seaborn, with specialized libraries as needed). Compose each data plot from
its anatomy model in ${CLAUDE_SKILL_DIR}/references/exemplar_plots/ (index in its README.md);
for box/violin, bar and heatmap figures follow the per-type conventions in figure_specs.md. Flow
diagrams never use matplotlib — see the Tool Selection Guide.
Script structure:
"""
Figure: {description}
Date: {YYYY-MM-DD}
Target: {journal}
Dimensions: {width} x {height} inches @ {DPI} DPI
"""
import numpy as np
import matplotlib.pyplot as plt
import os
style_path = os.path.join(os.environ.get('CLAUDE_SKILL_DIR', '.'), '../analyze-stats/references/style/figure_style.mplstyle')
if os.path.exists(style_path):
plt.style.use(style_path)
# Wong colorblind-safe palette
WONG = ['#000000', '#E69F00', '#56B4E9', '#009E73',
'#F0E442', '#0072B2', '#D55E00', '#CC79A7']
np.random.seed(42)
Never fabricate data points. If sample data is needed for a template demo, label it "example data".
When a figure is produced by a data-driven .py/.R script (ROC, forest, KM, calibration, heat
maps), lint that script before finalizing with the /analyze-stats code-quality gate
(check_generated_code.py {script} --strict): it catches a missing plotting seed for any
bootstrapped CI band, a hardcoded absolute data path, or a hand-typed data literal that should have
been read from the analysis CSV.
Step 4: Review
Present the figure to the user (layout, labels and annotations, colors/sizing/emphasis) and iterate until the user approves.
Step 4b: Critic Loop (self-critique before final export)
Before Step 5 Export, run two stages — deterministic quantitative checks, then your own qualitative review — to decide whether to re-render or hand off to the user.
Stage 1: Quantitative checks (critic_figure.py)
python ${CLAUDE_SKILL_DIR}/scripts/critic_figure.py \
figures/fig1_stard.png \
--type stard \
--spec-min-dpi 600 \
--spec-width-in 7.0 \
--source-text figures/fig1_stard.txt \
--out figures/fig1_stard.critique.json
--source-text is optional (expected strings for OCR coverage). The JSON report covers DPI and
physical width vs. the journal spec, the dominant-color breakdown and out-of-Wong-palette fraction,
and OCR word count, minimum text height, and source-word coverage. When the image carries no DPI
metadata, the DPI check uses the resolution at the spec width (width_px / --spec-width-in). A check
that cannot run (physical width without DPI metadata, OCR without pytesseract) is listed under
not_run and the summary reads INCOMPLETE, never PASS; --strict exits 3 on INCOMPLETE.
Stage 2: Qualitative review
- Use the Read tool to load the generated PNG.
- Read the corresponding rubric file (sections A–G in each):
- Flow diagrams:
${CLAUDE_SKILL_DIR}/references/critic_rubrics/flow_diagram.md - Data plots:
${CLAUDE_SKILL_DIR}/references/critic_rubrics/data_plot.md - For PRISMA / CONSORT / STARD / STROBE, also read
${CLAUDE_SKILL_DIR}/references/flow_diagram_lessons.md(official-template fidelity, PDF export via VML fallback, docx XML escape, sequential placeholder mapping, frozen-version sync with the manuscript). - For AI-extension guidelines or pipeline figures, also check against the
reporting_guideline_figure_map.mdrow /pipeline_concepts_medical_ai.mdloaded in Step 1.
- Flow diagrams:
- Read the
_why.mddesign notes in${CLAUDE_SKILL_DIR}/references/exemplar_diagrams/{type}/— hierarchy, whitespace, typography, emphasis, colour. They are the anchors. Where a rendered exemplar is bundled (template_output*.png), Read 1–2 of those too. Figures cropped from published papers are not shipped (see that directory's README); a user's own local exemplars can be used instead. For a non-flow data plot, read the matching anatomy model in${CLAUDE_SKILL_DIR}/references/exemplar_plots/(e.g.,forest_plot.md). - Score every rubric item as PASS / PARTIAL / FAIL with a one-line note, using the format at the bottom of the rubric file.
- Emit a "Required edits before next render" list of concrete source-code changes (node/label renames, count corrections, matplotlib parameter tweaks).
Refinement loop
- If all items are PASS → proceed to Step 5 Export with
critic_pass: yes. - If any item is FAIL → apply the required edits to the source (flow-diagram config/R script or matplotlib script), re-render, and re-run Stage 1 + Stage 2. Default maximum is T=2 rounds; the user may request up to T=3.
- If after the max rounds some items remain PARTIAL, proceed with
critic_pass: partialand record the residual items in the manifest'scritic_notesfield.
Record the final state in _figure_manifest.md (see Study-Type Figure Sets) so downstream steps
(/write-paper Phase 2 embedding and Phase 7 DOCX build) and future critic passes can see the
history.
Step 5: Export
Save final outputs to analysis/figures/ (analysis/figures/*.pdf, analysis/figures/*.png):
- PDF (vector format, preferred for journal submission)
- PNG (300 DPI raster, for review and presentation)
- TIFF (if the journal requires it, 300 DPI LZW compression) — produce it with
export_portal_tiff.py, not a rawmagick ... output.tiff: that keeps the alpha channel (transparent regions print black on many production pipelines) and stays uncompressed (a 600-dpi RGBA TIFF blows past a portal's 25 MB cap). The script white-flattens RGBA→RGB, LZW-compresses, and verifies the result is pixel-identical to that flatten. Use it when a portal accepts only.tiff/.jpeg/.eps(Springer Nature SNAPP) or caps figure size (JACC: Asia):python3 ${CLAUDE_SKILL_DIR}/scripts/export_portal_tiff.py --in figure.png --out figure.tiff --max-mb 25 # exit 1 if still over the cap
Name files descriptively: fig1_roc_curve.pdf, fig2_consort_flow.pdf, etc.
For PPTX outputs (visual abstract, central illustration, or any deck the figure
will live in): run the Mac-compatibility validator before delivery. PowerPoint
Mac silently drops TIFF, renders <a:sp3d> 3-D bevels as red outlines that PDF
export does not show, and refuses to open files whose app.xml slide count
disagrees with the actual slide XML files.
python ${CLAUDE_SKILL_DIR}/scripts/validate_pptx_mac_compat.py \
figures/visual_abstract.pptx \
--json figures/visual_abstract.mac_compat.json \
--strict
Exit code 1 means at least one FAIL — fix per the fix: field in the JSON
report and re-render the PPTX before delivery. Exit code 0 with WARN is
acceptable. Skip this step when the figure is PNG/PDF only (no PPTX).
Step 6: Design QC Checklist
Before delivering the final figure, verify all items:
- Font: Sans-serif (Arial/Helvetica), minimum 7pt, axis labels ≥ 9pt
- Color: Wong/Okabe-Ito colorblind-safe palette used
- Colorblind test: Works for deuteranopia (no red-green only distinctions)
- Grayscale test: Information preserved when printed in black & white
- Alignment: All elements on a consistent grid; panels aligned
- Vector output: PDF/SVG saved (not just PNG)
- Resolution: ≥ 300 DPI for raster elements, ≥ 600 DPI for line art
- Journal specs: Dimensions, font, and format match the target journal (if a figure exceeds a limit, resize it and report the adjustment)
- No chartjunk: No 3D effects, unnecessary gridlines, gradient fills, or decorative elements
- Caption: Drafted per
figure_specs.md(Caption Writing Guidelines) with key finding, abbreviations, statistical details, and sample size
Study-Type Figure Sets
When the study type is known (from /write-paper Phase 0 or user specification), generate the complete required figure set without asking for each figure individually.
| Study Type (Guideline) | Required Figures |
|---|---|
| Diagnostic accuracy (STARD) | STARD flow diagram, ROC curve, confusion matrix, calibration plot |
| AI validation (TRIPOD+AI / CLAIM) | Flow diagram, ROC curve, confusion matrix, calibration plot, feature importance or SHAP, Grad-CAM (if imaging) |
| Meta-analysis (PRISMA) | PRISMA flow diagram, forest plot, funnel plot |
| DTA meta-analysis (PRISMA-DTA) | PRISMA flow diagram, paired forest plot (Se + Sp), SROC curve, Deeks funnel plot |
| Observational cohort (STROBE) | Flow diagram, Kaplan-Meier curves (if survival endpoint) |
| RCT (CONSORT) | CONSORT flow diagram, primary endpoint figure |
| Case report / series (CARE) | Clinical timeline figure (exemplar_plots/clinical_timeline.md), annotated multimodality imaging panel when visually load-bearing (exemplar_plots/imaging_panel.md); for a series, an all-cases summary table |
| Oncology treatment study (CONSORT RCT or single-arm phase II; REMARK for biomarker subsets) | CONSORT or cohort flow diagram, Kaplan-Meier PFS/OS with number-at-risk table (exemplar_plots/km_curve.md), waterfall plot of best response (exemplar_plots/waterfall_plot.md), swimmer plot for durability (exemplar_plots/swimmer_plot.md), spider plot when response kinetics matter (exemplar_plots/spider_plot.md), cumulative incidence instead of 1 - KM when competing risks are present (exemplar_plots/cumulative_incidence.md) |
The manifest is mandatory. After generating all figures, write
analysis/figures/_figure_manifest.md — one row per figure (Figure | Path | Type | Tool | Critic | Rounds | Description) plus a ## Critic notes section recording any residual PARTIAL items and
why they were accepted. It is consumed by /write-paper Phase 2 (figure embedding) and Phase 7
(DOCX build); verify it exists and is non-empty before finishing. Read
${CLAUDE_SKILL_DIR}/references/figure_manifest.md when writing it (format and field definitions).
Tool Selection Guide
Data plots: matplotlib/seaborn → PDF + PNG (this skill)
Flow diagrams: generate_flow_diagram.R (DiagrammeR + rsvg) → PDF + 300/600 dpi PNG
Final assembly: pandoc or python-docx (auto-embedded in DOCX)
Flow Diagrams → Dedicated Tools (NOT matplotlib)
STARD / CONSORT / PRISMA / STROBE flow diagrams MUST use the standardized R pipeline
scripts/generate_flow_diagram.R (DiagrammeR + Graphviz dot + rsvg) — the single canonical tool
for all four. Do NOT use matplotlib FancyBboxPatch (manual coordinates break when text
changes, and patches distort when embedded in DOCX). Do NOT use D2 for new flow diagrams (weak
font control, overlap needs manual post-processing); D2 is a legacy fallback only when R is
unavailable.
| Type | Recommended Tool | Why |
|---|---|---|
| STROBE (cohort / cross-sectional) | scripts/generate_flow_diagram.R --type strobe | Single canonical tool; auto-layout; vector PDF + 300/600 dpi PNG |
| CONSORT (RCT) | scripts/generate_flow_diagram.R --type consort | Same pipeline; monochrome Arial default |
| PRISMA 2020 (SR/MA) | scripts/generate_flow_diagram.R --type prisma | Faithfully implements PRISMA 2020 structure; avoids PRISMA2020 R package's webshot-based raster PDF issue |
| STARD (DTA) | scripts/generate_flow_diagram.R --type stard | Same pipeline; supports 2x2 reference-standard split |
| Pipeline Diagram | D2 (legacy) | Until pipeline-diagram support is added to the R script |
YAML config → Rscript ${CLAUDE_SKILL_DIR}/scripts/generate_flow_diagram.R --type <t> --config <yaml> --out <prefix> →
PDF + 300/600 dpi PNG. Templates in references/exemplar_diagrams/{strobe,consort,prisma,stard}/template_input.yaml.
Read ${CLAUDE_SKILL_DIR}/references/flow_diagram_recipe.md when generating a flow diagram (YAML
schema, fixed style, per-project create_figure1.R pattern, D2 fallback); a ROC curve or forest
plot needs none of it.
Every box label contains its count (e.g., "Assessed for eligibility\n(n = 450)"). Numbers in
labels must be CSV-derived — author the YAML from an R/Python script that reads the upstream data —
or hand-written only when the value lives in a commit-tracked data artifact (cite the source file
in a comment).
Caption ↔ flow-SSOT reconciliation (before Step 5 Export). The flow-diagram config is the single source of truth for participant counts; a hand-written Figure 1 caption drifts from it when the cohort is re-locked but the caption is not ("caption says n = 1,284 analytic, diagram box says n = 998"). Re-derive the caption counts from the flow config and reconcile:
python3 ${CLAUDE_SKILL_DIR}/scripts/derive_figure_legend_counts.py \
--flow-config figures/figure1_strobe_graphviz.yaml \
--manuscript manuscript/index.qmd \
--out qc/figure_legend_counts.json --strict
Any n = N in the caption that is not a box count in the flow config is a MISMATCH (stale
caption) — update the caption from the config, never the reverse. The reconciler is stdlib-only and
parses the config as text, so it works regardless of the flow tool. When no Figure 1 caption is found
or the caption carries no n = N, the verdict is NOT_CHECKED (exit 2 under --strict), not OK.
Known limits: only n = N notation is read (a bare "1,150 patients" is not), and agreement is set
membership, so a count moved to the wrong box (excluded and analysed swapped) is not detected; check
those by eye against the diagram.
Official Reporting Guideline Templates → templates/official/
When a journal requires the canonical, statement-issued template (rather than the auto-laid-out R
version), use the bundled official files in
templates/official/{prisma2020,consort2010,stard2015,spirit2013}/.
| Guideline | What ships | When to use |
|---|---|---|
| PRISMA 2020 | Locally built .pptx (4 variants) + fill_prisma_template.py | Reviewer asks for the official PRISMA 2020 layout, or you want editable PowerPoint instead of an R-rendered PDF. |
| STROBE (cohort) | Parametric .pptx builder build_strobe_template.py (YAML config) | Cohort/case-control Figure 1 when co-authors want PowerPoint they can hand-edit. Optional left-side phase column: omit stages: for the plain STROBE convention; include it for the PRISMA-style Identification/Screening/Inclusion/Analysis column. Pair with generate_flow_diagram.R --type strobe for the vector PDF/TIFF submission file. |
| CONSORT 2025 | Official .docx flow diagram + checklist | RCT submissions to journals that mandate the consort-spirit.org template. |
| STARD 2015 | Official .pdf flow diagram + .docx checklist | Diagnostic accuracy studies; flow diagram is fixed PDF, checklist is editable. |
| SPIRIT 2025 | Official .docx participant timeline + checklist | Trial protocols. |
Refresh / fill workflow:
# Refresh from canonical sources (CC-BY 4.0 / public-statement licenses)
bash ${CLAUDE_SKILL_DIR}/scripts/fetch_official_templates.sh
# Build PRISMA 2020 .pptx (one-time; site blocks programmatic .docx fetch)
python3 ${CLAUDE_SKILL_DIR}/scripts/build_prisma2020_template.py \
--variant new \
--out ${CLAUDE_SKILL_DIR}/templates/official/prisma2020/PRISMA_2020_flow_new_v1.pptx
# Fill counts — positional 10-tuple matching most SR/MA workflows:
# n_db, n_dup, n_screened, n_screen_excluded,
# n_sought, n_assessed, n_excl_r1, n_excl_r2, n_excl_r3, n_studies
python3 ${CLAUDE_SKILL_DIR}/scripts/fill_prisma_template.py \
--template ${CLAUDE_SKILL_DIR}/templates/official/prisma2020/PRISMA_2020_flow_new_v1.pptx \
--counts "315,122,186,7,111,204,102,84,3,15" \
--out fig1_prisma_filled.pptx
# Or use full JSON mapping for studies with non-standard PRISMA splits
python3 ${CLAUDE_SKILL_DIR}/scripts/fill_prisma_template.py \
--template ${CLAUDE_SKILL_DIR}/templates/official/prisma2020/PRISMA_2020_flow_new_v1.pptx \
--counts-file my_counts.json \
--out fig1_prisma_filled.pptx
# STROBE — parametric single-script builder (cohort study; spine structure varies per study).
# YAML schema: stages, spine (id/stage/text), exclusions (after/text). Consecutive same-stage
# rows share one phase label automatically.
python3 ${CLAUDE_SKILL_DIR}/scripts/build_strobe_template.py \
--config figures/figure1_strobe.yaml \
--out figures/figure1_strobe.pptx
The STROBE builder checks that the exclusion cascade closes: for every link that declares an
exclusion, the spine box count minus the exclusions after it must equal the next spine box
(A - Σ(exclusions after A) == B). It warns loudly on any imbalance and, with --strict-cascade,
refuses to build — catching figure arithmetic drift (a dropped exclusion leaving the figure short of
the analytic N) that text and prose gates miss. Run python3 ${CLAUDE_SKILL_DIR}/scripts/_strobe_cascade.py --config figure1_strobe.yaml --strict to check a config without rebuilding the diagram. A config in neither
schema exits 2; under --strict, a spine config with no checkable exclusion link also exits 2 rather
than reporting OK.
Known limits: the generate_flow_diagram.R nodes/edges schema is recognised but not evaluated; the
helper prints NOT_ASSESSED and exits 0. Reading which box a dashed exclusion is subtracted from was
tried and flagged correct diagrams (an exclusion beside the box just before an exposed/unexposed or
index-positive/negative split, when the previous step has its own exclusion), so that finding is left
open. Check every dashed exclusion against its adjacent boxes by eye.
For STROBE the canonical KJR/Radiology/BMJ submission flow is:
- Render the vector submission file via the auto-fitting Graphviz path:
Rscript ${CLAUDE_SKILL_DIR}/scripts/generate_flow_diagram.R --type strobe --config figures/figure1_strobe_graphviz.yaml --out figures/figure1 - Build the editable PowerPoint companion via
build_strobe_template.pyso co-authors and senior reviewers can adjust prose/positioning before sign-off. - Re-export the final PPTX to PDF/TIFF only after co-author edits are integrated.
See templates/official/NOTES.md for licenses, attribution, and refresh notes.
Style Rules
Palettes, font sizes, panel layouts, and caption format are in figure_specs.md. In addition:
- All figure text (labels, legends, annotations) is in English; medical terminology is always English.
- Never use red-green only distinctions. Use line style (solid, dashed, dotted) in addition to color for line plots, and marker shape in addition to color for scatter plots.
- Panel labels: uppercase bold letter (12 pt), top-left of each panel.
- No figure titles in the plot itself (the title goes in the caption).
- Align multi-panel figures on a grid, with consistent axis ranges across comparable panels.
- Significance stars (* p<0.05, ** p<0.01, *** p<0.001) go above comparison brackets; report the exact p-value in the figure legend or caption, not in the plot.
- For AUC, correlation, or agreement: display in the legend with 95% CI.
Skill Interactions
| When | Call | Purpose |
|---|---|---|
| Need statistical values for plot | /analyze-stats | Get computed values (AUC, CI, p-values) |
| Flow diagram for manuscript | /write-paper Phase 2 | Coordinate with Tables & Figures plan |
| Caption review | /write-paper Phase 7 | Final polish pass |
Manage-Project Skill -- Research Project Management
Commands
/manage-project init {name} --type {type} --journal {journal} [--ssot] [--zotero-collection NAME]
Parameters:
{name}-- Project identifier (e.g.,nnunet-skull-fracture,rfa-meta-analysis)--type-- Paper type:original | meta | case | animal | technical | ai_validation | letter--journal-- Target journal:RYAI | AJR | Radiology | European_Radiology | KJR | INSI | AJNR | generic--ssot-- EmitSSOT.yaml(schema v1) fromtemplates/SSOT.yaml.templateinstead of legacyproject.yaml. Required, but not sufficient, for Phase 1C auto-enforce (the PostToolUse verify-refs hook blocks instead of warns): enforce also needsqc/migration_complete, which init does not write. After init, runmigrate_project_to_ssot.py --project-root {target_dir} --mark-completeto validate the SSOT.yaml and write the marker. New projects should pass--ssot; legacy in-flight projects stay onproject.yamluntil/manage-project migrate-ssotis run.--zotero-collection NAME-- Optional. Create a Zotero collection via pyzotero and populatelibrary_id+collection_keyin the contract. RequiresZOTERO_API_KEY+ZOTERO_LIBRARY_ID(optionallyZOTERO_LIBRARY_TYPE, defaultuser). With pyzotero or the credentials missing, the contract is scaffolded withlibrary_id: null/collection_key: nulland a WARN is printed.
SSOT template substitutions: {{PROJECT_ID}} → {name}; {{PROJECT_TYPE}} → the SSOT
project_type enum mapped from --type (original → original_research, meta → meta_analysis,
case → case_report, ai_validation → ai_validation, else other). Without
--zotero-collection, library_id / collection_key stay null until the owner links an
existing collection.
Run scripts/init_project.py:
python3 "${CLAUDE_SKILL_DIR}/scripts/init_project.py" \
--name {name} --type {type} --journal {journal} [--ssot] \
--project-root {target_dir}
It writes the contract file (SSOT.yaml with --ssot, else legacy project.yaml), the
directory scaffold, the minimal stubs scripts/validate_project_contract.py requires
(manuscript/index.qmd, artifact_manifest.json, qc/status.json), the memory-file stubs,
and project_state.json. qc/migration_complete is NOT written by init — that marker belongs
to the migrate script. For an --ssot project, run it with --mark-complete (no project.yaml
needed): it validates the existing SSOT.yaml and writes the marker only on PASS. Until then the
reference hook stays in warn mode.
Do not hand-build the scaffold. The script is the source of truth for the tree and for
project_state.json; a hand-built one drifts from what the contract validator expects. Read
${CLAUDE_SKILL_DIR}/references/init_scaffold.md only when you need to know where a scaffolded
file lands or what a project_state.json field means.
After the script, copy the matching reporting-guideline checklist from
${CLAUDE_SKILL_DIR}/../check-reporting/references/checklists/ into references/checklist_{GUIDELINE}.md.
/manage-project migrate-ssot [--no-mark-complete]
Converts a legacy project.yaml project into SSOT.yaml form and, by default, touches qc/migration_complete so Phase 1C auto mode switches from warn to enforce.
On an SSOT-native project (SSOT.yaml present, no project.yaml — e.g. init --ssot) there is nothing
to convert: --mark-complete validates the existing SSOT.yaml and writes the marker on PASS;
SSOT.yaml is not rewritten.
python3 "${CLAUDE_SKILL_DIR}/scripts/migrate_project_to_ssot.py" \
--project-root . --write --mark-complete
--no-mark-complete: run with--writeonly. Use it when the project still has open QC failures — enforcement is deferred until the migration is validated.- The migrate script refuses to touch
qc/migration_completeunless the generated SSOT.yaml passesvalidate_project_contract.pyANDcontract_mode=ssot. Do nottouch qc/migration_completemanually.
Re-run after resolving failures; the script is idempotent.
/manage-project status
Report current progress: read project_state.json, scan the existing files, and check whether the
project memory files are present and aligned. Take word-count limits from the target journal's
profile, ${CLAUDE_SKILL_DIR}/../write-paper/references/journal_profiles/{JOURNAL}.md, not from
memory. Lay the report out as in ${CLAUDE_SKILL_DIR}/references/status_output_format.md.
/manage-project sync-memory
Audit and refresh the project memory files:
- check for
PROJECT.md,STATUS.md,CLAIMS.md,DATA_DICTIONARY.md,ANALYSIS_PLAN.md,REVIEW_LOG.md - identify stale or contradictory project metadata
- propose the minimum files to create or update; create them from
${CLAUDE_SKILL_DIR}/references/scaffold_templates.md - align
project_state.jsonwith the current manuscript phase
/manage-project checklist
Read the pre-submission checklist from ${CLAUDE_SKILL_DIR}/references/pre_submission_checklist.md
and write it to submission/pre_submission_checklist.md. Then recommend /self-review (manuscript
quality gate), /verify-refs (citation verification) and AI pattern removal (/write-paper Phase 7).
/manage-project timeline {submission_date}
Generate a backwards timeline from the submission date, in the layout of
${CLAUDE_SKILL_DIR}/references/timeline_example.md.
Project State Conventions
Phase numbers:
- 0 = Init (scaffold created)
- 1 = Outline approved
- 2 = Table/Figure shells approved
- 3 = Methods (critic >= 85)
- 4 = Results (critic >= 85)
- 5 = Discussion (critic >= 85)
- 6 = Introduction + Abstract (critic >= 85)
- 7 = Polish complete (AI pattern removal + checklist + self-review >= 80)
Phase status values: pending | in_progress | complete | blocked
Manage-Refs Skill
Pick the tool from the Decision Tree; do not invent a parallel pipeline. References are never
hand-typed: the list is always rendered by pandoc citeproc + journal CSL or by the Zotero Word
plugin (CWYW). This skill writes (renders, injects, converts); bibliographic correctness against
PubMed/CrossRef stays in /verify-refs — one read-only audit, one writer.
Decision Tree
| Situation | Tool | Why |
|---|---|---|
Validate [@bibkey] ↔ refs.bib (UNDEFINED / UNUSED keys) | scripts/check_citation_keys.py | Hard build gate, runs in seconds |
| Single-author submission lockdown, frozen output | scripts/render_pandoc.sh -j <journal> | Reproducible, CI-friendly |
| Cascade rejection (e.g., ER → JVIR → CVIR) | render_pandoc.sh with new -j | CSL swap reformats references in seconds |
| Verify a journal CSL renders the in-text format / DOI / journal-name style the author guide actually requires | scripts/check_csl_render.py --csl <x>.csl --bib refs.bib --journal <key> --md manuscript.md | A stub/"dependent" CSL inherits its parent's format, which may differ from the guide (parenthetical vs superscript, DOI kept, full journal names). --md judges only the keys pandoc reads as cited (nocite: '@*' = whole bib); a preprint server named as the journal (journal = {arXiv}) is exempt from the abbreviation check; with no --journal/--expect-* the run is NOT_ASSESSED (exit 2), never PASS. Spec keys and what is not checked: references/REFERENCE_STYLE_SPECS.md. Run BEFORE submission, not after the proof PDF |
| Reference list prints FULL journal names but the journal wants NLM abbreviations | scripts/fill_journal_abbrev.py --bib refs.bib --out refs_abbrev.bib | Writes shortjournal from PubMed (PMID, else DOI → PMID), then CrossRef short-container-title, named per entry (often ISO 4 with periods, not NLM: check those). Inserts after the journal field in the entry's own indentation, never overwrites, re-parses before writing; exit 1 = some lookups failed. In a Better BibTeX-managed bib, fill Zotero's "Journal Abbr" instead |
| Reviewer revision: add 1–2 refs to a Word doc with co-authors live | Zotero Word plugin (user GUI) | Minimal disruption to track-changes flow |
| Reviewer revision: bulk reference change | Edit markdown SSOT, re-run render_pandoc.sh | Consistency, no cherry-pick risk |
Migrate [N] numeric markers → [@key] for pandoc | scripts/md_marker_convert.py --to-keys | Mapping-driven, partial conversion safe |
Convert [@key] → [N] for round-trip / debug | scripts/md_marker_convert.py --to-numbers | Same map, opposite direction |
| Wire native Zotero CWYW field codes into a .docx (live Refresh in Word) | scripts/inject_zotero_cwyw.py | Co-author Word workflow, post-circulation editability |
| Manuscript ↔ rendered DOCX cross-reference QC | scripts/check_xref.py --strict | Submission gate (P0 blocker on mismatch) |
| Figures/tables submitted as separate attachments (radiology, most medical journals) | check_xref.py --strict --allow-separate-attachments | Downgrades MISSING_DOCX to WARN; MISSING_BODY/MISMATCH remain P0 |
| v_(N+1) docx build-time regeneration check | check_xref.py --vN-docx-md5 <prev>.docx [--vN-md <prev>.md] | Identity = unmodified seed copy; missing diff lines = body not regenerated |
| Duplicate bibliography in the built artifact | scripts/check_reference_duplication.py --docx <built>.docx (or --text <rendered>.md) | DUP_REF_HEADING / REF_NUMBER_RESTART / REF_SIGNATURE_DUP (Major). Catches a hand-typed ## References list plus the pandoc --citeproc auto-bibliography, which renders two lists (the second often after the legends). Run after any citeproc build |
Publisher markup in a .bib title (renders as garbage) | scripts/check_bib_title_markup.py --bib refs.bib --strict | CrossRef titles carry <scp>WHO</scp> / <i>IDH</i>; BBT escapes them ({$<$}scp{$>$}) or strips them without restoring the space (andTERTPromoter). verify_refs proves the reference is true; this proves it will print. TITLE_MARKUP / TITLE_FUSION (Major) |
| Master pre-submission gate (recommended before any submission) | scripts/pre_submission_gate.sh | Chains check_citation_keys → check_bib_title_markup → verify_refs --strict → render_pandoc (optional) → check_xref --strict; single artifact qc/pre_submission_gate.json |
| Direct render with a built-in reference audit | scripts/render_pandoc.sh (audits the .bib via /verify-refs first; blocks on FABRICATED/MISMATCH/duplicates; reports UNVERIFIED rows as not clean, without blocking; after the render, warns about cited journal entries without shortjournal, which the style prints in full) | Best-effort (skips with a warning if /verify-refs is not alongside); opt out with -S. The master gate passes -S because it runs verify_refs in its stage 3 |
| Bibliographic audit against PubMed / CrossRef | delegate to /verify-refs | Audit-only — keep writer/auditor separation |
Workflows
A. Pandoc citeproc (default for solo authors and final submissions)
User provides manuscript.md with [@bibkey] citations + refs.bib.
- Gate:
python3 "${CLAUDE_SKILL_DIR}/scripts/check_citation_keys.py" manuscript.md refs.bib— exits non-zero on UNDEFINED keys. Fix and re-run. A[@NEW:topic_slug]drafting placeholder from/write-paperis reported as UNDEFINED too: resolve each by adding the citation to Zotero (/lit-syncthen refreshesrefs.bib) and replacing the placeholder with the real[@bibkey]. Never let a[@NEW:...]reach a rendered DOCX. - Render:
For the current inventory and what each style renders, read"${CLAUDE_SKILL_DIR}/scripts/render_pandoc.sh" \ -j european-radiology \ -i manuscript.md \ -b refs.bib \ -o manuscript_final.docxcitation_styles/README.md— that table is the registry.render_pandoc.shalso lists what is on disk when-jnames a style it cannot find, so ask the script rather than trusting a list written here. Two standing fallbacks: useradiologyfor RYAI andvancouverfor JVIR (neither has a dedicated CSL). A cited key that is not in the.bibmakes the render exit 5 and name the key: pandoc prints it as(key?)in the output and does not fail on its own. - QC:
Treatpython3 "${CLAUDE_SKILL_DIR}/scripts/check_xref.py" \ --md manuscript.md --docx manuscript_final.docx \ --out qc/xref_audit.json --strictsubmission_safe: falseas a halt. Route fixes by symptom — see the table inreferences/check_xref_symptoms.md. - Audit hand-off: invoke
/verify-refsfor the PubMed/CrossRef audit before sign-off.
B. Zotero CWYW (co-author Word workflow)
User has a markdown SSOT and wants reviewers to edit citations directly in
Word. Each reference must already exist as a Zotero item; the user supplies
a [N] → ZoteroKey mapping.
- Convert markers:
The conversion never guesses a Zotero key for a number: unmapped markers stay aspython3 "${CLAUDE_SKILL_DIR}/scripts/md_marker_convert.py" \ --input manuscript.md --output manuscript_keys.md \ --map ref_map.json --to-keys[N], are reported on stderr, and make the script exit 1 (the partial output is still written; pass--allow-partialto accept it). Ranges such as[1-3]are expanded. Any bracketed number counts as a marker, so a year in brackets ([2019-2021]) is reported UNMAPPED too. Optionally stage with--active-ns 1,2,3,4,19for a sample build first (a 5-reference sample limits the Word Refresh blast radius while debugging). - Render to .docx with pandoc (workflow A) so the body has plain text
[@key]markers, OR pre-build a .docx some other way that still contains plain[@key]text. - Inject CWYW:
The script fetches item metadata live from the local Zotero connector (port 23119), so Zotero must be running locally; there is no web-API fallback. Any HTTP failure aborts with a non-zero exit, so a partial bibliography never reaches the user. Never invent Zotero metadata.python3 "${CLAUDE_SKILL_DIR}/scripts/inject_zotero_cwyw.py" \ --input manuscript_keys.docx --output manuscript_cwyw.docx \ --user-id <zotero-user-id> --keys-from keys.txt - First-build instruction (REQUIRED): the script writes an empty
ADDIN ZOTERO_BIBLstub, which Word's Zotero Refresh treats as user-customized and refuses to populate. Open the output in Word → Zotero tab → Add/Edit Bibliography once. After that, Refresh keeps citations and bibliography in sync as authors edit. - Surgical patches are unsafe: for ref additions in later rounds, edit
the markdown SSOT and rebuild the whole .docx instead of regex-patching
the post-CWYW file. Zotero's rendered
[N]superscripts can collide with plain[N]markers and corrupt the field codes.
C. Cascade rejection re-render (find-journal hand-off)
User got rejected from journal A and /find-journal recommended journal B.
- Confirm the new CSL exists in
citation_styles/(or fetch from https://citationstyles.org/styles and drop in). - Re-run
render_pandoc.sh -j <new-csl>against the samemanuscript.md+refs.bib. - Re-run
check_xref.py --strict. - Re-run
/verify-refsif any new references were added during the inter-journal revision.
D. Cross-reference QC only
User shipped a manuscript and a reviewer flagged a Table/Figure mismatch.
-
Run
check_xref.py --stricton the currentmanuscript.md+.docx. -
Inspect
qc/xref_audit.json. Body caption is the SSOT — fixmanuscript.mdand rebuild, never patch the .docx by hand. -
See
references/check_xref_symptoms.mdfor theMISSING_BODY/MISSING_DOCX/MISMATCHtriage table. -
For journals that accept figures and tables as separate attachment files (European Radiology, Radiology, AJR, JVIR, KJR, and most medical journals), pass
--allow-separate-attachments. It downgrades two rows, reported apart because their evidence differs:MISSING_DOCX— a--docxwas supplied and proved the float is not in the rendered main document. That is what a separate attachment looks like.MISSING_BODYwith no--docxsupplied — nothing was checked. Excused on your word, printed asEXCUSED WITHOUT EVIDENCE, and counted insummary.downgraded_unchecked.
MISMATCHstays P0. So doesMISSING_BODYwhen the float is in the rendered DOCX — that is SSOT drift, which no attachment policy excuses.Run once with
--docxbefore submitting. The flag is a declaration, not a verification; supplying the DOCX is what converts an excuse into evidence.
D'. v_(N+1) docx regeneration check (build-time companion)
When building v_(N+1) from a frozen v_N, the v_(N+1) docx MUST differ
from v_N docx by content — a byte-identical copy is a silent seed-copy
that will revert markdown edits at peer review. check_xref.py carries
two flags for the build-time companion to the submission-time gate
in /sync-submission's scripts/verify_package_integrity.py --assert-vN-docx-changed:
python3 "${CLAUDE_SKILL_DIR}/scripts/check_xref.py" \
--md manuscript_v2.md \
--docx manuscript_v2.docx \
--vN-docx-md5 manuscript_v1.docx \
--vN-md manuscript_v1.md \
--strict
--vN-docx-md5alone: MD5 identity check. Identical bytes = FAIL.--vN-docx-md5 + --vN-md: additionally extracts the markdown-only diff between v_N and v_(N+1) and verifies each ≥40-char diff line appears verbatim (whitespace-normalized, case-insensitive) in the new docx body XML. Missing diff lines = body did not pick up the markdown edits.
Output records the result under vN_docx_check in qc/xref_audit.json.
Either failure mode causes a non-zero exit even without --strict.
E. Master pre-submission gate (recommended end-to-end chain)
The single entry point that combines workflows A and D plus /verify-refs
into one aborting chain. Use this immediately before submission or before
circulating a v_N package to senior co-authors.
bash "${CLAUDE_SKILL_DIR}/scripts/pre_submission_gate.sh" \
--md manuscript/manuscript.md \
--bib manuscript/_src/refs.bib \
--docx submission/<journal>/manuscript.docx \
--allow-separate-attachments # omit if the journal accepts inline figures/tables
Stage order (first failure aborts):
check_citation_keys.py manuscript.md refs.bib— UNDEFINED / UNUSED keyscheck_bib_title_markup.py --strict— publisher markup / tag-strip fusion in.bibtitlesverify_refs.py refs.bib --strict— PubMed / CrossRef per-entry verificationrender_pandoc.sh -S -j <csl> -i ... -b ... -o ...— invoked only when--docxis omitted (--journal <csl>, defaultvancouver)check_xref.py --md ... --docx ... --strict [--allow-separate-attachments]
On success the chain writes qc/pre_submission_gate.json (plus the
per-stage artifacts qc/bib_title_markup.json, qc/reference_audit.json and
qc/xref_audit.json) with submission_safe: true. On any failure the JSON records the failing
stage and exit code, and the script exits non-zero — do not submit until
the failing stage passes.
The gate does not reimplement any check; it calls the existing scripts as subprocesses. A new check belongs in the underlying script, which the gate then picks up.
F. BibTeX author-format corruption (rendered-name check)
Entries written as author = {Surname AB and Surname2 CD} (family + initials, no comma) make BibTeX treat the last token as the family name, rendering "AB S, CD S2". Always store author = {Family, Full Given}. Concatenated initials even with a comma (Family, AB) still collapse to a single initial under CSL initialize-with, so use the full forename from PubMed efetch.
/verify-refs compares bib content against PubMed but does not see the rendered output; grep the rendered docx and the bib separately:
unzip -p out.docx word/document.xml | sed 's/<[^>]*>//g' | grep -oE "[A-Z]{2} [A-Z], [A-Z]{2} [A-Z]" # corruption signature in output
grep -nE 'author\s*=\s*\{[A-Z][a-z]+ [A-Z]{1,3}( |\})' refs.bib # no-comma source entries
Quality Gates
This skill defines three submission gates and one user approval gate:
- Gate 1 (citekey integrity):
check_citation_keys.pyexits non-zero on UNDEFINED keys. The pipeline halts; the user reviews and fixes. - Gate 2 (cross-reference integrity):
check_xref.py --strictexits 1 on anyMISSING_DOCX/MISSING_BODY/MISMATCHrow (a float the markdown defines but the DOCX lacks isMISSING_DOCXeven when no in-text citation of it was recognised; a legend inside an HTML comment or fenced code is not rendered, so it staysUNCITED). With--docxand nopython-docxit exits 2: the DOCX audit did not run. The user reviewsqc/xref_audit.jsonand resolves before proceeding. Under--allow-separate-attachments, checksummary.downgraded_uncheckedas well assubmission_safe: a non-zero count means rows passed without being checked. - Gate 3 (audit hand-off): before sign-off, the user must run
/verify-refsand confirmsubmission_safe: trueinqc/reference_audit.json. This skill never marks the bibliography audited on its own. - User approval gate (CWYW first build): the user must perform Word →
Zotero → Add/Edit Bibliography manually after the first
inject_zotero_cwyw.pybuild. The skill cannot automate this and warns on stderr that it is required.
Provenance
scripts/_vendor_citation_writer.py is vendored from
alisoroushmd/zotero-mcp @ ed5dfb71, MIT licensed. See
NOTICE.md and LICENSE.zotero-mcp.
Known Limitations
- Webpage / non-journal item types: handled by the patched
zotero_to_csl_jsonthat fetches Zotero's native CSL-JSON; do not bypass this patch. - "Table 1 and 2" (singular kind word, number list):
check_xref.pyreads onlyTable 1from it; parsing a bare number after "and" in prose would read "Table 1 and 2 patients" as a citation. A float defined in the markdown but absent from the DOCX is still blocked asMISSING_DOCX; one that is neither defined nor rendered is not seen. Write "Tables 1 and 2" or "Table 1 and Table 2".
Meta-Analysis Skill
Meta-Analysis Types
| Type | RoB Tool | Statistical Model | Reporting Guideline |
|---|---|---|---|
| DTA (diagnostic test accuracy) | QUADAS-3 (QUADAS-2 for legacy reviews) | Bivariate / HSROC | PRISMA-DTA |
| Intervention (treatment effect) | RoB 2 (RCT) / ROBINS-I (NRSI) | Random-effects (REML, Hartung-Knapp CI) | PRISMA 2020 |
| Prognostic factor (association of one factor with outcome) | QUIPS | Random-effects | PRISMA 2020 |
| Prediction model (model performance) | PROBAST | Random-effects | PRISMA 2020 |
| Observational (prevalence/association) | NOS / JBI | Random-effects | MOOSE |
If the type is ambiguous (DTA vs intervention), ask the user to clarify before proceeding.
Workflow Phases
Phase 1: Protocol Development
Goal: Produce a PROSPERO-ready protocol document. Write the protocol, extraction forms and manuscript text in English whatever language the user writes in — PROSPERO records and the target journals are English-language.
-
Research question: PIRD (Population, Index test, Reference standard, Diagnosis) for DTA; PICO (Population, Intervention, Comparator, Outcome) for intervention.
-
DTA only — do QUADAS-3 phases 1 and 2 now, not at risk-of-bias time. They are review-level and belong in the protocol: phase 1 states the synthesis question(s) (population, index test(s), target condition — a review may have more than one); phase 2 defines the ideal test accuracy trial for each (objective, participants, index test(s), definition of the target condition, analysis). Every later risk-of-bias and applicability judgement is made against that trial. Write the review-specific guidance for answering each signalling question here too, with clinical and methodological input, and publish it as a web appendix. Defining the ideal trial after seeing the studies is a judgement fitted to the results, not an assessment. See
references/checklists/QUADAS3.md. -
Eligibility criteria: study design, population, index test / intervention, comparator / reference standard, outcomes (Se/Sp for DTA; effect size for intervention), and exclusion criteria with justification.
-
Search plan: at least 3 databases — PubMed, Embase, and Cochrane CENTRAL (add Scopus / Web of Science as needed); a Boolean strategy from the PIRD/PICO components; a grey-literature plan (conference abstracts, trial registries); language restrictions and date range stated explicitly, with justification.
-
RoB plan: tool by type (table above), at least 2 independent assessors, and the disagreement-resolution method (consensus, third reviewer).
-
Synthesis plan: bivariate random-effects (Reitsma) or HSROC (Rutter & Gatsonis) for DTA; random-effects for intervention (Phase 6); heterogeneity, subgroup / sensitivity, and publication-bias plans.
-
PROSPERO registration document: read
${CLAUDE_SKILL_DIR}/references/PROSPERO_template.mdand follow its field guide, word limits, output format (Markdown + DOCX via pandoc), and Common Pitfalls Checklist. Save to the project's7_Submission/or equivalent directory.- Submit before the formal search, and record the date. The user submits the registration;
record the initial submission date (not the later date the record is published) in the
project's status file (
STATUS.md,project_state.jsonorproject.yaml, whichever the project keeps). The Phase 2 entry check looks for it. A pilot search is fine before submission; the formal search, screening and extraction, AI first-pass drafts included, are not. - Registration-ID format gate. A PROSPERO ID is
CRD42+ 9 digits (14 characters total), e.g.CRD42024500001. Validate any ID that appears in the manuscript or registration doc withgrep -oE 'CRD42[0-9]+'and assert a 14-character length /^CRD42\d{9}$— a 15-character ID (a stray digit) is a transcription error a reviewer will check against the live record. - Review-type selection. Pick the least-wrong portal review type for the actual design and state any portal constraint in the protocol. A descriptive single-arm proportion synthesis is not an "Intervention review"; choosing that type only to satisfy a portal field contradicts a later GRADE / effect-certainty statement. Whatever certainty language the protocol commits to (GRADE vs "evidence statements only") must match the manuscript verbatim — a guideline-style "we recommend" is not licensed by a descriptive review type.
- Submit before the formal search, and record the date. The user submits the registration;
record the initial submission date (not the later date the record is published) in the
project's status file (
Phase 2: Search Strategy
Goal: Develop and validate reproducible search strategies.
Entry check — registration before the formal search. Look for the registration submission
date recorded in Phase 1 step 7. If none is recorded, stop and ask the user to choose: submit the
registration now and run the search after that date, or proceed without it, in which case Phase 8
item 5 limits what the manuscript may claim. Record the choice. Screening or extraction done before
submission, AI first-pass drafts included, cannot be made prospective afterwards. Under
/orchestrate --e2e this stop is the REGISTRATION_PENDING halt; no autonomous default passes it.
- Build search blocks from the PIRD/PICO components and execute them per database with
/search-lit(PubMed: MeSH + free text; Embase: Emtree + free text; further databases as the protocol specifies). Searching only: selecting records and extracting results belong to Phases 3–4, not to/search-lit's own selection and deep-read steps. - Report per PRISMA-S (Rethlefsen et al. 2021, PMID:33499930): one section per database with date of search, number of results, and any limits applied.
- Merge and deduplicate into a single spreadsheet: by DOI first, then PMID. Save raw counts for the PRISMA flow.
Phase 3: Screening & Selection
Goal: Systematic title/abstract and full-text screening with two independent reviewers. Read
references/phase3_screening_detail.md when executing a round (exclusion-code sets, AI pre-screening
template and Methods boilerplate) or when a 3f/3f.5 gate fires (set algebra, reconciliation table).
3a. Round 1 — title/abstract (single reviewer). Define the exclusion codes from the protocol.
Mark every record INCLUDE / EXCLUDE / MAYBE with a reason code → round1_{date}.tsv.
3b. Round 2 — dual independent title/abstract. A second independent reviewer (or AI as a
documented second-pass tool with human verification) re-screens all R1 records. Report Cohen's κ
in Methods. round2_tag = INCLUDE / EXCLUDE / MAYBE (MAYBE = disagreement or either reviewer
flagged uncertainty), plus round2_reason.
3c. Round 3 — adjudication (first reviewer). MAYBE records first, then INCLUDE records for a
brief confirmation pass → round3_decision (plus round3_reason only when overturning R2).
Optional AI pre-screening may compress the effort, but AI suggestions are not decisions: the
reviewer independently confirms or overturns every one.
3d. Round 4 — full text (/fulltext-retrieval) for round3_decision = INCLUDE: full-text
exclusion codes, two independent reviewers, Cohen's κ, consensus or a third reviewer. Flag
comparative studies for priority extraction.
3e. PRISMA flow. Track counts at every stage (R1 → R2 → R3 → R4 → final included); draw it with
/make-figures once final.
3f. Post-consensus count reconciliation gate (MANDATORY before Phase 5 write-up). Reconcile counts from the raw ID sets, never from prose summaries, into one source-of-truth file:
python "${CLAUDE_SKILL_DIR}/scripts/screening_reconcile.py" \
--screening 2_Screening/fulltext_screening.tsv \
--consensus 2_Screening/consensus_decisions.tsv \
--table1 6_Tables/table1_studies.csv \
--output 2_Screening/screening_consensus.json
Downstream stages consume screening_consensus.json for counts and ID sets; the Markdown consensus
document remains the human explanation. Three hard rules:
- List the narrative-only IDs explicitly. The highest-yield red flag is a numeric claim ("10
narrative-only studies") that does not match the enumerable set
(A ∪ C) \ B \ T. - No "N → M" transition without ID receipts. "k rose from 30 to 32 after FLAG consensus" must cite the added/removed IDs. A transition claim with no enumerable ID set is a P0 and blocks the Phase 5 hand-off.
STAGE_TRANSFER_LOSSis a P0. Exit 1 when a record is included at screening but absent from the consensus artifact altogether — no adjudication was ever recorded. An exclusion is a decision; silence is a gap. Never let it settle into narrative-only.
Two more gates at 3f; a non-zero exit from either blocks the Phase 5 write-up:
# every applied exclusion code vs the *registered* eligibility criteria
python3 ${CLAUDE_SKILL_DIR}/scripts/check_exclusion_code_validity.py --protocol 0_Protocol/protocol.md --screening 2_Screening/*.tsv --strict
# DI-6: PRISMA numbers on 5 surfaces vs YAML SSOT; re-run on every revision touching PRISMA numbers
python3 ${CLAUDE_SKILL_DIR}/scripts/prisma_5way_consistency.py --ssot prisma.yaml
Exclusion-code verdicts: CODE_CONTRADICTS_ELIGIBILITY (a code excludes a design the protocol
includes — bulk study loss no other gate can see), CODE_NOT_REGISTERED, CODE_RENUMBERED. Verdicts,
NOT_ASSESSED cases and known limits of all 3f gates: references/phase3_screening_detail.md §3f.
- Known limit: PRISMA 5-way still skips
after_dedup → full_text_assessed; optionalprisma2020:counts add report-level identities (§3f).
3f.5 Pool composition lock (MANDATORY at adjudication freeze). Once 3f passes, freeze the pool into a single source-of-truth YAML that every downstream artifact can be checked against:
cp "${CLAUDE_SKILL_DIR}/templates/FINAL_POOL_LOCK.yaml.template" 2_Data/FINAL_POOL_LOCK.yaml
# fill counts + UID lists from 3f, compute the SHA-256 over the sorted UID list,
# and COMMIT THE LOCK before any Phase 4 extraction
- Never re-derive
k includedfrom the extraction TSV at manuscript build time — always referencefinal_pool_nfrom the lock. - Aggregate patient/lesion totals are locked too, not just study counts. Distinguish arm-separable from both-arm rows: a study contributing one arm must not have its full-cohort count folded into a pooled total. A hand-carried headline total that does not re-derive from the locked per-study values is a P0.
- A late post-freeze change to the pool is a formal PROSPERO amendment: file it, re-freeze as
FINAL_POOL_LOCK_v2.yaml, and propagate to every artifact.
Phase 4: Data Extraction
Goal: Create standardized extraction forms and extract 2x2 or effect-size data. Read
references/phase4_extraction_detail.md when building the form (DTA / intervention field lists),
when an AI draft was shared, for the optional extract_assist.py suggestions (AI_SUGGESTED, a
human confirms each before dta_extraction_qc.py), or when a QC flag fires.
4.0 Entry gate (MANDATORY) — pool composition lock ↔ adjudication TSV. Before any extraction
work begins, confirm the round-3 adjudication TSV and FINAL_POOL_LOCK.yaml (Phase 3f.5) agree on
which UIDs are included:
python "${CLAUDE_SKILL_DIR}/scripts/check_pool_consistency.py" \
--lock 2_Data/FINAL_POOL_LOCK.yaml \
--adjudication-tsv 2_Screening/round3_adjudication.tsv \
--decision-col round3_decision --uid-col uid \
--include-labels "INCLUDE,INCLUDE_MIXED" \
--out qc/pool_consistency.json
The gate fails closed: any UID disagreement blocks extraction. Resolve by re-freezing the lock with the corrected UID set (and propagating downstream) or by correcting a mis-labelled TSV row. Do NOT proceed with a mismatch — the extraction matrix will not align with the locked pool, and the drift surfaces as a fabrication-grade red flag at peer review.
# before the first extraction row (DI-1): comparative arm rows never live in R-script comments
python3 ${CLAUDE_SKILL_DIR}/scripts/extraction_consensus_log_init.py --output 2_Data/extraction_consensus_log.md
Failure-mode cross-ref →
references/data_integrity_checklist.mdDI-1~DI-5 are mandatory during extraction (2x2 arm-swap, KM audit trail, methodology mismatch, PRISMA 5-way drift, single-source k).
Extraction form. Read ${CLAUDE_SKILL_DIR}/references/empirical_lessons.md before designing
it. For high-impact radiology / medical AI targets use
${CLAUDE_SKILL_DIR}/templates/extraction_form_v2.md: its dual-extractor, source-page-reference,
and verbatim-quote columns close the 2x2 cell-swap and cohort-overlap blind spots.
AI-drafted starting document — treat as hallucination-suspect. If a mentor or collaborator
shared an AI-drafted study list, 2x2 set, or effect estimates (even flagged "for reference
only"): save it with a _DO_NOT_USE_VERBATIM suffix and re-verify every N, denominator, event
count, OR/CI, and author/year against the source PDF. Trust hierarchy: source PDF + own analysis
stdout > the mentor's direct text > the attached AI draft — never promote a draft up that ladder.
4b. Special cases (KM reconstruction, composite exposure). When studies report outcomes only as
Kaplan-Meier curves, or the intervention is a composite of techniques, load
${CLAUDE_SKILL_DIR}/references/phase4_km_composite.md for the WebPlotDigitizer → IPDfromKM
procedure (cite Guyot et al. 2012, doi:10.1186/1471-2288-12-9) and the 4-path composite-exposure
decision tree. Pre-specify a sensitivity analysis excluding composite-exposure studies.
Cross-verification (≥2 independent reviewers). Report inter-reviewer agreement (% or Cohen's
κ) at title/abstract and full-text stages. Verify denominator consistency — the denominator may
differ across outcomes within one study, so for each outcome back-calculate event ÷ denominator
and confirm it reproduces the paper's reported percentage. Distinguish KM-curve estimates from raw
event counts and record the data source (Table / KM / text). Log every consensus decision in
2_Data/extraction_consensus_log.md, then lock the dataset; later changes need a dated
justification. If 2x2 cells are missing, suggest contacting the authors or a sensitivity analysis
with imputed values.
4c. Extraction QC & cohort overlap. After dual-extractor consensus, run both before locking:
# 2x2 cell integrity: validates TP/FN/TN/FP against source-reported sens/spec (catches arm-swap)
python3 "${CLAUDE_SKILL_DIR}/scripts/dta_extraction_qc.py" \
--input 2_Extraction/extraction.csv --tolerance 0.02 \
--out 2_Extraction/qc/dta_extraction_qc.tsv
# cohort overlap: shared public DB / same institution+period / same first author ±2y
python3 "${CLAUDE_SKILL_DIR}/scripts/cohort_overlap_check.py" \
--input 2_Extraction/studies.csv --enrich \
--out 2_Extraction/qc/cohort_overlap.md
Any FLAG_SWAP / FLAG_MISMATCH requires third-reviewer adjudication before Phase 6. A
confirmed flag is not resolved until the extraction form itself is edited — a flag corrected only
in a review note silently re-enters synthesis, so re-run the QC and confirm zero open flags before
locking. HIGH-confidence overlap pairs require a Limitations acknowledgment plus a sensitivity
analysis excluding one of the pair.
Phase 5: Risk of Bias Assessment
Goal: Guide structured RoB assessment with the appropriate tool.
DTA: this phase runs QUADAS-3 phases 3–6 (flow diagram, identify the estimates to assess, assess, overall judgement). Phases 1–2 — the synthesis question and the ideal test accuracy trial — were written in Phase 1 above. If they were not, stop and write them before judging anything; they are the comparator every judgement is made against.
Select the tool by meta-analysis type (see table above), then read its checklist:
| Tool | Checklist File |
|---|---|
| QUADAS-3 (DTA, current) | ${CLAUDE_SKILL_DIR}/references/checklists/QUADAS3.md |
| QUADAS-2 (DTA, legacy) | ${CLAUDE_SKILL_DIR}/references/checklists/QUADAS2.md |
| RoB 2 (RCT) | ${CLAUDE_SKILL_DIR}/references/checklists/RoB2.md |
| ROBINS-I (NRSI) | ${CLAUDE_SKILL_DIR}/references/checklists/ROBINS_I.md |
| PROBAST (Prediction) | ${CLAUDE_SKILL_DIR}/references/checklists/PROBAST.md |
| NOS (Observational) | ${CLAUDE_SKILL_DIR}/references/checklists/NOS.md |
| JBI (Case Series) | ${CLAUDE_SKILL_DIR}/references/checklists/JBI_Case_Series.md |
For AI/ML prediction models, also apply PROBAST+AI extensions.
Output: Summary table + traffic light plot (use /make-figures).
Phase 6: Statistical Synthesis
Goal: Execute meta-analysis and generate publication-ready outputs.
Failure-mode cross-ref →
references/data_integrity_checklist.mdDI-6/DI-7/DI-9 are the consistency gate (CSV ↔ script ↔ prose; single-source k; 3-way numeric reconciliation before Stage 4).
Always use R (packages: meta, metafor, mada); every reported estimate, CI, p-value, and
sample size comes from executed code output (Phase 6b audits this). Never guess dataset column names
or codings — if a mapping is uncertain, output [VERIFY: variable_name] and ask the user to confirm
against the data dictionary.
| Analysis family | Primary tool | Key output |
|---|---|---|
| DTA | mada::reitsma() (bivariate) | Pooled Se/Sp + SROC with confidence/prediction regions |
| Intervention | meta::metagen() / meta::metabin() | Pooled OR/RR, τ² + I² + prediction interval, measure-matched funnel test (k ≥ 10), leave-one-out |
| Dual (comparative + single-arm) | metabin + metaprop | PRIMARY vs SECONDARY per pre-specified protocol |
Read ${CLAUDE_SKILL_DIR}/references/phase6_statistical_synthesis.md before running the pooled
analysis — full R code templates (companion: ${CLAUDE_SKILL_DIR}/references/r_templates.md), the
dual-approach decision table (comparative vs single-arm), practical cautions (method.tau, HK CI,
zero-cell correction), publication-bias test power, the sensitivity-analysis menu, and
error-handling rules. Write the pooled estimates, heterogeneity statistics, and k for each analysis,
taken from the executed R output, to analysis/meta_analysis_outputs.json.
Three checks before the pool is written up — each is a Methods sentence, not only a setting. R and detail in the same reference:
- Is the event rare? A pooled event rate < 1%, or any zero-event arm, moves the analysis off the inverse-variance default onto Peto / Mantel-Haenszel without a zero-cell correction / GLMM. Inverse-variance methods including DerSimonian-Laird are to be avoided for rare events, and so are 0.5 continuity corrections with them.
- Why this model? Fixed vs random is a judgment about whether one common true effect exists — never derived from Cochran's Q or I². "A random-effects model was used because I² was 65%" is a reviewer catch, not a rationale.
- Does one study contribute several correlated effect sizes? Multiple outcomes, readers, thresholds, or time points from the same participants need one pre-specified estimate per study, a multivariate model, or robust variance estimation — not independent pooling.
Before pooling transcribed ratio CIs (and back-deriving SEs), check them (columns study, measure, estimate, lower, upper, optional ci_level in percent and ci_method); RATIO_CI_IMPOSSIBLE is
Major, RATIO_CI_ASYMMETRIC (Minor) means verify the transcription or declare ci_method:
python3 ${CLAUDE_SKILL_DIR}/scripts/check_ratio_ci_symmetry.py --extraction 2_Extraction/extraction.csv --out 2_Extraction/qc/ratio_ci.json --strict
Phase 6b: Post-Analysis Source Fidelity Audit (MANDATORY)
Goal: Catch numerical hallucinations that survived the forward pipeline (CSV → .R → manuscript).
When it runs: every time Phase 6 outputs change (first draft, revision, reviewer-requested
re-analysis) — including "minor" re-runs. The precedent: in a minor revision-era re-analysis, a
safety outcome's arm-level events (and so its p-value) were reported direction-reversed because a
Fisher matrix() was hand-typed from a misread source Table while the extraction CSV was correct.
Every downstream artifact echoed the wrong number, so internal consistency checks passed; only a
random back-check against the primary paper caught it.
Non-negotiable rules:
-
No hand-typed numerical matrices when a CSV exists. Use
read.csv(...)+ subset / filter; never copy a 2x2 table from a paper intomatrix(c(...), ...)by eye. If hand entry is truly unavoidable (e.g., text-only extraction), thematrix,c(), ordata.frameline MUST carry a comment citing the exact CSV row + column OR the exact primary-source Table/Page coordinate:# source: data_extraction_final.csv row <N> (<first-author> <year>), cols <event_arm1>=0, <event_arm2>=1 # verified against primary source Table <X>, page <P> fisher.test(matrix(c(0, 45, 1, 55), nrow = 2, byrow = FALSE)) -
Comparative-arm subsets are a separate consensus-log row. When one study's arm-specific values are used in a comparative analysis while its full cohort appears elsewhere,
extraction_consensus_log.mdmust carry an explicit row for the arm-specific values. Pooled totals and arm-specific values MUST NOT share a row. -
Random 3-claim back-check before closing Phase 6. After the forest/funnel/subgroup outputs stabilize, randomly sample 3 numerical claims from the draft Results and trace each back to (a) the R output log and (b) the original paper's Table/Figure. Record it in
peer_review_<vN>_internal.md:Claim (manuscript line) R output file:line Primary source (paper, Table/Fig, page) Match? A single mismatch is a P0 blocker — do not advance to Phase 7 until resolved.
-
Revision-introduced numbers must be tagged. Any new number added after v1 — including numbers from a new comparative / subgroup / sensitivity script — MUST be wrapped inline as
[VERIFY-CSV]in the manuscript until the Phase 2.5a audit in/self-reviewclears it. -
Sensitivity analyses must be recomputed on the modified data, not copied. Every reported effect size in a sensitivity / leave-one-out / erosion / alternative-model analysis (Cohen's dz/f, AUC, OR, HR, β, sens/spec, ICC) MUST be re-derived from the modified dataset. If a sensitivity-table effect size is identical to the primary analysis to two decimals across ≥4 values while the underlying means/SDs/counts differ, the recomputation may not have run (small leave-one-out shifts can round to the same value, so confirm from the script output rather than assume) and the primary values may have been transcribed — re-run the script on the modified data.
-
A "fixed" / "resolved" audit note requires re-run evidence, not a claim. A number recorded as
fixed,resolved, orcorrectedcounts only with a timestamp and the stdout / output-file line showing the corrected value, or the commit that changed it. A bare "fixed in v10" does NOT clear the finding — re-run the script and attach the output. The outcome-denominator cross-check (/self-reviewPhase 2.5b, the cohort-arithmetic / pool-lock assertions) must pass against the current outputs before any "fixed" status is accepted.
Phase 7: GRADE / Certainty of Evidence
Goal: Assess certainty of the body of evidence.
DTA: GRADE-DTA — risk of bias (from QUADAS-3, or QUADAS-2 for a legacy review), indirectness (applicability concerns), inconsistency (heterogeneity), imprecision (wide CIs, small sample), publication bias. Intervention: standard GRADE.
Certainty is assessed per outcome, not once for the review. The domains resolve differently for each outcome — one pooled from 12 studies with narrow CIs and one pooled from 3 with a wide CI do not share a rating, and a single review-level "moderate certainty" sentence tells a reader nothing about the outcome they came for. Rate every outcome carried into the Summary of Findings table, and state the reason for each downgrade (which domain, why), not only the resulting label.
Output: Summary of Findings table — one row per outcome, carrying the pooled estimate with its precision alongside the certainty rating (high / moderate / low / very low).
Phase 8: Reporting & Manuscript
Goal: Generate PRISMA-compliant manuscript sections.
Failure-mode cross-ref →
references/submission_package_drift.md— apply the_build.shpattern +DO_NOT_EDIT_HEREgate when staging multi-journal submission folders.
Re-read references/empirical_lessons.md before submission.
-
Check reporting compliance:
/check-reportingwith PRISMA-DTA (bundled copy:references/checklists/PRISMA_DTA.md) or PRISMA 2020, then a second, separate run over the abstract with PRISMA 2020 for Abstracts (PRISMA_2020_Abstracts.md, 12 items, its own denominator). Report that score separately: item 2 of the main checklist only defers to it, so a manuscript can satisfy all 42 main-text items and still fail most of the twelve, and folding them into one total is how they stay invisible. -
Write the manuscript:
/write-paperwith the meta-analysis type →manuscript/manuscript.md. Never generate references from memory; use/search-litfor all citations. -
Figures (
/make-figures): PRISMA flow diagram, forest plots (paired for DTA), SROC curve (DTA), funnel plot (Deeks' for DTA — see DTA pitfalls), RoB summary (traffic light plot). -
Tables: characteristics of included studies; 2x2 data per study (DTA); RoB assessment results; Summary of findings / GRADE table (one row per outcome — Phase 7).
-
The items published radiology SR/MAs most often drop — check these by hand before the compliance run. Park 2022 (Korean J Radiol; PMID:35213097) scored 24 SR/MAs (18 with meta-analysis) against PRISMA 2020, with each item's percentage on its own denominator (MA-only items out of 18), and found 24 of 42 items reported by fewer than 80%:
PRISMA item What is missing Observed 20a For each synthesis, a brief summary of the contributing studies' characteristics and risk of bias — not one global paragraph covering all pools 0/24 27 Data availability: which of the extraction forms, extracted data, analysis dataset, and analytic code are public, and where 0/24 24a–c Registration number, where the protocol can be read, and any amendment — an explicit "not registered" satisfies 24a 0/24 22 / 15 Certainty of evidence per outcome, and the method used to assess it 9% 13f / 20d Sensitivity analysis: method and result 28% (5/18) 18 Risk of bias per study, shown study-by-study rather than as a pooled proportion 32% 13d Rationale for the synthesis model (see Phase 6 check 2) 28% (5/18) 16b Studies that look eligible but were excluded, cited individually with the reason 25% Abstract #3, #12 Eligibility criteria and registration inside the structured abstract 0/24 each Word the registration from the initial submission date (not the record's later publication date), never from a template: "prospectively registered" only when it predates the start of screening; "registered retrospectively" when screening or extraction had already begun; "This review was not registered" when there is no record (an explicit statement satisfies 24a). Name the registry actually used (PROSPERO, OSF Registries). The same words go into every copy of the claim: abstract, Methods, cover letter, and the portal's ethics and registration fields.
-
Data availability statement: name what is being shared (extraction template, locked dataset, analysis code, RoB judgments) and where — repository, DOI, or supplementary file. "Available from the corresponding author on reasonable request" satisfies few journals now and no longer satisfies item 27. If a Zenodo DOI is minted post-acceptance,
references/post_submission_release_ops.mdcovers propagating it back into this statement. -
Supplementary & analysis-code pre-submission gate (before Phase 9 circulation and before portal upload). Presence of the 8-file package (Empirical Lesson 5) is necessary but not sufficient — each item must also be reviewer-ready:
- De-scaffold: strip internal-QC / tool artifacts — raw
/check-reportingoutput ("Assessed by: ", JSON blocks, "READY FOR SUBMISSION" verdicts, action-item lists), search-development planning docs (decision logs, expected-yield estimates,[Check on execution]placeholders, version-history dev notes), and stale version stamps. Ship a clean PRISMA 2020 checklist (27-item / 42-subitem table only) and an executed-method search-strategy doc, not the working drafts. - Blind: remove author names/initials and sibling-project cross-references ("Designed by: ", "identical to a sibling review") — same standard as the blinded manuscript.
- Cross-consistency: every supplementary number matches the main text — PRISMA counts, pool k/N, the Cochrane/CENTRAL search description, RoB counts.
- Reproducible, self-contained analysis code: run it from a clean copy of the bundle. It must read the bundled locked dataset (not an out-of-bundle path), write to the working directory, and regenerate every pool in the results table. A hard-coded study-id subset that drifts from the manuscript (a pool over k=7 while the manuscript reports k=9) is a P0 — fix and re-run; never ship stale code or figures derived from it.
- Supplementary-only review pass: the manuscript self-review does not see the supplement; mirror
/self-reviewPhase 2.5c–2.5d (reference + cross-reference QC) over the supplementary files.
- De-scaffold: strip internal-QC / tool artifacts — raw
-
Submission gates (on Phase 8 pre-submission and every journal retarget; a non-zero exit blocks submission):
/sync-submissionSR-MA gate: the supplementary package matches all 8 files intemplates/supplementary_8file_checklist.md(PRISMA, PROSPERO, search strategy, exclusion list, extraction table, per-study x per-domain RoB, subgroup forests, sensitivity / publication bias); AI Disclosure is present (cross-link/peer-reviewPhase 2A P8); no duplicate PMID/DOI in the cite list (/verify-refsGate 5).- DI-8 tag gate — fails if
VERIFY-CSV/TODO/FIXME/XXXsurvive in7_Manuscript,supplement,SUBMISSION, etc.:bash ${CLAUDE_SKILL_DIR}/scripts/tag_cleanup_gate.sh - SPD package integrity — checksum-based drift detection between the master manuscript and the
built
SUBMISSION/{journal}/folder (journal-editable files — cover letter, response, MANIFEST,DO_NOT_EDIT_HERE.md— are auto-excluded). On the first build per journal runpython3 ${CLAUDE_SKILL_DIR}/../sync-submission/scripts/verify_package_integrity.py --record --journal <name>, then--verify --journal <name>before every re-submission. - ICMJE COI forms for every author:
${CLAUDE_SKILL_DIR}/references/icmje_coi_guide.md.
Phase 9: Co-author Circulation
Goal: Pre-submission circulation to co-authors and a senior methodologist / reviewer, with a bounded review window and a controlled attachment scope.
Trigger: Phase 8 is complete, and the draft has cleared the Phase 6b source-fidelity audit.
Summary: Reply to the prior-version email thread to preserve In-Reply-To continuity
(v1 → v2 → v3 tracked in one place). Attach the manuscript body with figures inline and,
for v≥2, a change summary — exclude graphical abstract, cover letter, COI forms, and
supplementary until the target journal is confirmed. TO = corresponding author + one
senior methodologist; CC = remaining co-authors. Set a 7-day deadline (5 business days +
weekend). Ask the corresponding author for target-journal preference, reviewer candidates,
and cover-letter framing.
Load-on-demand procedural detail (thread continuity, attachment scope rationale,
size-to-method table, journal-undetermined framing, response-tracking log):
${CLAUDE_SKILL_DIR}/references/phase9_circulation.md.
Failure-mode cross-ref →
references/review_orchestration.mdRO-1~RO-5 (dual-rating completeness, defensive-tone bias audit, response-matrix numeric tracking, 2nd-reviewer availability blocking).
Phase 10: Self-Audit Recovery (v{N} → v{N+1} sprint)
Goal: When an audit uncovers a structural data or protocol-application error, withdraw the current version, rebuild, and re-circulate with a transparent audit trail.
Trigger conditions (any one):
| # | Trigger | Source |
|---|---|---|
| T1 | Extraction CSV ↔ primary source disagreement for a cell feeding a pooled/subgroup estimate or reported proportion | Phase 6b audit |
| T2 | Included/excluded study violates the pre-specified criteria on re-read | Protocol review |
| T3 | Hand-typed numerical literal in the analysis script traces to a wrong value | Phase 6b audit |
| T4 | PROSPERO protocol ↔ delivered analysis disagreement on outcome, subgroup, or eligibility | Protocol ↔ analysis diff |
| T5 | Dual-reviewer consensus record ↔ locked dataset disagreement on inclusion | Consensus log diff |
Non-negotiable rule: if the trigger fires after Phase 9 circulation but before journal submission, withdraw the current version within 24 hours. Reviewer discovery is a strictly worse failure mode than self-withdrawal.
Sprint: read ${CLAUDE_SKILL_DIR}/references/phase10_recovery.md and run its 12 steps, from
10.1 (audit log at qc/audit_vN_to_vNplus1.md) through the PROSPERO amendment (application
correction, not criteria change) and re-circulation in the Phase 9 thread to 10.12 (post-recovery
loop).
Failure-mode cross-ref →
references/post_submission_release_ops.mdGate 4 covers reject/revise Zenodo versioning, tag-cleanup gate, and re-target workflow (avoid "new version" misuse on re-target).
DTA-Specific Pitfalls (Always Check)
| Pitfall | Problem | Solution |
|---|---|---|
| Separate pooling of Se/Sp | Ignores correlation | Use bivariate/HSROC model |
| Ignoring threshold effect | False heterogeneity | Judge from the SROC plot and the bivariate Se–FPR correlation (Spearman is descriptive only) |
| Standard funnel plot for DTA | Inappropriate | Use Deeks' funnel plot |
| I-squared only for heterogeneity | Doesn't capture threshold effect | Use prediction region on SROC |
| Missing GRADE | Common omission in DTA MA | Apply GRADE-DTA. If <4 studies, assess each domain narratively and state the limitation explicitly |
| Partial verification bias | Inflates sensitivity | QUADAS-3 3.2 (target condition assessed in all participants). QUADAS-3 has no Flow & Timing domain — that was QUADAS-2 |
| Differential verification bias | Distorts both Se and Sp | QUADAS-3 3.3 (target condition assessed the same way in all participants) |
| Unevaluable results excluded | Biases accuracy estimates | Report intent-to-diagnose analysis |
Small Study Considerations
When the number of included studies is small (< 10):
- Bivariate/HSROC model may not converge (warn the user when a DTA review has fewer than 4 studies) — consider univariate random-effects as fallback
- Publication bias tests are underpowered — state this limitation
- Subgroup/meta-regression analysis not recommended
- Wide prediction regions expected — emphasize uncertainty in conclusions
- Consider narrative synthesis as alternative/complement
MLLM-Eval Skill
Purpose
This skill makes an LLM / MLLM clinical evaluation defensible: a real adjudicated reference standard, faithfulness measured not assumed, clinical-efficacy metrics beyond n-gram overlap, a pretraining- contamination check, prompt-sensitivity disclosed, and a reader study where text is generated. It is model-agnostic — every check applies to a closed API and to open weights — and read-only (an advisory design/audit skill): it audits the evaluation design and specifies and routes the clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 via their published extractors) rather than running the model or computing the metrics itself.
It is the LLM/MLLM evaluation-design counterpart in the lane — an auditor that hands the specified
metrics to their extractors and /analyze-stats, parallel to how /model-assessment audits an imaging
model's design and computes its metrics. The reviewer-side
audit of a finished manuscript uses the mllm_evaluation.md (ME0–ME8) probe via /self-review and
/peer-review; this skill is the author-side harness design. It routes the reader study to
/design-ai-benchmarking, the sizing to /calc-sample-size, and TRIPOD-LLM / MI-CLEAR-LLM compliance to
/check-reporting.
When to use
- You are designing or auditing an evaluation of an LLM/MLLM on a clinical task and want it to cover the axes a reviewer will check (reference standard, faithfulness, contamination, prompt sensitivity, reader study).
When NOT to use
- AI-vs-human-expert benchmark with a rated rubric →
/design-ai-benchmarking. - Imaging prediction/segmentation model →
/model-assessment. - Image-to-image generative model → the
image_synthesisprobe. - Training / serving the LLM → out of scope.
- Item-level TRIPOD-LLM / MI-CLEAR-LLM audit of a finished manuscript →
/check-reporting.
Workflow
Phase 1 — Pin the task, model, comparator, decoding (ME0)
State the task (report generation / VQA / extraction-classification), the exact model + version/date (closed API or open-weights id), the decoding settings (temperature, seed, max tokens), and what the outputs are scored against.
Phase 2 — Reference standard + metrics (ME1, ME2)
Require an adjudicated expert reference (not a single unverified report or a model-derived label). For report generation, report a clinical-efficacy metric — RadGraph-F1 (Jain et al., NeurIPS 2021) or CheXbert-F1 (Smit et al., 2020), or the composite RadCliQ (Yu et al., Patterns 2023) — alongside any BLEU/ROUGE, with CIs. For VQA/classification, state the answer-matching rule and report per-class sensitivity/specificity (or precision/recall/F1) and PPV at the real prevalence, with CIs; accuracy only alongside them — at 2% prevalence, answering "negative" every time scores 98% accuracy.
Phase 3 — Faithfulness + contamination (ME3, ME4)
Add an atomic-fact faithfulness measure + a false-premise / abstention probe (MedVH, Med-HALT) — report a hallucination rate, not just accuracy. For any public benchmark (VQA-RAD, SLAKE, MIMIC-CXR- derived, MedQA), add a contamination statement: training cutoff vs benchmark release, a held-out / post-cutoff set, or a contamination probe.
Phase 4 — Prompt sensitivity + reader study (ME5, ME7)
Disclose the exact prompt(s), temperature/seed, ≥ 3 runs with variance, and a prompt-robustness
check. For a deployment/utility claim, design a blinded reader study with an error taxonomy (route
the rubric/IRR to /design-ai-benchmarking, ICC/κ to /analyze-stats, sizing to /calc-sample-size).
Phase 5 — Gate the plan (deterministic)
Declare the axes in eval_manifest.json (copy ${CLAUDE_SKILL_DIR}/templates/eval_manifest.json;
fields and allowed values in references/eval_manifest_schema.md), then:
python3 ${CLAUDE_SKILL_DIR}/scripts/check_mllm_eval_completeness.py \
--manifest eval_manifest.json --strict
Use "none" for an axis not done and "other:<description>" for a method not listed; any other
value exits 2. --plan plan.md --task report_generation|vqa|classification still runs the older
keyword check on prose (see Known limits).
NGRAM_ONLY / FAITHFULNESS_MISSING / REFERENCE_STANDARD_MISSING / CONTAMINATION_UNADDRESSED /
READER_STUDY_MISSING must be resolved. A classification manifest always reports
CLASSIFICATION_METRICS_NOT_ASSESSED (Minor): the manifest has no metric field for it, so check
per-class sensitivity/specificity and PPV at the real prevalence by eye.
Phase 6 — Hand off
Methods/Results → /write-paper; compliance (TRIPOD-LLM / MI-CLEAR-LLM) → /check-reporting; reviewer
audit → /self-review (loads ME0–ME8).
Anti-Hallucination
- Never fabricate model outputs, reference labels, or metric scores. Compute only what the supplied outputs allow; if a reference standard or outputs are missing, say so and stop.
- Never report n-gram overlap (BLEU/ROUGE) as clinical correctness — pair it with a clinical-efficacy metric, and flag the n-gram score for what it is.
- Never claim "no contamination" without a stated check when a public benchmark is used.
- If a metric (RadGraph-F1 / CheXbert-F1) or its extractor is uncertain, flag
[VERIFY]and ask rather than inventing a number.
Deterministic gate
scripts/check_mllm_eval_completeness.py — task-aware presence gate on the evaluation plan (stdlib,
network-free). Reproducible challenge:
bash ${CLAUDE_SKILL_DIR}/scripts/mllm_eval_completeness_challenge/verify.sh.
Known limits. Manifest mode checks what is declared, not that the work was done; keep the
manifest in step with the Methods. It also does not check that a declared metric suits the task:
report-generation metrics (BLEU, RadGraph F1) declared for a VQA task are accepted without
comment. Prose mode (--plan) checks that a term is present; it does not read negation or sense.
"No human evaluation was performed" or "hallucination was not assessed" still counts as covering
that axis, and a word used in another sense still clears it: "green" anywhere clears the clinical-metric check, "unsupported"
clears faithfulness, "data leakage" from a patient split clears contamination, and "ground truth"
clears the reference standard whatever it refers to. Read each cleared axis in the plan yourself
before treating an exit 0 as clean.
Boundaries
mllm-eval (this skill: harness design + completeness gate, model-agnostic)
├─ design-ai-benchmarking (reader-study rubric / IRR)
├─ calc-sample-size (reader + case sizing)
├─ write-paper + check-reporting (TRIPOD-LLM / MI-CLEAR-LLM)
└─ self-review / peer-review (ME0–ME8 reviewer probe)
Reference Files
${CLAUDE_SKILL_DIR}/references/evaluation_axes.md— the why behind the ME2–ME7 axes: clinical-efficacy metrics beyond n-gram overlap (e.g. RadGraph-F1 / CheXbert-F1 vs BLEU/ROUGE), faithfulness & hallucination, pretraining/benchmark contamination, prompt-sensitivity & determinism, answer-matching, and the reader study — each mapped to its gate verdict. Load on demand during Phases 2–4.
Model-Assessment Skill
Assess a trained medical-imaging model (in-house, vendor or open-weights; segmentation, classification or detection) in the order the evidence is built: Part A designs and audits the validation study, Part B computes task-correct held-out metrics, Part C adds the uncertainty / OOD / abstention layer a deployment claim needs, and Part D makes an explainability analysis survive review. Run the parts the request needs (a Grad-CAM question starts at Part D), but no metric headline is reported before the Part A split gate is green. Each part ends in a stdlib gate whose verdict is reproduced from a file — never report a pass without running it.
Numbers come only from code executed on the supplied predictions, split table, or the researcher's executed UQ/XAI code; if predictions or ground truth are missing, say so and stop. Integrate MONAI / nnU-Net, MAPIE, captum, pytorch-grad-cam and pretrained OOD scorers by reference — never reimplement them, never build or train the model, never run a model on real patient data.
Elsewhere: building/training → /model-scaffold; choosing or vetting the model →
/model-selection; data-stage preprocessing leakage → /imaging-data; paired model comparison /
added value over a baseline / decision curves / MRMC / ICC / calibration tables → /analyze-stats
(added value: incremental_value.md); AI-vs-expert reader rubric
and IRR → /design-ai-benchmarking; LLM/MLLM → /mllm-eval; general validity → /design-study;
classical-ML tabular calibration → /radiomics-ml; item-by-item reporting audit →
/check-reporting; a finished manuscript → /self-review or /peer-review (the MD0–MD11
model_development.md probe).
Part A — Validation design
The rationale behind Phases 2–7 — the full leakage taxonomy, the internal-vs-external ladder,
comparator design, run variance, test-set sizing, the reporting map — is in
${CLAUDE_SKILL_DIR}/references/validation_design.md (load on demand). Verify citations via
/search-lit (confirmed DOI/PMID), else mark [UNVERIFIED - NEEDS MANUAL CHECK]; flag an uncertain
CLAIM 2024 / TRIPOD+AI / Metrics Reloaded item [VERIFY] and ask.
Phase 1 — Task, intended-use horizon, analysis unit
State the task, the intended-use horizon (screening, triage, pre-procedure, post-hoc), the single headline metric the conclusion leans on, and the analysis unit (per-patient / per-lesion / per-image). Everything downstream is read against this; a per-lesion metric is never reported as per-patient.
Phase 2 — Leakage audit (run first)
Produce the emitted split-assignment table (patient_id,split) and run:
python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \
--splits <split_assignment.csv> --out qc/split_leakage.json --strict
PATIENT_OVERLAP (a patient in ≥ 2 partitions) and MISSING_SEED (an unreproducible split) are
Major, SINGLE_PARTITION Minor — proven by set arithmetic on the ID column the gate prints as
id_col (pass --id-col with the patient identifier when it is not the one; an auto-picked
column whose name is not patient-level, such as image_id, gets a Minor ID_COL_NOT_PATIENT_LEVEL). A design with patient overlap is never
approved. Then walk the leakage the table cannot show (Kapoor & Narayanan, Patterns 2023):
preprocessing fit before the split (normalisation, resampling, foundation-model embeddings, ComBat
harmonisation over the whole cohort — /imaging-data gates the declared pipeline), site / scanner /
burned-in-label shortcuts, and temporal leakage (a random split where future and past coexist).
The decisive question: could any value used in training have been computed only with knowledge of
a test case?
Phase 3 — Validation tier
Classify honestly: apparent → internal random split → cross-validation → temporal → geographic / external (different site, scanner, vendor) → multi-site external. Cross-validation and bootstrap are development-time optimism corrections, not external validation. Flag a generalisability or deployment claim that outruns the design, and "external validation" where the single external set was used for tuning. Confirm the test set was touched once — no architecture search, hyperparameter sweep, early stopping, or threshold choice read it.
Phase 4 — Comparator
Clinical-only baseline, incremental value over an existing score, or reader comparison (hand the
rubric / inter-rater design to /design-ai-benchmarking).
Phase 5 — Test-set sizing
Count events per class in the test set, not the cohort total: a sparse positive set gives a CI
spanning much of the usable range, and calibration needs roughly ≥ 100 events. Hand formal sizing to
/calc-sample-size.
Phase 6 — Prospective evaluation and deployment-monitoring horizon
Retrospective external validation shows accuracy transfers, not that the model is safe and useful
in the workflow. For a clinical-use claim design the higher tier explicitly — silent / shadow
deployment → prospective comparative or impact study / RCT on a clinical endpoint →
post-deployment monitoring with recalibration-or-withdrawal triggers and subgroup audit
(references/validation_design.md §2b). Scope the claim to the tier reached: a retrospective
external study never claims deployment readiness or outcome benefit.
Phase 7 — Reporting-guideline fit
Map via /check-reporting: CLAIM 2024 (diagnostic imaging AI), TRIPOD+AI (prediction model),
STARD-AI (diagnostic accuracy), PROBAST+AI (risk of bias), and for a prospective/live
evaluation DECIDE-AI or CONSORT-AI / SPIRIT-AI.
Part B — Held-out metrics
Phase 8 — Compute task-correct metrics
Generate and execute evaluation code on the held-out predictions (Metrics Reloaded — Maier-Hein & Reinke et al., Nat Methods 2024):
- segmentation — Dice/IoU and a boundary metric (HD95 / NSD), per structure, 95% CIs by patient-level bootstrap (resample patients, not pixels or slices);
- classification — AUROC and AUPRC with patient-level bootstrap CIs, sensitivity/specificity, and PPV/NPV at the deployment prevalence, never bare accuracy on a balanced set. Report AUPRC with the test-set prevalence, which is its no-skill value: AUPRC moves with prevalence, so a value from an enriched or case-control test set does not carry over to deployment or across datasets. For multiclass, state the aggregation (one-vs-rest / macro / micro / pairwise / Obuchowski);
- detection — FROC / mAP with the IoU match criterion stated. Lesions and false positives cluster within patients, so CIs come from a patient-level bootstrap (resample patients, carrying all their lesions and false positives), not a Wilson/binomial interval over lesions, which is too narrow; compare two detectors' FROC curves with JAFROC (RJafroc), not per-lesion tests;
- interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive) — the segmentation
metrics plus Dice-vs-interactions / number-of-clicks (NoC) to a target threshold,
initial-vs-converged (or peak) Dice, and per-case interaction/inference time. With two arms
(simulated prompting + human operator), record protocol fidelity — identical prompt types,
stopping rule, target threshold, seeds — because arm-to-arm comparability is what lets the human arm
validate the simulated one (human-arm design:
/design-study); - generative / synthesis — full-reference (MSE/RMSE/PSNR/SSIM) or no-reference (SNR/CNR, visual scores) quality plus a downstream-task evaluation: image quality is not clinical utility (Park et al., Radiol Med 2024);
- time-to-event discrimination (Harrell's C, time-dependent ROC) →
/analyze-stats.
Report the headline as the point estimate with a patient-level bootstrap 95% CI over the test
cases: that is the uncertainty of the test-set estimate. Seed-to-seed SD across training runs is a
different quantity (training-run variability, usually smaller) — report it separately, over ≥ 5
runs, for a training-recipe or model-comparison claim, and never present it as the CI. A frozen
vendor or open-weights model has no training runs to vary; its uncertainty is the test-set CI. Add
calibration — for a binary risk or diagnostic output, calibration-in-the-large (intercept), the
calibration slope and a flexible (loess) calibration curve, plus the Brier score; ECE only as a
supplementary top-label summary for multi-class confidence, with its binning stated — and
subgroup slices (the Model Card Factors). Emit results.md (metrics report) and a per-case
CSV for /analyze-stats. Load
${CLAUDE_SKILL_DIR}/references/metric_guide.md for the per-task checklist and
${CLAUDE_SKILL_DIR}/references/metric_selection_grounding.md for why each pairing is required and
the CLAIM 2024 fit map.
Phase 9 — Gate the metric choice
Declare the reported metrics in metrics_manifest.json (copy
${CLAUDE_SKILL_DIR}/templates/metrics_manifest.json; fields and allowed values in
references/metrics_manifest_schema.md), then:
python3 ${CLAUDE_SKILL_DIR}/scripts/check_metric_reporting.py \
--manifest metrics_manifest.json --out qc/metric_reporting.json --strict
PIXEL_ACCURACY_SEG / NO_BOUNDARY_METRIC / ACCURACY_ONLY / DETECTION_METRIC_MISSING /
INTERACTIVE_NO_INTERACTION_COUNT / GENERATIVE_NO_DOWNSTREAM / CLASSIFICATION_METRIC_MISSING /
SEGMENTATION_METRIC_MISSING (every Major) must be zero; the last two (no headline metric declared)
are manifest-only.
An off-list value exits 2; use "none" or "other:<description>". --report results.md --task <task> still runs the older keyword check on prose.
Known limits: manifest mode checks what is declared, not the reported numbers. Prose mode
(--report) tests keyword presence with a short negation window: "MSD" counts as mean surface
distance even when it names the Medical Segmentation Decathlon, "we did not compute the Hausdorff
distance or HD95" still counts HD95, "sensitivity and specificity were not reported" or "FROC was
not performed" still count as reported, a bare "map" ("saliency map") counts as mAP, and a wrapped
"mean average\nprecision" is not seen.
Part C — Uncertainty, OOD and selective prediction (deployment claims)
A deployment-framed model must say what it does when unsure or off-distribution. Read
${CLAUDE_SKILL_DIR}/references/uncertainty_guide.md for method choice and the manifest schema.
Phase 10 — Choose the uncertainty method, OOD guard and abstention rule
- Conformal (MAPIE) — prediction sets/intervals at nominal coverage; the strongest default with a calibration set. Its coverage guarantee is finite-sample but needs exchangeability, which can fail on clinical data, so measure achieved coverage on a test split disjoint from the calibration split and report it with its binomial CI — never report it as guaranteed.
- Deep ensemble — K ≥ 2 independent members (distinct seeds/inits); shared seeds underestimate epistemic uncertainty.
- MC-dropout — dropout active at inference, T passes; off, every pass is identical and the estimate collapses to a point prediction.
- Bayesian / last-layer Laplace — a light option.
- OOD guard — energy score, feature Mahalanobis, ODIN or max-softmax, evaluated on a held-out OOD set (different scanner/site/pathology) with detection AUROC and the operating point.
- Selective prediction — abstain at a pre-specified coverage/risk target; report the risk–coverage curve. A post-hoc threshold inflates accuracy-at-coverage.
- Under shift — report calibration/coverage on shifted or external data, not in-distribution only (Ovadia 2019).
Phase 11 — Emit and gate the uncertainty manifest
Write uncertainty_manifest.json:
{
"task": "classification",
"deployment_claim": true,
"uncertainty_method": "conformal",
"coverage_target": 0.90,
"coverage_validated": true,
"ood_method": "mahalanobis",
"ood_heldout_set": "external-ood-cohort",
"selective_prediction": true,
"selective_target": 0.95,
"calibration_under_shift": true
}
python3 ${CLAUDE_SKILL_DIR}/scripts/check_uncertainty_reporting.py --manifest uncertainty_manifest.json \
--out qc/uncertainty_reporting.json --strict
Verdicts: POINT_PREDICTION_NO_UNCERTAINTY, CONFORMAL_NO_COVERAGE_VALIDATION, OOD_NO_HELDOUT_SET
(Major); ENSEMBLE_NOT_INDEPENDENT, MCDROPOUT_DISABLED_AT_INFERENCE, SELECTIVE_NO_TARGET,
NO_CALIBRATION_UNDER_SHIFT (Minor). It audits the declared spec; it complements, not replaces,
Phase 8's executed calibration. Report TRIPOD+AI / DECIDE-AI deployment-monitoring fit via /check-reporting.
Part D — Explainability
A saliency / Grad-CAM map is the most over-interpreted artifact in imaging AI: Adebayo et al.
(NeurIPS 2018) showed many methods produce convincing maps independent of the model's weights and
labels. Read ${CLAUDE_SKILL_DIR}/references/explainability_guide.md for method by architecture,
sanity checks, localisation metrics and framing.
Phase 12 — Produce, sanity-check and quantify the maps
Choose the method for the architecture — Grad-CAM / Grad-CAM++ for CNNs, attention rollout for ViTs, integrated gradients / SHAP for attribution — wired through captum or pytorch-grad-cam. Run the Adebayo model-parameter and data (label) randomisation tests; a faithful map degrades when they are randomised, and both axes are the minimum bar. If the map is claimed to localise the finding, compute IoU / pointing game / Dice against ground-truth masks over the cohort — not eyeballed, cherry-picked cases. Frame a map as attribution ("where signal is attributed"), never as proof the model is correct or of causation.
Phase 13 — Emit and gate the explainability report
Write explainability_report.json:
{
"method": "grad-cam++",
"n_examples": 200,
"cohort_level": true,
"localization_metric": "iou",
"localization_value": 0.63,
"sanity_checks": ["model_randomization", "data_randomization"],
"interpretation": "localization"
}
interpretation: attribution / localization / faithfulness — never validation / causal.
python3 ${CLAUDE_SKILL_DIR}/scripts/check_explainability_report.py --manifest explainability_report.json \
--out qc/explainability_report.json --strict
Verdicts: SALIENCY_AS_VALIDATION, NO_SANITY_CHECK, NO_LOCALIZATION_METRIC (Major);
INSUFFICIENT_SANITY, CHERRY_PICKED_EXAMPLES, MISSING_METHOD (Minor).
Outputs and hand-off
- Part A: validation-design decision notes (leakage, tier, comparator, metric, sizing handoff, reporting
fit) and
qc/split_leakage.json. - Part B:
results.md, the per-case CSV, andqc/metric_reporting.json. - Part C:
uncertainty_manifest.json+qc/uncertainty_reporting.json. - Part D:
explainability_report.json+qc/explainability_report.json.
The per-case table → /analyze-stats (paired ΔAUC of frozen models on the same test patients —
DeLong or bootstrap; added value over a baseline per incremental_value.md; decision curves;
publication tables);
figures → /make-figures; numbers and subgroup performance → /model-card; Methods/Results →
/write-paper; compliance → /check-reporting; sizing → /calc-sample-size; the reviewer-side audit
of the draft → /self-review, whose ai_overclaiming / image_synthesis probes also check saliency
claims. Gate regression (${CLAUDE_SKILL_DIR}/): scripts/check_split_leakage_challenge/verify.sh,
scripts/metric_reporting_challenge/verify.sh, scripts/check_uncertainty_reporting_challenge/verify.sh,
scripts/check_explainability_report_challenge/verify.sh, and tests/test_*.sh.
Model-Card Skill
Purpose
This skill produces the documentation an engineer-built medical-imaging model must carry: a
Model Card (intended use, out-of-scope use, training data, per-subgroup performance, caveats), a
Datasheet for its dataset (provenance, composition, collection, labelling, consent), and a
METRIC-informed data-quality pass. It fills the templates from facts the user supplies — it
never invents a number, a provenance detail, a consent status, or a licence — and ships a deterministic
gate that no required section is missing or left as an unfilled [NEEDS INPUT] placeholder.
It is the reporting seam of the model-engineering lane: after /model-assessment audits the design
and produces the numbers, this skill records them in a portable, auditable card that
/write-paper and /check-reporting consume. It mirrors /version-dataset structurally (generate +
deterministic verify).
When to use
- A trained model needs a Model Card / Datasheet for a repo, Hugging Face card, or manuscript supplement.
When NOT to use
- Auditing the validation design / metrics →
/model-assessment. - Versioning the dataset bytes →
/version-dataset; tabular variable docs →/generate-codebook. - Item-by-item reporting-guideline compliance of the manuscript →
/check-reporting. - Building / training the model →
/model-scaffold.
Workflow
Phase 1 — Collect the facts
Gather, from the user / the model's developers: task + architecture + provenance + licence; intended use
and out-of-scope use; training and evaluation cohorts; the reference standard and inter-reader agreement;
overall and per-subgroup performance; data collection, consent, and de-identification. Anything not
supplied stays [NEEDS INPUT] — never guess.
Phase 2 — Fill the Model Card
Copy ${CLAUDE_SKILL_DIR}/references/model_card_template.md to MODEL_CARD.md and fill each section
from the facts. Keep the headings. Numbers come only from /model-assessment / executed results.
Phase 3 — Fill the Datasheet
Copy ${CLAUDE_SKILL_DIR}/references/datasheet_template.md to DATASHEET.md and fill the seven
question groups (Motivation, Composition, Collection, Preprocessing/Labeling, Uses, Distribution,
Maintenance).
Phase 4 — METRIC data-quality pass
Walk ${CLAUDE_SKILL_DIR}/references/metric_dimensions.md (completeness, correctness, consistency,
representativeness, timeliness, provenance, label provenance, fairness/coverage, leakage safety) and
record each finding in the Datasheet. Anything that affects the headline metric's validity is also a
/model-assessment finding — cross-check there.
Phase 5 — Verify completeness (deterministic gate)
python3 ${CLAUDE_SKILL_DIR}/scripts/check_model_card_complete.py \
--card MODEL_CARD.md --datasheet DATASHEET.md --strict
MISSING_SECTION / EMPTY_REQUIRED_SECTION / UNFILLED_FIELD must be zero before the card ships.
UNFILLED_FIELD names each field of a required section still left as [NEEDS INPUT] / [VERIFY]
(e.g. License or subgroup performance), even when a sibling field is filled. An explicit N/A /
None counts as an answer only as a field's whole value, and never in Intended Use, Training Data,
Evaluation Data, Metrics or Quantitative Analyses. Only the template tokens as written
([NEEDS INPUT ...], [VERIFY], [VERIFY: ...], upper case) count; markdown link text such as
[verify the protocol](https://...) or [Verify][ref], and anything inside an HTML comment or a
fenced code block, is not a placeholder.
Known limits: a field left as a hand-written TODO / TBD / <...> / XXXX is caught only when
the whole section is unfilled; keep the template's bracketed [NEEDS INPUT] markers for open fields.
Phase 6 — Hand off
Carry the card into /write-paper (the Methods / supplement reference it), /check-reporting
(CLAIM 2024 / TRIPOD+AI item audit of the manuscript), and /self-review.
Anti-Hallucination
- Never invent evaluation numbers, subgroup results, or dataset provenance. Every figure comes from
/model-assessmentor the user's executed results; every provenance / consent / licence statement is user-confirmed. Unknown →[NEEDS INPUT], which the gate flags. - Never mark a section complete without user-supplied content, and never auto-fill a placeholder to pass the gate.
- Never assert a licence or consent status the user did not confirm.
- The gate checks presence, not truth — a complete card can still contain a wrong number; validity
is
/model-assessmentand the human's responsibility.
Deterministic gate
scripts/check_model_card_complete.py — verifies every required Model Card / Datasheet section is
present and non-empty (stdlib, network-free). Reproducible challenge:
bash ${CLAUDE_SKILL_DIR}/scripts/check_model_card_complete_challenge/verify.sh.
Note on classification
Model Cards (Mitchell et al. 2019) and Datasheets (Gebru et al. 2021) are documentation standards,
not clinical reporting guidelines, so they live here as references/ templates (uncounted), not in
/check-reporting's counted checklist set — the same way appraisal_tools/METRICS.md is kept separate.
/check-reporting still owns the manuscript-level CLAIM 2024 / TRIPOD+AI item audit.
Boundaries
model-assessment (audit design + metrics)
└─ model-card (this skill: Model Card + Datasheet + METRIC pass, completeness-gated)
└─ write-paper + check-reporting (manuscript) ; version-dataset (dataset bytes)
/model-evaluation was renamed to /model-assessment in v6.0; this redirect is removed in v7.
Run /model-assessment now, passing the same arguments the user gave /model-evaluation.
Start at /model-assessment Part B, Phases 8–9 (task-correct held-out metrics and the metric-reporting gate).
Model-Scaffold Skill
Purpose
This skill stamps out a runnable PyTorch training repo for a medical-imaging task — --task
segmentation (U-Net), classification (CNN / timm backbone), detection (torchvision Faster
R-CNN / FPN), synthesis (Pix2Pix generator + PatchGAN), ssl (SimCLR encoder), or finetune
(transfer-learning a pretrained backbone with a frozen→unfrozen schedule + a provenance record) —
with the reproducibility guarantees baked in by construction — so the build is leakage-safe and
reproducible before a single epoch runs. It is the imaging analogue of how /analyze-stats generates
runnable statistical code: the generator produces the repo, you run the training on your GPU / Colab,
and the lane's deterministic gates verify the network-free parts.
It is the missing middle link in the lane: /model-selection (choose) → model-scaffold (build)
→ /model-assessment (validate the split / design, compute metrics) → /analyze-stats
→ /write-paper + /check-reporting (publish). It integrates MONAI / nnU-Net / TorchIO (referenced
in the generated requirements.txt); it does not reimplement them.
When to use
- You have a data manifest (one row per image, with a patient/subject ID) and want a reproducible, leakage-safe starting repo for a segmentation, classification, detection, synthesis or self-supervised model.
- You want to fine-tune a pretrained backbone (transfer learning — the common clinician workflow:
a
timm/ MONAI / MedSAM checkpoint adapted to your collected clinical data) with the freeze schedule, discriminative learning rates, and pretrained-weight provenance recorded (--task finetune).
When NOT to use
- Auditing an already-trained model's validation design →
/model-assessment. - Held-out metrics / calibration / bootstrap CIs →
/model-assessmentthen/analyze-stats. - Choosing the architecture for the research question →
/model-selection. - Reimplementing MONAI / nnU-Net → out of scope (the scaffold integrates them).
- LLM / MLLM evaluation →
/mllm-eval.
Workflow
Phase 1 — Prepare the manifest
A CSV with one row per image and a patient/subject ID column (patient_id / subject_id /
case_id), plus image and label path columns. The ID column is load-bearing: the split is done at the
patient level off this column. IDs are compared after stripping surrounding whitespace (P01 and
P01 are one patient), in the split and in the generated dataset.py alike.
Phase 2 — Generate the repo
python3 ${CLAUDE_SKILL_DIR}/scripts/scaffold.py \
--manifest <manifest.csv> --task segmentation --out model_repo --seed 42 \
--in-channels 1 --out-channels 1
# --task = segmentation | classification | detection | synthesis | ssl | finetune
# (out-channels = num classes for classification/finetune, target channels for synthesis;
# finetune uses a softmax CrossEntropy head, so it refuses --out-channels < 2 — binary = 2)
# fine-tuning a pretrained backbone (transfer learning) on collected clinical data:
python3 ${CLAUDE_SKILL_DIR}/scripts/scaffold.py \
--manifest <manifest.csv> --task finetune --out model_repo --seed 42 \
--out-channels <num_classes> --from-pretrained timm:resnet50.a1_in1k
# emits PRETRAINED.md (provenance) + a frozen→unfrozen train.py with discriminative LRs;
# record the exact pretrained source so the fine-tune is reproducible. build_model(pretrained=True)
# raises if timm is missing (never a silent random-init stand-in); best.pt records backbone_class.
# reuse the split /imaging-data's preprocessing gate checked (do not draw a new one):
python3 ${CLAUDE_SKILL_DIR}/scripts/scaffold.py \
--manifest <manifest.csv> --preprocessing-manifest preprocessing_manifest.json --out model_repo
# copies its split_assignment + split_seed into splits/, reading rows with the gate's own rules
# (patient key patient_id/subject_id/patient/id; split synonyms such as training/validation/holdout);
# exits 2 if a manifest patient has no split there, a patient sits in two splits, a split does not
# map to train/val/test, or split_seed is absent.
The imaging-data QC handoff is enforced, not advisory. With --preprocessing-manifest the
scaffold also reads /imaging-data's gate reports (check_dataset_profile,
check_preprocessing_leakage, check_normalizer_domain JSON) from <manifest dir>/qc/ and
<manifest dir>/../qc/ (the manifest path is resolved first). --imaging-qc <file|dir> (repeatable)
replaces that search: pointed at an empty directory, every gate is recorded NOT ASSESSED. A
leakage report is skipped only when its recorded manifest has a different file name, resolves to an
existing file, and no candidate path holds a byte-identical copy of the scaffolded manifest (the
reason is shown); anything else — a copied project, a same-named manifest elsewhere — is read.
- A Major claim — or any severity that is not plainly Minor/Flag, or a report whose
summary.n_majorexceeds its listed Majors — refuses: exit 1, nothing written, each code + report listed. Resolve it upstream and re-run the gate, or pass--ack-qc CODE='reason'once per code. - Minor / Flag claims never block; they are carried forward as warnings.
- A gate with no readable report is NOT ASSESSED; an unparseable or off-shape file in
qc/is listed as UNREADABLE (stderr + record), never dropped.
All of it lands in model_repo/IMAGING_QC.md, referenced from config.yaml (imaging_qc:) and
REPRODUCIBILITY.md, so training, evaluation and the Methods read what /imaging-data found. Never
write an --ack-qc reason the user has not given. Without either flag the output is unchanged.
This writes model_repo/ with config.yaml, model.py (the task's model — U-Net / CNN / Faster R-CNN
/ Pix2Pix / SimCLR encoder), dataset.py (reads the frozen split), losses.py (task-appropriate),
train.py, evaluate.py, requirements.txt,
REPRODUCIBILITY.md, methods_stub.md (+ IMAGING_QC.md when imaging-data outputs are given), and — the key artifact — splits/split_assignment.csv +
splits/split_seed.txt. The split is patient-disjoint by construction (a deterministic group split)
and the emitted code seeds every RNG, sets cuDNN deterministic, builds the training loader from the
train split only, and infers under model.eval() + torch.no_grad().
Phase 3 — Verify the build (network-free)
# this skill's own training-hygiene gate
python3 ${CLAUDE_SKILL_DIR}/scripts/check_training_hygiene.py --repo model_repo --strict
# the split-leakage gate (proves patient disjointness) — owned by /model-assessment
Route the emitted splits/split_assignment.csv to /model-assessment
(check_split_leakage.py --splits model_repo/splits/split_assignment.csv --strict) for the
patient-disjointness proof, and (optionally, locally with torch installed)
bash ${CLAUDE_SKILL_DIR}/scripts/scaffold_challenge/verify.sh to smoke the forward pass.
Phase 4 — Plug in your data and train
Implement dataset.py's _load_image / _load_label for your modality (DICOM / NIfTI / TIFF via
nibabel / pydicom / tifffile / TorchIO / MONAI transforms). For production, swap model.py for MONAI
UNet / SegResNet or an nnU-Net plan (see ${CLAUDE_SKILL_DIR}/references/training_guide.md). For a
fine-tuning repo (--task finetune), fill PRETRAINED.md and set the freeze schedule / discriminative
learning rates (see ${CLAUDE_SKILL_DIR}/references/finetuning_guide.md, which also covers MedSAM/SAM
adaptation and train-only diffusion augmentation). Run python train.py (best model selected on the
val split), then python evaluate.py (predictions on the test split, touched once).
Phase 5 — Validate, evaluate, publish
Hand off to /model-assessment (validation-tier + comparator + metric-selection audit; Dice +
HD95/NSD with CIs) + /analyze-stats, /make-figures, and /write-paper
(fill the methods_stub.md [VERIFY] placeholders) + /check-reporting (CLAIM 2024 / TRIPOD+AI). For
reproducibility-safe wiring of experiment tracking (W&B / MLflow), config / data / environment
versioning, and the MLOps reporting checklist, see ${CLAUDE_SKILL_DIR}/references/mlops_guide.md
(a wiring + reporting reference — it points to the frameworks, it does not replace them).
Runnability — honest contract
The generated repo is runnable, but runnability is not a CI guarantee. The default gates prove
the network-free properties (the emitted split is patient-disjoint + seeded; the emitted training code
is hygienic) by parsing the produced artifacts — no torch is executed. A torch forward-pass smoke
(build + forward shape + gradients flow + reproducible loss) is a self-skipping tier in the
challenge verify.sh and a documented local command; it is never counted as CI coverage of
runnability.
Anti-Hallucination
- Never fabricate training or evaluation metrics. The scaffold emits
[VERIFY]placeholders; every number must come from the user's executed run and from/model-assessment+/analyze-stats. - Never emit a split that is not patient-disjoint or not seed-locked. The generator does this by construction; do not hand-edit the split table to introduce overlap or remove the seed.
- Never claim the generated repo was trained or that it achieved a result — it is a starting point the user runs.
- If a library API, default, or architecture detail is uncertain, flag
[VERIFY]and ask rather than guessing.
Deterministic gates
scripts/scaffold.py— the generator (stdlib + numpy; deterministic given manifest + seed); also the imaging-data QC handoff (tests/test_imaging_qc_handoff.sh).scripts/check_training_hygiene.py— AST linter: all RNGs seeded, cuDNN deterministic,eval()+no_grad()inference, no training on a non-train split, and (fine-tuning) a recorded pretrained-weight provenance when pretrained weights are loaded (PRETRAINED_PROVENANCE_MISSING).scripts/scaffold_challenge/verify.sh— the build → validate chain, network-free (torch tier self-skips).
Known limits of check_training_hygiene.py
- It counts seeding / cuDNN /
eval()/no_grad()only in code a run reaches from the script's import-time statements. A file with no import-time call into its own functions is entered through every public top-level function nothing references, so an uncalled publicseed_everythingin such a file still counts. Every decorated function or method (a click/typer command, a route, a@staticmethod) is also an entry point, so an uncalled decorated seeding helper still counts. - It does not check statement order: inference placed before
model.eval()in the same function is not detected. --repowithouttrain.py/evaluate.pyreports those checks as NOT CHECKED; with--strictit exits 2 rather than clearing them.- Dataset variables are resolved by one name-to-split map for the whole file (later assignment in source order wins, across functions), not per scope; a shuffled loader that combines train with val only (a train+val refit) is not flagged.
Boundaries
model-selection (choose)
└─ model-scaffold (this skill: generate the reproducible repo)
├─ check_training_hygiene.py (training-code hygiene)
├─ model-assessment -> analyze-stats (split-leakage proof, validation design, metrics + CIs)
└─ write-paper + check-reporting (Methods stub -> compliant manuscript)
Model-Selection Skill
Two questions, in this order. Phases 1–3 answer a literature question with a stable answer: which architecture family suits the task. Phases 4–6 answer a provenance question: which concrete artifact — repository, revision, checkpoint — will be run, and what its numbers may claim. The skill writes decision notes and runs one stdlib gate; it never downloads, runs, trains or benchmarks a model, and it describes archetypes, not a live SOTA leaderboard.
Elsewhere: building the repo → /model-scaffold; profiling data and planning preprocessing →
/imaging-data; validation design, metrics, uncertainty, explainability → /model-assessment;
documenting a model you built → /model-card; an LLM/MLLM, including benchmark contamination in
that setting → /mllm-eval; AI-vs-expert reader study → /design-ai-benchmarking.
Workflow
Phase 1 — Frame the question
State the task (classification / segmentation / detection / synthesis / transfer), the modality and dimensionality (2-D vs 3-D volume), the labelled-data scale (events / structures, not just images), label availability (lots / few / unlabelled pool), and the constraints (class imbalance, small structures, interpretability, deployment compute).
Phase 2 — Walk the decision tree, read the family card
Follow ${CLAUDE_SKILL_DIR}/references/index.md (task → constraints → default pick). It routes to
one card; each gives the source paper, core idea, when-to-use, medical-imaging use, reference
implementation, and the typical validation setup for that class.
| Card | Families |
|---|---|
${CLAUDE_SKILL_DIR}/references/classification.md | ResNet / DenseNet / EfficientNet / Inception / ConvNeXt / ViT / Swin / DeiT |
${CLAUDE_SKILL_DIR}/references/segmentation.md | U-Net / 3-D U-Net / V-Net / Attention & Residual U-Net / nnU-Net (+ ResEnc, MedNeXt, STU-Net) / SegResNet / Swin-UNETR / Mask R-CNN |
${CLAUDE_SKILL_DIR}/references/detection.md | nnDetection / Faster R-CNN + FPN / Mask R-CNN / RetinaNet / YOLO / DETR |
${CLAUDE_SKILL_DIR}/references/synthesis.md | Pix2Pix / CycleGAN / SPADE / diffusion (DDPM, latent) / VAE / fastMRI reconstruction |
${CLAUDE_SKILL_DIR}/references/foundation_models.md | SAM / MedSAM / MedSAM2 / nnInteractive / VISTA3D / TotalSegmentator / SegVol / BiomedCLIP / DINO / MAE / SimCLR / MoCo |
${CLAUDE_SKILL_DIR}/references/graph.md | GCN / GraphSAGE / GAT / GIN / BrainGNN for connectomes and population graphs (PyTorch Geometric / DGL; not scaffolded by /model-scaffold) |
Never recommend an architecture for a modality or data scale it does not suit (a from-scratch ViT on a few hundred images, 2-D slices for a volumetric structure) — the decision-tree constraints exist to prevent exactly that. If asked for "the best" model, say the zoo is a curated archetype map, not a current SOTA ranking.
Phase 3 — Write the architecture decision note
Write decisions/architecture_choice.md: the task, the chosen architecture, its source paper
(mandatory — the Methods cite it), the reason against the constraints, the runner-up and why not,
and the matching /model-scaffold template. Never invent a benchmark number or paper claim: cite it
(verify via /search-lit), or write [VERIFY] and ask. A reference without a /search-lit-confirmed
DOI/PMID is marked [UNVERIFIED - NEEDS MANUAL CHECK].
Phase 4 — Write the model dossier for the concrete artifact
When a concrete candidate exists (a GitHub repo, a Hugging Face checkpoint, a paper's released
weights), write model_dossier.json recording what is known; leave unknowns unstated, never
guessed — the gate turns an absence into a finding.
{
"model": "OrganSeg-3D v2.5.1",
"source": {"kind": "github", "url": "...", "version": "v2.5.1", "commit": "abc1234"},
"licence": {"spdx": "Apache-2.0", "verified_from": "LICENSE at commit abc1234"},
"intended_use": "research",
"weights": {"pretrained": false},
"task": {"model": "3d_ct_organ_segmentation", "study": "3d_ct_organ_segmentation"},
"reported_validation": [{"dataset": "ExampleBench", "metric": "Dice", "source": "J Ex 2021"}],
"developed_on": ["ExampleBench"],
"evaluation_arms": [{"name": "external", "dataset": "OtherCohort-2026"}],
"hardware": {"claimed": "any CUDA GPU", "verified_on": "GTX 1080 Ti", "verified": true}
}
Read each field from the artifact, not from memory:
- Licence — only from the licence file at the pinned revision, and record which file
(
verified_from). Never from a README badge, a model-card summary, or memory; an unstated licence isLICENCE_UNSTATED, never "probably MIT". developed_on— the paper's own account of where the method was built and tuned. It is the field people skip and the one the gate needs: a method that won a challenge was tuned on it.hardware.verified— true only after it actually ran. A support matrix is a different claim: a CUDA capability one compiler accepts may still be refused by another in the same stack.
Phase 5 — Gate the dossier
python3 ${CLAUDE_SKILL_DIR}/scripts/check_model_provenance.py --dossier model_dossier.json \
--out qc/model_provenance.json --strict
Stdlib-only and network-free: nothing is fetched and no licence resolved online.
| Verdict | Severity | Fires when |
|---|---|---|
BENCHMARK_PROVENANCE_CONFLICT | Major | an evaluation arm uses a dataset the model was developed or tuned on |
EVAL_DATA_IN_TRAINING | Major | an evaluation arm's dataset is inside the pretraining corpus |
LICENCE_UNSTATED | Major | no licence recorded — not the same as a permissive one |
LICENCE_INCOMPATIBLE | Major | non-commercial / research-only licence under commercial or deployment intent |
WEIGHTS_PROVENANCE_UNKNOWN | Major | pretrained weights whose training corpus is not stated |
DEVELOPED_ON_UNSTATED | Major | the developed_on key is absent, so the conflict check could not run |
EVALUATION_ARMS_UNSTATED | Major | the evaluation_arms key is absent, so neither provenance check could run |
INTENDED_USE_UNSTATED | Major | intended_use is absent, so licence compatibility could not be checked |
TASK_MISMATCH | Minor | the model's task is not the study's task |
NO_VERSION_PIN | Minor | no commit, tag or revision |
VALIDATION_UNREPORTED | Minor | no reported validation (dataset + metric + source) |
HARDWARE_UNVERIFIED | Minor | hardware support claimed but never executed |
LICENCE_UNVERIFIED | Minor | a licence is named but not the file it was read from |
The gate flags a relationship, not a reputation: developed_on: ExampleBench passes cleanly as
long as no evaluation arm uses ExampleBench (the clean fixture proves this). Dataset names match as
token sequences with a small family-alias table — MSD Task09 Spleen matches MSD,
MS Cohort 2026 does not — never by substring.
An explicit empty list ("developed_on": []) states "none" and is not a finding; an absent key is.
intended_use must be research, commercial or clinical_deployment, and developed_on /
weights.trained_on must be lists of strings; anything else is an input error (exit 2), never a pass.
Known limits. LICENCE_INCOMPATIBLE recognises non-commercial / research-only markers in the
licence string (NC, non-commercial, research-only, in SPDX or spaced form). A free-text licence
that restricts use without those markers (e.g. "academic use only"), or a copyleft licence under
closed deployment, is not detected: read the licence file and record the conclusion yourself.
Phase 6 — Turn a Major into a study decision
A BENCHMARK_PROVENANCE_CONFLICT rarely means abandoning the model (it is often the best-engineered
option precisely because it was tuned hard). It changes what the arm may claim:
- Report that arm as a demonstration that the pipeline runs end to end, not as evidence the method works.
- Put the evidential weight on an arm whose data post-dates the model, and say so with dates.
- State the conflict in Methods and Limitations rather than leaving a reviewer to find it.
Never report a flagged arm as independent validation. An EVAL_DATA_IN_TRAINING arm is different in
kind: it produces a training-set score and cannot be reported as validation at all. Write this
arm-by-arm decision into the study record.
Outputs and hand-off
decisions/architecture_choice.md— the architecture decision note (Phase 3).model_dossier.json— the provenance record (Phase 4);qc/model_provenance.json— the audit (Phase 5).- The arm-by-arm decision (Phase 6).
Carry them to /model-scaffold (instantiate the template), /model-assessment (arm and validation
design, what each arm may claim, metrics), /model-card (provenance section), and /write-paper
(Methods cite the source paper; Limitations state any conflict). Gate regression:
bash ${CLAUDE_SKILL_DIR}/scripts/check_model_provenance_challenge/verify.sh and
bash ${CLAUDE_SKILL_DIR}/tests/test_model_provenance.sh.
/model-sourcing was renamed to /model-selection in v6.0; this redirect is removed in v7.
Run /model-selection now, passing the same arguments the user gave /model-sourcing.
Start at /model-selection Phases 4–6 (write and gate the model dossier for the concrete repository or checkpoint).
/model-validation was renamed to /model-assessment in v6.0; this redirect is removed in v7.
Run /model-assessment now, passing the same arguments the user gave /model-validation.
Start at /model-assessment Part A, Phases 1–7 (validation design and the split-leakage gate).
Sponsors
Details
- Marketplace
- Aperivue/medsci-skills
- Author
- Aperivue
- Category
- Community
- Keywords
- medical-research, clinical-research, presentations, pptx, pdf