
science-superpowers
Computational-science methodology for Claude Code: research framing, pre-registration, reproducible analysis, anomaly investigation, and red-team review
- version
- 0.1.0
- GitHub stars
- 375
Install
/plugin install science-superpowers@science-superpowers-devRun it inside Claude Code after adding the marketplace with /plugin marketplace add K-Dense-AI/science-superpowers.
Contents
Discovered from sourceSkills 16
Designing the Analysis
Overview
Write a comprehensive analysis plan assuming the analyst has zero context for this project and questionable statistical taste. Document everything they need: which datasets and variables, how each construct is computed, the exact model or test, the sample size / power justification, which confounds are handled and how, the decision rules, and the planned figures. Give them the whole thing as bite-sized steps. DRY. YAGNI. Pre-register. Validate pipelines on known data. Commit frequently.
Assume they are a capable programmer but know almost nothing about this domain, this dataset, or good statistical design.
Announce at start: "I'm using the designing-the-analysis skill to create the analysis plan."
Save plans to: docs/science-superpowers/plans/YYYY-MM-DD-<topic>.md
Scope Check
If the question document still bundles several independent investigations, stop and split it — one plan per question. Each plan should produce an interpretable, self-contained result.
Inputs You Need First
- The approved question document (
docs/science-superpowers/questions/...) - The prior-work survey: the adopted method, the confound list, and a prior effect size for powering
If you don't have a prior effect size, the design must include how you will justify the sample size anyway (smallest effect of interest, precision target, or a sensitivity analysis).
Artifact & Data-Flow Structure
Before defining steps, map the pipeline. Data flows one direction: raw → cleaned → derived → results.
- List the scripts/notebooks you will create and what each is responsible for. One responsibility each.
- Name the data artifacts at each stage. Raw data is immutable; every transform writes a new artifact.
- Identify where the train/test or discovery/confirmation split happens (if any) and lock it before any modeling.
- Keep files focused. A 600-line "do everything" notebook is a design smell.
Confound & Validity Plan
For each known confound from the survey, state how it is handled: measured and adjusted for, stratified, matched, design-excluded, or explicitly acknowledged as a limitation. "We'll see" is not a plan.
State the threats to validity you are accepting and why.
Power / Sample Size
State the target effect size, alpha, desired power, and the resulting required N — or, for a fixed existing sample, the minimum detectable effect at the planned power. If underpowered, say so and decide with your human partner whether to proceed (e.g., reframe as estimation, not a hypothesis test).
Bite-Sized Step Granularity
Each step is one action (2-5 minutes). Every step that touches data is paired with a validation — the science analog of watching a test fail then pass:
- "Write the data-loading code" — step
- "Run it; confirm row/column counts match the known dataset shape" — step
- "Write the cleaning transform" — step
- "Validate: assert no rows silently dropped, ranges within expected bounds" — step
- "Validate the pipeline on simulated data with a known ground-truth effect; confirm it recovers the effect" — step
- "Commit" — step
The simulated-data validation step is mandatory for any nontrivial estimator or model: if you never watched your pipeline recover a known signal, you don't know it works.
Plan Document Header
Every plan MUST start with this header:
# [Question] Analysis Plan
> **For agentic workers:** REQUIRED SUB-SKILL: pre-register this plan with science-superpowers:preregistering-analysis BEFORE execution. Then use science-superpowers:subagent-driven-analysis (recommended) or science-superpowers:executing-analysis to run it step-by-step. Steps use checkbox (`- [ ]`) syntax for tracking.
**Question:** [the falsifiable question, one sentence]
**Design:** [observational/experimental; cross-sectional/longitudinal; the comparison]
**Data:** [datasets, sample, unit of analysis]
**Primary analysis:** [the one model/test that answers the question]
**Decision rule:** [exactly what result confirms vs. disconfirms H1]
---
Task Structure
### Task N: [Analysis component]
**Artifacts:**
- Create: `analysis/exact_script.py`
- Reads: `data/raw/exact_file.csv` (immutable)
- Writes: `data/derived/exact_output.parquet`
- [ ] **Step 1: Write the loading/transform code**
```python
df = pd.read_csv("data/raw/exact_file.csv")
clean = df[df["value"].between(0, 100)]
```
- [ ] **Step 2: Validate the step**
Run: `python analysis/exact_script.py --check`
Expected: `rows in: 10342, rows out: 10298, dropped: 44 (out-of-range)` — dropped count matches the known data-quality issue, not silent loss.
- [ ] **Step 3: Run the primary model exactly as specified**
```python
model = smf.ols("outcome ~ exposure + age + site", data=clean).fit()
```
- [ ] **Step 4: Apply the pre-registered decision rule**
The estimate for `exposure` is interpreted against the rule fixed in the pre-registration — not re-decided here.
- [ ] **Step 5: Commit**
```bash
git add analysis/exact_script.py data/derived/exact_output.parquet
git commit -m "analysis: primary model for exposure effect"
```
No Placeholders
Every step must contain the actual content the analyst needs. These are plan failures — never write them:
- "TBD", "choose an appropriate test", "handle outliers as needed", "tune the model"
- "Run statistics on the above" without naming the test and the variables
- "Adjust for confounders" without naming which and how
- "Similar to Task N" (repeat the specifics — steps may be read out of order)
- Any decision rule that depends on the result you haven't seen yet (that is HARKing; fix it in the design)
Self-Review
After writing the plan, re-read the question document with fresh eyes and check the plan against it:
- Coverage: Does a task answer the question? Does every hypothesis have an analysis?
- Researcher degrees of freedom: Is every analytic choice (test, covariates, exclusions, transformations, cutoffs) specified in advance? List any that are still open and close them.
- Confound coverage: Is every surveyed confound handled by a task?
- Placeholder scan: Any of the failures above? Fix them.
- Consistency: Do variable names, file paths, and the decision rule match across tasks?
Fix issues inline. If a question requirement has no task, add the task.
Handoff
The plan is not ready to execute until its predictions and decision rules are locked.
REQUIRED NEXT SKILL: Use science-superpowers:preregistering-analysis to freeze the confirmatory hypotheses, predictions, and decision rules before any outcome is observed. Execution happens only after that.
Dispatching Parallel Investigations
Overview
You delegate investigations to specialized agents with isolated context. By precisely crafting their instructions, you keep each focused and preserve your own context for synthesis. They never inherit your session history — you construct exactly what each needs.
When you have multiple independent investigations (different datasets, different sub-topics in a literature survey, different pre-specified robustness checks), running them sequentially wastes time. Each is independent and can run in parallel.
Core principle: One agent per independent investigation. Let them run concurrently, then synthesize.
The Pre-Registration Guard (read first)
Parallelism multiplies researcher degrees of freedom. If you dispatch 20 specifications and report the one that "works," you have p-hacked at scale — parallelism made it faster, not more honest.
- Pre-specified robustness checks are fine in parallel — they were registered, and you report all of them.
- Exploratory analyses are fine in parallel — as long as every result is labeled exploratory and none is promoted to confirmatory.
- Never use parallel dispatch to search the specification space for significance and report the survivor.
When to Use
digraph when_to_use {
"Multiple investigations?" [shape=diamond];
"Independent?" [shape=diamond];
"Pre-specified or all-reported?" [shape=diamond];
"Single agent / sequential" [shape=box];
"STOP: this is spec-hunting" [shape=box];
"Parallel dispatch" [shape=box];
"Multiple investigations?" -> "Independent?" [label="yes"];
"Independent?" -> "Single agent / sequential" [label="no - shared state"];
"Independent?" -> "Pre-specified or all-reported?" [label="yes"];
"Pre-specified or all-reported?" -> "Parallel dispatch" [label="yes"];
"Pre-specified or all-reported?" -> "STOP: this is spec-hunting" [label="no - cherry-picking"];
}
Use when:
- Surveying several independent sub-topics of prior work
- Replicating one analysis across several independent datasets
- Running a pre-specified set of robustness/sensitivity checks (all reported)
- Investigating several independent anomalies with different root causes
Don't use when:
- The investigations share state or feed each other
- You'd be selecting which result to report based on what comes back
The Pattern
1. Identify Independent Investigations
Group by what's being examined. Each must be understandable without the others.
2. Create Focused Agent Tasks
Each agent gets:
- Specific scope: one dataset, sub-topic, or check
- Clear goal: the question it must answer
- Constraints: for robustness checks, the exact pre-specified specification — not "find what works"
- Expected output: the result AND the method, so you can verify
3. Dispatch in Parallel
Task("Survey prior effect sizes for X in domain A")
Task("Survey known confounds for X")
Task("Replicate the primary model on dataset B, exact spec")
4. Synthesize and Verify
- Read each summary
- Verify results independently (don't trust "done")
- Report ALL pre-specified/robustness results together — including the ones that disagree
- Check for conflicts and reconcile
Agent Prompt Structure
- Focused — one investigation
- Self-contained — all context to understand it, including the exact specification for a robustness check
- Specific about output — the result, the method used, and enough to reproduce
Replicate the primary model on dataset B.
Use EXACTLY this pre-registered specification (do not alter it to improve fit):
outcome ~ exposure + age + site, OLS, exclude rows with missing exposure
Dataset B is at data/raw/site_b.csv (immutable). Set seed 20260528.
Validate the loaded shape, run the model, report:
- the coefficient on exposure with 95% CI and p
- N used and any rows excluded (with reason)
Do NOT try alternative specifications. Report this one result.
Common Mistakes
- Too broad: "Analyze everything" -> agent flails. Give one scope.
- No spec for robustness checks: the agent improvises -> you've created forking paths. Specify exactly.
- Trusting summaries: verify each result against the committed output.
- Reporting only the agreeable results: report all pre-specified investigations, including disconfirming ones.
Verification
After agents return:
- Read each summary
- Verify each result independently (
science-superpowers:verifying-results-before-claiming) - Check for conflicts between investigations
- Report the full set — agreements and disagreements alike
Integration
- science-superpowers:surveying-prior-work — parallel literature survey
- science-superpowers:subagent-driven-analysis — single-investigation execution (sequential within one investigation)
- science-superpowers:investigating-anomalous-results — parallel investigation of independent anomalies
Establishing Feasibility First
Overview
When it is not yet known whether the computation can run at all, the decisive experiment is not scientific — it is whether the thing executes at scale. Run that experiment first, on the smallest honest version of the real path, and let its measured answer shape everything downstream.
Core principle: Process built on an unmeasured feasibility number is process built on a guess. When the guess is wrong, every plan, prediction, and decision rule resting on it was written for a study that does not exist.
This is the deliberate exception to the standard sequence. It exists because the standard sequence assumes the analysis can be run, and pays its full cost before testing that assumption.
The Iron Laws
NOTHING IS FEASIBLE UNTIL IT HAS RUN
NOTHING IS BOUNDED UNTIL THE BOUND HAS FIRED
An estimate is not a measurement. A declared kill criterion is not a kill criterion. Both must be demonstrated by execution before any work is built on top of them.
The Entry Gate — Opt-In Only
Only your human partner can enter this mode. You may offer it; you may never declare it yourself.
Offer it when you notice the triggers: an unrun pipeline, an unbenchmarked solver, an estimated rather than measured ceiling, or a plan whose largest configuration has never executed. Make the offer concrete and wait:
"Whether this can run at scale is still unknown, and everything downstream depends on the answer. I can enter feasibility mode: a minimal end-to-end runner, three scaling probes, then one exploratory campaign — deferring the survey, the analysis plan, and pre-registration until we know what is runnable. That defers confirmatory claims; nothing produced in this mode can support one. Do you want that, or the standard confirmatory sequence?"
If they decline, or do not answer, you are in the standard sequence. Silence is not opt-in. Use science-superpowers:framing-research-questions and proceed normally.
You may not enter this mode to escape rigor. "This feels exploratory" is not opt-in. "The pre-registration is getting tedious" is not opt-in. Only an explicit choice by your human partner opens this path.
Before Anything: A Short Framing
Feasibility mode defers the survey and the analysis plan. It does not skip framing — it shrinks it.
Write a few sentences, not a document, and get agreement:
- The feasibility question — the specific thing that must be true for the science to be possible, stated as a measurable threshold ("a 30M-row assembly fits in under 60% of RAM and solves in under 8 hours")
- The scientific question it gates — one sentence, held for later
- The abandonment condition — what measured result means this approach is not viable, decided now
Keep these separate. Tangling them is what produces pre-registrations written over runs that cannot happen.
Get explicit agreement on the abandonment condition specifically — not an inference from "go ahead". It is necessarily a judgement call made before any measurement exists, so it is both the item most likely to be wrong and the one that later fires and stops the work. Revise it freely before the probes run; never after a probe has reported.
Phase 1: The Minimal End-to-End Runner
Build the thinnest thing that exercises the real code path end to end, at a trivial problem size.
- Real solver, real assembly, real I/O — a stub anywhere means the phase proved nothing about the real path
- Smallest size that still touches every stage
- One ground-truth check: a case whose answer is known analytically, asserted to tolerance
If the thing under test does not exist yet, build only the adapter to it. Do not build the study around a component that has never been called. Where the real component is genuinely unavailable, a reference implementation is acceptable — but label every number it produces as surrogate, and never let a surrogate result gate a decision without saying so.
Beware the degenerate test case. A manufactured problem whose solution is an eigenvector of the operator converges in a fixed number of iterations regardless of size, and will report flat, beautiful, meaningless scaling. If every probe converges in the same handful of iterations, suspect the test problem before believing the result.
A clean number is not a correct number. Convergence, a plausible order, and a tidy extrapolation are all available to a quantity measured at the wrong location, in the wrong component, or under a broken preconditioner — and they look exactly like success. Check the measured quantity against an independent reference (an analytic solution, a published value, a different implementation) before you believe it, and treat a large disagreement as an anomaly to root-cause rather than a calibration offset to absorb.
Phase 1 is done when the real path has run end to end once and produced a correct known answer.
Phase 2: Three Scaling Probes
Three measured points, then at most one inference hop.
First, write the feasibility budget to docs/science-superpowers/feasibility/YYYY-MM-DD-<topic>.md — before any probe runs:
# Feasibility: <topic>
**Feasibility question:** <the measurable threshold>
**Scientific question it gates:** <one sentence, deferred>
## Budget
| Resource | Per-run limit | Whole-sweep limit | On breach |
|----------|---------------|-------------------|-----------|
| Wall clock | <...> | <...> | record status=timeout, stop |
| Peak memory | <...> | — | record status=oom, stop |
| Iterations | <...> | — | record achieved residual, stop |
## Kill criteria
- <exact condition> -> <exact action>
- Consecutive failures before abandoning the ladder: <N>
## Not a stop criterion
- Sunk cost. Time already spent is not an input to any rule above.
- "We're close." Position in the case list is not fraction of work done.
## Abandonment condition
- <measured result that means this approach is not viable>
## Kill-criteria verification
- <criterion> -> fired deliberately at <when>, observed <what>
The budget is your partner's, not yours. Declaring an amendment in advance is necessary but not sufficient: granting yourself more compute because the sweep ran out is the one move that turns a budget into a formality. Refining how a limit is enforced is yours to fix. Enlarging the total envelope is your partner's call — ask, in one line, and record the answer.
Then prove every kill criterion fires. Construct a case that breaches each limit and confirm the limit produces a record, not a hang, not a crash, and not a silent success. This costs about a minute and is not optional.
Untested limits fail in ways that are invisible until they matter: a memory rlimit that is a no-op on the host OS, a timeout binary that does not exist on the platform so the job never starts, a killed run that records success with a residual computed from a zeroed solution vector. Each of those looks exactly like a working budget right up until it is load-bearing.
Then run three probes — three measured points, geometrically spaced, positioned as high as the budget allows. Cheap calibration runs to find where the ladder should sit do not count against the three; they are how you position it. If the work has several implementation paths (assembled vs matrix-free, direct vs iterative), each path that could decide feasibility gets its own three, because their ceilings differ by orders of magnitude.
- Fit on the largest rungs. Small rungs are pre-asymptotic and will bend the exponent; a fit that moves substantially when small points are dropped is not a fit.
- Recompute the convergence residual explicitly. Do not trust a solver's own success code — a stopping rule of the form
||r|| <= btol*||b|| + atol*||A||*||x||reports success at a residual floor that rises with problem size, so "converged" can mean an error two orders of magnitude off target. - Record the extrapolation factor: target size divided by largest measured size. Beyond about 10×, you do not have a feasibility answer, you have a hope. Push the ladder higher or say so plainly.
- Persist an artifact per probe. A case with no artifact is not a completed case, whatever the log says.
If a probe says infeasible, test the obvious mitigation before accepting it. The most expensive mistake available here is to accept a ceiling derived from one implementation path and build a study around it. If you can name a mitigation — a matrix-free operator, a different factorization, chunking, a preconditioner — measure it now, in this phase. Do not write it into a recommendations section and continue as if the ceiling were real. A ceiling that has not survived its own obvious mitigation is not a ceiling.
When a mitigation works, re-measure where the ceiling went. It usually moves rather than vanishes: relieve the assembler and the limit reappears in a boundary-condition matrix copy, relieve that and it reappears in the solve. Read the artifact to find which stage actually died — a run that recorded nnz before dying was killed after assembly, and attributing that ceiling to the assembler would send you optimizing the wrong stage.
A failing mitigation may be a broken test rather than a real limit. Before reporting that an approach does not work, confirm you invoked it correctly — a preconditioner that violates the solver's symmetry requirement produces divergence that looks like an unsuitable method. Reporting a bug in your test as a property of the pipeline is how a viable path gets abandoned.
Phase 3: One Exploratory Campaign
One campaign. Its purpose is to find out what the system does, not to establish anything.
- Run it inside the budget, with the verified kill criteria armed
- Span enough range to distinguish alternatives. A campaign's value is set by the width of the range it sweeps, not the number of configurations in it: many points packed into one decade cannot separate a power law from a gently curved alternative, while a handful spread over three decades can. Before running, ask what competing explanation this campaign could rule out — if the answer is none, widen the range rather than adding configurations
- Keep provenance: environment, seeds, code version, per-case artifacts
- Report everything under an Exploratory heading
- Anomalies still get root-caused — use
science-superpowers:investigating-anomalous-results. Feasibility mode relaxes sequencing, never honesty
Nothing from this campaign is confirmatory. Ever. Not the striking effect, not the clean scaling law, not the result that matches the hypothesis. These are leads for a pre-registration that has not been written yet.
Resist growing the campaign. "One more configuration" is how one campaign becomes the study you were supposed to pre-register.
Phase 4: The Exit Gate — Your Human Partner Decides
Feasibility mode does not graduate itself. When the campaign completes, report and stop:
"Feasibility settled: <measured result, with the extrapolation factor>. Exploratory campaign found <leads, labeled>. Nothing here is confirmatory. Options: (a) pre-register a confirmatory study now that we know what is runnable, (b) another exploratory campaign, (c) abandon this approach — the abandonment condition <did/did not> trigger. Which?"
Only your human partner chooses. Do not begin surveying, designing, or pre-registering on your own initiative because the campaign looked promising. The transition out of this mode is theirs, exactly as the transition in was.
On exit to confirmatory work, the standard sequence resumes at full strength: science-superpowers:surveying-prior-work, then science-superpowers:designing-the-analysis, then science-superpowers:preregistering-analysis. The feasibility measurements are legitimate inputs to that design — that is what they were for. The exploratory results are not evidence; they are hypotheses. Say so in the pre-registration: record that the design was informed by exploratory work, and which parts.
What Stays On, What Defers
| Discipline | In feasibility mode |
|---|---|
| Honest labeling of exploratory vs confirmatory | On — the whole mode is labeled exploratory |
| Root-causing anomalies | On — never tune past a surprising number |
| Pinned environment, recorded seeds, provenance | On — an unreproducible probe is not a measurement |
| Verifying a claim before making it | On |
| Short framing of the feasibility question | On — a few sentences |
| Prior-work survey | Deferred to exit |
| Detailed analysis plan | Deferred to exit |
| Pre-registration and freezing | Deferred to exit — deferred, not cancelled |
| Red-team review, archival reporting | Deferred to exit |
| Confirmatory claims of any kind | Forbidden until exit |
Common Rationalizations
| Excuse | Reality |
|---|---|
| "This feels exploratory, so I'm in feasibility mode" | Only your human partner opts in. Feeling exploratory is not opt-in. |
| "The extrapolation is solid, no need to run it" | An estimate that gates work must be measured. Ceilings are wrong by factors of 50. |
| "I'll write the plan while the probes run" | A plan written against an unmeasured ceiling is rework. Wait for the number. |
| "The budget is written, the limits are set" | Written is not fired. Prove each one kills something. |
| "The solver reported convergence" | Solver success codes encode their own stopping rule, not yours. Recompute the residual. |
| "It's infeasible, so let's redesign around that" | Test the mitigation you just named. Then it is a ceiling. |
| "The mitigation worked, that ceiling is gone" | Ceilings move rather than vanish. Re-measure which stage dies now. |
| "That approach diverged, it's unsuitable" | Confirm you invoked it correctly first. A broken test is not a property of the method. |
| "The sweep needs a bit more compute, I'll amend the budget" | Enforcement fixes are yours; a larger envelope is your partner's call. Ask. |
| "It converged cleanly with a sensible order, so it's right" | Wrong quantities converge tidily too. Check against an independent reference. |
| "More configurations will make the campaign conclusive" | Range separates hypotheses; count does not. Widen instead. |
| "This exploratory result is too clean to waste" | It is a lead. Pre-register it and test it on runs you have not used. |
| "We're 14 of 20 cases in, let's finish" | Position in a list is not fraction of work. The remaining cases may be most of the compute. |
| "One more configuration in the campaign" | That is the study. Exit the mode and pre-register it. |
| "The campaign went well, I'll start the survey" | The exit gate is your partner's. Report and stop. |
Red Flags — STOP
- About to write a plan, criterion, or prediction for a configuration that has never run
- A number that gates downstream work came from extrapolation, not execution
- A kill criterion exists in a document but has never been observed firing
- A named mitigation sits in a recommendations section, untested, while work proceeds around it
- Every probe converges in suspiciously few iterations
- A clean, converged, plausibly-ordered number that has never been checked against an independent reference
- Granting yourself additional compute after the declared budget is exhausted
- Attributing a ceiling to a stage without reading the artifact to see which stage actually died
- Cases marked complete with no persisted artifact
- Extrapolating more than ~10× beyond the largest measured point
- Reaching for the words "confirmatory", "significant", or "we found" before the exit gate
- Entering or leaving this mode without your human partner saying so
The Bottom Line
Can it run? -> measure it, at the top of the ladder, before building on the answer
Is it bounded? -> fire the bound and watch it catch
Is it a finding? -> not in this mode. It is a lead.
Handoff
Staying in feasibility mode: science-superpowers:investigating-anomalous-results for surprising probe results; science-superpowers:setting-up-reproducible-analysis if the workspace is not yet pinned and seeded.
On exit, only when your human partner says so: science-superpowers:surveying-prior-work -> science-superpowers:designing-the-analysis -> science-superpowers:preregistering-analysis.
Executing the Analysis
Overview
Load the pre-registered plan, review it critically, execute all steps exactly as registered, report when complete.
Announce at start: "I'm using the executing-analysis skill to run this analysis."
Note: Tell your human partner that Science Superpowers works much better with access to subagents — the protocol-compliance and rigor reviews catch more when run by fresh agents. If subagents are available, use science-superpowers:subagent-driven-analysis instead.
Prerequisite
The plan MUST be pre-registered and frozen (science-superpowers:preregistering-analysis) and the workspace set up (science-superpowers:setting-up-reproducible-analysis). If either is missing, stop and do it first. Executing before freezing turns the analysis exploratory.
The Process
Step 1: Load and Review
- Read the analysis plan and the frozen pre-registration.
- Review critically — any concern about the design, the decision rule, or feasibility?
- If concerns: raise them with your human partner before starting. A change to the registered analysis is a documented deviation and renders that part exploratory.
- If no concerns: create a TodoWrite with all steps and proceed.
Step 2: Execute Steps
For each step:
- Mark in_progress.
- For nontrivial estimators: validate the pipeline on known/simulated data first.
- Run the step exactly as registered. Keep the seed. Read immutable raw / named derived; write a new artifact.
- Validate the output (shape, ranges, counts) — nothing changes silently.
- Apply the registered decision rule as written; do not re-decide it after seeing the number.
- Use
science-superpowers:verifying-results-before-claimingbefore marking done. - Commit. Mark completed.
Checkpoints
After each natural group of steps (e.g., data prep, then primary model), pause and report results to your human partner before continuing. Keep confirmatory and exploratory results clearly separated in every report.
Step 3: Complete
After all steps complete and verified:
- Use
science-superpowers:requesting-red-team-reviewon the whole result. - Then
science-superpowers:reporting-and-archiving-findings.
When to Stop and Ask
STOP immediately when:
- A result is surprising, impossible, or contradicts a sanity check → use
science-superpowers:investigating-anomalous-results(do NOT quietly drop data or tweak parameters) - The plan requires deviating from the pre-registration to proceed
- A validation fails repeatedly
- An instruction is unclear
Ask rather than guess. A guess that changes the analysis silently destroys the result's credibility.
Remember
- Pre-registration frozen before execution
- Validate pipelines on known data
- Run steps exactly as registered; never silently deviate
- Label anything unregistered as exploratory
- Don't skip validation or verification
- Stop on anomalies; investigate by root cause
Integration
Required workflow skills:
- science-superpowers:setting-up-reproducible-analysis — run first
- science-superpowers:preregistering-analysis — frozen before execution
- science-superpowers:investigating-anomalous-results — on surprising results
- science-superpowers:verifying-results-before-claiming — before any completion claim
- science-superpowers:reporting-and-archiving-findings — after all steps complete
Framing Research Questions Into Investigations
Help turn a fuzzy research interest into a precise, falsifiable question with explicit hypotheses, the data required, and what would count as an answer — through natural collaborative dialogue.
Start by understanding the context (the data on hand, the domain, what your human partner already knows), then ask questions one at a time to sharpen the interest. Once you understand what is being investigated, present the framing and get approval.
Do NOT load the dataset, compute any statistic, fit any model, plot any outcome, or invoke any execution skill until you have presented a research framing and your human partner has approved it. This applies to EVERY investigation regardless of perceived simplicity.Why this gate is strict for science specifically: Looking at outcomes before the question and predictions are fixed contaminates a confirmatory analysis. Once you have seen the data, you cannot un-see it, and every later choice (which test, which subgroup, which cutoff) becomes suspect. Framing first is what keeps a result confirmatory rather than a story told after the fact.
What this gate does NOT forbid: measuring whether the computation can run — assembly size, peak memory, wall clock, iteration counts, convergence behavior. Those are provenance and structure, not outcomes. Resource measurement never contaminates a confirmatory analysis, because the outcome variable is untouched.
When Feasibility Is Unknown
If it is not yet known whether the computation can run at all — an unbenchmarked solver, an unproven pipeline, a configuration whose memory or wall-clock ceiling is estimated rather than measured — say so during framing and offer feasibility mode:
"Whether this can run at scale is unknown, and the design depends on the answer. I can establish feasibility first — a minimal runner, three scaling probes, one exploratory campaign — and defer the survey, plan, and pre-registration until we know what is runnable. Nothing produced that way can support a confirmatory claim. Or we proceed with the standard sequence. Which?"
Only your human partner can choose it. If they do, use science-superpowers:establishing-feasibility-first after a short framing. If they decline or say nothing, continue the standard sequence below.
Separate the two questions in the framing either way: the feasibility question (a measurable threshold) and the scientific question it gates. Tangling them produces predictions written over runs that may not be possible.
Anti-Pattern: "This Is Too Simple To Need Framing"
Every investigation goes through this process. A single t-test, a quick correlation, a "just look at the trend" — all of them. "Simple" questions are where unexamined assumptions and undeclared researcher degrees of freedom do the most damage. The framing can be short (a few sentences for a truly simple question), but you MUST present it and get approval.
Checklist
You MUST create a task for each of these items and complete them in order:
- Explore context — what data exists, its shape and provenance, the domain, recent related work in the repo
- Ask clarifying questions — one at a time; understand the phenomenon, the population, the unit of analysis, and what decision the answer informs
- Propose 2-3 framings — different ways to make the question precise, with trade-offs and your recommendation
- Present the framing — question, hypotheses, data needed, success/answer criteria, scope; get approval after each section
- Write the question document — save to
docs/science-superpowers/questions/YYYY-MM-DD-<topic>.mdand commit - Self-review — quick inline check for vagueness, unfalsifiable claims, undefined measures, scope creep (see below)
- Your human partner reviews the written framing — ask them to review the file before proceeding
- Transition — invoke surveying-prior-work to ground methods, then designing-the-analysis
Process Flow
digraph framing {
"Explore context" [shape=box];
"Ask clarifying questions" [shape=box];
"Propose 2-3 framings" [shape=box];
"Present framing sections" [shape=box];
"Partner approves framing?" [shape=diamond];
"Write question doc" [shape=box];
"Self-review (fix inline)" [shape=box];
"Partner reviews doc?" [shape=diamond];
"Invoke surveying-prior-work" [shape=doublecircle];
"Explore context" -> "Ask clarifying questions";
"Ask clarifying questions" -> "Propose 2-3 framings";
"Propose 2-3 framings" -> "Present framing sections";
"Present framing sections" -> "Partner approves framing?";
"Partner approves framing?" -> "Present framing sections" [label="no, revise"];
"Partner approves framing?" -> "Write question doc" [label="yes"];
"Write question doc" -> "Self-review (fix inline)";
"Self-review (fix inline)" -> "Partner reviews doc?";
"Partner reviews doc?" -> "Write question doc" [label="changes requested"];
"Partner reviews doc?" -> "Invoke surveying-prior-work" [label="approved"];
}
The terminal state is invoking surveying-prior-work (then designing-the-analysis). Do NOT jump to loading data or fitting models. The ONLY skills you invoke after framing are surveying-prior-work and designing-the-analysis — or, if and only if your human partner opted into feasibility mode, establishing-feasibility-first.
The Process
Understanding the interest:
- Inspect the context first: what data is available, its schema, size, and where it came from. You may look at structure and provenance (column names, row counts, collection method) — but do NOT look at outcome distributions or relationships you might later test.
- Before asking detailed questions, assess scope: if the request bundles several independent questions ("does the treatment work, and what drives churn, and can we forecast revenue"), flag it immediately. Decompose into separate investigations, each with its own question → design → pre-registration cycle.
- For appropriately-scoped interests, ask questions one at a time.
- Prefer multiple-choice questions when possible; open-ended is fine too.
- Only one question per message.
- Focus on: the phenomenon, the population/sample, the unit of analysis, the comparison, and what decision or understanding the answer serves.
A good research question is:
- Specific — names the variables, the population, and the relationship being examined
- Falsifiable — there is an outcome that would prove it wrong
- Operationalized — every construct maps to something measurable in the data ("engagement" → what column, computed how)
- Scoped — answerable with one coherent analysis, not five
Exploring framings:
- Propose 2-3 ways to make the question precise (e.g., association vs. causal effect; continuous outcome vs. threshold; population-level vs. subgroup).
- Lead with your recommended framing and explain the trade-offs (what each can and cannot conclude, what data each needs).
Presenting the framing:
- Once you understand the question, present it in sections scaled to complexity.
- Cover: the question, the hypotheses (including the null and a directional prediction if appropriate), the data and key variables, what would count as a confirmatory answer, and explicit scope/exclusions.
- Ask after each section whether it looks right.
After the Framing
Documentation — write the approved framing to docs/science-superpowers/questions/YYYY-MM-DD-<topic>.md:
# <Question title>
**Research question:** <one precise, falsifiable sentence>
**Background / motivation:** <why this matters, what decision it informs>
**Hypotheses:**
- H0 (null): <...>
- H1 (alternative, directional if justified): <...>
**Population & unit of analysis:** <who/what, the sample, the unit>
**Key variables (operationalized):**
- Outcome: <construct> → <measure / column / computation>
- Predictor(s) / exposure: <...> → <...>
- Covariates / potential confounders: <...>
**What counts as an answer:** <the confirmatory criterion, stated qualitatively here; exact decision rules come later in pre-registration>
**Scope & exclusions:** <what is explicitly out of scope>
**Open questions for prior-work survey:** <methods to check, known confounds to look up>
Commit the document to git.
Self-Review — look at the document with fresh eyes:
- Vagueness scan: Any construct not mapped to a measure? Any "better", "improves", "related to" without specifying direction or magnitude? Fix them.
- Falsifiability: Is there an outcome that would disconfirm H1? If not, reframe.
- Scope check: Is this one investigation, or several wearing a trench coat? Decompose if needed.
- Unit/population clarity: Is the unit of analysis unambiguous (person? session? cluster?)? Fix if not.
Fix issues inline. No need to re-review — just fix and move on.
Partner Review Gate — after the self-review passes:
"Framing written and committed to
<path>. Please review it and let me know if you want changes before we survey prior work and design the analysis."
Wait for the response. If they request changes, make them and re-run the self-review. Only proceed once they approve.
Transition:
- Invoke surveying-prior-work to ground the question and methods in existing knowledge.
- Then designing-the-analysis. Do NOT invoke any other skill or touch outcomes yet.
Key Principles
- One question at a time — don't overwhelm
- Falsifiable or it's not a question — there must be an outcome that proves it wrong
- Operationalize everything — constructs must map to measures
- YAGNI ruthlessly — cut sub-questions that don't serve the decision
- Don't peek — structure and provenance are fair game; outcomes are not, until pre-registration is done
Investigating Anomalous Results
Overview
Random tweaks waste time and manufacture false findings. Quietly dropping the inconvenient data point, nudging the cutoff, or re-running until it "works" doesn't fix the problem — it fabricates a result.
Core principle: ALWAYS find the root cause before adjusting anything. An adjustment made before you understand the cause is, at best, noise and, at worst, fraud.
Violating the letter of this process is violating the spirit of it.
The first question for any anomaly: is this a code bug, a data issue, or a real finding? You cannot answer by guessing, and you must not "fix" it until you know — because a real finding is not something to fix.
The Iron Law
NO ADJUSTMENTS WITHOUT ROOT-CAUSE INVESTIGATION FIRST
No dropping data, changing a test, transforming a variable, re-running with new parameters, or "cleaning" until Phase 1 is complete. If you haven't found the cause, you cannot justify the adjustment.
When to Use
- A result is surprising, too good, or contradicts what's known
- An impossible value (negative count, probability > 1, age of 500)
- A pipeline errors or a model won't converge
- A result contradicts a sanity check or a simple cross-tab
- A replication or robustness check disagrees with the primary result
- Performance that's suspiciously high (often leakage)
Use this ESPECIALLY when:
- The anomaly is convenient (it supports your hypothesis) — that's when the urge to skip investigation is strongest
- Under time pressure
- You've already tried one quick adjustment
The Four Phases
Complete each before the next.
Phase 1: Characterize the Anomaly
Before any adjustment:
- Read the output/error carefully. The exact value, the exact message, the stack trace. Note shapes, counts, and where in the pipeline it appears.
- Reproduce it deterministically. Fixed seed, same data version. Does it happen every time? If not reproducible, gather more data — don't guess.
- Check recent changes. What changed in the code, the data version, or the environment since it last looked right?
git diff, data provenance, package versions. - Gather evidence at each pipeline stage. Instrument raw → cleaned → derived → result. Print shapes, ranges, null counts, and key summaries at each boundary. This reveals where the anomaly enters.
raw: N=10342, outcome mean=0.31, nulls=0
cleaned: N=10298, outcome mean=0.31, nulls=0 <- 44 dropped, expected
derived: N=10298, feature mean=4e7 <- WRONG: unit blew up here
result: coefficient enormous <- symptom; cause is upstream
- Classify the cause with the evidence: code bug, data issue, or real finding. Do not proceed assuming one without evidence.
Phase 2: Pattern Analysis
- Find a working comparison. A subset, variable, site, or time period that behaves as expected. What's different about the anomalous one?
- Compare against a reference. A known correct result, a textbook expectation, a prior estimate from the survey. Read the reference method completely — don't skim.
- List every difference between working and anomalous, however small. Don't assume "that can't matter."
Phase 3: Hypothesis and Test
- Form a single hypothesis: "The anomaly is caused by X because Y." Write it down. Be specific.
- Test minimally on the smallest slice that exhibits it. One variable at a time.
- Verify before continuing. Confirmed → Phase 4. Not confirmed → new hypothesis. Don't stack adjustments.
- When you don't know, say so. "I don't understand why the derived feature inflates" — then investigate or ask. Don't pretend.
Phase 4: Resolution
Resolve according to the cause you established:
- Code bug: Write a minimal failing check that reproduces it, fix the code at the source (not a downstream patch), re-run the affected steps, confirm the anomaly is gone and other results unchanged.
- Data issue: Handle it per a documented rule. If the rule was not pre-registered, the decision is a deviation — record it, and the affected analysis becomes exploratory. Dropping observations is legitimate ONLY when root cause shows they are invalid measurements (e.g., a sensor error), and that justification is documented. Never drop data because it's inconvenient.
- Real finding: It is a result, not a defect. Do NOT "fix" it. If it was not what you pre-registered, it is an exploratory finding — report it as such, and pre-register a fresh confirmatory test (ideally on new/held-out data) before claiming it.
If 3+ adjustments fail: STOP. The pipeline or the design may be wrong — repeated failures that each reveal a new problem indicate a structural issue, not a series of small bugs. Discuss the design with your human partner before another attempt. Re-opening the design or pre-registration must be documented as a deviation.
Red Flags - STOP and Investigate
- "Let me just drop that outlier and re-run"
- "I'll nudge the threshold until it's significant"
- "Probably a fluke, let me re-run with a different seed" (changing the seed to change the result)
- "The result I want appeared, no need to dig into why the first run didn't"
- "Quick adjustment now, understand it later"
- Proposing an adjustment before tracing where the anomaly enters the pipeline
- An adjustment that happens to make the result support your hypothesis
All of these mean: STOP. Return to Phase 1.
Your Human Partner's Signals You're Doing It Wrong
- "Is that even possible?" — you reported a value without sanity-checking it
- "Why did that point get dropped?" — you adjusted without documenting the cause
- "Did you check leakage?" — suspiciously good performance you didn't investigate
- "That seems too clean" — question the pipeline, not just the result
Common Rationalizations
| Excuse | Reality |
|---|---|
| "It's clearly an outlier, just remove it" | "Clearly" is a guess. Find why it's extreme; document if you remove it. |
| "Re-running with a new seed fixed it" | You changed the result by chance, not the cause. Investigate. |
| "The good result is the right one" | Convenient results need MORE scrutiny, not less. |
| "Emergency, no time to investigate" | Investigation is faster than retracting a wrong finding. |
| "Tweak the model, then understand it" | The first tweak sets a false trail. Understand first. |
| "Performance is just high, ship it" | Suspiciously high performance is usually leakage. Check. |
Quick Reference
| Phase | Activities | Success Criteria |
|---|---|---|
| 1. Characterize | Read, reproduce, check changes, instrument stages, classify | Know WHERE and WHICH (bug/data/finding) |
| 2. Pattern | Working comparison, reference, list differences | Identify the difference |
| 3. Hypothesis | Single theory, minimal test | Confirmed or new hypothesis |
| 4. Resolution | Fix bug / document data rule / report real finding | Anomaly explained, not just hidden |
Related Skills
- science-superpowers:verifying-results-before-claiming — confirm the resolution actually holds
- science-superpowers:preregistering-analysis — a real but unregistered finding needs a fresh confirmatory pre-registration
- science-superpowers:setting-up-reproducible-analysis — deterministic reproduction depends on fixed seeds and pinned environments
Pre-Registering the Analysis
Overview
Write down the hypotheses, the exact analysis, the directional predictions, and the decision rules — and freeze them — before you look at the outcomes. Then run exactly that.
Core principle: If you decided the analysis after seeing the data, you cannot tell whether the data shaped the answer. A prediction is only evidence if it was made before the result.
Violating the letter of this rule is violating the spirit of this rule.
This is the pre-registration discipline — the science analog of test-driven development. In TDD you write the test before the code so you know the test tests something. Here you write the prediction before the result so you know the result confirms something.
The Iron Law
NO CONFIRMATORY CLAIM WITHOUT A PRE-REGISTERED PREDICTION FIRST
Looked at the outcomes before locking the analysis? That analysis is now exploratory. You cannot relabel it confirmatory. To make a confirmatory claim, pre-register and test on data you have not yet used.
No exceptions:
- Don't "lightly" reorder the analysis after a peek
- Don't add the covariate that "obviously" should be there once you saw the result
- Don't move the cutoff to where it "works"
- Don't keep an unregistered analysis and call it confirmatory because "it would have been my prediction anyway"
Looked? It's exploratory. Period.
When to Use
Always, before:
- Any hypothesis test you intend to report as confirmatory
- Any p-value, confidence interval, or effect estimate that will support a claim
- Loading or plotting the outcome variable in a way that reveals the relationship you'll test
Exploratory work is allowed and valuable — but it must be labeled exploratory and never dressed up as confirmation. Exploratory findings are hypotheses for the next pre-registration.
Exceptions (decide with your human partner):
- Purely descriptive reporting with no inferential claim
- Pure method development on simulated data
- Re-analysis of already-public results
- Feasibility mode, if your human partner opted into it (
science-superpowers:establishing-feasibility-first) — registration is deferred to the exit gate, not cancelled, and nothing produced before it may be claimed as confirmatory
Don't Register Over Runs That Have Never Happened
A registration whose configurations have never executed is not protection — it is rework waiting to happen. Before freezing, confirm:
- Every configuration in the plan has run at least once, or is known to be runnable from a measured point (not an extrapolation)
- Every acceptance criterion has been observed on a real run of the pipeline it judges. Writing "error decreases monotonically" for a pipeline you have never executed is writing a pass/fail rule for behavior you have not seen
- The cheapest run that could falsify your prediction has been executed
That last one matters most. If a run costing seconds could disconfirm the prediction you are about to freeze, run it first. A validation benchmark on different data or a different geometry cannot contaminate the confirmatory test — it is a separate measurement — so there is no rigor argument for freezing before it. Freezing a prediction that a cheap run would have falsified costs a re-registration and buys nothing.
Where feasibility is genuinely unknown, that is the signal to offer feasibility mode rather than freeze early.
Thinking "I'll just check the result first, then pre-register"? Stop. That's the rationalization this skill exists to stop.
Lock — Execute — Separate
digraph prereg_cycle {
rankdir=LR;
lock [label="LOCK\nhypotheses, analysis,\npredictions, decision rules", shape=box, style=filled, fillcolor="#ffcccc"];
falsifiable [label="Disconfirming\nresult specified?", shape=diamond];
freeze [label="FREEZE\ncommit + timestamp", shape=box];
execute [label="EXECUTE\nexactly as registered", shape=box, style=filled, fillcolor="#ccffcc"];
separate [label="SEPARATE\nlabel anything else\nexploratory", shape=box, style=filled, fillcolor="#ccccff"];
lock -> falsifiable;
falsifiable -> freeze [label="yes"];
falsifiable -> lock [label="no - unfalsifiable,\nreframe"];
freeze -> execute;
execute -> separate;
}
LOCK — Write the prediction before the result
Specify, in writing:
- Hypotheses: H0 and a directional H1 where justified
- The primary analysis: the one model/test, the exact variables, transformations, covariates, exclusions
- The prediction: what you expect, with direction (and magnitude if the survey gave you one)
- The decision rule: the exact result that confirms vs. disconfirms — e.g., "confirm H1 if the coefficient on
exposureis positive with p < .005 in the pre-specified model" - Stopping rule: the sample size is fixed in advance; you will not peek-and-extend
- Multiplicity plan: how many tests, and the correction (so you don't harvest one "significant" result from twenty)
- Secondary/exploratory analyses: listed and labeled as such
Verify it is falsifiable — the "watch it fail" step
A registration is only real if some possible result would disconfirm the prediction. State that result explicitly. If no result could disconfirm it, the prediction is unfalsifiable — reframe before freezing. This mirrors verifying a test fails for the right reason before trusting it.
FREEZE — Make the timestamp real
Write the pre-registration to docs/science-superpowers/preregistrations/YYYY-MM-DD-<topic>.md, then freeze it with prereg.sh (it ships alongside this skill's SKILL.md) — before touching outcomes:
# <skill-dir> = this skill's directory, e.g. $CLAUDE_PLUGIN_ROOT/skills/preregistering-analysis
<skill-dir>/prereg.sh freeze \
docs/science-superpowers/preregistrations/YYYY-MM-DD-<topic>.md \
data/raw/<raw-data-file>...
The freeze commit is your timestamp: it proves the prediction preceded the result. Passing the raw data files records their checksums, so post-freeze data edits are detectable. If the script is unavailable, commit the registration alone — its first-ever commit, message prereg-freeze: <path> — and never touch the file again; audit derives everything from git history.
A freeze is a claim the repo must be able to prove. prereg.sh audit verifies from git history that the registration was never edited after its first commit, that no output was committed at-or-before the freeze, and that frozen data checksums still match. A registration commit that merely precedes the results proves nothing if the file changed afterward — "frozen" means unchanged since, and only the audit shows that.
EXECUTE — Run exactly what you registered
Run the pre-registered analysis unchanged. Resist every "while I'm here" addition. The discipline of science-superpowers:designing-the-analysis and the validation-on-known-data steps apply here.
SEPARATE — Confirmatory vs. exploratory, always distinguishable
Anything you do that was not registered — a follow-up, a subgroup, a different model — is exploratory. Report it under an "Exploratory" heading. Exploratory results are leads, not conclusions.
Pre-Registration Template
# Pre-registration: <topic>
**Frozen at commit:** <stamped by prereg.sh freeze>
**Question doc:** docs/science-superpowers/questions/<...>
**Analysis plan:** docs/science-superpowers/plans/<...>
## Hypotheses
- H0: <...>
- H1 (directional): <...>
## Primary analysis (exact)
- Model/test: <e.g., OLS: outcome ~ exposure + age + site>
- Variables & operationalizations: <...>
- Transformations: <...>
- Inclusion/exclusion criteria: <fixed now>
- Covariates: <fixed now>
## Prediction
- Direction: <sign>
- Expected magnitude (from prior work): <effect size + source, if available>
## Decision rule
- Confirm H1 if: <exact criterion>
- Disconfirm / null if: <exact criterion>
## Sample size & stopping
- N (fixed): <...> ; Power: <...> at alpha <...> for effect <...>
- No optional stopping. No peeking-and-extending.
## Multiplicity
- Number of confirmatory tests: <...>
- Correction: <e.g., Bonferroni / pre-specified primary only>
## Secondary & exploratory (labeled)
- <listed; reported separately, never as confirmatory>
## Planned deviations handling
- Any deviation will be documented in the report and renders the affected analysis exploratory.
Why Order Matters
"I'll just look at the data first, then write the prediction"
A prediction written after seeing the result always "fits." That is HARKing — Hypothesizing After Results are Known. It feels like insight; it is circular. The data cannot test a hypothesis it generated.
"I'll try a few specifications and report the one that works"
That is p-hacking / the garden of forking paths. With enough analytic choices, a "significant" result is almost guaranteed by chance. Pre-specifying the one analysis removes the forking.
"I'll keep collecting until it's significant"
Optional stopping inflates false positives dramatically. Fix N in advance, or use a pre-specified sequential design with proper corrections.
"Re-deriving the analysis with TDD-style discipline is wasteful, I already saw the answer"
Sunk cost. The peeked analysis is exploratory — that's not waste, it's a lead. The cost of relabeling it "confirmatory" is a result you cannot trust and a finding that won't replicate.
"Pre-registration is dogmatic; pragmatism means adapting to what the data shows"
Pre-registration IS pragmatic: it is the difference between a finding that replicates and one that evaporates. Adapting to what the data shows is exactly the bias it prevents. Adapt in the next study, with a new pre-registration.
Common Rationalizations
| Excuse | Reality |
|---|---|
| "Too simple to pre-register" | Simple analyses have forking paths too. Locking takes minutes. |
| "I'll register after a quick look" | A look makes it exploratory. The look is the violation. |
| "This covariate obviously belongs" | Add it to the registration before seeing the result, or it's a researcher degree of freedom. |
| "The cutoff that works is the right one" | Choosing a cutoff by its result is p-hacking. Fix it first. |
| "It would have been my prediction anyway" | Then writing it down first cost nothing. You didn't, so it's exploratory. |
| "Just one more specification" | One more fork. Pre-specify the primary; label the rest exploratory. |
| "Deleting the peeked analysis is wasteful" | Don't delete it — relabel it exploratory. Don't promote it to confirmatory. |
| "Stopping early since it's already significant" | Optional stopping inflates false positives. Honor the fixed N. |
| "I'll freeze now and find out later if it runs" | A registration over unrunnable configurations is rework. Measure feasibility first. |
| "Running the benchmark first would contaminate my prediction" | A validation benchmark is not the confirmatory test. Run it; an informed prediction beats a falsified one. |
Red Flags - STOP and Re-frame
- Looking at the outcome distribution / the relationship before the registration is frozen
- Choosing the test, covariates, exclusions, or cutoff after seeing results
- Adding analyses because the first wasn't significant
- Extending the sample because p was "almost there"
- Reporting an unregistered result as confirmatory
- "It's about the spirit, this peek doesn't really count"
- "This is different because..."
All of these mean: that analysis is exploratory. Pre-register a fresh confirmatory test on unused data.
Statistical Fallacies Reference
When specifying the analysis and decision rules, read @statistical-fallacies.md to avoid the traps pre-registration is designed to prevent: p-hacking, HARKing, the garden of forking paths, optional stopping, multiple comparisons, data leakage, overfitting, and p-value misinterpretation.
Verification Checklist
Before executing:
- Hypotheses written, with a directional H1 where justified
- The primary analysis fully specified (model, variables, transforms, exclusions, covariates)
- A disconfirming result is stated explicitly (it is falsifiable)
- The decision rule is exact and result-independent
- Sample size fixed; no optional stopping
- Multiplicity handled
- Exploratory analyses listed and labeled
- Pre-registration frozen with
prereg.sh freeze(or an equivalent lone first commit) BEFORE any outcome was observed -
prereg.sh auditpasses — and it is re-run before any confirmatory claim is reported
Can't check all boxes? You're not ready for a confirmatory claim.
The Bottom Line
Confirmatory claim → prediction was registered and frozen first
Otherwise → it's exploratory, label it so
No exceptions without your human partner's permission.
Handoff
Once frozen, set up the workspace and execute: use science-superpowers:subagent-driven-analysis (recommended) or science-superpowers:executing-analysis. Both invoke science-superpowers:setting-up-reproducible-analysis first.
Receiving Critical Review
Overview
Critique of an analysis requires technical evaluation, not emotional performance.
Core principle: Verify before changing. Ask before assuming. Methodological correctness over social comfort.
A reviewer pointing at a confound is doing you a favor only if you check whether the confound is real. Reflexively agreeing — or reflexively defending — both skip the verification that makes review worthwhile.
The Response Pattern
WHEN receiving review feedback:
1. READ: The complete feedback without reacting
2. UNDERSTAND: Restate the methodological concern in your own words (or ask)
3. VERIFY: Check it against the data, code, and pre-registration
4. EVALUATE: Is it correct FOR THIS analysis?
5. RESPOND: Technical acknowledgment or reasoned pushback
6. ACT: One item at a time, re-running and re-verifying each
Forbidden Responses
NEVER:
- "You're absolutely right!" (before you've checked)
- "Great catch!" / "Excellent point!" (performative)
- "Let me fix that now" (before verification)
INSTEAD:
- Restate the methodological concern
- Ask a clarifying question
- Push back with reasoning if it's wrong for this analysis
- Or just verify and report what you found
Handling Unclear Feedback
IF any item is unclear:
STOP - do not change anything yet
ASK for clarification
WHY: Methodological items interact. Adjusting for the wrong confound
because you misread the concern can bias the result further.
Source-Specific Handling
From your human partner
- Trusted — verify, then act
- Still ask if scope is unclear
- No performative agreement; skip to verification or a technical acknowledgment
From an external or red-team reviewer
BEFORE changing anything:
1. Is it methodologically correct for THIS data and design?
2. Would the change itself introduce bias (e.g., adding a collider as a covariate)?
3. Is there a pre-registered reason the analysis is the way it is?
4. Does the reviewer have the full context (the pre-registration, the data structure)?
IF the suggestion seems wrong:
Push back with technical reasoning and evidence
IF you can't verify:
Say so: "I can't verify this without [X]. Should I [investigate/ask]?"
IF it conflicts with the pre-registration:
A change to the registered analysis is a deviation. It renders that
analysis exploratory. Flag this explicitly before making the change.
The Pre-Registration Guard
A reviewer may suggest a "better" analysis. Before adopting it:
- If it changes the pre-registered confirmatory analysis, you are moving from confirmatory to exploratory. That may be the right call — but say so, and report it accordingly.
- Don't let good-faith "improve it" feedback silently convert a confirmatory result into a post-hoc one.
When To Push Back
- The suggestion would introduce bias (conditioning on a collider, adjusting away a mediator you care about, leakage)
- It treats observational data as if it supported causal claims, or vice versa
- The reviewer lacks context the pre-registration provides
- It contradicts your human partner's design decisions (raise with them)
How: technical reasoning, not defensiveness. Show the diagnostic, the DAG, the pre-registration line, or the reproduced number.
Acknowledging Correct Feedback
When feedback IS correct:
✅ "Verified — site does confound this. Added it; estimate drops to 0.09 [−0.01, 0.19]. Now inconclusive."
✅ "Confirmed leakage: the scaler was fit before the split. Refit on train only; AUC falls to 0.71."
✅ [Just fix it, re-run, and show the new number]
❌ "You're absolutely right!"
❌ "Great catch!"
❌ "Thanks for spotting that!"
❌ ANY gratitude or praise performance
Why no thanks: the corrected, re-verified result shows you heard the feedback. State the fix and the new evidence.
Gracefully Correcting Your Pushback
If you pushed back and were wrong:
✅ "You were right — I checked and the assumption is violated. Re-running with the robust estimator."
❌ Long apology or defense of why you pushed back
Common Mistakes
| Mistake | Fix |
|---|---|
| Performative agreement | State the concern or verify |
| Blind change | Check against data/code/prereg first |
| Batch edits without re-running | One at a time, re-verify each |
| Assuming the reviewer is right | Check whether the change biases the result |
| Silently changing the registered analysis | Flag it as a deviation → exploratory |
| Avoiding pushback | Methodological correctness > comfort |
The Bottom Line
External feedback = hypotheses to verify, not orders to follow.
Verify against the data and the pre-registration. Question. Then act — and re-verify with science-superpowers:verifying-results-before-claiming.
Reporting and Archiving Findings
Overview
Complete an investigation by confirming reproducibility, presenting clear options, handling the chosen one, and archiving everything needed to reproduce the result.
Core principle: Confirm reproducibility -> separate confirmatory from exploratory -> present options -> execute choice -> archive code + data + environment + pre-registration.
Announce at start: "I'm using the reporting-and-archiving-findings skill to complete this work."
Step 1: Verify Reproducibility
Before reporting anything, confirm the whole analysis reproduces from immutable raw data with the fixed seed in the pinned environment.
# From a clean state: re-run the pipeline end to end
# Confirm headline numbers match what you intend to report
Use science-superpowers:verifying-results-before-claiming. If it doesn't reproduce, stop — fix reproducibility (possibly via science-superpowers:investigating-anomalous-results) before reporting. Don't report a number you can't regenerate.
For any confirmatory claim, also run the pre-registration audit and keep its output for the report:
# ships with science-superpowers:preregistering-analysis
<skills root>/preregistering-analysis/prereg.sh audit
If the audit fails, the confirmatory label is indefensible — the registration changed after its freeze, or outputs predate it. Relabel the affected analyses exploratory (or pre-register a fresh test on unused data) before reporting.
Step 2: Detect Environment
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
GIT_DIR == GIT_COMMON: normal repo, no worktree cleanup neededGIT_DIR != GIT_COMMON, named branch: worktree, provenance-based cleanup (Step 5)- Detached HEAD: reduced menu (no local merge), no cleanup (externally managed)
Step 3: Determine Base Branch
git merge-base HEAD main 2>/dev/null || git merge-base HEAD master 2>/dev/null
Or ask: "This branch split from main — correct?"
Step 4: Present Options
Normal repo / named-branch worktree — present exactly these 4 options:
Analysis complete and reproducible. What would you like to do?
1. Merge the analysis back to <base-branch> locally
2. Write up and share (report / preprint / pull request)
3. Keep the branch as-is (I'll handle it later)
4. Discard this work
Which option?
Detached HEAD — present these 3 (no local merge):
Analysis complete and reproducible (externally managed workspace).
1. Push as a new branch and open a pull request / share
2. Keep as-is
3. Discard this work
Which option?
Don't add explanation — keep options concise.
Step 5: Execute Choice
Option 1: Merge Locally
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
cd "$MAIN_ROOT"
git checkout <base-branch> && git pull && git merge <feature-branch>
# Re-run the pipeline on the merged result; confirm it still reproduces
Then cleanup worktree (Step 6), then git branch -d <feature-branch>.
Option 2: Write Up and Share
Produce the report (see "Report Content" below). If sharing via PR:
git push -u origin <feature-branch>
gh pr create --title "<title>" --body "$(cat <<'EOF'
## Question
<the research question>
## What was done
<the pre-registered analysis, and any documented deviations>
## Findings
<confirmatory results with effect sizes + intervals>
## Exploratory (not confirmatory)
<clearly separated leads>
## Reproducibility
- Pre-registration: <path/commit>
- Environment: <lockfile>
- Seed: <value>
- Re-run: <command>
EOF
)"
Do NOT clean up the worktree — it's needed for iteration on feedback.
Option 3: Keep As-Is
Report: "Keeping branch . Worktree preserved at ." No cleanup.
Option 4: Discard
Confirm first:
This will permanently delete:
- Branch <name>
- All commits: <list>
- Worktree at <path>
Type 'discard' to confirm.
Wait for the exact word. Then cleanup worktree (Step 6) and git branch -D <feature-branch>.
Step 6: Cleanup Workspace
Only for Options 1 and 4. Options 2 and 3 preserve the worktree.
WORKTREE_PATH=$(git rev-parse --show-toplevel)
GIT_DIR == GIT_COMMON: normal repo, nothing to clean up.- Worktree under
.worktrees/,worktrees/, or~/.config/superpowers/worktrees/: we own it.MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel) cd "$MAIN_ROOT" git worktree remove "$WORKTREE_PATH" git worktree prune - Otherwise: the harness owns it. Don't remove it.
Report Content
Every report MUST:
- Separate confirmatory from exploratory. Confirmatory results are those that match the frozen pre-registration, run as registered. Everything else goes under an "Exploratory" heading and is described as a lead, not a conclusion.
- Report effect sizes and intervals, not just p-values. Describe null results as inconclusive vs. evidence of absence honestly.
- Document every deviation from the pre-registration and note that the affected analysis is exploratory.
- State limitations and threats to validity carried from the design.
- Avoid over-claiming — no causal language from observational data; no generalizing past the sample.
Archive for Reproducibility
Whatever the option, ensure the archive contains everything needed to regenerate the result:
- The code (committed)
- The environment lockfile and runtime version
- The fixed seed
- The frozen pre-registration (commit reference) and the passing
prereg.sh auditoutput - Data provenance + checksums for raw inputs (and the data itself, or instructions to obtain it, per any data-sharing constraints)
- The exact command(s) to re-run
Red Flags
Never:
- Report a number you haven't just reproduced
- Blur confirmatory and exploratory results
- Report a p-value without an effect size and interval
- Use causal language for an observational finding
- Hide a deviation from the pre-registration
- Remove a worktree before confirming a merge succeeded
- Clean up a worktree you didn't create
Always:
- Verify reproducibility before offering options
- Present exactly 4 options (3 for detached HEAD)
- Get typed confirmation for discard
- Archive code + env + seed + pre-registration together
Requesting Red-Team Review
Dispatch a reviewer subagent whose explicit job is to find why your conclusion is wrong — before you believe it or report it. The reviewer gets precisely crafted context for evaluation, never your session's history. This keeps it focused on the work product and your reasoning, not on agreeing with you, and preserves your own context.
Core principle: Try hardest to break a finding when it's most likely to be right. A result that survives a genuine attack is worth reporting; one that hasn't been attacked isn't.
When to Request Review
Mandatory:
- After each analysis step in subagent-driven-analysis (the rigor reviewer plays this role per step)
- After completing the full analysis, before writing it up
- Before claiming any confirmatory finding
Especially valuable:
- When the result is surprising, important, or convenient
- When performance is suspiciously good (possible leakage)
- After resolving an anomaly
How to Request
1. Get the git SHAs that bound the work:
BASE_SHA=$(git rev-parse HEAD~N) # before the analysis
HEAD_SHA=$(git rev-parse HEAD)
2. Dispatch the reviewer subagent with the Task tool (general-purpose), filling the template at reviewer.md.
Placeholders:
{DESCRIPTION}— what you analyzed and the conclusion you're considering{QUESTION}— the research question{PREREGISTRATION}— the frozen pre-registration (or note that this is exploratory){BASE_SHA}/{HEAD_SHA}— commit range
3. Act on feedback:
- Fix Critical issues (a confound, leakage, an assumption violation that changes the conclusion) before claiming anything
- Address Important issues before reporting
- Note Minor issues
- Push back with reasoning if the reviewer is wrong — but verify first
What the Reviewer Attacks
The reviewer is prompted to look for:
- Confounds and alternative explanations — what else could produce this result?
- Assumption violations — does the method fit the data?
- Data leakage — did test information contaminate the result?
- Researcher degrees of freedom — signs of p-hacking, HARKing, or post-hoc choices
- Multiplicity — uncorrected multiple comparisons
- Reproducibility — does it reproduce from raw + seed?
- Over-claiming — does the conclusion outrun the evidence (causal language from observational data, generalizing beyond the sample)?
Example
[Completed primary analysis: exposure → outcome, pre-registered]
You: Requesting red-team review before I report this.
BASE_SHA=a1b2c3d HEAD_SHA=e4f5g6h
[Dispatch reviewer subagent with reviewer.md filled in]
Reviewer returns:
Critical: Site is associated with both exposure and outcome but isn't in the model — likely confound (Simpson's risk).
Important: No check of the homoscedasticity assumption.
Minor: Figure axis unlabeled.
You: [Site WAS pre-registered as a covariate — verify the model actually included it]
[Check: the model formula dropped `site` due to a typo. Reviewer is right. Fix, re-run, re-verify.]
Red Flags
Never:
- Skip review because the result "is obviously right"
- Ignore a Critical issue
- Report a finding with unresolved Important issues
- Argue with valid methodological feedback
If the reviewer is wrong: push back with technical reasoning and show the verification (the diagnostic, the pre-registration line, the reproduced number) — see science-superpowers:receiving-critical-review.
Integration
- science-superpowers:subagent-driven-analysis — the per-step rigor review uses this pattern
- science-superpowers:receiving-critical-review — how to respond to what the reviewer finds
- science-superpowers:reporting-and-archiving-findings — only after the result survives review
See template at: requesting-red-team-review/reviewer.md
Setting Up a Reproducible Analysis
Overview
Ensure the analysis happens in an isolated workspace that another person (or future you) can reproduce exactly: same code, same environment, same seed, same immutable input data.
Core principle: A result you cannot reproduce is not a result. Set up reproducibility before you run anything.
Announce at start: "I'm using the setting-up-reproducible-analysis skill to set up an isolated, reproducible workspace."
Step 0: Detect Existing Isolation
Before creating anything, check whether you're already in an isolated workspace.
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
Submodule guard: GIT_DIR != GIT_COMMON is also true in submodules. Verify you're not in one:
git rev-parse --show-superproject-working-tree 2>/dev/null
If GIT_DIR != GIT_COMMON (and not a submodule): already in a linked worktree. Skip to Step 2 (Environment). Don't create another.
If GIT_DIR == GIT_COMMON: normal checkout. If your human partner hasn't already stated a preference, ask before creating a worktree:
"Would you like me to set up an isolated worktree? It keeps this analysis separate from your current branch."
Honor any declared preference. If declined, work in place and continue to Step 2.
Step 1: Create Isolated Workspace
Native tools first. If you have a native worktree tool (EnterWorktree, WorktreeCreate, a /worktree command, a --worktree flag), use it and skip to Step 2. Using git worktree add when a native tool exists creates phantom state the harness can't manage.
Git fallback (only if no native tool):
# Prefer an existing .worktrees/ (must be git-ignored), else default to it
git check-ignore -q .worktrees 2>/dev/null || echo ".worktrees/" >> .gitignore
git worktree add ".worktrees/$BRANCH_NAME" -b "$BRANCH_NAME"
cd ".worktrees/$BRANCH_NAME"
If git worktree add fails with a sandbox permission error, tell your human partner and work in the current directory instead.
Step 2: Pin the Environment
A reproducible result needs a pinned environment. Detect and set one up:
# Python
if [ -f environment.yml ]; then conda env create -f environment.yml || conda env update -f environment.yml; fi
if [ -f requirements.txt ]; then python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt; fi
if [ -f pyproject.toml ]; then python -m venv .venv && . .venv/bin/activate && pip install -e .; fi
# R
if [ -f renv.lock ]; then Rscript -e 'renv::restore()'; fi
If no environment file exists, create one and record exact versions (a lockfile, pip freeze > requirements.txt, conda env export, or renv::snapshot()). Record the language runtime version too. An analysis run against unpinned dependencies is not reproducible.
Step 3: Fix Random Seeds
Set a single global seed and record it (it belongs in the pre-registration too). Anything stochastic — sampling, splits, bootstrap, model init, cross-validation folds — must use it.
SEED = 20260528
import os, random, numpy as np
os.environ["PYTHONHASHSEED"] = str(SEED)
random.seed(SEED); np.random.seed(SEED)
# torch.manual_seed(SEED); etc.
Step 4: Data Provenance & Immutability
- Raw data is immutable. Put it under
data/raw/and treat it as read-only. Every transform writes a NEW artifact underdata/derived/. Never edit raw in place. - Record provenance: where each raw file came from, when, which version/query, and a checksum.
mkdir -p data/raw data/derived
shasum -a 256 data/raw/* > data/raw/CHECKSUMS.txt # detect silent changes later
chmod -w data/raw/* 2>/dev/null || true # best-effort read-only
- Keep raw and derived separate so a reviewer can see exactly what came in and what you made.
Step 5: Verify a Clean Baseline
Before adding new work, confirm the workspace starts clean and existing analyses reproduce:
# Run existing tests / re-run an existing analysis and confirm outputs match
pytest 2>/dev/null || true
If existing results don't reproduce: report it and ask whether to fix the baseline before proceeding. You can't attribute new findings to your work if the starting point was already broken.
Report
Workspace ready at <path> (branch <name>)
Environment pinned: <python 3.x / conda env>, lockfile recorded
Seed fixed: <SEED>
Raw data: <N files>, checksummed, read-only
Baseline: <reproduces / N tests pass>
Ready to execute <topic>
Quick Reference
| Situation | Action |
|---|---|
| Already in a linked worktree | Skip creation (Step 0) |
| In a submodule | Treat as normal repo |
| Native worktree tool available | Use it |
| No environment file | Create one + record a lockfile |
| Stochastic step | Use the recorded global seed |
| Need to transform raw data | Write a new derived artifact; never edit raw |
| Baseline doesn't reproduce | Report + ask before proceeding |
Red Flags
Never:
- Run an analysis against unpinned dependencies
- Leave seeds unset for anything stochastic
- Edit raw data in place
- Skip the baseline reproducibility check
- Use
git worktree addwhen a native worktree tool exists
Always:
- Detect existing isolation first
- Pin the environment and record a lockfile
- Fix and record the seed
- Keep raw immutable and provenance documented
- Confirm the baseline reproduces
Subagent-Driven Analysis
Execute a pre-registered analysis plan by dispatching a fresh subagent per analysis step, with two-stage review after each: protocol-compliance review first (did it run exactly what was pre-registered, nothing more), then statistical-rigor review (assumptions, leakage, correctness, reproducibility).
Why subagents: You delegate steps to specialized agents with isolated context. By precisely crafting their instructions, you keep them focused and prevent them from improvising analyses you didn't pre-register. They never inherit your session's history — you construct exactly what they need. This also preserves your own context for coordination.
Core principle: Fresh subagent per step + two-stage review (protocol then rigor) = trustworthy, reproducible results.
Continuous execution: Do not pause to check in between steps. Execute the whole plan. The only reasons to stop: a BLOCKED status you cannot resolve, an anomaly that needs science-superpowers:investigating-anomalous-results, a genuine ambiguity, or all steps complete.
Prerequisite
The analysis plan MUST be pre-registered and frozen (science-superpowers:preregistering-analysis) before any step runs. If it is not frozen, stop and pre-register first. Executing before freezing turns the whole thing exploratory.
When to Use
digraph when_to_use {
"Pre-registered plan?" [shape=diamond];
"Steps mostly independent?" [shape=diamond];
"Stay in this session?" [shape=diamond];
"subagent-driven-analysis" [shape=box];
"executing-analysis" [shape=box];
"Pre-register first" [shape=box];
"Pre-registered plan?" -> "Steps mostly independent?" [label="yes"];
"Pre-registered plan?" -> "Pre-register first" [label="no"];
"Steps mostly independent?" -> "Stay in this session?" [label="yes"];
"Stay in this session?" -> "subagent-driven-analysis" [label="yes"];
"Stay in this session?" -> "executing-analysis" [label="no - parallel session"];
}
The Process
digraph process {
rankdir=TB;
"Set up reproducible workspace" [shape=box];
"Read plan + frozen prereg, extract all steps, create TodoWrite" [shape=box];
"Dispatch analyst subagent (./analyst-prompt.md)" [shape=box];
"Analyst asks questions?" [shape=diamond];
"Answer, provide context" [shape=box];
"Analyst runs step, validates, commits, self-reviews" [shape=box];
"Dispatch protocol-compliance reviewer (./protocol-compliance-reviewer-prompt.md)" [shape=box];
"Ran exactly the pre-registered analysis?" [shape=diamond];
"Analyst fixes deviations" [shape=box];
"Dispatch rigor reviewer (./rigor-reviewer-prompt.md)" [shape=box];
"Rigor reviewer approves?" [shape=diamond];
"Analyst fixes rigor issues" [shape=box];
"Mark step complete" [shape=box];
"More steps?" [shape=diamond];
"Use requesting-red-team-review on whole result" [shape=box style=filled fillcolor=lightgreen];
"Use reporting-and-archiving-findings" [shape=box style=filled fillcolor=lightgreen];
"Set up reproducible workspace" -> "Read plan + frozen prereg, extract all steps, create TodoWrite";
"Read plan + frozen prereg, extract all steps, create TodoWrite" -> "Dispatch analyst subagent (./analyst-prompt.md)";
"Dispatch analyst subagent (./analyst-prompt.md)" -> "Analyst asks questions?";
"Analyst asks questions?" -> "Answer, provide context" [label="yes"];
"Answer, provide context" -> "Dispatch analyst subagent (./analyst-prompt.md)";
"Analyst asks questions?" -> "Analyst runs step, validates, commits, self-reviews" [label="no"];
"Analyst runs step, validates, commits, self-reviews" -> "Dispatch protocol-compliance reviewer (./protocol-compliance-reviewer-prompt.md)";
"Dispatch protocol-compliance reviewer (./protocol-compliance-reviewer-prompt.md)" -> "Ran exactly the pre-registered analysis?";
"Ran exactly the pre-registered analysis?" -> "Analyst fixes deviations" [label="no"];
"Analyst fixes deviations" -> "Dispatch protocol-compliance reviewer (./protocol-compliance-reviewer-prompt.md)" [label="re-review"];
"Ran exactly the pre-registered analysis?" -> "Dispatch rigor reviewer (./rigor-reviewer-prompt.md)" [label="yes"];
"Dispatch rigor reviewer (./rigor-reviewer-prompt.md)" -> "Rigor reviewer approves?";
"Rigor reviewer approves?" -> "Analyst fixes rigor issues" [label="no"];
"Analyst fixes rigor issues" -> "Dispatch rigor reviewer (./rigor-reviewer-prompt.md)" [label="re-review"];
"Rigor reviewer approves?" -> "Mark step complete" [label="yes"];
"Mark step complete" -> "More steps?";
"More steps?" -> "Dispatch analyst subagent (./analyst-prompt.md)" [label="yes"];
"More steps?" -> "Use requesting-red-team-review on whole result" [label="no"];
"Use requesting-red-team-review on whole result" -> "Use reporting-and-archiving-findings";
}
Model Selection
Use the least powerful model that can handle each role.
- Mechanical steps (load data, apply a fixed transform, produce a planned figure): fast, cheap model.
- Integration/judgment steps (fitting the primary model, handling missingness per plan): standard model.
- Rigor review and anomaly judgment: most capable model.
Handling Analyst Status
DONE: Proceed to protocol-compliance review.
DONE_WITH_CONCERNS: Read the concerns. If about correctness or an unexpected data issue, address before review (may need science-superpowers:investigating-anomalous-results). If an observation ("this file is getting large"), note and proceed.
NEEDS_CONTEXT: Provide the missing information and re-dispatch.
BLOCKED: Assess. Context problem → provide more and re-dispatch. Needs more reasoning → more capable model. Step too large → split. Plan itself is wrong → escalate to your human partner. A wrong plan may require re-opening the pre-registration, which must be documented as a deviation.
Never silently let a subagent change the registered analysis to make a step "work." A deviation is documented and renders that analysis exploratory.
Prompt Templates
./analyst-prompt.md— dispatch the analyst subagent./protocol-compliance-reviewer-prompt.md— did it match the pre-registration?./rigor-reviewer-prompt.md— is the statistics correct, reproducible, leak-free?
Red Flags
Never:
- Execute before the pre-registration is frozen
- Skip either review (protocol OR rigor)
- Let a subagent add an unregistered analysis and report it as confirmatory
- Run rigor review before protocol compliance is green (wrong order)
- Accept a silently dropped outlier or changed cutoff (route to anomaly investigation instead)
- Dispatch multiple analyst subagents on the same artifacts in parallel (conflicts)
- Make the subagent read the whole plan (provide the step's full text + the relevant pre-registration excerpt)
If a reviewer finds issues: the same analyst subagent fixes them, then the reviewer reviews again. Repeat until approved.
Integration
Required workflow skills:
- science-superpowers:setting-up-reproducible-analysis — isolated, seeded, reproducible workspace (run first)
- science-superpowers:preregistering-analysis — must be frozen before execution
- science-superpowers:investigating-anomalous-results — when a step's output is surprising or impossible
- science-superpowers:requesting-red-team-review — adversarial review of the whole result
- science-superpowers:reporting-and-archiving-findings — after all steps complete
Subagents should use:
- science-superpowers:verifying-results-before-claiming — before any subagent reports a step as done
Alternative:
- science-superpowers:executing-analysis — inline execution for harnesses without subagents
Surveying Prior Work
Overview
Before designing an analysis, ground the question and your chosen methods in what is already known. Most questions have established methods, known confounds, and prior effect sizes. Reinventing a method badly — or rediscovering a known artifact and reporting it as a finding — wastes effort and erodes credibility.
Core principle: Find out what is already known before you generate new claims.
This is the science analog of reading the existing codebase before writing new code. It is a flexible skill — adapt depth to the stakes of the investigation.
When to Use
- After
framing-research-questions, beforedesigning-the-analysis - When selecting a statistical method or model and unsure what is standard
- When a result looks novel — check whether it is a known effect or artifact first
- When you need a plausible prior effect size to power the study
- When you suspect confounds but don't know which are established in the field
What to Ground
Survey four things:
- Established methods — what is the standard, accepted way to analyze this kind of question? What are its assumptions and failure modes?
- Known confounds and artifacts — what variables are known to drive this outcome? What measurement artifacts are common in this data type?
- Prior effect sizes — what magnitude have others found? This feeds the power/sample-size calculation in the analysis design.
- Relationship to prior work — is this a replication, an extension, a new population, or genuinely novel? Be honest about which.
The Process
- List what needs grounding — turn the "open questions for prior-work survey" from the question document into concrete search targets.
- Search — use web search and any available literature tools for published methods and effect sizes; search the repo and your human partner's prior analyses for in-house precedent and conventions.
- For broad surveys, dispatch parallel subagents — one per independent sub-topic (a method, a confound, a prior estimate). REQUIRED: use
science-superpowers:dispatching-parallel-investigationsfor the workflow. This keeps your own context clean. - Synthesize — write a short prior-work note (or a section appended to the question document):
- Established method(s) you will adopt and why
- Confounds you must measure or control for
- Prior effect size(s) and the source, for powering the design
- Honest statement of how this investigation relates to prior work
- Cite sources — record where each fact came from so the design and final report can reference it. When you present findings from web search, follow your platform's citation standards.
Red Flags
- Reaching for a method you invented when a standard one exists → find and use the standard one (or justify the deviation explicitly)
- Designing the analysis with no idea what effect size to expect → you skipped the power input; survey first
- About to report something as novel without having checked → search first; novelty is a claim that needs evidence
- Ignoring a confound because checking is inconvenient → known confounds are not optional
Handoff
After grounding, invoke science-superpowers:designing-the-analysis. Bring forward the adopted methods, the confound list, and the prior effect size for powering the design.
IF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.
This is not negotiable. This is not optional. You cannot rationalize your way out of this.
Instruction Priority
Science Superpowers skills override default system prompt behavior, but your human partner's instructions always take precedence:
- Your human partner's explicit instructions (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority
- Science Superpowers skills — override default system behavior where they conflict
- Default system prompt — lowest priority
If CLAUDE.md, GEMINI.md, or AGENTS.md says "skip pre-registration" and a skill says "always pre-register," follow your human partner's instructions. They are in control.
How to Access Skills
In Claude Code: Use the Skill tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.
In Copilot CLI: Use the skill tool. Skills are auto-discovered from installed plugins. The skill tool works the same as Claude Code's Skill tool.
In Gemini CLI: Skills activate via the activate_skill tool. Gemini loads skill metadata at session start and activates the full content on demand.
In Google Antigravity: Skills are Antigravity Agent Skills (same SKILL.md format). Antigravity equips a skill automatically when your request matches its description. The always-on bootstrap rule keeps the discipline active from the first message, before any skill is equipped. See references/antigravity-tools.md.
In Pi: Skills are discovered natively (same SKILL.md format) and listed in your system prompt; when a task matches, read the skill's SKILL.md and follow it. Pi also exposes each skill as a /skill:name command. The bootstrap extension keeps the discipline active from the first message. Pi ships without sub-agents and a to-do tool — see references/pi-tools.md.
In other environments: Check your platform's documentation for how skills are loaded.
Platform Adaptation
Skills use Claude Code tool names. Non-CC platforms: see references/copilot-tools.md (Copilot CLI), references/codex-tools.md (Codex), references/antigravity-tools.md (Google Antigravity), references/pi-tools.md (Pi) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.
Using Skills
The Rule
Invoke relevant or requested skills BEFORE any response or action. Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.
digraph skill_flow {
"User message received" [shape=doublecircle];
"About to investigate/analyze?" [shape=doublecircle];
"Already framed the question?" [shape=diamond];
"Invoke framing-research-questions skill" [shape=box];
"Might any skill apply?" [shape=diamond];
"Invoke Skill tool" [shape=box];
"Announce: 'Using [skill] to [purpose]'" [shape=box];
"Has checklist?" [shape=diamond];
"Create TodoWrite todo per item" [shape=box];
"Follow skill exactly" [shape=box];
"Respond (including clarifications)" [shape=doublecircle];
"About to investigate/analyze?" -> "Already framed the question?";
"Already framed the question?" -> "Invoke framing-research-questions skill" [label="no"];
"Already framed the question?" -> "Might any skill apply?" [label="yes"];
"Invoke framing-research-questions skill" -> "Might any skill apply?";
"User message received" -> "Might any skill apply?";
"Might any skill apply?" -> "Invoke Skill tool" [label="yes, even 1%"];
"Might any skill apply?" -> "Respond (including clarifications)" [label="definitely not"];
"Invoke Skill tool" -> "Announce: 'Using [skill] to [purpose]'";
"Announce: 'Using [skill] to [purpose]'" -> "Has checklist?";
"Has checklist?" -> "Create TodoWrite todo per item" [label="yes"];
"Has checklist?" -> "Follow skill exactly" [label="no"];
"Create TodoWrite todo per item" -> "Follow skill exactly";
}
Red Flags
These thoughts mean STOP—you're rationalizing:
| Thought | Reality |
|---|---|
| "This is just a quick data peek" | A peek can poison a confirmatory analysis. Check for skills first. |
| "I need to look at the data first" | Skills tell you WHEN looking is safe. Check first. |
| "This is just a simple question" | Questions are tasks. Check for skills. |
| "Let me just run the analysis" | Skills tell you HOW to run it honestly. Check first. |
| "I'll frame the question after I explore" | Framing comes BEFORE exploration. Check first. |
| "This doesn't need a formal skill" | If a skill exists, use it. |
| "I remember this skill" | Skills evolve. Read current version. |
| "This doesn't count as a task" | Action = task. Check for skills. |
| "The skill is overkill" | Simple analyses become p-hacked papers. Use it. |
| "I'll just do this one thing first" | Check BEFORE doing anything. |
| "This feels productive" | Undisciplined action produces unreproducible results. Skills prevent this. |
| "I know what that means" | Knowing the concept ≠ using the skill. Invoke it. |
Skill Priority
When multiple skills could apply, use this order:
- Process skills first (framing-research-questions, investigating-anomalous-results) - these determine HOW to approach the task
- Execution skills second (designing-the-analysis, subagent-driven-analysis) - these guide carrying it out
"Let's investigate X" → framing-research-questions first, then execution skills. "This result looks wrong" → investigating-anomalous-results first.
Feasibility mode is the one opt-in branch. If whether the computation can run at all is unknown, framing-research-questions offers establishing-feasibility-first. Only your human partner can choose it, and only they can end it. Never enter or leave that mode on your own initiative — and never enter it to avoid pre-registration.
Skill Types
Rigid (pre-registration, anomaly investigation, verification): Follow exactly. Don't adapt away discipline.
Flexible (patterns): Adapt principles to context.
The skill itself tells you which.
User Instructions
Instructions say WHAT, not HOW. "Analyze X" or "Check whether Y" doesn't mean skip workflows.
Verifying Results Before Claiming
Overview
Claiming a finding without fresh verification is dishonesty, not efficiency.
Core principle: Evidence before claims, always.
Violating the letter of this rule is violating the spirit of this rule.
The Iron Law
NO CLAIMS WITHOUT FRESH REPRODUCED EVIDENCE
If you haven't run the analysis in this state and read its actual output, you cannot claim its result. "It was significant earlier" is not evidence now.
The Gate Function
BEFORE claiming any result or expressing satisfaction:
1. IDENTIFY: What command/analysis proves this claim?
2. RUN: Execute it fresh and complete (from the immutable raw data, fixed seed)
3. READ: The actual output — the estimate, the interval, the p-value, the diagnostics
4. CHECK: Do the method's assumptions hold? Does it reproduce?
For a CONFIRMATORY claim: does the pre-registration audit pass?
(prereg.sh, ships with science-superpowers:preregistering-analysis:
<skills root>/preregistering-analysis/prereg.sh audit)
5. VERIFY: Does the output actually support the claim?
- If NO: state the real result with evidence
- If YES: state the claim WITH the evidence (number + interval)
6. ONLY THEN: make the claim
Skip any step = asserting, not verifying
Common Failures
| Claim | Requires | Not Sufficient |
|---|---|---|
| "The effect is significant" | Fresh run; read estimate, CI, and p | "It was significant before" |
| "There's no effect" | Effect size + interval showing precision | A non-significant p (could be underpowered) |
| "The result reproduces" | Re-run from raw data + fixed seed → same number | "It ran fine earlier" |
| "Assumptions are met" | The diagnostic output, read | "It's probably fine" |
| "The model is good" | Out-of-sample metric | In-sample fit / training accuracy |
| "Data cleaned correctly" | Validation counts (rows in/out, ranges) | "The script ran without error" |
| "Confirmatory finding" | prereg.sh audit passes AND re-run as registered | The prereg file was committed before the results (it may have been edited after) |
| "The subagent finished" | Inspect the committed artifacts/diff | The subagent said "done" |
Red Flags - STOP
- Using "should", "probably", "seems to", "looks significant"
- Expressing satisfaction before verification ("Great, it worked!", "Perfect!", "Confirmed!")
- About to write up / report / commit a conclusion without a fresh run
- Reporting a p-value you computed before the latest code change
- Calling a result "reproducible" without having re-run it from raw
- Trusting a subagent's success report
- Calling a finding confirmatory without a passing
prereg.sh audit— commit order alone is not a freeze - Describing a non-significant result as "no effect" without checking power/interval
- ANY wording implying a finding without having just produced the evidence
Rationalization Prevention
| Excuse | Reality |
|---|---|
| "It was significant last run" | Re-run it now. Code/data may have changed. |
| "I'm confident in the result" | Confidence is not evidence. |
| "The script ran without errors" | Running ≠ correct. Read the output. |
| "p < .05, so it's real" | p is not the probability the effect is real. Report effect + interval. |
| "p > .05, so no effect" | Could be underpowered. Absence of evidence ≠ evidence of absence. |
| "The subagent reported success" | Verify the artifacts independently. |
| "The prereg was committed before the results, so it's frozen" | Commit order misses post-freeze edits to the registration. Run prereg.sh audit; read the frozen-vs-current diff. |
| "It reproduces, I'm sure" | Re-run from raw + seed and show the same number. |
| "Different words, so the rule doesn't apply" | Spirit over letter. |
Key Patterns
Significance / effect:
✅ [Re-run the model] [Read: beta=0.23, 95% CI [0.08, 0.38], p=.002] "Exposure raises the outcome; CI excludes zero."
❌ "The effect looked significant."
Reproducibility:
✅ Fresh env → re-run from data/raw with seed → same estimate to reported precision → "Reproduces."
❌ "It ran earlier, so it reproduces."
Null result:
✅ [Read: beta=0.01, 95% CI [-0.12, 0.14]] "No detectable effect; the interval rules out effects larger than ~0.14."
❌ "p=0.3, so there's no effect."
Model performance:
✅ [Held-out test set, used once] [AUC=0.78] "Out-of-sample AUC 0.78."
❌ "Training accuracy is 0.97, the model is great."
Confirmatory label:
✅ [Run prereg.sh audit] [Read: RESULT: PASS] → apply the frozen decision rule to the fresh output
❌ "The pre-registration predates the results commit, so the finding is confirmatory."
Subagent delegation:
✅ Subagent reports done → inspect committed code + output artifact → confirm → report actual state
❌ Trust the report
When To Apply
ALWAYS before:
- Any statement of a result, effect, or significance
- Any claim that an analysis reproduces or that assumptions hold
- Any expression of satisfaction about a finding
- Writing up, reporting, committing a conclusion, or requesting review
- Trusting a subagent's output
Applies to: exact phrases, paraphrases, synonyms, and any implication of a verified finding.
The Bottom Line
Run it fresh. Read the output. Check it reproduces. THEN state the result — with the number and the interval.
This is non-negotiable.
Related Skills
- science-superpowers:investigating-anomalous-results — if verification reveals the result doesn't hold
- science-superpowers:requesting-red-team-review — independent scrutiny after you've verified
Writing Science Skills
Overview
Writing a skill is an experiment, not an essay. You form a hypothesis (this guidance will change agent behavior), run a baseline (watch agents fail without it), write the skill, and test that behavior changes.
Core principle: If you didn't watch an agent fail without the skill, you don't know whether the skill teaches the right thing — the same reason you pre-register before analyzing rather than rationalizing a result after.
REQUIRED BACKGROUND: Understand science-superpowers:preregistering-analysis first. The discipline of fixing a prediction before observing the outcome is exactly the discipline here: observe baseline behavior before writing the skill, so the skill is shaped by evidence, not by what you imagine agents do.
What is a Skill?
A skill is a reference guide for a proven technique, pattern, or discipline that future agents can find and apply.
Skills are: reusable techniques, patterns, disciplines, reference guides.
Skills are NOT: narratives about how you did something once.
The Empirical Mapping
| Experiment concept | Skill creation |
|---|---|
| Hypothesis | "This guidance will fix behavior X" |
| Baseline / control | Agent behavior WITHOUT the skill |
| Observed failure (the signal) | The exact rationalizations agents use |
| Intervention | The skill document |
| Effect | Agent now complies |
| Replication | Re-test under different pressures until robust |
When to Create a Skill
Create when:
- The technique wasn't obvious and you'd reference it again across projects
- It applies broadly (not project-specific)
Don't create for:
- One-off solutions
- Standard practice documented elsewhere
- Project-specific conventions (put those in CLAUDE.md)
- Anything enforceable mechanically (validate it in code instead of documenting a judgment call)
Skill Types
- Technique — a concrete method with steps (e.g., investigating-anomalous-results)
- Pattern — a way of thinking (e.g., separating confirmatory from exploratory)
- Discipline — a rule that must resist rationalization (e.g., preregistering-analysis)
- Reference — API/command/method documentation (e.g., statistical-fallacies)
Directory Structure
skills/
skill-name/
SKILL.md # Main reference (required)
supporting-file.* # Only for heavy reference or reusable tools
Flat namespace. Keep principles and short patterns inline; split out only heavy reference (100+ lines) or reusable tools/templates.
SKILL.md Structure
Frontmatter (YAML): two required fields, name and description, max 1024 characters total.
name: letters, numbers, hyphens onlydescription: third person, describes ONLY WHEN to use (not what it does); start with "Use when..."
Description = When to Use, NOT What the Skill Does
This is the single most important authoring rule. If the description summarizes the workflow, the agent follows the summary and skips the skill body.
# BAD: summarizes workflow — agent follows this instead of reading the skill
description: Use when analyzing data - pre-register, run the test, check assumptions, then report
# GOOD: triggering conditions only
description: Use before running any confirmatory analysis or looking at outcome data
Use concrete triggers and symptoms. Describe the problem (a convenient result, a surprising number), not just the topic. Keep it technology-agnostic unless the skill is technology-specific.
Claude Search Optimization (CSO)
Future agents must FIND your skill.
- Rich description — the triggering conditions an agent would recognize
- Keyword coverage — error messages, symptoms ("leakage", "won't converge", "p-hacked"), synonyms, tool/library names
- Descriptive, verb-first naming —
investigating-anomalous-resultsnotanomaly-utils; gerunds work well for processes - Token efficiency — frequently-loaded skills should be tight; move detail to references, use cross-references instead of repetition
Cross-Referencing Other Skills
Use the skill name with an explicit requirement marker:
- Good:
**REQUIRED SUB-SKILL:** Use science-superpowers:preregistering-analysis - Bad:
See skills/.../SKILL.md(unclear if required) - Bad:
@skills/.../SKILL.md(force-loads, burns context)
Never use @ links to other skills — they load immediately and consume context before it's needed.
Flowchart Usage
Use a small graphviz dot flowchart ONLY for a non-obvious decision point or a loop where the agent might stop too early. Never for reference material (use tables/lists), code (use code blocks), or linear steps (use numbered lists). Keep node labels semantic.
Code & Examples
One excellent, complete, runnable example beats five mediocre ones. For a computational-science framework, prefer Python for data work and shell for environment/pipeline work. Comment WHY, not what. Don't write fill-in-the-blank templates.
The Iron Law
NO SKILL WITHOUT AN OBSERVED BASELINE FAILURE FIRST
Applies to NEW skills AND EDITS. Wrote the skill before observing the baseline? You're guessing at what agents need. Delete it, run the baseline, start over.
No exceptions — not for "simple additions", not for "just a section", not for "documentation updates". Don't keep untested changes as "reference".
Testing All Skill Types
Different types need different tests. Dispatch fresh subagents (no shared context) and observe.
- Discipline skills (preregistering-analysis, verifying-results-before-claiming): pressure scenarios — time pressure, a convenient result, sunk cost, authority. Success = agent holds the line under combined pressure. Capture every rationalization and add an explicit counter.
- Technique skills (investigating-anomalous-results): application scenarios and edge cases. Success = agent applies the technique correctly to a new case.
- Pattern skills: recognition + application + counter-examples. Success = agent knows when it applies and when it doesn't.
- Reference skills (statistical-fallacies): retrieval + application. Success = agent finds and correctly uses the right entry.
RED — GREEN — REFACTOR for Skills
- RED (baseline): run the scenario WITHOUT the skill. Document the exact choices and rationalizations verbatim.
- GREEN (minimal skill): write the skill addressing those specific failures. Re-run; confirm compliance.
- REFACTOR (close loopholes): new rationalization appears? Add an explicit counter. Re-test until robust.
Bulletproofing Discipline Skills
- Close every loophole explicitly — don't just state the rule; forbid the specific workarounds ("Don't relabel the peeked analysis confirmatory").
- State the foundational principle — "Violating the letter is violating the spirit" cuts off a whole class of rationalizations.
- Build a rationalization table — every excuse from baseline testing, with its reality.
- Create a Red Flags list — the thoughts that signal the agent is about to violate the rule.
Common Rationalizations for Skipping Testing
| Excuse | Reality |
|---|---|
| "Skill is obviously clear" | Clear to you ≠ clear to another agent. Test it. |
| "It's just a reference" | References have gaps. Test retrieval. |
| "No time to test" | Deploying an untested skill costs more time later. |
| "Academic review is enough" | Reading ≠ using. Test application. |
Checklist (use TodoWrite for each item)
RED:
- Write pressure/application scenarios (3+ combined pressures for discipline skills)
- Run WITHOUT the skill; document baseline verbatim
- Identify rationalization/failure patterns
GREEN:
-
nameuses only letters/numbers/hyphens -
descriptionstarts with "Use when", third person, triggers only, no workflow summary - Keywords for search throughout
- Addresses the specific baseline failures
- One excellent example
- Re-run WITH the skill; confirm compliance
REFACTOR:
- Add counters for new rationalizations
- Rationalization table + Red Flags list
- Re-test until robust
Quality:
- Small flowchart only if a decision is non-obvious
- No narrative storytelling
- Cross-references use
science-superpowers:<name>, no@links - Run
scripts/bump-version.sh --checkif you touched a manifest
STOP: Before Moving to the Next Skill
After writing ANY skill, STOP and complete its testing before starting another. Don't batch. Deploying an untested skill is deploying an untested intervention.
The Bottom Line
Creating a skill IS running an experiment on agent behavior. Same discipline as pre-registration: observe the baseline before you intervene, and let evidence — not your imagination — shape the result.
Hooks 1
Sponsors
Details
- Marketplace
- K-Dense-AI/science-superpowers
- Author
- K-Dense, Inc.
- Category
- Community