Drop-in agentic loop skills (autoresearch, literature & writing, data, optimization, red-team, power analysis). Skills install namespaced as agent-loops:<name>.

GitHub stars
183

Install

/plugin install agent-loops@agent-loop-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add gaasher/Agent-Loop-Skills.

Contents

Discovered from source

Skills 25

Alpha-Evolve

Reference (read if you need the algorithm's details): AlphaEvolve — https://arxiv.org/abs/2506.13131 · OpenEvolve (open-source impl) — https://github.com/algorithmicsuperintelligence/openevolve

A population-based evolutionary loop over a program. The artifact is the editable model code; a child is one analysis-informed SEARCH/REPLACE diff to a parent, and the feedback signal is a cascade-evaluated training run (<metric>, smoke→full). Children are placed in a MAP-Elites archive across islands (complexity × diversity axes), so a child survives by being either better or more novel, not just better. The discipline this enforces: diversity is preserved, not collapsed — diverse high performers co-exist instead of one local optimum winning. You are the controller: sample a parent + inspirations, spawn parallel Mutators to propose and evaluate children, place them, migrate between islands, checkpoint. Loops to a fixed compute budget or until interrupted.

When to use

Use this for parallel, diversity-preserving search over a model/program where many variants explore at once and the archive keeps the illuminated frontier. Default to broad island coverage; if quality stalls, bias selection toward exploiting top elites; if coverage stalls, bias toward empty cells. Not for the sequential autoresearch loops (one change at a time), and not for fixing a known anomaly.

The cast (both in this folder): roles/Mutator.md produces + cascade-evaluates one child (the generation step); schemas/result.schema.json is the result a Mutator returns.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available, <host> = claude-code) infer a likely value for each binding and present it as the recommended option; on other hosts (<host> = other) ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files. <host> also decides execution: Claude Code spawns real Agent Mutators in parallel (capped at <concurrency>); other hosts degrade to running a generation's children serially (identical algorithm).

Probe the box first (mandatory — measure, never assume <concurrency>). Record and report:

  • CPU cores → <cores>: python3 -c "import os; print(os.cpu_count())".
  • RAM → <ram_gb>: macOS sysctl -n hw.memsize; Linux grep MemTotal /proc/meminfo.
  • Accelerator → <accelerator>/<vram>/<gpu_count>: nvidia-smi --query-gpu=name,memory.total,count --format=csv (NVIDIA); else macOS Apple GPU/MPS; else CPU-only.
bindingmeaningdefaulthow to infer
<metric> + <metric_direction>scalar to optimize; min/maximize—ask; scan eval output for the reported metric
<run_cmd> / <entrypoint>command for one training run (the evaluator)—pyproject.toml/.venv/uv/README
<editable_files>the program being evolved (e.g. model.py, config.yaml); never the harness or data—ask explicitly — this is the code that gets mutated; do not default it (multi-select on Claude Code)
<sandbox_root>where lae/ is created./sandbox—
<gate> + <budget>one full run's size: time/epochs + amount; the FIXED eval budget applied to every program—identify the duration key now (e.g. train.epochs) so the controller can override it
<total_budget>total compute = number of full training runs (or wall-clock minutes); the single cost dial—ask
<concurrency>parallel evaluations Cderived from the probeCPU-only → max(1, <cores>//4); single GPU/MPS → 1 (ask if more fit <vram>); multi-GPU → <gpu_count> (pin one child/GPU)

num_generations is derived: ceil(<total_budget> / <concurrency>). The cascade is derived from <budget> (not asked): smoke = ~1 epoch / a small subset, full = <budget>, gate = child's smoke <metric> ≥ parent's smoke. <budget>/<metric>/eval split are FIXED — never mutation targets (a child may not "train longer" to look better); changing them means re-running the whole loop.

Advanced (opt-in). Ask one yes/no: "Use defaults for the evolutionary settings, or customize?" Defaults are faithful to AlphaEvolve/OpenEvolve — use them and ask nothing more. Only on "customize" ask for each (showing the default as recommended): num_islands (4), num_top (3), num_diverse (2), num_bins (10), migration_interval (5), diversity_reference_size (10), pop_per_island (40), seed (42). Axes are fixed: complexity × diversity. See examples/run.example.yaml for the shape.

Print the resolved bindings + the probe + derived num_generations, and do not create files or launch until the user confirms. Then initialise the sandbox (header rows only; programs/ is created as children are evaluated):

<sandbox_root>/lae/
├── archive.tsv     ← current elites = program database + checkpoint
├── history.tsv     ← append-only record of every child
├── leaderboard.md  ← rendered UI
└── programs/       ← one self-contained dir per program

The controller (loop)

You maintain num_islands MAP-Elites maps in archive.tsv, the append-only history.tsv, running per-axis percentile stats, and leaderboard.md. You are the sole writer of all shared logs — Mutators only return results, so there are no write races. Copy this checklist and tick items off:

  • Setup done: probe recorded, bindings confirmed, sandbox initialised, num_generations derived.
  • GEN 0 — in each island, create the baseline program (a copy of <editable_files>) + optionally a few stochastic variants; cascade-evaluate; place in the archive.
  • Per generation: build EXACTLY <concurrency> tasks (round-robin island, seeded-rule parent, top num_top + num_diverse most-diverse inspirations); make each child dir by copying the parent program + harness.
  • Run the C Mutators (spawn-or-degrade), each with roles/Mutator.md, parent code, inspirations, parent artifacts, its child dir, and the smoke/full budgets.
  • For each returned child: append a history.tsv row; if evaluated, compute its niche → cell and place it in the island map iff <metric> is better (kept=y); record smoke_dropped/crash without placing.
  • Re-render leaderboard.md; checkpoint (archive.tsv is the checkpoint); print a status line.
  • Every migration_interval generations: ring-migrate top elites island k → k+1.
  • Stop at <total_budget> (reserve a little for synthesis), then synthesize the final report.

Niche computation (you do this, from a child's sandbox):

  • complexity = trainable param count (fallback: total LOC of the editable files + any files the child added), log10-scaled.
  • diversity = average normalized edit distance of the program's concatenated code (editable + added files) to a random sample of diversity_reference_size programs from its island (vs the baseline if the island is near-empty). Higher = more novel.
  • Normalize each axis with running ~5th/95th percentiles (not raw min/max, so one outlier can't collapse the range): scaled = clamp01((v − p5)/(p95 − p5)); bin = min(num_bins−1, int(scaled × num_bins)); cell = (complexity_bin, diversity_bin). Re-bin existing elites when a percentile shifts enough to move an edge (keep the higher <metric> on collisions; the archive is small).

The Mutator's prompt (the sampler): parent code + inspirations + the parent's rendered artifacts (<metric>, per-class accuracy, loss curve, stderr) + the instruction to return one SEARCH/REPLACE diff. Single harness model — no LLM ensemble. The Mutator applies its diff in the child dir, cascade-evaluates at the FIXED <budget> (the controller injects/caps the duration key on the run command), and returns a result validated against schemas/result.schema.json:

{"child_id": "g3-i1-a2", "parent_id": "g1-i1-a0", "approach_summary": "add BatchNorm after conv2",
 "sandbox_path": "<sandbox_root>/lae/programs/g3-i1-a2", "status": "evaluated",
 "smoke_metric": 0.61, "metric": 0.71}

status ∈ {evaluated, smoke_dropped, crash}; metric is null unless evaluated. Mutators compute nothing about the archive — the controller derives every niche from the sandbox.

Program sandboxes. A parallel population doesn't map onto branches, so every program is a self-contained, fully-runnable dir <sandbox_root>/lae/programs/<child_id>/; the archive references it by id. Build each child dir by copying real files (the parent's <editable_files>, then apply the diff, plus the harness/entrypoint code it imports) and evaluate from inside it (cd <child_dir> && <entrypoint>). Symlink only large read-only data, never the entrypoint or any imported .py: Python resolves a symlinked script's __file__ to the link target, so sys.path[0] becomes the original dir and the child's model.py/dataset.py are silently shadowed by the baselines — every architecture/data mutation becomes a no-op (tell-tale: identical loss curves across different "architectures"). Isolation sanity gate: the harness logs the param count / a code fingerprint; flag any child whose code changed but whose metric/loss curve is identical to its parent's (shadowed), and fix the sandbox before placing it. The repo working tree is never mutated.

Final synthesis. Report the global-best program + its lae/programs/<id>/ path, the illuminated complexity×diversity map (coverage + who won each region), per-island bests, and 2–3 notably diverse runners-up.

Ledger

All three logs live under <sandbox_root>/lae/, tab-separated, never commas in free text. The controller is the sole writer; resume from archive.tsv + history.tsv if interrupted.

archive.tsv — current elites + checkpoint. Header island cell metric child_id parent_id sandbox_path complexity diversity:

island	cell	metric	child_id	parent_id	sandbox_path	complexity	diversity
0	(2,7)	0.7100	g4-i0-a1	g2-i0-a3	lae/programs/g4-i0-a1	2.1M	0.71

history.tsv — every child, append-only. Header gen island parent_id child_id smoke_metric full_metric status kept cell:

gen	island	parent_id	child_id	smoke_metric	full_metric	status	kept	cell
4	0	g2-i0-a3	g4-i0-a1	0.61	0.71	evaluated	y	(2,7)
4	1	g2-i1-a0	g4-i1-a2	0.40	-	smoke_dropped	n	-

leaderboard.md — re-rendered each generation: global best + per-island coverage + the archive ranked by <metric>. Report the best program at stop (not the last), the archive coverage, and a few diverse runners-up. Leave lae/ untracked.

Constraints

  • A child works only inside its own lae/programs/<child_id>/ dir — it may edit the copied <editable_files> and create new files there, but never modify any file outside it (the repo, the read-only harness, the data, other programs' dirs are ground truth or shared state).
  • The controller is the sole writer of archive.tsv/history.tsv/leaderboard.md, so parallel Mutators never race on the logs.
  • <concurrency> comes from the probe + the user's confirmation — never assume the box; pin one child per GPU on multi-GPU; if a run OOMs/thrashes, lower C and say so (don't rewrite a child's config to fit), because the box's limit is real and rewriting the child corrupts the comparison.
  • <budget> (epochs/time), <metric>, and the eval/test split are FIXED and out-of-bounds for mutation. The controller injects <budget> on every run, overriding any duration the child set — so "train longer" / change-the-metric / change-the-test-set can never win. Evolve the model/optimizer/data pipeline, not the compute or the scoring; comparability across programs depends on it.
  • Never symlink the entrypoint or any imported .py into a child dir (it shadows the child's code via sys.path[0]); copy harness code, symlink only data, and run the isolation sanity gate before placing a child — a shadowed result is a phantom.
  • Do not install new packages or modify the evaluation harness — <metric> is ground truth.
  • Do not pause to ask "should I continue?" Run until <total_budget> (reserving a little for synthesis) or interrupt; if coverage stalls bias toward empty cells, if quality stalls exploit top elites. A child that overruns its gate is killed and recorded as crash.

Anomaly Investigation Loop

A form → test → eliminate → confirm loop — root-cause analysis as a search. The artifact is an investigation log; the feedback signal is the count of live candidate explanations, driven down toward a single cause that is confirmed, not merely consistent. Each iteration you test one candidate against the data and drop the ones the data refutes, narrowing the field until one survives.

The discipline this enforces: a cause is "root" only when it both survives an honest attempt to refute it and makes a positive prediction that checks out (e.g. "if this is the cause, removing it restores normal" — and it does). A story that merely could explain the anomaly is a hypothesis, not a finding.

When to use

Use this when an anomaly is already in hand — you know roughly what looks wrong and want the cause diagnosed by elimination against the data. Default to a broad initial slate of mutually distinguishable causes, then test the one that splits the field fastest; if the anomaly is vague, your first job is to make it precise (iteration 0). Not for open-ended exploration of a dataset with no anomaly to chase (use data-analysis), and not for verifying an external claim against the literature (use claim-verify).

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

bindingmeaningdefaulthow to infer
<dataset>data (or logs) to investigate; read-only ground truth—scan the working dir for a data/log file
<anomaly>what looks wrong: the metric, where/when, and how big the deviation is—ask the user; make precise in iter 0
<analysis_cmd>interpreter that runs analysis snippets in the user's envpython3pyproject.toml/.venv/uv in the working dir
<log>output investigation log<sandbox_root>/investigation.md—
<sandbox_root>where snippets + ledger live./sandbox—
<budget>max iterations8—

Analysis snippets run in the user's environment via <analysis_cmd>, so they may use whatever the user has installed. Keep helper code stdlib-first (csv, statistics): if a snippet needs pandas/numpy, probe with try/except ImportError and degrade to a stdlib path, or offer a consented uv pip install "pandas==<ver>" — never assume the package is installed.

The loop

Copy this checklist and tick items off:

  • Iteration 0 — characterize the anomaly precisely; form an initial slate of candidate causes in <log>.
  • Pick a candidate to test (the one whose test most cleanly splits the remaining field).
  • Test it: write <sandbox_root>/iter<N>/test.py, run with <analysis_cmd>, redirect to out.txt.
  • Eliminate (data refutes it → drop from the live set) or advance (data supports it → keep it live).
  • Confirm the survivor: when one candidate leads, run a positive test only it predicts, after a refutation attempt.
  • Append a ledger row; stop when one cause is confirmed or at <budget>.

Iteration 0 — characterize. Quantify the anomaly precisely: write and run a snippet that pins down what deviated, where/when, and how big the deviation is against the normal baseline (the same metric on surrounding periods/segments). Then form an initial slate of candidate causes — mutually distinguishable explanations, broad enough to contain the truth (a real change, a composition/mix shift, a data-quality bug, a measurement change, seasonality, an outlier segment). List them in <log> as the live candidates. Record nothing as confirmed yet.

Then, until stop (one confirmed cause, or budget):

  1. Pick a candidate to test. Ideally the one whose test most cleanly splits the remaining field, so each iteration removes as many live candidates as possible.
  2. Test it. Write <sandbox_root>/iter<N>/test.py that computes the thing that would refute or support it (slice by segment/source/time, recompute the metric, compare distributions). Run it with <analysis_cmd>, redirecting output to <sandbox_root>/iter<N>/out.txt (never flood your context).
  3. Eliminate or advance.
    • Data refutes it → mark it eliminated in <log> with the evidence; drop it from the live set.
    • Data supports it → keep it live, and if it is now the leading candidate run a confirming test: a positive prediction it uniquely makes (e.g. "remove / seasonally-adjust the suspected factor → the anomaly disappears"). Also try to refute it — a leading candidate that survives a genuine refutation attempt and passes its confirming test is the root cause.
  4. Log one ledger row and continue, narrowing the live set.

Observational equivalence. Two mechanistically different candidates can make identical predictions in the data you have (e.g. a bot flood and a pipeline double-count both look like "sessions spike, conversions flat" in daily aggregates). When that happens you cannot separate them here — do not pick one arbitrarily. Report them as a single confirmed cause at the resolution of the available data, and name the additional data that would distinguish them (finer-grained logs, raw event records, an upstream check). Distinguish, too, the mechanism (how the metric moved) from the root cause (why the inputs were wrong) — confirming the mechanism is progress, but is not the cause.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header:

iter	candidate_tested	verdict	live_candidates

verdict ∈ {characterize, refuted, supported, confirmed}. Example:

iter	candidate_tested	verdict	live_candidates
0	characterize anomaly + slate	characterize	5
1	real drop across all segments	refuted	4
2	one segment's conversions fell	refuted	3
3	one source's sessions inflated	supported	2
4	removing that source restores normal	confirmed	1

Report the confirmed root cause with its confirming evidence, the alternatives and how each was ruled out, and — if you stop without a single confirmed cause — the remaining live candidates and the test that would separate them.

Constraints

  • Confirm, don't just fit. The root cause must survive an honest refutation attempt and pass a positive confirming test; "consistent with the data" is not enough, since several stories usually are.
  • Test against the data, not intuition — every elimination and the final confirmation is backed by a computation you ran, recorded in <log>.
  • Keep candidates distinguishable and prefer the test that splits the field fastest, so the live count falls; do not chase one pet theory while leaving alternatives untested.
  • Only read <dataset> — never modify it, because it is the ground truth every test is checked against. The sandbox is self-contained (no ../ escapes).
  • Do not pause the loop to ask whether to continue; run until a cause is confirmed or the budget is hit.

Stops

  • Confirmed — exactly one candidate survived refutation and passed a positive confirming test.
  • Budget — <budget> iterations reached without a single confirmed cause; report the live set.

Blue Team

A defensive fixer loop — the inverse of red-team. The artifact is the target, now writable; the feedback signal is two-part, like optimize-loop: a gate that must hold (nothing that passed before regresses) and a metric that must drop (the count of open failure classes, toward zero). You point it at a set of concrete failed cases and fix them one root-cause class at a time. Each iteration you patch one class, then run tools/verify.py, and keep the patch only if it closes the class with no regression, else revert. You loop until every class is closed (dry) or the budget runs out, then hand the patch set off as a pull request. This is the fix half of a find→fix setup (see Pairing).

The failed cases come from a real source; tools/ingest.py normalizes any of them into one catalogue:

  • oracle mode — a red-team failures.jsonl: each case is an input where the target's verdict disagrees with a ground-truth oracle. A case is closed when target and oracle now agree; a regression is a benign <holdout> input that newly disagrees (most often a new over-block).
  • tests mode — a CI/CD test-failure report (pytest --junitxml / JUnit XML, or a list of failing node ids): each case is a failing test. A case is closed when its test now passes; a regression is any other test that was passing and now fails.

When to use

Use to fix a concrete set of failing cases in code or a guardrail/classifier/filter/prompt/API the user owns — a red-team catalogue, or the failing tests from a CI run — driving the open-class count to zero without breaking what worked. A class is the root-cause group the loop closes as a unit (a red-team technique, or a CI failure area / test class).

Default: pick the mode that matches the source (oracle for red-team, tests for CI/CD). Escape hatch: in oracle mode with no separate functional test suite, the <holdout> alone is the regression guard; in tests mode the suite's own previously-passing tests are the guard. Not for finding new failures (run red-team), and not for editing the ground truth (the oracle, the tests, or the holdout).

Setup

Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write loop.run.yaml and confirm before creating any other files. Two worked configs: examples/run.example.yaml (oracle mode) and examples/tests.run.yaml (tests mode).

bindingmeaningdefaulthow to infer
<source>where the failed cases come from: oracle (red-team) or tests (CI/CD)—red-team failures.jsonl → oracle; failing pytest/JUnit → tests
<target_files>the file(s) the loop may edit to fix the target—the source/guardrail/classifier behind the failures
<catalogue>the failed cases to close, JSONL; build it with tools/ingest.py (see below)<sandbox_root>/catalogue.jsonlred-team's <failures_log>, or a JUnit report
<oracle_cmd>(oracle mode) ground-truth verdict (frozen), same stdin→verdict contract as red-team—a reference checker / policy impl
<holdout>(oracle mode) benign inputs that must keep passing (regression guard)<sandbox_root>/holdout.jsonlknown-good inputs the oracle agrees on
<test_cmd>(tests mode) runs the suite and writes a JUnit XML; regressions read from it—pytest --junitxml=<junit> (or any runner that emits JUnit)
<junit>(tests mode) path to the JUnit XML <test_cmd> writes<sandbox_root>/junit.xml—
<iter_strategy>branches (one commit per kept fix → feeds the PR) or snapshots (folder per iter)branchesdirty / non-git tree → snapshots
<pr_branch>branch the fixes land on and the PR opens fromblue-team/<tag>today's date as <tag>
<sandbox_root>where snapshots + the ledger live./sandbox—
<budget>max iterations8—
<patience>give up on one class after N failed attempts → mark it a residual3—

<skill_dir> is this skill's installed folder; substitute the real path when writing loop.run.yaml.

Build the catalogue first with tools/ingest.py, which normalizes any source into {id, ..., class}:

python3 <skill_dir>/tools/ingest.py --from red-team --in <failures.jsonl> --out <catalogue>   # oracle mode
python3 <skill_dir>/tools/ingest.py --from junit    --in <report.xml>     --out <catalogue>   # tests mode

The signal each iteration is tools/verify.py, in the mode matching <source>:

# oracle mode — <target_cmd> runs <target_files>, e.g. "python3 ./guardrail.py"
python3 <skill_dir>/tools/verify.py --target "<target_cmd>" --oracle "<oracle_cmd>" \
  --catalogue <catalogue> --holdout <holdout>
# tests mode — <test_cmd> writes the JUnit report verify.py then reads
python3 <skill_dir>/tools/verify.py --test-cmd "<test_cmd>" --junit <junit> --catalogue <catalogue>

Either way it prints one JSON object: {mode, open_classes, closed_classes, open_count, closed_count, regressions, regression_count, still_failing}.

The loop

Copy this checklist and tick items off:

  • Iteration 0 — baseline: run tools/verify.py in the <source> mode; record the open classes (should match the catalogue) as the current state and confirm regression_count is 0 — if it is not, the catalogue or holdout is dirty, so fix that before fixing the target. Log the baseline row. Save a pristine copy of <target_files> to <sandbox_root>/iter0/ (snapshots mode) or note the branch base (branches mode) — this is the baseline the final handoff diffs against, and also the snapshot iteration 1 reverts to.
  • In branches mode, open the run on a fresh branch: git checkout -b <pr_branch>.
  • For iteration N (≥1): snapshot the current, pre-patch <target_files> to iter<N>/ (or note the git HEAD) before editing, so a discard can restore exactly this state.
  • Pick one open class. Read its still_failing examples + the suggested fix from the catalogue, and patch <target_files> at the root cause — one fix should close all payloads of that class (e.g. normalize case once, not per-keyword). One class per iteration so each delta is attributable.
  • Check the signal: run tools/verify.py. Discard — restore the snapshot / git reset --hard — if regression_count > 0 (the gate) or the targeted class is still open. verify.py's regression check is the gate: in oracle mode a regression is a newly-broken <holdout> case (e.g. a fix that closes a bypass by over-blocking benign inputs); in tests mode it is any previously-passing test the patch broke.
  • Keep if there are no regressions and open_count strictly dropped. In branches mode commit it: git commit -am "close <class>: <one-line fix>". Append a ledger row.
  • If a class resists <patience> attempts, mark it an open residual and move on rather than thrashing. Stop when open_count = 0 (dry), at <budget>, or when every remaining class is a residual. <budget> counts attempts (each keep or discard is one iteration), not classes closed — a discard still consumes the budget.

On stop, restore the working files to the best iteration (most classes closed, zero regressions) and report: classes closed vs residual, the failures resolved (oracle mode: the bypass/over-block split), and regressions avoided. Then open the pull request (see Handoff).

Fix toolkit. In tests mode the patches are ordinary bug-fixes, grouped by failure area and applied one area per iteration. In oracle mode (hardening a guardrail/filter), reach for these root-cause patterns, mirroring red-team's attack toolkit:

  • Normalize before matching — case-fold, de-leet (homoglyph/leet → letters), strip spacing and punctuation, NFKC-normalize unicode. One normalization step closes case / leetspeak / spacing classes.
  • Broaden the policy — add missing synonyms/expansions to the blocked set (the missing-synonym class), keyed to the oracle's categories, not ad-hoc strings.
  • Tighten over-broad rules — scope a match to whole words / the right context so benign inputs stop tripping it (the overblock class), the most common source of regressions.

Mind the interaction order (both modes): make the narrowing/over-broad fix before a sweeping one. A fix that strips separators (closing spacing) can re-collapse a benign input into an over-broad substring and silently reopen an overblock class — and likewise a broad code change can reopen a test a narrower fix had to protect. Fix the narrow/over-broad case first, then generalize.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the description. regr = regression_count this iteration (the gate: 0 is clean); status ∈ {keep,discard,baseline,residual}. Header iter class_targeted regr open_classes status description:

iter	class_targeted	regr	open_classes	status	description
0	-	0	5	baseline	catalogue: 5 open classes
1	case-bypass	0	4	keep	case-fold the input before matching
2	leetspeak	1	4	discard	de-leet regex also over-blocked a holdout input (regression)
3	leetspeak	0	3	keep	de-leet via translate table, holdout clean
4	missing-synonym	0	2	keep	add passphrase/credentials/api-key to the policy set
5	overblock	0	1	keep	require whole-word "secret key", not bare "secret"
6	spacing	0	0	keep	strip non-alphanumerics before matching — dry

Report the best iteration (open_classes lowest with regr 0), not necessarily the last.

Constraints

  • Only edit <target_files>. The ground truth — the oracle + <holdout> (oracle mode) or the test suite (tests mode) — and tools/verify.py are frozen; editing what measures the fix manufactures a pass (same rule as red-team and optimize-loop). If the oracle or a test is itself wrong, that is a finding to report, not something to patch here.
  • Fix the root cause, not the payload. One fix should close every item of a class; patching a single example string while siblings still fail means the class is not closed. This mirrors red-team's class accounting, so the two loops agree on what "closed" means.
  • The regression gate is non-negotiable. A patch that breaks something that passed before — a new over-block/bypass (oracle mode) or a previously-passing test (tests mode) — is a regression, not progress; revert it regardless of how many classes it closes. Prefer a narrower fix over a sweeping one.
  • One class per iteration, so each delta is attributable and a bad fix is cheap to revert.
  • Keep changes within the project's existing dependency set (stdlib is fine); stay inside the repo and <sandbox_root>; do not pause the loop to ask whether to continue — run until dry, budget, or residuals.

Handoff: the pull request

The deliverable is a human-reviewable pull request — the communication interface to whoever owns the target (a human keeps final approval, as with Copilot Autofix). On stop, with the working tree at the best iteration:

  • In branches mode each kept fix is already a commit on <pr_branch> whose message names the class + the one-line fix. Push and open the PR with gh pr create, body = the ledger (catalogue → fixes, classes closed, residuals listed, regressions avoided) and one reproducible example per closed class.
  • Confirm once before opening the PR — pushing a branch and creating a PR is outward-facing; never auto-push silently.
  • Degrade, don't fail: if there is no remote or gh is unavailable/unauthed, still produce the handoff — leave the commits on <pr_branch>, write git format-patch output and a PR_BODY.md into <sandbox_root>, and tell the user the single command to open the PR themselves. In snapshots mode (no git), emit a unified diff of <target_files> against the iter0/ baseline copy, plus PR_BODY.md, instead.

Pairing

This skill is the fixer — the back half of a find→fix loop. It assumes a catalogue already exists; by itself it closes classes but does not search for new ones.

  1. Find — red-team runs against the frozen target → failures.jsonl (distinct classes).
  2. Fix (this loop) — patch the target to close those classes under the gate, between red-team runs, never inside one.
  3. Re-verify — a fresh red-team run against the patched target confirms each class is closed and surfaces any new class the fix introduced (the regression gate already guards the over-block direction within this loop).

Keep the two agents independent — the attacker that wrote the catalogue should not grade its own patch. The purple-team loop orchestrates the full find → fix → re-verify cycle until a fresh find stays dry; this skill deliberately covers only the fix phase.

Claim Verify Loop

A claim-by-claim adversarial verification loop over a results draft. The artifact is the draft; the feedback signal is the count of unverified claims — claims not yet checked, or checked but not yet survived a stress test. You drive it to zero: each claim ends verified (reproduces and survives the obvious threats) or appropriately qualified (hedged, scoped, or retracted with the reason).

The discipline: a number that merely reproduces is not trustworthy — most wrong findings reproduce fine. A claim is verified only when it also survives the threat most likely to kill it: an outlier, a confound, a subgroup too small to mean anything, a sign that flips under stratification. This loop is a gate on an existing draft, not a generator of new findings.

When to use

Use this when you have a draft (or a list of claims) drawn from a dataset and want each claim red-teamed before it goes out. Default to verifying every discrete claim in the draft; if the user flags a few high-stakes claims, prioritize those but still sweep the rest. Not for open-ended discovery of new findings (that is the data-analysis loop) and not for diagnosing one known anomaly.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

bindingmeaningdefaulthow to infer
<draft>results/claims document to verify (markdown/text)—scan the working dir for a results/report file
<dataset>data the claims were drawn from; read-only ground truth—scan the working dir for the data file
<analysis_cmd>interpreter that runs check snippets in the user's envpython3pyproject.toml/.venv/uv in the working dir
<report>the verified/revised draft this loop produces<sandbox_root>/verified.md—
<sandbox_root>where check snippets + ledger live./sandbox—
<budget>max iterations10—

Check snippets run in the user's environment via <analysis_cmd>, so they may use whatever the user has installed. Keep helper code stdlib-first (csv, statistics): if a snippet needs pandas/numpy, probe with try/except ImportError and degrade to a stdlib path, or offer a consented uv pip install "pandas==<ver>" — never assume the package is installed.

The loop

Copy this checklist and tick items off:

  • Iteration 0 — extract the discrete checkable claims from <draft>; record nothing as verified.
  • Pick one unverified claim.
  • Reproduce its exact number from <dataset>; if it does not reproduce → refuted.
  • Stress-test against the threat(s) most likely to kill it; classify verified / fragile.
  • Revise <report>: keep verified, hedge/scope/retract fragile, correct refuted.
  • Append a ledger row; the claim leaves the unverified set. Stop when none remain or at <budget>.

Iteration 0 — extract claims. Read <draft> and list its discrete, checkable claims, each with the number/effect it asserts and its claim type (a group difference, a correlation, a causal/policy claim, a subgroup result, a rate). These are the live unverified set. If the draft is prose, splitting it into discrete claims is the first job.

Then, until stop (all claims resolved, or budget):

  1. Pick one unverified claim.

  2. Reproduce — the first gate. Write <sandbox_root>/iter<N>/check.py to recompute the exact statistic the claim states from <dataset>. Run it with <analysis_cmd>, redirecting output to <sandbox_root>/iter<N>/out.txt (never flood your context). If the number does not reproduce → refuted (the number is wrong); skip to step 4.

  3. Stress-test — the second gate. Hit the claim with the one or two threats most likely to kill it for its claim type:

    • Outlier sensitivity — recompute dropping extreme points / using a robust statistic. Does the effect survive, or was it driven by a handful of rows?
    • Confound & Simpson's reversal — stratify by the obvious confounder; does the effect hold within strata, or flip? A causal/policy claim that reverses within subgroups is not supported.
    • Subgroup size & multiplicity — how large is the subgroup? Is the result one of many comparisons? A striking rate on n=5 is noise.
    • Alternative specification — a defensible different cut (different bins, controlling for a covariate). Does the sign/size stay?

    Classify: verified (reproduces and survives) or fragile (reproduces but collapses or flips under a reasonable stress). A claim whose number reproduces but whose implied interpretation is not supported — a descriptive gap dressed up as causal ("treatment works"), a tiny-n rate sold as "superior", a one-point correlation called an "early-warning signal" — is fragile, not verified: the statistic is fine, the conclusion drawn from it is not.

  4. Revise the draft. Update <report>:

    • verified → keep the claim, noting the robustness check it passed.
    • fragile → hedge, scope, or down-weight it to what the data supports (e.g. "descriptively higher, but the within-stratum comparison reverses — not evidence the treatment causes recovery"), or retract it. Never leave a fragile claim standing as first written.
    • refuted → correct it with the right number, or remove it. Record the verdict and the evidence.
  5. Log one ledger row and continue; the claim leaves the unverified set.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header:

iter	claim	verdict	threat	resolution

verdict ∈ {extract, verified, fragile, refuted}. Example:

iter	claim	verdict	threat	resolution
0	claims extracted	extract	-	7 claims listed
1	treatment recovery rate > control (70.6 vs 55.0)	verified	reproduced; holds	kept
2	treatment causes higher recovery (+16pp)	fragile	Simpson: control >= treatment within both age groups	rescoped to descriptive; causal claim retracted
3	biomarker correlates with recovery_days (r=0.16)	fragile	one outlier drives it (r=0.16 -> 0.02 without it)	retracted
5	pilot site 100% recovery (superior)	fragile	n=5 subgroup	hedged: too small to conclude

Report the outcome: the <report> path, the per-claim verdicts, and a summary — how many claims were verified, hedged, or retracted, and the single most important fragility found.

Constraints

  • Reproduce and stress-test — both. A claim that only reproduces is not verified; it must survive the threat most likely to kill it. Skipping the stress test is the failure mode this loop exists to prevent.
  • Every verdict is backed by a re-run recorded in <report>; no claim is waved through or condemned on intuition.
  • A fragile claim is changed, never left standing — hedge it to what the data supports or retract it, because leaving it as first written is exactly what shipped the unverified draft.
  • Distinguish description from causation — "treatment arm recovered more" can be true while "treatment causes recovery" is refuted by a confound; say exactly what the data supports.
  • Only read <dataset> — never modify it, because it is the ground truth every claim is checked against. The sandbox is self-contained (no ../ escapes).
  • One claim per iteration, so each verdict is attributable.
  • Do not pause the loop to ask whether to continue; run until all claims are resolved or <budget>.

Stops

  • Resolved — no unverified or unresolved-fragile claims remain.
  • Budget — <budget> iterations reached.

Data Analysis Loop

A hypothesis → verify reflection loop over a dataset. The artifact is a findings report; the feedback signal is verification — a finding only counts if re-running the computation confirms it at a meaningful effect size. The discipline this enforces: no insight without a number behind it. A plausible claim the data does not support is discarded, not softened; every line in the report can be reproduced from the dataset.

When to use

Use this for open-ended, self-checking exploration of a bound dataset where each finding must survive an independent re-computation. Default to broad exploration across the columns; if the user gives a focus question, let it steer the hypotheses. Not for diagnosing one known anomaly or for checking an external claim against the literature.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

bindingmeaningdefaulthow to infer
<dataset>data file to analyze (CSV/TSV/Parquet/…); read-only ground truth—scan the working dir for a data file
<question>optional analysis focus; omit to explore broadly—ask the user; else leave unbound
<report>output findings file<sandbox_root>/findings.md—
<analysis_cmd>interpreter that runs analysis snippets in the user's envpython3pyproject.toml/.venv/uv in the working dir
<sandbox_root>where snippets + ledger live./sandbox—
<budget>max iterations8—
<patience>stop after N consecutive iters with no new verified finding2—

Analysis snippets run in the user's environment via <analysis_cmd>, so they may use whatever the user has installed. Keep helper code stdlib-first (csv, statistics): if a snippet needs pandas/numpy, probe with try/except ImportError and degrade to a stdlib path, or offer a consented uv pip install "pandas==<ver>" — never assume the package is installed.

The loop

Copy this checklist and tick items off:

  • Iteration 0 — profile <dataset> (shape, types, ranges, missingness); record nothing as a finding.
  • Propose one specific, checkable hypothesis (steered by <question>; not already settled).
  • Compute it: write <sandbox_root>/iter<N>/analysis.py, run with <analysis_cmd>, redirect to out.txt.
  • Verify: re-derive the key number a second way; judge against a stated effect-size bar.
  • Supported → append finding to <report> (verified); else log refuted, do not add it.
  • Append a ledger row; stop on plateau (<patience>) or <budget>.

Iteration 0 — profile. Write and run a snippet that reports the shape of <dataset>: columns, inferred types, row count, and a quick summary (ranges, category counts, missingness). This grounds the hypotheses; record nothing as a finding yet.

Then, until stop (dry or budget):

  1. Propose one hypothesis. A single, specific, checkable claim — e.g. "enterprise orders average higher value than consumer", "mobile has a higher return rate than other channels", "order value rises with signup tenure". Let <question> steer it; do not repeat a hypothesis already settled.
  2. Compute it. Write <sandbox_root>/iter<N>/analysis.py that loads <dataset> and computes the relevant statistic plus an effect size (a group-mean difference, a rate gap, a correlation — not just a yes/no). Run it with <analysis_cmd>, redirecting output to <sandbox_root>/iter<N>/out.txt (never flood your context).
  3. Verify — the gate. Re-derive the key number a second, independent way (a different grouping, a recount, or a sanity cross-check) and confirm the two agree. Then judge honestly: does the result support the hypothesis at a meaningful effect size, or is it negligible / within noise? Decide "meaningful" against a bar you state up front and apply consistently — a minimum effect size scaled to the group sizes and noise (e.g. roughly |Cohen's d| ≳ 0.2, risk ratio ≳ 1.5, or |r| ≳ 0.1, tightened when groups are small) — so the keep/refute threshold does not drift between iterations.
    • Supported → append a finding to <report>: the claim, the exact numbers, the effect size, and the method (so it is reproducible). Mark it verified.
    • Not supported / negligible → record it as refuted in the ledger and do not add it to the report. A null result is a real outcome, not a failure to hide.
  4. Log one ledger row and continue.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header:

iter	hypothesis	effect	status

status ∈ {profile, verified, refuted}. Example:

iter	hypothesis	effect	status
0	dataset profile	-	profile
1	enterprise orders average higher value than consumer	185 vs 109 (+70%)	verified
2	returns differ by region	North 0.16 vs South 0.14 (negligible)	refuted
3	mobile has a higher return rate than web/store	0.30 vs 0.10	verified

Report the best outcome: the <report> path, the count of verified findings, and the hypotheses refuted (so the user sees what was checked and ruled out, not just what survived).

Constraints

  • No claim without a computed number. Every finding in <report> carries the figures and the method that produced it; if you cannot compute it, you cannot claim it.
  • Verify before recording. The independent re-derivation in step 3 is the gate — a finding that does not reproduce, or whose effect is within noise, does not enter the report.
  • Report effect sizes, not just direction, and do not inflate a correlation into a causal claim — say "associated with", and note confounders when the data cannot separate them.
  • One hypothesis per iteration, so each finding is attributable, and skip hypotheses already settled.
  • Only read <dataset> — never modify it, because it is the ground truth every finding is checked against. The sandbox is self-contained (no ../ escapes).
  • Do not pause the loop to ask whether to continue; run until it goes dry or hits the budget.

Stops

  • Dry — <patience> consecutive iterations add no new verified finding.
  • Budget — <budget> iterations reached.

Dueling Autoresearch Loop

Two lanes work the same objective in parallel and race the same metric — by default a classical/algorithmic lane against an ML/learned lane (the lanes are user-named). Each lane runs its own analysis-first iteration via roles/TrackAgent.md, confined to its lane. Every round both lanes post to a shared duel_log.md scoreboard and may borrow ideas across the lane boundary — but each stays in its lane. The feedback signal is the shared <metric> on a shared eval: if the classical lane wins, that is a real result. Lanes support mixed code locations — a codebase lane edits existing repo files, a sandbox lane authors its own code — and an eval-parity gate keeps the scores comparable.

You are the orchestrator: each round you advance both lanes, update the scoreboard, and keep both honest. Do not pause for permission once the loop is running.

When to use

Use this to race two genuinely different approaches on one metric and keep them honest against the same eval — classical vs learned, two model families, two query strategies. Default to spawning both lanes in parallel and letting the scoreboard drive cross-lane idea borrowing; if a lane runs dry, push it to a more radical in-lane change or to borrow a fresh idea from the log. Not for tuning a single approach (use a single-track loop), and not for a one-shot comparison of two finished things.

The cast (all in this folder):

  • roles/TrackAgent.md — the per-lane researcher, instantiated once per lane.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

The host also decides spawn-or-degrade: on Claude Code spawn a real Agent per lane so the two run in parallel; otherwise adopt roles/TrackAgent.md inline and run the lanes sequentially.

Shared bindings (identical for both lanes — the honesty anchor):

bindingmeaningdefaulthow to infer
<metric>the single metric both lanes race; ground truth of the duel—ask; scan run logs for a printed score
<metric_direction>minimize or maximize—from the metric's nature (loss vs accuracy)
<gate>run budget unit: time or epochsepochsthe artifact's runner
<budget>epochs per run (or minutes if gate: time)5—
<sandbox_root>where snapshots, ledgers, and the duel log live./sandbox—
<iter_strategy>snapshots or branches (snapshots recommended — two lanes on one branch is simplest)snapshots—

Per-lane bindings (two lanes, default names classical and learned). Each lane has a code_location that decides which other fields it needs — never add code to the codebase:

fieldmeaningwhen
namelane namealways
code_locationcodebase or sandboxalways
run_cmdexisting entrypoint to run, e.g. python train.pyif codebase
editable_filesexisting repo files this lane may editif codebase
entrycommand run from inside <sandbox_root>/<lane>/iter<N>/, e.g. python run.pyif sandbox
  • codebase — the lane maps to existing code: edits its editable_files and runs run_cmd. Two codebase lanes must have non-overlapping editable_files.
  • sandbox — no implementation exists and none is added to the repo: the lane authors and runs its code inside <sandbox_root>/<lane>/iter<N>/ via entry.

Typical duel on a repo with one existing model: the learned lane is codebase (edits model.py/config.yaml, runs train.py); the classical lane is sandbox (authors its own code under <sandbox_root>/classical/iter<N>/). Nothing is added to the codebase, yet the classical lane is still built and iterated.

Eval-parity gate (the honesty anchor). Before starting, confirm both lanes report <metric> on the same held-out set, computed the same way, so the scores are comparable — state how each lane emits it (e.g. both print <metric>: to their run log). If they don't match, fix it first; the duel is meaningless otherwise. If gate: time, write a run_with_timeout.sh wrapper per lane (timeout $(( <budget> * 60 )) <entry-or-run_cmd> "$@").

Initialise the sandbox (after confirmation):

<sandbox_root>/
├── duel_log.md          ← shared channel + scoreboard (## Scoreboard, ## Round log; headers only)
├── <laneA>/results.tsv  ← lane A ledger, header only
└── <laneB>/results.tsv  ← lane B ledger, header only

Each lane's per-iteration work lives in <sandbox_root>/<lane>/iter<N>/ (analysis/, results/, the run log). A codebase lane's iter dir also holds code_snapshot/ (the pre-change copy for revert); a sandbox lane's iter dir holds the lane's actual code for that iteration (a kept iteration carries forward as the next one's starting point).

The loop (duel)

Each round advances both lanes by one iteration. On Claude Code, spawn the two TrackAgents in parallel (one turn, two Agent calls); otherwise run lane A then lane B inline. A track is one analysis-first iteration confined to its lane — the 8 steps in roles/TrackAgent.md. Round 1 is each lane's baseline (a codebase lane runs unmodified; a sandbox lane authors its initial implementation in iter1/). One change per lane per round, so each metric delta is attributable.

Copy this checklist and tick items off each round:

  • State — note round N; read duel_log.md (both lanes' latest posts + the scoreboard).
  • Advance each lane — run a TrackAgent (roles/TrackAgent.md) per lane, given its lane bindings, the shared <metric>/<metric_direction>/<gate>/<budget>, and duel_log.md.
  • Track posts — each lane appends its round entry to duel_log.md (best <metric>, one key finding, any dead end, one idea the other lane could borrow).
  • Scoreboard — update ## Scoreboard: best <metric> per lane and the current leader (per <metric_direction>); optionally flag one cross-pollination suggestion for next round.
  • Continue — go to the next round; never pause to ask whether to continue.

Spawn-or-degrade per lane. Where the host supports it, spawn a real isolated TrackAgent per lane (Claude Code: an Agent per lane, both launched in one turn for parallelism). Otherwise adopt roles/TrackAgent.md inline and run the lanes sequentially. Each TrackAgent is confined to its lane and returns its iteration summary to the orchestrator.

Ledger

Two ledgers: a per-lane results.tsv for each lane's experiments, and the shared duel_log.md scoreboard + round posts.

Per-lane <sandbox_root>/<lane>/results.tsv (tab-separated, never commas in free text):

iter	<metric>	status	analysis_summary	description
1	0.6320	keep	baseline; classical features, logistic head	baseline
2	0.6610	keep	added HOG features; per-class gains on textured classes	add HOG feature extractor

status ∈ {keep, discard, crash} (0.000000 for <metric> on crash).

Shared <sandbox_root>/duel_log.md — scoreboard + per-round posts:

## Scoreboard
round	classical_best	learned_best	leader
1	0.6320	0.6480	learned
2	0.6610	0.7050	learned

## Round log
### Round 2
- **classical** — best 0.6610 (this iter 0.6610). Finding: HOG helps textured classes
  (results/per_class.txt). Dead end: raw-pixel kNN plateaus. Borrow: learned's augmentation
  could expand classical's training set.
- **learned** — best 0.7050. Finding: BN fixed conv2 saturation. Dead end: dropout hurt at this
  budget. Borrow: classical's HOG features as an aux input channel.

Report the current leader (per <metric_direction>), never a final winner — a lane that is behind can still come back. Leave results.tsv, duel_log.md, and iter*/ untracked (do not commit them).

Constraints

  • Never add code to the codebase. A codebase lane edits only its own <editable_files>; a sandbox lane lives entirely in <sandbox_root>/<lane>/. Lanes never touch each other's files — every other file is the evaluation ground truth.
  • Stay in lane. A lane borrows ideas, never converts into the other approach — a classical lane stays classical even if it borrows a loss/target idea from the learned lane.
  • Same metric, same eval. Both lanes optimize <metric> on the same held-out set computed the same way; never compare otherwise. Do not modify the evaluation/metric — it is the shared ground truth that makes the duel honest.
  • One change per lane per round, so each metric delta is attributable.
  • Do not install new packages or add dependencies the project lacks; helper code stays stdlib-only.
  • Redirect each run's output to its lane's run log; never use tee. The sandbox is self-contained — no ../ escapes.
  • Do not pause the loop to ask for direction; once running, keep both lanes iterating until manually interrupted, and report the current leader rather than declaring a final winner.

Exploratory Autoresearch Loop

This loop runs hot. Like the standard ml-autoresearch, every experiment is followed by a diagnostic analysis pass. Unlike it, the type of change at each iteration is set by a temperature scheduler, not the agent's intuition: it forces wide, diverse swings early (full rewrites, fundamentally different architectures and training regimes), then drops into an adaptive phase that chooses between swing (a fresh wild approach), merge (combine two registered approaches), or exploit (a focused tweak of the best). A stagnation guard bans exploit once it has run <stagnation_limit> times in a row, forcing a pivot back to swing or merge so the loop never gets stuck hill-climbing. The feedback signal is <metric> read from the run log; an approaches.md registry and a move_type per iteration are what make the scheduler work.

You are the researcher. Do not pause to ask for permission once the loop is running.

When to use

Use for an open-ended ML campaign where you want forced breadth before refinement — the scheduler guarantees you sample several distinct families before converging, and the stagnation guard prevents endless small steps. Default to <swing_budget> = 3 and <stagnation_limit> = 3; raise <swing_budget> for wider initial exploration. Not for the standard analysis-first ml-autoresearch (use that when you want the analysis alone to drive each change, with no forced-swing scheduler), not for a single training run or a fixed sweep, and not for tasks with no measurable scalar metric.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available — record <host> = claude-code) infer a likely value for each binding from the project and present it as the recommended option; on other hosts (<host> = other) ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value with the user before creating any other files. For branches strategy, create git checkout -b autoresearch/<run_tag> (tag from today's date; branch must not exist). For time gating, write <sandbox_root>/run_with_timeout.sh (timeout $(( <budget> * 60 )) <entrypoint> "$@") and use it as the run command, hard-killing at 2 × <budget> min; for epochs, patch the epoch cap in an <editable_files> file.

bindingmeaningdefaulthow to infer
<metric> / <metric_direction>scalar to optimize + minimize/maximize—scan editable files + README for metric names
<run_cmd> / <entrypoint>command that runs one experiment end to end—pyproject.toml / .venv / README
<editable_files>files fair game to edit (never the eval harness)—model / config / train scripts; exclude data, logs, env, harness
<sandbox_root>where snapshots + ledgers live./sandboxnext to the editable files
<iter_strategy>snapshots or branchessnapshotsis the working dir a clean git repo?
<gate> / <budget>time (min) or epochs, and the limit—existing time/epoch settings in config
<swing_budget>forced wild swings before adaptive mode3 (3–5)wider = more initial breadth
<stagnation_limit>max consecutive exploits before a forced pivot3—

FILE EDIT GUARD: before touching any file at any point — setup or loop — confirm it is in <editable_files>, because everything else is read-only ground truth (the eval harness defines <metric>). No exceptions.

Initialise the sandbox

Create the layout and write the ledger headers:

<sandbox_root>/
├── loop.run.yaml      ← resolved bindings (written now)
├── results.tsv        ← experiment ledger, header only (written now)
├── approaches.md      ← registry of every distinct approach (header only, written now)
└── iter1/             ← created at loop start

results.tsv header (tab-separated; move_type ∈ {swing, merge, exploit}):

iter	<metric>	status	move_type	analysis_summary	description

approaches.md header: # Approach Registry plus a one-line note that the merge step consults it to find complementary approaches to combine.

The loop (LOOP FOREVER — until interrupted)

Iteration 1 is always the unmodified baseline (it does not count as a swing): skip move-selection and change-planning, but still run the mandatory analysis — it is the first empirical anchor iteration 2 builds on. Everything in <editable_files> is fair game (architecture, optimizer, hyperparameters, data pipeline, loss, init, eval); on swings especially, full rewrites are encouraged. The only constraints are that the code runs and finishes within <budget>. Epoch efficiency is part of the objective — a change that reaches the same score in fewer effective steps is a real win. Simplicity criterion: all else equal, simpler is better — a 0.001 gain that adds 20 lines of hacky code is not worth it; a 0.001 gain (or an equal metric) from deleting code is a keep.

The scheduler keeps two counters in memory across iterations: swings_taken (total swing iterations, excludes the baseline) and consecutive_exploit (exploits since the last swing/merge; resets to 0 on any swing or merge).

Copy this checklist each iteration and tick items off:

  • 1. Look at the state. branches: git log --oneline -5. snapshots: confirm iter<N>/ doesn't exist. Read iter N-1's analysis summary and the two counters.
  • 2. Pick move_type (iteration 1: SKIP — baseline). Apply the scheduler below, then record the move before touching any file.
  • 3. Form the hypothesis (iteration 1: SKIP). State the move and why (cite the rule or the analysis), what you will do, and which <editable_files> it touches. See The three moves.
  • 4. Snapshot / commit, then apply the change. snapshots: create iter<N>/{code_snapshot,analysis,results}/, copy every <editable_files> into code_snapshot/, copy loop.run.yaml to iter<N>/, then apply. branches: apply, then git commit -am "<move_type>: <desc>".
  • 5. Run the experiment, redirecting everything (never tee): <entrypoint> > <sandbox_root>/iter<N>/<run_log> 2>&1 (or run_with_timeout.sh when time-gated). If it overruns, kill it and treat as a crash.
  • 6. Read the metric: grep '^<metric>:' <sandbox_root>/iter<N>/<run_log>. If empty, tail -n 50 <run_log>, read the trace, attempt one trivial fix (typo/import); if fundamentally broken, log crash and continue.
  • 7. Analyse the results — MANDATORY, produces real artifact files. See Analysing.
  • 8. Update approaches.md (swing and merge moves only). See The registry.
  • 9. Log to results.tsv (untracked — never commit). See Ledger.
  • 10. Keep or revert (the change ran this iteration). Improved per <metric_direction> → keep, update current-best. Equal/worse/crash → discard/crash; branches git reset --hard HEAD~1, snapshots restore <editable_files> from iter<N>/code_snapshot/. Apply the simplicity criterion before logging discard. On a crash/OOM, fix with the minimal change that preserves the intent (OOM → smaller batch + grad-accum to hold effective batch) — never mutate the experiment.
  • 11. Update counters (below) and go to step 1.

The scheduler (step 2 — this is the loop's identity)

Follow the rules exactly, in order — they are hard constraints, not suggestions:

IF   iter == 1                                  → baseline   (run unmodified; no move)
ELIF swings_taken < <swing_budget>              → swing      (forced exploration)
ELIF consecutive_exploit >= <stagnation_limit>  → swing OR merge  (forced pivot — exploit BANNED)
ELSE                                            → agent chooses: swing / merge / exploit

On the free ELSE branch, let iter N-1's analysis decide:

  • swing if the current family has a fundamental ceiling — e.g. all top results share a failure mode.
  • merge if two+ approaches.md entries have distinct, non-overlapping strengths (prefer parents that changed different axes — they combine additively rather than interfere).
  • exploit if the current best has obvious analysis-suggested headroom not needing a new architecture.

Counter update (step 11)

if move_type in {swing, merge}:  swings_taken += 1 (swing only); consecutive_exploit = 0
elif move_type == exploit:       consecutive_exploit += 1

The three moves (step 3)

  • Swing — fundamentally different from every previous swing (not a tweak; the diff should look obviously different from the current best). Most people swing on architecture by reflex — fight that. These axes are equally valid and underexplored: architectural family (how information flows, depth vs width, skip connections, local vs global); initialization (magnitude-based, structure-preserving, input-statistics-driven, sparse — different early dynamics); data pipeline (ordering, sampling, determinism, coverage of the view space — not just augmentation flavours); per-component LR decoupling (early/late layers, norms, biases, heads each have their own optimal step); evaluation (single pass, multi-view, checkpoint averaging, calibration); objective (loss shape, target sharpness, auxiliary/consistency signals). A genuine swing explores one of these in a way not yet tried.
  • Merge — select two+ entries from approaches.md and name what is taken from each; the result is a new approach that is not a minor variant of either parent. Prefer components from different axes.
  • Exploit — a targeted, focused change to the current best, grounded in a specific analysis finding. One or two things at a time; decouple the axes (test a new optimizer and a new LR as separate iterations so you know which caused the result). Never a different architecture.

Analysing (step 7 — MANDATORY; produces real artifacts)

This is the spine that feeds the next move. Run whatever analysis most increases your understanding of why this result happened. Every analysis script goes in iter<N>/analysis/; every output (plots, CSVs, text) goes in iter<N>/results/, redirecting stdout there. Do not proceed until the results exist — analysis that wrote no file did not happen. Dimensions to draw from (choose what fits): gradient norms/flow, activation stats/saturation, embeddings (PCA/CKA/collapse), error & confusion analysis, loss dynamics & headroom (was it still improving at cutoff?), weight/parameter stats, data profiling (often the highest-yield), compute profiling.

Write a concise analysis summary (3–8 bullets): what you examined, the single most important finding, and what it implies for the next move (whether it favours swing / merge / exploit).

Ablation discipline (after any keep that touched more than one axis): you don't yet know which part caused the gain — flag it and consider an ablation exploit next, reverting one component at a time. Building on an unablated multi-axis change is building on an unknown.

Forward-looking instrumentation. After analysing, ask "what would I wish I had logged?" — if the producing script is in <editable_files>, add it now (best-epoch checkpoint, per-layer grad norms, per-class accuracy). Richer logs improve every future analysis; adding instrumentation is a valid iteration on its own (log it as move_type=exploit).

The registry (step 8 — approaches.md, swing and merge only)

Append an entry; the axes changed field is what the merge step uses to find complementary parents (two that changed different axes beat two that both changed the architecture):

## Approach <N>: <short name>
- **move_type**: swing | merge
- **iter**: <N>
- **<metric>**: <value>  (status: keep | discard | crash)
- **axes changed**: architecture | initialization | data pipeline | optimizer | lr schedule | evaluation | objective | other
- **key ideas**: <what makes this approach distinct>
- **strengths** / **weaknesses** (from analysis): <what worked / what the analysis says is missing>
- **ablated?**: yes (which components isolated, what was found) / no
- **parents** (merge only): Approach X + Approach Y — axes taken from each

Ledger

<sandbox_root>/results.tsv, tab-separated, never commas in free text. Status ∈ {keep, discard, crash}; use 0.000000 for the metric on crashes. For branches, the first column is the 7-char commit instead of the iter number.

iter	<metric>	status	move_type	analysis_summary	description
1	0.6320	keep	swing	baseline; grad norms clean, no pathologies	baseline
2	0.5910	discard	swing	ResNet blocks; gradient flow good but overfit on 5 epochs	ResNet-style residual blocks
3	0.6890	keep	swing	MLP-Mixer; feature mixing effective, less overfit	MLP-Mixer token+channel mix
4	0.7120	keep	merge	Mixer channel-mix + ResNet skips; best of both	merge: Mixer + ResNet skips
5	0.7250	keep	exploit	train/val gap small; warmup helped stability	add 2-epoch LR warmup
6	0.7240	discard	exploit	no gain from dropout; val acc unchanged	add dropout 0.1
7	0.6800	discard	swing	forced pivot after stagnation guard; ViT too data-hungry	ViT patch-16 from scratch

approaches.md registry — one entry per swing/merge, format above. Example:

# Approach Registry
The merge step consults this file to find complementary approaches to combine.

## Approach 3: MLP-Mixer
- **move_type**: swing
- **iter**: 3
- **val_acc**: 0.6890  (status: keep)
- **axes changed**: architecture
- **key ideas**: token mixing + channel mixing, no convolutions
- **strengths**: less overfit than ResNet, fast per-epoch
- **weaknesses**: spatial locality not exploited; edges/textures underused
- **ablated?**: no

Report the best iteration (not necessarily the last) when summarising. Do not commit results.tsv or approaches.md — leave them untracked.

Constraints

  • Only edit files in <editable_files> — confirm before every edit, because everything else is read-only ground truth and the eval harness defines <metric>.
  • One change per iteration, so each metric delta is attributable to one move.
  • The scheduler rules are hard constraints, not suggestions. When consecutive_exploit >= <stagnation_limit> you MUST choose swing or merge — exploit is unavailable regardless of what the analysis suggests; this guard is what prevents endless hill-climbing.
  • Analysis is mandatory every iteration and must produce real files in iter<N>/results/; a move with no analysis behind it degrades the loop into blind iteration and starves the next decision.
  • A swing must be fundamentally different from every prior swing; a merge must name what it takes from each parent — otherwise the registry and the breadth guarantee mean nothing.
  • Do not install packages or add dependencies the project lacks; helper scripts stay stdlib-only.
  • Always redirect training output to <run_log>; never use tee. The sandbox is self-contained — no ../ escapes.

Stops

This loop runs forever until the human interrupts it — do not pause to ask "should I continue?" or "is this a good stopping point?". The human may be away and expects autonomous work indefinitely. A working result is the start of the next iteration, not the end. If ideas run dry: re-read the in-scope files for missed angles, go deeper on the analysis (gradients/activations/embeddings/errors always surface something), combine previous near-misses from approaches.md, or take a more radical swing.

Hypothesis Generation Loop

A multi-agent, literature-grounded generation loop. The artifact is a growing pool of hypotheses; the feedback signal is the count of strong, distinct hypotheses that clear the bar — where "strong" is decided against real retrieved literature, not assertion. Each round: generate → ground → judge → keep → mutate toward the gaps, until the pool stops growing (saturation).

The discipline: a hypothesis enters the pool only if the literature says it is not already established (novelty), prior work makes it plausible (grounding), and a feasible test exists. Generating is not confirming — the output is a ranked set of strong candidates to test, each stated with how to test it.

The cast (all in roles/):

  • roles/Generator.md — proposes a batch of candidate hypotheses aimed at the open gaps.
  • roles/LiteratureScout.md — grounds each candidate in real literature (novelty · support · gap); emits litscout.json (validates schemas/litscout.schema.json).
  • roles/Judge.md — scores each against the fixed rubric and decides keep/kill/dedupe; emits verdict.json (validates schemas/verdict.schema.json).

Spawn-or-degrade. On Claude Code, spawn Generator / LiteratureScout / Judge as real Agent subagents each round; otherwise adopt each role inline in this context. You are the orchestrator.

When to use

Use when the user wants candidate hypotheses generated and vetted for a question or domain. Default: run the full generate→ground→judge loop below until saturation. Escape hatch: if the user only wants a single batch (no looping), run one round and report the kept hypotheses. Not for sharpening or decomposing a question, and not for grading an existing written proposal.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value plus the live/degraded literature tier before creating any other files.

bindingmeaningdefaulthow to infer
<question>the research question / domain, plus any scope (field, population, constraints)—ask the user
<gen_n>candidate hypotheses the Generator proposes per round6—
<keep_threshold>rubric score (0-100) a hypothesis must clear to enter the pool65—
<eval_scale>LiteratureScout grounding depth (low/medium/high, see below)medium—
<sandbox_root>where rounds, ledger, and lit cache live./sandbox—
<budget>max rounds6—
<patience>stop after this many rounds with no new kept hypothesis2—
<report>final ranked hypothesis set<sandbox_root>/hypotheses.md—

Grounding depth dial (<eval_scale> caps per round — candidates examined deeply · queries each · papers read full-text):

presetcandidates deepqueries eachfulltext reads
low210 (snippet/abstract only)
medium (recommended)all21
highall33

Literature toolchain. Paper search goes through the sibling literature-search skill — resolve <lit_skill_dir> (it installs as a sibling, e.g. ~/.claude/skills/literature-search/), <lit_py> = python3, and <lit> = <lit_skill_dir>/tools/lit_search.py (note the tools/ segment); append --cache-dir <sandbox_root>/literature/.cache after a subcommand to reuse the cache. Confirm <lit> --help works at setup; if the skill is absent, tell the user and either install it (copy the repo's loops/literature-search folder into ~/.claude/skills/) or degrade all retrieval to WebSearch/WebFetch (tag that evidence source:"web"). The keyless S2 + arXiv core needs no setup; a free S2_API_KEY makes snippet/cite reliable.

API key (optional, never block). The literature-search skill owns the key convention: run <lit> keys --init, then have the user fill the printed keys.env themselves and never paste secrets into chat. Re-run <lit> keys to record the tier in loop.run.yaml (literature_tiers, presence only). A missing key just degrades to the keyless pool → WebSearch.

Initialise the sandbox once bindings are confirmed:

<sandbox_root>/
├── loop.run.yaml        ← resolved bindings + literature_tiers
├── ledger.tsv           ← header only (see Ledger)
└── literature/.cache/   ← lit_search on-disk cache

Start with an empty pool and gaps seeded from <question>; create no round files until the loop runs.

The loop

pool = the kept hypotheses (starts empty). gaps = open questions the LiteratureScout has surfaced (starts empty; seed from <question>). dry = consecutive rounds with no new keep (starts 0). <N> starts at 1.

Copy this checklist and tick items off:

  • Generate — run roles/Generator.md (spawn-or-degrade) with <question>, the current pool, gaps, and <gen_n>; it writes round<N>/candidates.json (<gen_n> specific, testable, plausibly-novel candidates aimed at the gaps, none duplicating the pool).
  • Ground — run roles/LiteratureScout.md (spawn-or-degrade) on candidates.json with <lit> and the <eval_scale> caps; it writes round<N>/litscout.json (validates schemas/litscout.schema.json) — per candidate: novelty + closest prior work, support, gap, testability, each citing real evidence.
  • Judge — as roles/Judge.md, apply the fixed rubric + evidence gate to litscout.json, checking each candidate against the pool for duplicates; write round<N>/verdict.json (validates schemas/verdict.schema.json): scores, total, keep, duplicate_of.
  • Update — add every keep:true non-duplicate to pool (with scores + grounding + how-to-test); add this round's gap points to gaps. If ≥1 new keep, dry = 0; else dry += 1.
  • Log one ledger row (see Ledger); N = N + 1.
  • Stop check — dry == <patience> (saturation) or N > <budget> → stop (see Stops).

Re-ground every round — novelty is judged from a fresh literature check each round, never carried over, so "the literature already covers this" reliably kills a crowded idea. Every cited snippet comes from a real retrieval that round; on {"error","fallback"} fall back to WebSearch/WebFetch — never invent a paper.

schemas/litscout.schema.json gates the LiteratureScout output — a generic instance:

{"round": 2,
 "evaluations": [{"hid": "r2h1", "novelty_assessment": "novel",
   "closest_prior_work": [{"claim": "X tested for facts, not skills", "cites": ["E1"]}],
   "support": [{"claim": "spacing aids motor consolidation", "cites": ["E2"]}],
   "gap": [{"claim": "long-term procedural retention untested at scale", "cites": ["E1"]}],
   "testability_note": "RCT: spaced vs massed schedule, 1-month retention."}],
 "evidence": [{"key": "E1", "title": "...", "source": "s2", "id": "a1b2", "snippet": "...verbatim..."}]}

schemas/verdict.schema.json gates the Judge output — a generic instance:

{"round": 2,
 "verdicts": [{"hid": "r2h1",
   "scores": {"novelty": 4, "grounding": 4, "testability": 5, "specificity": 4, "significance": 4},
   "total": 82.0, "keep": true, "gate_failures": [], "duplicate_of": null,
   "rationale": "Closest work shows X untested for skills -> novel; supported; clean RCT named."}]}

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in free text. Header:

round	generated	kept_new	pool_size	top_kept

Example:

round	generated	kept_new	pool_size	top_kept
1	6	3	3	spaced practice aids procedural (not just declarative) retention [82]
2	6	2	5	sleep-timed review beats time-of-day-matched review [78]
3	6	0	5	-

Per-round candidates.json / litscout.json / verdict.json live in round<N>/. Report the best state of the pool when stopping, not just the last round. Leave ledger.tsv, round*/, and literature/ untracked.

Constraints

  • Never fabricate citations or snippets — every evidence entry is a real <lit>/WebFetch retrieval from that round, verbatim; the evidence gate exists to catch fabrication.
  • Novelty is decided by the literature, not assertion — a hypothesis the search shows is already established is killed, however appealing; "I think it's novel" with no closest-work search caps novelty.
  • Generate ≠ confirm — kept hypotheses are strong candidates to test, each stated with its test; never report them as established findings.
  • The rubric is fixed and the keep-bar is stable across rounds, so saturation is a meaningful stop.
  • Reward distinct hypotheses, not volume — duplicates and rewordings are cut.
  • No installs — the sibling literature-search skill is stdlib-only; never print or commit API keys (keys.env stays gitignored at the project root). The sandbox is self-contained — no ../ escapes.
  • Do not pause the loop to ask whether to continue; run until saturation or budget.

Stops

The loop stops on the first of:

  • Saturation — dry == <patience> consecutive rounds add no new kept hypothesis.
  • Budget — <budget> rounds reached.

End with the ranked hypothesis set (<report> path) — each hypothesis with its statement, novelty assessment + closest prior work (cited), supporting evidence (cited), the gap it fills, and how to test it — plus the pool-size trajectory from ledger.tsv and the strongest unexplored gaps, so the user sees both the vetted hypotheses and where a deeper run would look next.

Karpathy Autoresearch

This is an experiment to have the LLM do its own research. You are a completely autonomous researcher: you hack the training code with an idea, run it, keep the change if the metric improves and revert it if it doesn't, advancing a branch as you go — and you repeat forever, until the human interrupts you. The artifact is the <editable_files>; the feedback signal is one scalar <metric> (lower is better, e.g. val_bpb) read from the run. Training runs in the user's own environment via <run_cmd> — this skill installs nothing and imports nothing; it edits code, shells out, and reads the metric from the log.

When to use

Use this to leave an agent running on a single training script, optimizing one scalar metric hands-off, where any improvement is kept and the loop never stops on its own. Default to broad freedom inside <editable_files>; the only hard limit is that the run finishes within the budget without crashing. Not for the analysis-first variant that reasons about the data before each edit (that is ml-autoresearch).

Setup

Resolve bindings interactively (load loop.run.yaml and skip if it already exists; else, on Claude Code infer + recommend each via AskUserQuestion, otherwise ask as quoted prompts; write loop.run.yaml). Then work with the user to set up a fresh run:

  1. Choose <iter_strategy> — branches (one git commit per run; the original) or snapshots (one folder per run under <sandbox_root>/). Snapshots are safer on a dirty or gitignored tree; branches mirror Karpathy. Either is fully supported throughout the loop.
  2. Open the run — branches: agree on a run tag from today's date (e.g. mar5) and create the branch git checkout -b autoresearch/<tag> (it must not already exist; this is a fresh run). snapshots: no branch — each iteration gets its own <sandbox_root>/iter<N>/.
  3. Read the in-scope files — the repo is small; read them for full context: the README, the read-only harness that defines the metric (the <metric> ground truth — do not modify), and the <editable_files> you will hack (model/optimizer/training loop).
  4. Verify the env/data exists — confirm <run_cmd> can run (data shards, tokenizer, deps present). If not, tell the human the one command to prepare it (e.g. uv run prepare.py).
  5. Initialize results.tsv — create it with just the header row; the baseline is recorded after the first run. Leave it untracked (never commit it).
  6. Confirm and go — confirm the setup looks right, then kick off the experimentation.
bindingmeaningdefaulthow to infer
<metric>scalar to minimize; must be printed by the run (e.g. val_bpb)—grep the in-scope files / README for val_bpb, val_loss, error…
<run_cmd> / <entrypoint>the command that launches one training run—e.g. uv run train.py; from pyproject.toml/.venv/README
<editable_files>the file(s) you may hack — everything else is read-only—the training script(s); exclude data, the eval harness, configs you must not touch
<sandbox_root>where results.tsv (+ snapshots) live./sandbox—
<iter_strategy>branches (a git commit per run) or snapshots (a folder per run)snapshotssnapshots is safer off a dirty/gitignored tree; branches mirrors Karpathy
<gate> / <budget>time (wall-clock) or epochs, and its value—the run's existing time/epoch setting

The experiment loop

Each experiment is one training run on a fixed budget (<gate>/<budget> — wall-clock time or a fixed epoch count, excluding startup/compile). You launch it simply: <run_cmd> (e.g. uv run train.py). Because the budget is fixed you don't need to worry about training time — every run gets the same budget.

What you CAN do: Modify <editable_files> — this is the only file you edit. Everything is fair game: model architecture, optimizer, hyperparameters, training loop, batch size, model size, etc.

What you CANNOT do: modify the read-only harness or the evaluation (the <metric> is the ground truth); install new packages or add dependencies (use only what's already available).

The goal is simple: get the lowest <metric>. Since the budget is fixed, you don't need to worry about training time — it's always the budget. Everything is fair game: change the architecture, the optimizer, the hyperparameters, the batch size, the model size. The only constraint is that the code runs without crashing and finishes within the budget.

VRAM is a soft constraint. Some increase is acceptable for meaningful <metric> gains, but it should not blow up dramatically.

Simplicity criterion: All else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it. Conversely, removing something and getting equal or better results is a great outcome — that's a simplification win. When evaluating whether to keep a change, weigh the complexity cost against the improvement magnitude. A 0.001 <metric> improvement that adds 20 lines of hacky code? Probably not worth it. A 0.001 <metric> improvement from deleting code? Definitely keep. An improvement of ~0 but much simpler code? Keep.

The first run: Your very first run should always be to establish the baseline, so you will run the training script as is.

Copy this checklist; LOOP FOREVER:

  • 1. Look at the git state — the branch/commit you're on (snapshots: the next iter<N>/).
  • 2. Tune <editable_files> with one experimental idea by directly hacking the code.
  • 3. Commit it (git commit -am "<idea>"; snapshots: copy <editable_files> into iter<N>/code_snapshot/ first).
  • 4. Run the experiment: <run_cmd> > run.log 2>&1 (redirect everything — do NOT use tee or let output flood your context).
  • 5. Read the result: grep "^<metric>:" run.log (also grab peak memory if printed).
  • 6. If the grep is empty the run crashed — tail -n 50 run.log, read the trace, fix if it's something dumb (typo/missing import), else give up after a couple of tries.
  • 7. Record the result in results.tsv (do NOT commit it — leave it untracked).
  • 8. If <metric> improved (lower), advance — keep the commit.
  • 9. If it's equal or worse, git reset back to where you started (snapshots: restore from code_snapshot/). Go to 1.

The idea is that you are a completely autonomous researcher trying things out. If they work, keep. If they don't, discard. And you're advancing the branch so that you can iterate. If you feel like you're getting stuck in some way, you can rewind but you should probably do this very very sparingly (if ever).

Output format. When the run finishes it prints a summary; the exact lines depend on what the user's script prints (<metric> is whatever you bound — val_bpb, val_loss, a perplexity, an error rate, …), e.g.:

<metric>:         0.997900
peak_vram_mb:     45060.2
num_params_M:     50.3

The numbers vary by machine since each run stops at the budget. Extract the metric with grep "^<metric>:" run.log.

Timeout. A run should take ~its budget plus a little eval overhead. If a time-gated run exceeds 2× <budget> minutes, kill it and treat it as a failure (discard and revert).

Crashes. Use judgement: something dumb and easy (a typo, a missing import) — fix it and re-run; an idea that's fundamentally broken — skip it, log crash as the status, and move on.

results.tsv (logging results)

<sandbox_root>/results.tsv, tab-separated (NOT comma-separated — commas break in descriptions). Header

  • 5 columns: the git commit (short, 7 chars; or iter in snapshots mode), <metric> (e.g. 1.234567, or 0.000000 for a crash), peak memory in GB (.1f, peak_vram_mb/1024; 0.0 for a crash), status ∈ {keep, discard, crash}, and a text description of what the experiment tried.
commit	<metric>	memory_gb	status	description
a1b2c3d	0.997900	44.0	keep	baseline
b2c3d4e	0.993200	44.2	keep	increase LR to 0.04
c3d4e5f	1.005000	44.0	discard	switch to GeLU activation
d4e5f6g	0.000000	0.0	crash	double model width (OOM)

Report the best run when interrupted, not necessarily the last.

Constraints

  • Only edit <editable_files>. The read-only harness that produces <metric> is the ground truth — editing it (or the eval) would corrupt the signal the loop is scored against.
  • One change per iteration, so each <metric> delta is attributable to a single idea.
  • Run in the user's env via <run_cmd>. Install nothing, add no dependencies — shell out and read the log. Always redirect output to run.log; never tee, never flood your context.
  • Don't commit results.tsv — leave it untracked. <sandbox_root>/ is self-contained (no ../).

Stops — NEVER STOP

Once the experiment loop has begun (after the initial setup), do NOT pause to ask the human if you should continue. Do NOT ask "should I keep going?" or "is this a good stopping point?". The human might be asleep, or gone from a computer and expects you to continue working indefinitely until you are manually stopped. You are autonomous. If you run out of ideas, think harder — read papers referenced in the code, re-read the in-scope files for new angles, try combining previous near-misses, try more radical architectural changes. The loop runs until the human interrupts you, period.

Literature Search (shared toolchain)

This skill is not a loop — it is the literature-retrieval dependency several loops use for paper discovery and novelty checks. It bundles tools/lit_search.py (a stdlib-only entrypoint) and the tools/lit/ package: Semantic Scholar (S2) for semantic search + snippets + the citation graph and arXiv for full-text reads (the keyless core), plus optional OpenAlex, Perplexity Sonar (ask), and bgpt.pro (bgpt) when their keys are set. Every subcommand prints JSON; on a terminal failure it prints {"error","fallback"} and exits non-zero so the caller degrades to its built-in WebSearch/WebFetch. No installs, Python ≥3.9.

How a loop uses it

A consuming loop resolves these once at setup and reuses them:

  • <lit_skill_dir> — where this skill is installed. After cp -r loops/* ~/.claude/skills/ it sits as a sibling of the calling loop, e.g. ~/.claude/skills/literature-search/ (adjust per host). If it cannot be resolved, the caller falls back to WebSearch/WebFetch — depending on it never hard-breaks a loop, it only enriches it.
  • <lit> — the literature CLI as a single command token: <lit_skill_dir>/tools/lit_search.py. The entrypoint is executable and carries a #!/usr/bin/env python3 shebang, so it runs directly with no python3 prefix. To reuse a cache across calls, append --cache-dir <sandbox_root>/literature/.cache after the subcommand (a per-subcommand flag, not global).
  • <lit_py> — only needed for the fallback form on a host that doesn't honour the execute bit: <lit_py> <lit_skill_dir>/tools/lit_search.py, with <lit_py> = python3 (any ≥3.9 interpreter).

Invoke <lit> as one command — keep it a single shell token. In particular zsh does not word-split unquoted variables, so storing the two-word python3 .../lit_search.py in a variable and calling $VAR tries to exec one long (nonexistent) filename. Use the executable single-token form above, call the script path directly, or split explicitly (${=VAR} in zsh).

Subcommands

SubcommandUse
<lit> search "<q>" [--year 2020-] [--min-citations N] [--limit N] [--source s2|openalex|both] [--sort relevance|citations|recent]semantic discovery — related papers (titles, TLDR, abstract, citations, arxiv_id). Defaults to --source s2 (relevance); add openalex/both to widen discovery
<lit> snippet "<q>" [--limit N]full-text passage search — pinpoint a verbatim fact/number without reading a paper
<lit> cite <paperId> --direction references|citations|recommendwalk the citation graph (backward / forward / "more like this")
<lit> fulltext <arxivId> [--mode auto|latex|pdf]deep-read one paper's methods/results (HTML > LaTeX > PDF)
<lit> ask "<question>" [--model sonar|sonar-pro|sonar-reasoning]Perplexity Sonar synthesis — a cited high-level answer (needs OPENROUTER_API_KEY; else use WebSearch)
<lit> bgpt "<q>" [--num N] [--days-back N]bgpt.pro structured experimental-result/limitations extraction (free for 50 results, then BGPT_API_KEY)
<lit> keys [--init]report which API keys are present (booleans only); --init writes/extends keys.env

The S2 + arXiv core (search/snippet/cite/fulltext) needs no keys. The optional extras (search --source openalex|both, ask, bgpt) widen discovery and synthesis when their keys are set.

API keys (the shared convention — optional, never block)

This skill is the canonical home for the project's key convention (it is the installed code that implements it). Keys live in one gitignored keys.env at the project root (the nearest .git ancestor of the working dir, else the CWD), shared by every skill in the project.

  • Plain KEY=VALUE lines; # comments; blank lines ignored; 0600 perms.
  • A real OS environment variable always wins over the file (CI/power users can just export).
  • All keys are optional — a missing key degrades that one feature gracefully, down to WebSearch.
  • Secrets never enter the conversation — the user edits keys.env themselves; skills read only presence as booleans, never values.

Onboarding flow: <lit> keys (report what's present) → <lit> keys --init (create/append slots, append-only, never overwrites; prints the path) → user opens keys.env and pastes the keys they want (in-session: ! $EDITOR ./keys.env) → <lit> keys again, then proceed. Never block on keys.

  • S2_API_KEY (free) — the one most worth setting; a dedicated 1 req/s for search/snippet/cite.
  • OPENALEX_EMAIL (free, any email) — OpenAlex "polite pool" for faster --source openalex|both.
  • OPENROUTER_API_KEY (paid) — enables ask (Perplexity Sonar); without it ask is disabled.
  • BGPT_API_KEY (free for 50 results, then paid) — enables bgpt beyond the free tier.

Constraints

  • Stdlib only — never add a dependency or assume one is installed; it must run under bare python3.
  • Self-contained — everything lives under this folder; no ../ escapes, no docs/ links.
  • Fail closed to a fallback — on any source error, emit {"error","fallback"} and exit non-zero rather than raising, so callers degrade cleanly.

Literature Survey Loop

A search → extract → map → expand loop that builds an evidence/contradiction matrix and stops at saturation. The artifact is the matrix (claims × sources, with each source's stance); the feedback signal is how many new, matrix-changing sources a round adds — you keep expanding until that falls below <min_new> for <patience> rounds. Unlike a one-shot summary, the loop deliberately hunts contradictions and gaps and keeps pulling threads until the picture stops changing.

The discipline: every cell — a source's stance on a claim — is backed by a verbatim snippet from a real retrieval. The value is not a tidy narrative; it is an honest map of where the literature agrees, disagrees, and is silent.

When to use

Use this for a multi-source survey of a question where the deliverable is a structured map of the evidence, not a paragraph. Default: run the full expand→admit→map loop below until saturation. Escape hatch: if the user wants only a quick scan, run round 0 (seed) alone and hand back the seed matrix. Not for grading a written proposal against the literature, and not for proposing new hypotheses.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value plus the live/degraded literature tier before creating any other files.

bindingmeaningdefaulthow to infer
<question>the survey question/topic, with any scope (years, sub-fields, inclusion criteria)—ask the user; restate the scope back for confirmation
<eval_scale>depth per round (low/medium/high, see below)medium—
<matrix>structured output matrix (validates schemas/matrix.schema.json); survey.md written alongside<sandbox_root>/matrix.json—
<sandbox_root>where the matrix, survey.md, ledger, and lit cache live./sandbox—
<budget>max rounds6—
<patience>stop after this many consecutive "dry" rounds2—
<min_new>saturation threshold — a round is "dry" if it adds fewer than this many new, matrix-changing sources2—

Evaluation depth dial (<eval_scale> caps per round — queries · citation-walks · fulltext reads · new-source admit cap):

presetqueries · walksfulltext readsnew-source cap
low2 · 00 (snippet/abstract only)~6
medium (recommended)4 · 11~10
high6 · ≥23~16

Literature toolchain. Paper search goes through the sibling literature-search skill: resolve <lit_skill_dir> (it installs as a sibling, default ~/.claude/skills/literature-search/), <lit_py> = python3, and <lit> = <lit_skill_dir>/tools/lit_search.py (note the tools/ segment); append --cache-dir <sandbox_root>/literature/.cache after a subcommand to reuse the cache. Subcommands used here: search "<q>" (discover sources), snippet "<q>" (verbatim passage = the evidence for a cell), cite <paperId> --direction references|citations|recommend (walk the citation graph), fulltext <arxivId> (deep-read one key paper). Confirm <lit> --help works at setup; because this loop is literature retrieval, do not silently proceed if the skill is missing — tell the user and either install it or degrade all retrieval to WebSearch/WebFetch (no ranked snippets or citation-graph expansion), tagging that evidence source:"web".

S2 key (optional, never block). A free S2_API_KEY makes snippet/cite reliable. Run <lit> keys --init, have the user fill the printed keys.env themselves, never paste secrets into chat; a missing key just degrades to the keyless pool → WebSearch. Record presence (booleans only) in loop.run.yaml.

The loop

matrix = sources + claims + gaps (starts empty). dry = consecutive dry rounds (starts 0). <N> starts at 0.

Copy this checklist and tick items off:

  • Round 0 — seed. Decompose <question> into sub-topics; run one <lit> search each, admit the most relevant sources, extract each one's key claim(s) into the matrix with a verbatim snippet, note obvious gaps.
  • Expand. Pick the least-covered sub-topics and open contradictions/gaps; run new <lit> search queries and walk the citation graph (<lit> cite) from the 1-2 most central papers. Honor the <eval_scale> caps.
  • Admit & extract. Dedupe against existing sources (by title/id); for each genuinely new source extract its key claim(s) + a verbatim snippet.
  • Map. For each claim, record where each relevant source stands — supports / contradicts / qualifies — with its snippet; set is_contested when sources both support and contradict; add newly-exposed gaps.
  • Saturation check. Count new, matrix-changing sources this round: dry += 1 if fewer than <min_new>, else dry = 0. Steer the next round at whatever is still thin.
  • Log one ledger row; N = N + 1; stop on saturation (dry == <patience>) or <budget>.

On stop, write <matrix> (validates schemas/matrix.schema.json) and survey.md — a synthesis organized as consensus (well-supported claims), disputes (the contested claims and who is on each side), and gaps (open questions), each citing its sources, plus an honest coverage note naming which sub-topics are well covered and which are thin.

The matrix is the schema-validated artifact; a compact generic instance (see schemas/matrix.schema.json):

{
  "question": "<question>",
  "sources": [{"key": "S1", "title": "...", "source": "s2", "id": "...", "year": 2022}],
  "claims": [{
    "claim_id": "C1", "statement": "...", "is_contested": true,
    "positions": [
      {"source_key": "S1", "stance": "supports",    "snippet": "verbatim passage ..."},
      {"source_key": "S3", "stance": "contradicts",  "snippet": "verbatim passage ..."}
    ]
  }],
  "gaps": ["open question the survey surfaced"]
}

source ∈ {s2, arxiv, web}; stance ∈ {supports, contradicts, qualifies}.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header:

round	queries	new_sources	total_sources	new_claims	contested	dry

Example:

round	queries	new_sources	total_sources	new_claims	contested	dry
0	4	7	7	9	1	0
1	5	5	12	4	2	0
2	4	1	13	0	2	1
3	4	0	13	0	2	2

Report the new-sources trajectory so the reader sees saturation actually happen, not just the final count.

Constraints

  • Never fabricate. Every source, claim snippet, and stance comes from a real <lit> / WebFetch retrieval that round, and snippets are verbatim; on {"error","fallback"}, use WebSearch/WebFetch and tag the evidence source:"web" — the verbatim-snippet rule is what makes the matrix trustworthy.
  • Map disagreement, do not smooth it. When sources conflict, record the contradiction explicitly (is_contested, both snippets); surfacing disputes is the point, not picking a winner.
  • Saturation is measured, not guessed — stop because new-sources-per-round actually fell below <min_new>, and report coverage honestly rather than implying completeness the search did not reach.
  • Dedupe sources so "new sources" counts real additions, not re-finds of the same paper.
  • No installs — the sibling literature-search skill is stdlib-only; never print or commit API keys (keys.env stays gitignored at the project root). The sandbox is self-contained — no ../ escapes.
  • Do not pause the loop to ask whether to continue; run until saturation, patience, or budget.

Stops

The loop stops on the first of:

  • Saturation — dry == <patience> (each of those rounds added fewer than <min_new> new sources).
  • Budget — N == <budget> rounds reached.

Always end with: the <matrix> path, the synthesis (consensus / disputes / gaps), the source count and new-sources trajectory from ledger.tsv showing saturation, and the coverage note naming the thin spots.

ML Autoresearch Loop

This loop is analysis-first: every experiment is followed by a diagnostic pass that examines what happened inside the model, and the next change is a hypothesis grounded in that evidence — not a guess. The feedback signal is <metric> read from the run log; the analysis is the spine that decides what to change. A <literature> dial (on/off) optionally grounds each change in prior work via the sibling literature-search skill. One change per iteration, so each metric move is attributable.

You are the researcher. Do not pause to ask for permission once the loop is running.

When to use

Use for an open-ended, autonomous ML research campaign where you want each change motivated by analysis of the model's actual behaviour. Set <literature> = off for a self-contained analysis-and-score loop; set <literature> = on to additionally ground changes in the scientific literature (paper search, evidence grading, a reusable findings backlog). Not for a single training run, a fixed sweep, or tasks with no measurable scalar metric. Default to off unless the user wants literature grounding or the problem is a known, well-published one where prior recipes will pay off.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available — record <host> = claude-code) infer a likely value for each binding from the project and present it as the recommended option; on other hosts (<host> = other) ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value with the user before creating any other files. For branches strategy, create git checkout -b autoresearch/<run_tag> (tag from today's date; branch must not exist). For time gating, write <sandbox_root>/run_with_timeout.sh (timeout $(( <budget> * 60 )) <entrypoint> "$@") and use it as the run command, hard-killing at 2 × <budget> min; for epochs, patch the epoch cap in an <editable_files> file.

bindingmeaningdefaulthow to infer
<metric> / <metric_direction>scalar to optimize + minimize/maximize—scan editable files + README for metric names
<run_cmd> / <entrypoint>command that runs one experiment end to end—pyproject.toml / .venv / README
<editable_files>files fair game to edit (never the eval harness)—model / config / train scripts; exclude data, logs, env, harness
<sandbox_root>where snapshots + ledgers live./sandboxnext to the editable files
<iter_strategy>snapshots or branchessnapshotsis the working dir a clean git repo?
<gate> / <budget>time (min) or epochs, and the limit—existing time/epoch settings in config
<literature>on = literature-grounded; off = analysis-onlyoffdoes the user want paper grounding?
<research_scale> (on only)depth dial low/medium/high/x-highmediumsee roles/research-subagent.md
<domain> (on only)one-phrase problem domain; seeds query phrasing only, never filters—infer from data/model/task
<lit_skill_dir> (on only)install dir of the literature-search skillsibling of this loop~/.claude/skills/literature-search/ (adjust per host)
<lit_py> (on only)Python ≥3.9 interpreter for the lit helper (stdlib-only)python3independent of <run_cmd>

FILE EDIT GUARD: before touching any file at any point — setup or loop — confirm it is in <editable_files>, because everything else is read-only ground truth (the eval harness defines <metric>). No exceptions.

Literature toolchain (only when <literature> = on)

Paper search goes through the sibling literature-search skill (stdlib-only, no installs): <lit> = <lit_skill_dir>/tools/lit_search.py (note the tools/ segment). Reuse one cache by appending --cache-dir <sandbox_root>/literature/.cache after the subcommand. Subcommands print JSON; on failure they print {"error","fallback"} and exit non-zero — then degrade to the host's WebSearch/WebFetch (never fabricate citations). Smoke-test <lit> --help at setup; if the skill is absent, tell the user and offer to install it (cp -r <repo>/loops/literature-search ~/.claude/skills/) or proceed degraded. For onboarding and API keys (all optional; a free S2_API_KEY is recommended), run <lit> keys --init — it manages the shared gitignored keys.env at the project root and reports presence as booleans (secrets never enter chat). Persist the live tiers to loop.run.yaml.

Initialise the sandbox

Create the layout (extra literature/ tree only when <literature> = on) and write the ledger headers:

<sandbox_root>/
├── loop.run.yaml       ← resolved bindings (written now)
├── results.tsv         ← experiment ledger, header only (written now)
├── literature/         ← (on only)
│   ├── corpus.tsv      ← findings ledger, header only (written now)
│   ├── .cache/         ← lit_search on-disk cache
│   ├── pdfs/           ← fallback PDF reads
│   └── text/           ← extracted LaTeX section text
└── iter1/              ← created at loop start

results.tsv header (tab-separated; the literature_basis column exists only when <literature> = on):

iter	<metric>	status	analysis_summary	[literature_basis	]description

The loop (LOOP FOREVER — until interrupted)

Iteration 1 is always the unmodified baseline: skip change-planning and research (no diagnostics yet to ground a change), but still write a baseline plan.md and run the mandatory analysis — it produces the first empirical anchor that iteration 2 builds on. Everything in <editable_files> is fair game (architecture, optimizer, hyperparameters, data pipeline, loss); the only constraints are that the code runs without crashing and finishes within <budget>. Simplicity criterion: all else equal, simpler is better — a 0.001 gain that adds 20 lines of hacky code is not worth it; a 0.001 gain (or an equal metric) from deleting code is a keep.

Copy this checklist each iteration and tick items off:

  • 1. Look at the state. branches: git log --oneline -5. snapshots: confirm iter<N>/ doesn't exist. Read iter N-1's analysis summary; (on) skim corpus.tsv for unimplemented keepers.
  • 2. Plan one change (iteration 1: SKIP — run baseline unmodified). Grounded in iter N-1's analysis. See Planning a change below; (on) it also runs the literature step.
  • 3a. Snapshot / commit, then apply the one change. snapshots: create iter<N>/{code_snapshot,analysis,results}/, copy every <editable_files> into code_snapshot/, copy loop.run.yaml to iter<N>/, then apply. branches: apply, then git commit -am "<desc>". When implementing a published/library technique, ground it in a real current example or the actual library in the repo (read it first) — never write the API from memory.
  • 3b. Write the analysis plan BEFORE the run + add any instrumentation it needs. See The analysis plan below.
  • 4. Run the experiment, redirecting everything (never tee): <entrypoint> > <sandbox_root>/iter<N>/<run_log> 2>&1 (or run_with_timeout.sh when time-gated). If it overruns, kill it and treat as a crash.
  • 5. Read the metric: grep '^<metric>:' <sandbox_root>/iter<N>/<run_log>. If empty, tail -n 50 <run_log>, read the trace, attempt one trivial fix (typo/import); if fundamentally broken, log crash and continue.
  • 6. Analyse the results — MANDATORY, produces real artifact files. See Analysing below.
  • 7. Log to the ledger(s) (untracked — never commit). See Ledger.
  • 8. Keep or revert (the change ran this iteration). Improved per <metric_direction> → keep, update current-best. Equal/worse/crash → discard/crash; branches git reset --hard HEAD~1, snapshots restore <editable_files> from iter<N>/code_snapshot/. Apply the simplicity criterion before logging discard. On a crash/OOM, fix with the minimal change that preserves the intent (OOM → smaller batch + grad-accum to hold effective batch) — never mutate the experiment into something the plan didn't call for.
  • 9. Go to step 1 — the analysis from step 6 is the primary input to the next hypothesis.

Planning a change (step 2 — iterations 2+)

The latest analysis sets the direction; it is the master input every iteration. Decide exactly one lever, grounded in iter N-1's analysis. Other vetted ideas are queued, not bundled into one run.

State explicitly before applying:

  • the one change and which <editable_files> it touches;
  • the empirical anchor — a specific file + value/pattern from iter N-1's results/ that motivates it. Every non-simplification change must cite an anchor; theoretical reasoning alone is insufficient. A swing to a different architecture is anchored too (a ceiling/structural finding, e.g. "the family plateaus at X with headroom" or "it fails exactly on cases needing Y"), not a local pathology.
  • what you predict will happen and why the finding supports it.

When <literature> = off: that anchor is the whole basis — pick the change directly from the analysis. Before writing the plan, scan prior analyses (ls <sandbox_root>/iter*/analysis/*.py) so you don't repeat a diagnostic without a comparison reason.

When <literature> = on: after fixing the anchor, ground the change in the literature —

  • 2a. Retire drift, then consult the backlog as a cache. If the last kept change altered the architecture family (e.g. CNN→transformer), set result=stale for every unimplemented keeper whose scope is a non-matching architecture tag; scope=agnostic keepers (schedules, weight decay, augmentation, init philosophies) survive. Then check corpus.tsv for an unimplemented keep targeting the analysis's direction — reuse it only if it still passes the gate (2c) against the CURRENT architecture (re-validate now; a finding that no longer applies is retired, not forced in).
  • 2b. Research the direction (the default unless 2a yielded a still-valid lever). Turn the analysis's limitations into questions (tie limitations to questions), record them in iter<N>/questions.md, then dispatch research subagents — see roles/research-subagent.md (spawn-or-degrade: real isolated subagents on Claude Code, otherwise run the research inline in this context) at the dial's depth/effort (see that file's dial table for <research_scale>). Level 1 = high-level (architecture fit, prior approaches, does the literature show success); research L1 first — if it surfaces a compelling new direction that becomes the lever. Level 2 = specific micro-opts (init, weight-decay dynamics, attention/cache for the sequence length, norm placement, schedule). Anti-rut: a keeper passed over for ~3 iterations, or no longer on any live direction, is retired (result=stale) so it stops resurfacing.
  • 2c. Evidence gate (re-validated; consults history). Keep/reuse a finding only if all hold: evidence.strength is moderate/strong (not weak); applicability_to_our_setup transfers to the CURRENT data/model/budget; every files_to_edit is within <editable_files> and fits <budget>; and it is not materially equivalent to a change already logged no-effect/hurt/crashed in corpus.tsv (unless the finding explains why it differs now). Tag each kept finding's scope (agnostic or the current architecture tag) so drift retirement is mechanical. Record every finding in corpus.tsv (keep or reject); kept-but-unchosen findings stay queued (implemented=n, result=pending).
  • 2d. Choose the single lever from the keepers (backlog + new) — highest evidence × applicability × gain-per-complexity. Add to the explicit statement above: the literature basis (kept finding + citation(s)), or none for a pure analysis-driven simplification/ablation; and the verify-todo (the metric/log/analysis from the finding's how_to_verify that will tell whether it helped). The literature basis is additive grounding — it never replaces the empirical anchor.

The analysis plan (step 3b — written BEFORE the run)

Write <sandbox_root>/iter<N>/analysis/plan.md as a deliverables table, not prose — each row is a hard commitment: a script you will write and run, the exact output file it produces, the question it answers, and what each outcome would look like:

scriptoutput filequestion it answersexpected if change helpedexpected if not

The plan must include at least one diagnostic dimension not measured in any prior iteration (the history scan enforces this); the "expected if not" column is mandatory (it forces you to define falsification before seeing the result). When <literature> = on, also include the verify-todo from 2d, and periodically (and whenever a swing is on the table) a ceiling/headroom probe — is the current architecture near its limit? (train/val plateau, representation collapse, structural failure).

Instrumentation (mandatory when needed). If a planned measurement needs a metric/log not currently produced (e.g. per-layer grad norms, per-class accuracy, val_loss alongside val_acc) and the producing script is in <editable_files>, add that logging NOW, before the run. Writing the plan first is what forces this — don't discover at analysis time that the number you need wasn't logged.

Analysing (step 6 — MANDATORY; produces real artifacts)

This is the spine of the loop. Do not proceed to step 7 until every row in plan.md has a corresponding non-empty file in iter<N>/results/. Analysis that didn't write a file did not happen.

  • 6a. Execute the plan. For each row, write the script in iter<N>/analysis/ and run it, redirecting to iter<N>/results/. Diagnostic dimensions to draw from (choose what the hypothesis and results call for, not a fixed rotation): gradient norms/flow, activation stats/saturation, embeddings (PCA/CKA/collapse), error & confusion analysis, loss dynamics & headroom, weight/parameter stats, data profiling (auditing the data itself is one of the highest-yield diagnostics), compute profiling.
  • 6b. Checklist. ls iter<N>/results/ against plan.md; write and run any missing deliverable now.
  • 6c. Interpret against the plan. For each row, did it match "expected if helped" or "if not"? A mismatch is information, not a failure.
  • 6d. (on) Check the verify-todo from 2d — did the change do what the literature predicted? This sets the finding's result in step 7.
  • 6e. Opportunistic follow-up. If something unexpected appears (odd loss shape, a consistently-wrong class, a layer behaving differently), write one more script to chase it — the most novel findings come from here.

Write a concise analysis summary (3–8 bullets): what you examined, the single most important finding (expected or not), (on) whether the architecture is near its ceiling, and the specific empirical anchor (file + value/pattern) that will motivate the next plan.

Empirical-justification gate. Before step 7, you must be able to complete this sentence with a real file reference and a concrete value/pattern — not a theoretical argument:

"The next change will be X because the analysis showed Y (from results/<file>, value/pattern Z)."

If you cannot, go back to 6a and run more analysis until you can. Proceeding without an empirical anchor is the exact failure this loop is built to prevent.

Forward-looking instrumentation. After analysing, ask "what would I wish I had logged?" — if the producing script is in <editable_files>, add it now (best-epoch checkpoint, extra per-epoch diagnostics) so future analyses are richer. Adding instrumentation is a valid iteration on its own.

Ledger

<sandbox_root>/results.tsv, tab-separated, never commas in free text. The literature_basis column is present only when <literature> = on. Status ∈ {keep, discard, crash}; use 0.000000 for the metric on crashes. For branches, the first column is the 7-char commit instead of the iter number.

iter	<metric>	status	analysis_summary	literature_basis	description
1	0.6320	keep	grad norms clean; no pathologies	none (baseline)	baseline
2	0.7050	keep	overfit on small data per val/train gap	Dosovitskiy 2020 ViT — aug+warmup	add RandAug + 3-ep warmup
3	0.7010	discard	warmup ok but LR too high; unstable	Loshchilov 2017 — decoupled wd	switch to AdamW wd=0.05

<sandbox_root>/literature/corpus.tsv (only when <literature> = on; tab-separated): a backlog of vetted findings reused across iterations instead of re-researched. The orchestrator (not the subagent) writes it and applies the final evidence gate, so verdicts are consistent across questions.

iter	level	paper_id	title	scope	relevance	verdict	implemented	result
2	649def…	1	An Image is Worth 16x16 Words	vit	high	keep	y	helped
3	a1c2…	2	Decoupled Weight Decay Regularization	agnostic	high	keep	y	no-effect
4	7d9e…	1	ConvNeXt blocks	cnn	low	keep	n	stale

Columns: level ∈ {1,2}; scope = agnostic or an architecture tag (drives drift retirement); relevance ∈ {high,med,low}; verdict ∈ {keep,reject}; implemented ∈ {y,n}; result ∈ {helped,no-effect,hurt,pending,stale} (stale = retired on drift/expiry). Last row: a CNN keeper retired after the architecture moved to ViT; the agnostic weight-decay finding survives drift.

Report the best iteration (not necessarily the last) when summarising progress. Do not commit results.tsv or literature/ to git — leave them untracked.

Constraints

  • Only edit files in <editable_files> — confirm before every edit, because everything else is read-only ground truth and the eval harness defines <metric>.
  • One change per iteration, so each metric delta is attributable to one lever.
  • Analysis is mandatory every iteration and must produce real files in results/ (each plan.md row → a non-empty output file). A change with no analysis behind it is not allowed — it degrades the loop into blind iteration (or, when on, recipe-following that plateaus blind).
  • Every non-simplification change must cite an empirical anchor (file + value/pattern) from that analysis; when on, the literature basis is additive, never a substitute. Record both in results.tsv.
  • Do not install packages or add dependencies the project lacks; helper scripts (and the lit helper) stay stdlib-only.
  • Always redirect training output to <run_log>; never use tee. The sandbox is self-contained — no ../ escapes.
  • (on) Never print or commit API keys — keys reports presence only; keys.env lives at the project root and must stay gitignored. When a lit tool returns {"error","fallback"}, degrade to WebSearch/WebFetch; never fabricate citations — every cited paper must come from a real tool result.

Stops

This loop runs forever until the human interrupts it — do not pause to ask "should I continue?" or "is this a good stopping point?". The human may be away and expects autonomous work indefinitely. A working result is the start of the next iteration, not the end. If you run out of ideas: re-read the in-scope files for missed angles, go deeper on the analysis (gradients/activations/embeddings/errors always surface something), combine previous near-misses from results.tsv, try more radical changes — and when on, go back to the literature: crawl citation graphs deeper, read methodology of recent work citing your approach, mine corpus.tsv for queued keepers, and combine recipes across papers.

Optimize Loop

An evaluator-optimizer loop with a pluggable correctness gate + minimized metric. The artifact is some editable thing (a code module or a SQL query); the feedback signal is two-part: a bound gate that must pass (behaviour/results unchanged) and a bound metric that must drop (the cost you minimize). You apply one change, check the gate, measure the metric, and keep the change only if the gate passes AND the metric improves — otherwise you revert. Repeat until the metric stops improving or the budget runs out. Once the loop starts, do not pause for permission.

Two ready bindings ship in tools/ (both vendored, stdlib-only):

  • code mode — gate: <gate_cmd> (the test suite) exits 0; metric: tools/metrics.py prints complexity (primary), max_nesting, loc (lexicographic tie-breakers). Lower is better.
  • sql mode — gate: the result-set hash from tools/bench.py matches the baseline; metric: the same tool's median_ms. Lower is better.

The gate is non-negotiable in both modes: a change that fails it is a regression, not an improvement. Never edit the ground truth (the tests / tools/metrics.py in code mode, the database / tools/bench.py in sql mode) — editing what measures you to move the number defeats the loop.

When to use

Use when there is a clear correctness bound to hold and a number to minimize: refactoring code that has a passing test suite (cut complexity), or tuning a SQL query that has a fixed result-set (cut latency). The default is the matching shipped tool; the escape hatch is to bind any <gate_cmd> that exits 0 on pass and any <metric_cmd> that prints a single number to minimize (e.g. a linter's issue count, or a non-SQLite engine's timing + result fingerprint). Not for adding features or fixing bugs — those intend to change behaviour, which this loop is built to forbid.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values back in one line, and skip to the loop. Otherwise pick <mode> first (it selects the gate + metric), then on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml and confirm the values before creating any other files.

<gate_cmd> and <metric_cmd> are the pluggable core: bind them per <mode> from the table. In sql mode one bench command supplies both — its hash is the gate, its median_ms is the metric.

bindingmeaningdefaulthow to infer
<mode>code (refactor under test) or sql (query, fixed results)—the artifact's kind
<editable_files>the file(s) the loop may change—code: source files (not tests/configs/the tool); sql: the query file (+ optional indexes file)
<gate_cmd>the bound gate that must PASS, else revert—code: the test command (exits 0 on pass); sql: implicit — candidate hash must equal the baseline hash from the bench command
<metric_cmd>the bound metric printing a number to minimize—code: python3 <skill_dir>/tools/metrics.py <editable_files> (→ complexity, then max_nesting, loc); sql: python3 <skill_dir>/tools/bench.py --db <db> --query <query_file> --setup <indexes_file> --repeat 5 (→ median_ms, hash)
<sandbox_root>where snapshots + the ledger live./sandbox—
<budget>max iterations (hard cap)8—
<patience>stop after N consecutive no-improvement iterations3—

<skill_dir> is this skill's installed folder; substitute the real path when writing loop.run.yaml. For non-default engines/languages, bind any <gate_cmd>/<metric_cmd> meeting the contract above. Two worked configs: examples/refactor.run.yaml (code) and examples/sql.run.yaml (sql).

The loop (until plateau or <budget>)

Copy this checklist and tick items off:

  • Iteration 0 — baseline: run <gate_cmd> (code) — if not green, stop (the loop needs a passing gate to protect behaviour). Run <metric_cmd>; record the metric as the current best, and in sql mode record the baseline hash as the correctness reference. Log the baseline row.
  • Snapshot every file in <editable_files> to <sandbox_root>/iter<N>/ so the iteration reverts.
  • Apply one focused change (see the per-mode ideas below) — one idea per iteration so each metric delta is attributable.
  • Check the gate: code — run <gate_cmd>; sql — read the candidate's hash from <metric_cmd>.
  • If the gate fails (tests non-zero / hash ≠ baseline / the tool errored), discard: restore from the snapshot, log the reason, continue.
  • Measure the metric and compare to the best: code — the triple (complexity, max_nesting, loc) lexicographically (complexity first; only on a tie consult max_nesting, then loc); sql — median_ms, keeping only on a margin clear of timing noise (default ≥ 3% relative).
  • Keep if the metric strictly improves the best (update the best, leave the files in place), else discard (restore from the snapshot).
  • Append a ledger row; stop on plateau (<patience>) or <budget>.

Change ideas — code mode: flatten nested if/else into guard clauses, replace a hand-rolled loop with a stdlib call (sum, min, max, statistics.*), collapse duplicated branches, remove dead code. Preserve public behaviour — names, signatures, return shapes, raised exceptions; the test suite is the contract.

Change ideas — sql mode: add an index to <indexes_file> covering filtered/joined/grouped columns; rewrite the query (correlated subquery → JOIN + GROUP BY, hoist a repeated computation, replace SELECT * with needed columns, push a filter earlier, drop a redundant DISTINCT/sort). The hash is over the multiset of rows, so it does not catch a changed row order — if ORDER BY is part of the contract, eyeball that the rewrite preserves it.

Lexicographic keep (code mode), current best (18, 3, 64): (15, 3, 45) keep (lower complexity); (18, 2, 70) keep (tie complexity, lower nesting); (18, 3, 61) keep (tie, fewer lines); (18, 3, 64) discard (no progress); (19, 1, 20) discard (higher complexity outweighs simpler nesting/loc).

Plateau counting: increment the no-improvement counter on every iteration that does not set a new best — discarded for a failed gate, a broken change, or an insufficient metric gain — and reset it to 0 on each keep. <patience> fruitless iterations in a row ends the run; <budget> is the hard cap. On stop, restore the working files to the best iteration (if the latest was a discard) and report: baseline vs best metric (and, in sql mode, the speedup factor), the trajectory, and the winning change set. If you run low on ideas before the budget, look harder rather than stopping early.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the description. status ∈ {keep, discard, baseline}. Use the columns for the active <mode>.

Code mode header iter complexity max_nesting loc status description:

iter	complexity	max_nesting	loc	status	description
0	23	7	86	baseline	unmodified module
1	19	5	78	keep	flatten summarize guard clauses
2	19	5	80	discard	extract helper (no complexity gain)
3	13	3	40	keep	use statistics + min/max/median

SQL mode header iter median_ms rows hash_ok status description (hash_ok ∈ {yes,no,-}):

iter	median_ms	rows	hash_ok	status	description
0	1121.06	10	-	baseline	correlated subquery no index
1	6.82	10	yes	keep	rewrite correlated subquery as JOIN + GROUP BY
2	1.18	10	yes	keep	add index orders(customer_id, amount)
4	0.40	10	no	discard	drop ORDER BY — changed result set

Report the best iteration, not necessarily the last.

Constraints

  • Only edit files in <editable_files> — the gate's ground truth is read-only: the tests and tools/metrics.py (code), the database and tools/bench.py (sql). Editing what measures you to move the number invalidates the run.
  • The correctness gate is non-negotiable. A change that fails it (tests red, or hash ≠ baseline) is a regression, not an optimization — revert it regardless of the metric. A green gate after a behaviour change means the gate is too weak, not that the change is safe; prefer holding behaviour identical over trusting a thin gate.
  • One change per iteration, so each metric delta is attributable.
  • Measure every candidate the same way (code: the same <gate_cmd>; sql: the same --repeat); compare the metric, not a single noisy run.
  • Keep changes within the existing dependency set; do not add imports the project lacks (stdlib is fine).
  • The sandbox is self-contained — no ../ escapes beyond the bound <sandbox_root>.
  • Do not pause the loop to ask whether to continue; run until plateau or budget.

Plan Loop

A planning loop: it turns a prompt into a plan detailed and correct enough to hand to a lower-tier model. The artifact is the plan — plan.md (the layout) + tasks.json (PR-sized tasks, each with files, tests, dependencies, and atomic subtasks). The feedback signal is two-part, like the repo's other evaluator loops: an objective gate (tools/validate_plan.py — schema shape, an acyclic dependency graph, a valid topological order, full component coverage) and a qualitative gate (a separate principal engineer agent that critiques alignment, decomposition, testability, and whether a junior could execute each task without guessing). You build the plan from first principles, validate it, critique it, and revise until the critique passes. This is the plan stage of a larger prompt → plan → execute → debug pipeline; it stops once the plan is ready to delegate.

When to use

Use to convert a feature/bug/refactor prompt into an executable plan grounded in a real repository — when the goal is a hand-off artifact a downstream executor (or a smaller model) can implement task-by-task. The plan is only as good as its weakest task for a literal-minded implementer, so the loop optimizes for executability, not prose.

Default: ground the plan in the <repo> you are given and let the principal-engineer critique drive the revisions. Escape hatch: if a key decision can't be resolved from the prompt or the repo, record it as an open_question for the human rather than guessing. Not for writing the code (a downstream execute loop), and not for research/experiment proposals (use research-proposal).

Setup

Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm before creating any other files.

bindingmeaningdefaulthow to infer
<prompt>the task to plan — a file path or inline text—the user's request
<repo>the project the plan targets; read for ground-truth env + conventions (never edited).the repo being worked on
<plan_file>the human-readable plan layout (markdown)<sandbox_root>/plan.md—
<tasks_file>the structured tasks, validated against schemas/plan.schema.json<sandbox_root>/tasks.json—
<pr_loc>target lines of code per task (PR-sized; a task may exceed it)100-400—
<sandbox_root>where plan, tasks, and the ledger live./sandbox—
<budget>max refine cycles5—

<skill_dir> is this skill's installed folder; substitute the real path when writing loop.run.yaml. The objective gate runs each cycle:

python3 <skill_dir>/tools/validate_plan.py --tasks <tasks_file>

It prints one JSON object {ok, errors, warnings, stats} — ok must be true (no errors) before a plan is considered ready.

The loop

Copy this checklist and tick items off.

Build the plan (iteration 0 — first principles):

  • Ground. Read <prompt> and inspect <repo> for ground truth: language, package manager, runtime/OS, the test command, and existing modules/conventions to reuse. Check what you can; never assume what you can read.
  • Decompose. Lay out the components from first principles — objective; end state / definition of done; non-goals; environment; building blocks; interfaces/contracts between blocks; tools; packages; data/assets; reuse (existing code + installed skills); open questions. Recursively split any component too big to reason about in one piece.
  • Tasks (PR-sized). Break the work into tasks, each ~<pr_loc> and equivalent to one PR. For each record: which components it serves, a description, building_blocks/tools/packages, the exact files it creates/modifies, tests that prove it, acceptance_criteria, and estimated_loc.
  • Order. Fill each task's depends_on, then compute a topological order (every task after its dependencies).
  • Subtasks. Split each task into atomic subtasks a junior can do with no further decisions — "create file X", "define function f(args) -> T", "wire f into Y".
  • Write + validate. Write <plan_file> and <tasks_file>, then run tools/validate_plan.py; fix every error and weigh every warning before the first critique. Log the baseline ledger row.

Refine (repeat until the plan passes or <budget>):

  • Critique. Spawn the principal engineer (spawn-or-degrade, roles/principal-engineer.md) with the prompt, the repo, plan.md, tasks.json, the latest validate_plan.py output, and the list of installed skills on this host. It returns a structured critique (schemas/critique.schema.json): verdict, score, issues (blocking/major/minor, each with a fix), coverage gaps, and suggested skills.
  • Revise. Apply the fix for every blocking and major issue — re-decompose, re-size, re-order, add tests, pin packages, add concrete signatures/paths/data shapes so a junior can't go wrong. Re-run tools/validate_plan.py. Append a ledger row.
  • Stop when the critique returns pass (no blocking or major issues) and the validator is clean — the plan is ready to delegate. Else loop, up to <budget>; if the score plateaus with only minor issues, stop and record them as open notes.

On stop, the deliverable is <plan_file> + <tasks_file> (plus any open_questions for the human), built to be executed task-by-task in order by a downstream execute loop or a lower-tier model.

Plan artifacts

<tasks_file> (tasks.json) — the machine-executable plan; full contract in schemas/plan.schema.json. Compact shape:

{
  "objective": "...", "end_state": "...", "non_goals": ["..."],
  "environment": {"language": "python", "package_manager": "uv", "test_command": "pytest -q"},
  "components": [{"id": "c1", "name": "config", "description": "load + validate config"}],
  "open_questions": ["which auth provider?"],
  "tasks": [
    {"id": "T1", "title": "config loader", "serves": ["c1"], "description": "...",
     "building_blocks": ["dataclass Config"], "tools": ["pytest"], "packages": ["pyyaml"],
     "files": [{"path": "src/config.py", "action": "create", "what": "Config + load()"}],
     "subtasks": [{"id": "T1.1", "description": "define load(path) -> Config"}],
     "tests": [{"description": "load() parses a valid file", "kind": "unit"}],
     "acceptance_criteria": ["invalid config raises ConfigError"],
     "depends_on": [], "estimated_loc": 180, "suggested_skills": []}
  ],
  "order": ["T1"]
}

<plan_file> (plan.md) — the human-readable layout: objective, end state, non-goals, environment, the components, a task table (id · title · serves · depends_on · est. LOC) in order, risks/open questions, and a one-line "how to execute" pointer to tasks.json. It mirrors tasks.json; tasks.json is the source of truth the executor consumes.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in free text. Header iter phase verdict score blocking major change:

iter	phase	verdict	score	blocking	major	change
0	build	-	-	-	-	first-principles decomposition: 6 components, 8 tasks, validator ok
1	critique	revise	72	1	2	PE: T3 bundles 2 PRs; T5 has no real test; nothing covers config loading
2	revise	-	-	-	-	split T3 -> T3a/T3b; added retry test to T5; added T9 config loader; reordered
3	critique	pass	90	0	0	PE: solid; one minor naming nit recorded as an open note

Report the final plan at the cycle the critique passed (or the best score reached at <budget>).

Constraints

  • Plan only — do not implement. Write no production code, scaffolding, or files in <repo>; the output is plan.md + tasks.json. Execution and debugging are downstream loops.
  • Ground every claim in the real repo/environment. Do not invent packages, files, APIs, or commands; verify against <repo>. A genuine unknown is an open_question for the human, not a guess — a plan that confidently states something false is worse than one that flags the gap.
  • Keep tasks PR-sized and self-contained. Each task links to ≥1 component and has tests and acceptance criteria; each subtask is atomic. Optimize for a literal implementer who will not fill gaps.
  • Keep the critic independent. The principal engineer reviews the plan it did not write (spawn-or-degrade gives real isolation on Claude Code); never let the author pass its own plan.
  • Edit only inside <sandbox_root> (plan, tasks, ledger); <repo> is read-only context. Run the loop to a passing critique or <budget> without pausing to ask whether to continue.

Roles

roles/principal-engineer.md — the adversarial plan critic. Spawn-or-degrade: a real isolated subagent on Claude Code (the Agent/Task tool), else adopt the role inline. It is read-only, judges against a fixed rubric, and returns JSON validated against schemas/critique.schema.json. Pass it the host's installed-skill list so it can recommend reuse; if that list is unavailable, it simply skips suggested_skills.

Power Analysis Loop

A power-analysis-and-preregister loop for a two-arm comparison. The artifact is the study's statistical plan; the feedback signal is two parts — statistical power (estimated by Monte-Carlo simulation of the planned test) and a count of validity flaws. Each iteration simulates power, solves for the sample size that reaches the target, audits the design for flaws, and revises — until power clears the target and the flaw list is empty. The deliverable is a sample-size justification plus a preregistration that pins the hypothesis, primary outcome, analysis, sample size, and stopping rule before any data is seen.

Scope & limitations

This loop does exactly three things, in a loop: (1) computes power and required sample size for a two-group comparison by simulation, (2) runs a fixed validity checklist over the design, and (3) writes a preregistration. The vendored power model (tools/power_sim.py) covers two-sample mean (continuous outcome) and two-proportion (binary outcome) tests only.

It is not a general experiment designer. It does not handle factorial, repeated-measures, clustered/multilevel, time-series, adaptive, or survival designs; it does not pick your outcome measure or manipulation from domain knowledge; and it does not analyze data you have already collected. For those, the power numbers here do not apply — use a design-appropriate power method. If the study is not a simple two-arm comparison, say so and stop rather than reporting a power that does not match the planned analysis.

When to use

Use this to size and preregister one two-arm comparison whose primary outcome is a continuous mean or a binary rate. Default to powering for the minimal effect of interest the user states; if they are unsure of that effect, help them set it from a baseline and a smallest-meaningful difference rather than an optimistic guess — a design "powered" for an effect bigger than reality is a fiction. If the study is not a two-arm comparison, stop and point to a design-appropriate method.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

bindingmeaningdefaulthow to infer
<hypothesis>the claim the experiment tests—ask the user
<outcome>primary outcome type + minimal effect of interest: continuous (baseline_mean, sd, min_effect) or binary (baseline_rate, min_lift)—ask; this fixes the effect size power is computed at
<target_power>power the design must clear0.80—
<alpha>significance level0.05—
<power_cmd>invocation of the vendored simulatorpython3 <skill_dir>/tools/power_sim.py --design <two-sample-mean|two-proportion> --effect <e> [--sd <sd> | --baseline <p0>] --alpha <alpha> --n <n_per_group>—
<design_doc>output design + preregistration file<sandbox_root>/design.md—
<sandbox_root>where design + ledger live./sandbox—
<budget>max iterations8—

<power_cmd> prints one JSON object, {"power", "n_per_group", ...}. Run it to get the power; never estimate power by hand.

The loop

Copy this checklist and tick items off:

  • Iteration 0 — draft the design to <design_doc>; record nothing as final.
  • Simulate power: run <power_cmd> at the current n and the assumed effect.
  • Solve N: if power < <target_power>, re-run at larger n (step up, then bisect) until it clears.
  • Audit validity: list every flaw from the checklist below.
  • Revise: fix the highest-priority flaw, set n to the power-adequate value, update <design_doc> (+ Preregistration section).
  • Append a ledger row; stop when power clears the target and no flaws remain, or at <budget>.

Iteration 0 — draft. Write a first design to <design_doc>: the arms/conditions, the unit of analysis and how units are assigned, the primary outcome and the exact planned test, the assumed effect size (from <outcome>), and a first sample-size guess. Record nothing as final yet.

Then, until stop (power met + no flaws, or budget):

  1. Simulate power. Run <power_cmd> at the current per-group n and the assumed effect, with the --design matching the planned test. Record the achieved power.
  2. Solve N. If power < <target_power>, re-run the simulation at larger n — step up (e.g. double), then bisect — until power clears the target, and adopt that n.
  3. Audit validity. Check the design against the checklist and list every flaw found:
    • Confounding / no control — is there a concurrent control group, or is the comparison against a historical/other-source baseline that differs in other ways?
    • Randomization — are units randomly assigned? If not, selection bias threatens any effect.
    • Selection / sampling — is the sample representative of the population the claim is about?
    • Multiple comparisons — more than one outcome/subgroup tested without correction?
    • Optional stopping / peeking — is there a pre-specified stopping rule, or will analysis run repeatedly until significant?
    • Outcome & analysis pre-specification — are the primary outcome and its single planned test fixed in advance (not chosen after seeing data)?
    • Measurement — is the outcome measured reliably and blind to condition where possible?
  4. Revise. Fix the highest-priority flaw (or a tightly-coupled pair that cannot be fixed independently, such as adding a concurrent control and randomizing assignment to it) and set n to the power-adequate value. Update <design_doc>, including a Preregistration section: hypothesis, primary outcome, the one planned analysis, sample size + how it was derived, randomization scheme, and the stopping rule.
  5. Log one ledger row and continue.

Stop when power ≥ <target_power> and the flaw list is empty, or at <budget>. Report the final design + preregistration, the achieved power and required n, and — if stopping on budget — the flaws still outstanding.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header:

iter	n_per_group	power	open_flaws	change

Example:

iter	n_per_group	power	open_flaws	change
0	50	0.50	2	draft: volunteers vs last-year cohort, n=50
1	100	0.80	1	solved n for 80% power at d=0.4
2	100	0.80	0	randomized concurrent control; pre-specified single primary outcome + stopping rule

Report the best iteration: the final design, the achieved power and required n, and any flaws still open if stopping on budget.

Constraints

  • Power is computed at the minimal effect of interest, not an optimistic one, because a design powered for an effect bigger than reality detects nothing real — and the --design in the simulation must match the test named in the design. Do not edit tools/power_sim.py.
  • A design does not pass on power alone — an adequately powered but confounded or non-randomized design still fails; both gates (power and flaws) must clear.
  • Preregister before data, so the eventual test is confirmatory rather than chosen after seeing results: the analysis, outcome, sample size, and stopping rule are fixed in advance.
  • One primary outcome and one planned test drive the power and the verdict; secondary analyses are labeled exploratory.
  • The sandbox is self-contained — no ../ escapes. Do not pause the loop to ask whether to continue.

Prompt Optimize Loop

An evolutionary optimizer for a prompt (OpenEvolve / AlphaEvolve-style). The artifact is a prompt that feeds the user's system; the feedback signal is a scalar metric printed by the user's own evaluation command. Each iteration proposes one quality-focused edit, re-runs the eval, and keeps the edit only if the metric improves — evolving the prompt toward higher scores. The eval is a black-box oracle the loop runs but never edits, so the optimization tracks what actually matters rather than gaming a number.

When to use

Use this when the user has a prompt and a command that scores the system using it, and wants the prompt improved to raise that score. Default to diagnosing the prompt's biggest current weakness each round and applying the one operator that addresses it; if the eval feedback points elsewhere, follow the feedback. Not for authoring a prompt from nothing, tuning weights/hyperparameters, or making a single manual edit with no score to compare against.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

bindingmeaningdefaulthow to infer
<prompt_file>the prompt to optimize — the artifact the loop evolves—scan the working dir for the prompt/template file the eval reads
<eval_cmd>required. Command that scores the current <prompt_file>; prints the metric (see output convention below). Treated as a black box — never edited—ask the user; look for eval/score/bench scripts
<objective>maximize or minimize, plus one line on what the metric measuresmaximizeask the user
<target>optional score at which to stop early—ask the user; else leave unbound
<sandbox_root>where prompt snapshots + ledger live./sandbox—
<budget>max iterations10—
<patience>stop after N consecutive non-improving iterations (plateau)3—

Eval output convention. <eval_cmd> must print, on its last line, either a JSON object {"score": <number>, "feedback": "<optional notes/errors>", ...any extra metrics...} or a bare number. Higher is better unless <objective> is minimize. The feedback field, when present, is the richest signal — read it like AlphaEvolve's artifacts side-channel to decide the next edit.

Eval runs in the user's environment. <eval_cmd> may call an inference endpoint or any tooling the user has installed; the loop just shells out and reads the last line. If instead the prompt is executed by you (interactive development with no separate endpoint), first run the current prompt over the user's eval inputs to produce outputs, write them where <eval_cmd> reads, then run <eval_cmd> to score them.

The loop

Copy this checklist and tick items off:

  • Iteration 0 — baseline: run <eval_cmd> on <prompt_file>, record its score as the current best, snapshot the prompt.
  • Diagnose the prompt's single biggest weakness from the latest score, the eval feedback, and the history.
  • Apply one targeted edit (one operator from the toolkit) to <prompt_file>.
  • Measure: re-run <eval_cmd> and read the new score from its last line.
  • Keep if the metric improves (require a margin if the eval is stochastic), else revert to the best snapshot.
  • Append a ledger row; if stuck, branch from an earlier high-scoring variant; stop on <target>, plateau (<patience>), or <budget>.

Iteration 0 — baseline. Run <eval_cmd>, record its score as the best, snapshot <prompt_file> to <sandbox_root>/iter0/, and start the history ({iter, edit, score, feedback} per row).

Then, until stop (target, plateau, or budget):

  1. Diagnose. From the latest score, the <eval_cmd> feedback, and recent history, name the prompt's single biggest current weakness — the one thing most likely holding the metric back.

  2. Make one targeted edit — pick the toolkit operator that addresses that weakness:

    • Clarity — remove ambiguity, contradictions, and vague wording.
    • Context — supply missing domain knowledge, definitions, or background the task needs.
    • Specificity — make instructions concrete; pin down the output format; define what "good" is.
    • Structure — order the prompt into steps/sections; add a short checklist.
    • Examples — add one or two demonstrations of the desired input → output.
    • Decomposition — split a complex instruction into explicit ordered sub-steps.
    • Guardrails — state edge cases and what to avoid.

    One change per iteration, so its effect on the metric is attributable.

  3. Measure. Snapshot the edited prompt to <sandbox_root>/iter<N>/, run <eval_cmd>, and read the new score off the last line.

  4. Keep or revert. Keep if the metric improves per <objective> (if the eval is stochastic, require a small margin so noise alone does not drive a keep); otherwise revert <prompt_file> to the previous best snapshot. Append {edit, score, feedback} to the history either way.

  5. Escape local optima. If the score has not improved for a couple of iterations, stop making tiny tweaks — branch from an earlier high-scoring snapshot, or try a bolder restructuring (a different decomposition, a fresh set of examples). Diversity beats grinding the same local hill.

When stopping, restore the best prompt to <prompt_file> and report the score trajectory, which edits moved the metric (and which did not), and the final prompt.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header:

iter	score	status	edit

status ∈ {baseline, keep, revert}. Example (metric = task accuracy, maximize):

iter	score	status	edit
0	0.42	baseline	original prompt
1	0.61	keep	specificity: define each output label and the exact output format
2	0.61	revert	examples: add 3 few-shot demos — no metric gain
3	0.78	keep	context: add the domain rules the task assumes but never states

Report the best iteration, not necessarily the last.

Constraints

  • The metric is the user's. Never edit <eval_cmd>, its data, or its scoring — that games the number instead of improving the prompt, and the eval is the only ground truth the loop has.
  • Optimize the prompt only, and preserve the task's intent. Improve how the task is instructed, not what is being asked; do not tailor the prompt to exploit eval quirks that would break real use.
  • One edit per iteration, and compare the metric by re-running the full <eval_cmd>, not a single sample, so each score delta is attributable to that one edit.
  • Report the best variant, not the last. The sandbox is self-contained — no ../ escapes.
  • Do not pause the loop to ask whether to continue; run until target, plateau, or budget.

Stops

  • Target — the score reaches <target> (if set).
  • Plateau — no iteration improved the best for <patience> consecutive rounds (every non-improving iteration counts toward patience; a keep resets it).
  • Budget — <budget> iterations reached.

Purple Team

The combined red+blue loop — the outer orchestration the red-team skill says "lives outside it." The artifact is a target system (frozen within a phase, patched between phases); the feedback signal is how many new failure classes a fresh attack pass finds against the patched target. Each cycle runs three strictly separated phases — find (red-team), fix (blue-team), re-verify (a fresh red-team pass) — and you repeat until a fresh find stays dry (zero new classes), meaning the target is hardened. Red and blue run as independent agents so the attacker that wrote a catalogue never grades its own patch. On stop it opens a pull request with the cycle history and the patch set.

When to use

Use to harden a guardrail/classifier/filter/prompt/API the user owns or is authorized to test, when the goal is an actually-hardened target plus a reviewable patch — not just a catalogue (that is red-team alone) and not just closing a pre-existing catalogue (that is blue-team alone). It needs a runnable oracle for the objective signal, and the two sibling skills installed.

Default: spawn red and blue as separate subagents per phase. Escape hatch: on hosts without subagent dispatch, run each phase inline (serial) per the role files — still correct, but the same context plays both sides, so be deliberate about not letting the fix bias the re-verify. Not for unauthorized targets.

Setup

Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm before creating any other files.

The phases reuse the sibling skills, so the bindings are their union — one shared loop.run.yaml drives both. The same file is <target_files> to blue (writable) and the program behind <target_cmd> to red (read-only); the same path is red's <failures_log> and blue's <catalogue>.

bindingmeaningdefaulthow to infer
<target_cmd>run the target on one stdin input → a verdict (what red attacks)—the guardrail/classifier/API entrypoint
<target_files>the source file(s) blue may edit to fix the target—the file(s) behind <target_cmd>
<oracle_cmd>ground-truth verdict for the same input (frozen)—a reference checker / policy impl
<gate_cmd>functional tests that must stay green through a fix (exits 0)—the target's test command; else rely on <holdout>
<holdout>benign + clearly-correct inputs blue must not break<sandbox_root>/holdout.jsonlknown-good inputs the oracle agrees on
<catalogue>shared failures file: red writes it, blue closes it, JSONL {id,text,class,...}<sandbox_root>/failures.jsonl—
<iter_strategy>branches (commit per fix → PR) or snapshotsbranchesdirty/non-git tree → snapshots
<pr_branch>branch the fixes land on and the PR opens frompurple-team/<tag>today's date as <tag>
<sandbox_root>where catalogues, snapshots, ledgers live./sandbox—
<cycle_budget>max find→fix→re-verify cycles4—
<find_budget>, <fix_budget>inner per-phase budgets passed to red / blue8—

<skill_dir> is this skill's installed folder. The phases run the sibling skills' tools (red-team/tools/harness.py, blue-team/tools/verify.py); the role files name them.

The loop

Copy this checklist and tick items off each cycle. Cycles are numbered from 0; per-cycle artifacts go in <sandbox_root> with the cycle index in the name so nothing is overwritten: the find catalogue is failures.cycle<N>.jsonl (cycle 0 may use <catalogue> directly) and the re-verify result is failures.cycle<N>_reverify.jsonl. The catalogue blue fixes in cycle N is the failures file from that cycle's find/re-verify pass — never append back into one shared file, so each cycle's accounting is clean.

  • Cycle 0 — Find. Run the red-team phase against the frozen target (spawn-or-degrade, roles/red-find.md), writing failures.cycle0.jsonl. Read back its distinct failure classes.
  • If the catalogue is empty on cycle 0, stop — the target is already dry; report clean, no PR.
  • Fix. Run the blue-team phase on this cycle's failures file (spawn-or-degrade, roles/blue-fix.md): it patches <target_files> one class per iteration under the gate + regression guard, committing kept fixes on <pr_branch>. Read back the classes it closed and any residuals.
  • Re-verify. Run a fresh red-team phase against the patched target into failures.cycle<N>_reverify.jsonl. This confirms the closed classes no longer reproduce and surfaces any new classes the fix introduced (e.g. an over-block).
  • Append one cycle row to the ledger. If the re-verify pass found new classes, they become the next cycle's catalogue (failures.cycle<N+1>.jsonl) — loop to Find/Fix for cycle N+1. Repeat find→fix→re-verify until a re-verify pass stays dry (no new classes) or <cycle_budget> is hit.
  • Handoff. Open the pull request with the full cycle history (see Handoff).

Phase independence (why three separated phases). The target is read-only ground truth for the duration of a red pass, so patches happen only between passes — mutating it mid-pass would break reproducibility and the class accounting. Spawn red and blue as separate subagents so neither grades its own work; the orchestrator only passes artifacts (the catalogue, the closed/residual summary) between them. Launch a phase's subagent and wait for its structured return before the next phase.

Ledger

<sandbox_root>/cycle_ledger.tsv, tab-separated, never commas in free text. Header cycle found fixed residual regressions_introduced net_open:

cycle	found	fixed	residual	regressions_introduced	net_open
0	5	5	0	1	1
1	1	1	0	0	0

All counts are distinct classes, not inner iterations: found = classes red surfaced this cycle; fixed = classes blue closed (a class blue attempted several times still counts once); residual = classes blue could not close; regressions_introduced = new classes the re-verify pass found that the fix caused; net_open = classes still open entering the next cycle (residual + regressions_introduced, the next cycle's catalogue size). Convergence is a re-verify pass with net_open = 0. Report the cycle at which the target went dry (or the best net_open reached).

Constraints

  • Authorized targets only. This drives real attacks against the target; only run it on a system the user owns or is explicitly authorized to test (inherited from red-team).
  • Keep the phases independent and the ground truth frozen. Never let one phase edit the oracle, <gate_cmd> tests, <holdout>, or either tool; never patch the target inside a red pass. These keep the find/fix/re-verify accounting honest.
  • Re-verify is a fresh attack, not a replay. It must be free to find new classes (including ones the fix introduced), not just re-check the old list — concretely, at least half its candidates should be new payloads and it should try at least two attack angles not in the prior catalogue (this matters most in degrade mode, where the same context plays both sides). That is what makes the loop converge to dry rather than to "the original five are gone."
  • One change per inner iteration (handled by the sub-loops); the orchestrator changes nothing itself except moving artifacts between phases.
  • Stay inside the repo and <sandbox_root>; do not pause between phases to ask whether to continue — run until dry or <cycle_budget>.

Handoff: the pull request

The deliverable is one pull request for the whole hardening run — the communication interface to the target's owner, who keeps final approval. With the tree at blue's best state and the kept fixes already committed on <pr_branch>:

  • Open it with gh pr create; body = the cycle ledger (found → fixed → re-verified per cycle), one reproducible example per closed class, and any residuals still open. Confirm once before opening — it is outward-facing; never auto-push silently.
  • Degrade, don't fail: with no remote / no gh, leave commits on <pr_branch> and write git format-patch + PR_BODY.md into <sandbox_root>; in snapshots mode emit a unified diff + PR_BODY.md. Tell the user the one command to open the PR themselves.

Roles

  • roles/red-find.md — runs the red-team phase against the target and returns the catalogue + classes.
  • roles/blue-fix.md — runs the blue-team phase on the catalogue and returns closed classes + residuals.

Both are spawn-or-degrade: spawn a real isolated subagent on Claude Code (the Agent/Task tool), else adopt the role inline. Each delegates to the sibling skill (red-team / blue-team) bound to the shared loop.run.yaml; if a sibling is not installed, ask the user to install it (npx skills add gaasher/agent-loop-skills) rather than re-implementing it here.

Red Team

An adversarial loop-until-dry. The artifact is a target system; the feedback signal is the count of distinct failure classes you can surface. Each round you craft adversarial inputs aimed at new weaknesses and run them through the target and a ground-truth oracle via tools/harness.py, which records every disagreement as a failure and de-dupes by the class (technique) you label each input with. You loop until fresh rounds stop finding anything new. This is only the find half of a find→fix setup: it catalogues failures and never patches the target (see Pairing).

When to use

Use to harden a guardrail, classifier, content filter, prompt, or API that the user owns or is explicitly authorized to test — when the goal is a catalogue of distinct, reproducible failures, each an objective target-vs-oracle disagreement. A failure is a bypass (target allows what the oracle would block) or an over-block (target blocks what the oracle would allow).

Default: drive the loop with a runnable oracle so the signal is objective. Escape hatch: if the user has no runnable oracle, the oracle is your judgment against a written policy — apply it consistently and record the intended verdict per input. Not for fixing the target, and not for testing systems outside the user's authorization.

Setup

Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm before creating any other files.

bindingmeaningdefaulthow to infer
<target_cmd>system under test: reads one input on stdin, prints a verdict (BLOCK/ALLOW, a label, a score). Never edited.—the guardrail/classifier/API entrypoint the user names
<oracle_cmd>ground-truth verdict for the same input. A failure is target != oracle.—a reference checker / policy impl; else your judgment vs a written policy
<candidates_file>each round's candidates, JSONL {id, text, class}; class is the technique the harness de-dupes on<sandbox_root>/candidates.jsonl—
<failures_log>append-only log of confirmed failures<sandbox_root>/failures.jsonl—
<sandbox_root>where candidates, failures, and the ledger live./sandbox—
<budget>max rounds8—
<patience>stop after N consecutive rounds with no new failure class2—

The signal comes from tools/harness.py. Run it each round:

python3 <skill_dir>/tools/harness.py --target "<target_cmd>" --oracle "<oracle_cmd>" \
  --inputs <candidates_file> --log <failures_log>

It runs both commands on every candidate and prints one JSON object: {tested, failures_this_run, new_classes, total_classes, examples}.

The loop

Copy this checklist and tick items off each round:

  • Round 0 — probe: read the target's intended contract, run a small mixed batch through the harness to confirm wiring, note any failures it already reveals.
  • Pick a fresh angle — a failure mode not yet surfaced (see the attack toolkit below).
  • Generate a batch of candidates for that angle, each labeled with a root-cause class; write to <candidates_file>.
  • Run tools/harness.py; read new_classes and the example failures.
  • Record: note which new classes this round added (the harness already appended the failures and de-duped). Append one ledger row.
  • Stop when <patience> consecutive rounds add no new class, or at <budget>. Else steer the next round at an untried angle.

Attack toolkit (draw from these, and invent your own):

  • Obfuscation — case changes, spacing/punctuation, leetspeak, unicode homoglyphs, encoding.
  • Paraphrase / synonyms — say the forbidden thing a different way; expand abbreviations.
  • Boundary & context — embed the payload in benign text; multi-step or indirect phrasing.
  • Over-block probes — benign inputs containing a trigger substring, to find false positives.

Labeling classes. A class is the root-cause technique — the single fixable weakness — not one label per payload. Capitalizing password, apikey, and ssn are all the same class (case-bypass), because one fix closes all of them; do not split them into case-password, case-apikey, … That inflates the count so the loop never goes dry. Aim for a handful of root-cause classes (e.g. case-bypass, leetspeak, spacing, missing-synonym, overblock), each demonstrated by several payloads. Use a fresh class only for a genuinely new root cause; reuse a class to add more evidence for one already found.

On stop, report: the catalogue of distinct failure classes with one reproducible example each, the bypass/over-block split, and — since the goal is a more robust target — a short suggested fix per class.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header round angle tested new_classes total_classes:

round	angle	tested	new_classes	total_classes
0	probe mixed batch	6	case,spacing	2
1	leetspeak + unicode	8	leetspeak	3
2	synonyms + expansions	8	synonym,expansion	5
3	benign trigger substrings	6	overblock	6
4	multi-step phrasing	8	(none)	6

Report the catalogue at the round with the most distinct classes (the cumulative total_classes), not whichever round ran last.

Constraints

  • Never edit the target, the oracle, or tools/harness.py. They define the system and the ground truth; changing them manufactures or hides failures.
  • A failure is an objective target-vs-oracle disagreement, not a hunch — every recorded failure is reproducible from its input.
  • Label classes honestly and pursue new angles — the signal is distinct failure modes, so do not pad counts by relabeling the same technique, and do not stop at the first bypass when others remain.
  • Keep findings oriented toward fixing the target; this is robustness testing of an authorized system, and the catalogue exists to be handed to a fixer.
  • Stay inside <sandbox_root>; no path escapes outside it.
  • Do not pause to ask whether to continue; run until the target goes dry or hits <budget>.

Pairing

This skill is the attacker — half of a find→fix loop. By itself it tells you how the system fails but leaves it unfixed. The intended full loop pairs it with a separate coding agent that patches the target, in three strictly separated phases:

  1. Find (this loop) — run against the frozen target → a catalogue of distinct failure classes, each with a reproducible example and a suggested fix.
  2. Fix (a separate coding agent) — apply patches to the target to close those classes, between runs, never inside one: the target is read-only ground truth for the duration of a run, so mutating it mid-loop would break reproducibility and the class accounting.
  3. Re-verify (a fresh find run) — start a new run against the patched target. Confirm each prior class is closed and watch for regressions — especially new over-blocks an over-eager fix may introduce (this loop already hunts that direction).

Repeat find → fix → re-verify until a fresh run stays dry. Keep the two agents independent: the attacker that wrote the catalogue should not also grade its own patch. This skill deliberately stops at the end of phase 1; the fix/re-verify orchestration lives outside it.

Research Proposal Loop

The artifact is a research proposal; the feedback signal is ScholarEval (literature-grounded Soundness + Contribution) turned into a 0-100 grade by a Judge against the fixed rubric.md. Each iteration evaluates → grades → revises until the grade clears <pass_threshold> or the budget runs out.

North star. Clearing the threshold is the stopping condition, not the goal. The goal is the strongest, most novel, genuinely publishable version the proposal can honestly become — every revision should ask "does this make the work more significant and more novel?", not just "does this patch a flaw?". This is grounded ambition: the lift comes from better-justified methods, a sharper-but-defensible novelty claim, and stronger baselines, all backed by real retrieved evidence. Overclaiming lowers the grade (evidence gate + Contribution axis); it never raises it.

The cast (all in this folder):

  • roles/ScholarEval.md — the two-module literature-grounded evaluator; emits scholareval.json.
  • roles/Judge.md — grades the feedback → 0-100 + ranked fixes; emits verdict.json (decides pass).
  • roles/Reviser.md — rewrites the proposal to address the fixes (guards the research intent).
  • rubrics/rubric.md — the fixed grading rubric (the Judge never edits it).
  • schemas/scholareval.schema.json, schemas/verdict.schema.json — the two validated outputs.

Spawn-or-degrade. On Claude Code, spawn ScholarEval / Reviser as real Agent subagents; otherwise adopt each role inline. You are the orchestrator and the Judge.

When to use

Use when a written proposal exists and the user wants it pushed past a quality bar with literature-grounded critique. Default: run the full evaluate→grade→revise loop below. Escape hatch: if the user only wants the critique (no rewriting), run one ScholarEval + Judge pass and stop. Not for writing a proposal from a blank page, and not for a standalone literature survey.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value, <intent>, and the live/degraded literature tier before creating any other files.

bindingmeaningdefaulthow to infer
<proposal_path>file with problem + methodology + planned experiments—scan for a likely .md/.txt/.pdf; if only prose is pasted, save it to <sandbox_root>/iter1/proposal.md
<intent>core research question + headline contribution, 2-3 sentences — frozen; the Reviser may never change it—read the proposal, extract it, have the user confirm
<pass_threshold>grade (0-100) the proposal must reach to pass75a solid, well-grounded proposal without demanding perfection
<budget>max iterations6—
<patience>stop after this many consecutive no-improvement iterations2—
<eval_scale>how much literature ScholarEval pulls per iteration (low/medium/high, see below)medium—
grade_weightssoundness · contribution · evidence_quality (must sum to 1; frozen for the run)0.45 · 0.35 · 0.20rubric.md; recommend default
<sandbox_root>where snapshots, ledger, and lit cache live./sandbox—

Evaluation depth dial (<eval_scale> caps per iteration — methods · dimensions examined · queries each · papers read full-text):

presetmethods · dimsqueries eachfulltext readseffort
low2 · 210 (snippet/abstract only)low
medium (recommended)4 · 322medium
high6 · 535high

Literature toolchain. Paper search goes through the sibling literature-search skill — resolve <lit_skill_dir> (it installs as a sibling, e.g. ~/.claude/skills/literature-search/), <lit_py> = python3, and <lit> = <lit_skill_dir>/tools/lit_search.py; append --cache-dir <sandbox_root>/literature/.cache after a subcommand to reuse the cache. Confirm <lit> --help works at setup; if the skill is absent, tell the user and either install it (copy the repo's loops/literature-search folder into ~/.claude/skills/) or degrade all retrieval to WebSearch/WebFetch (no ranked snippets or citation-graph expansion). The keyless S2 + arXiv core (search/snippet/cite/fulltext) needs no setup; a free S2_API_KEY makes snippet+cite (ScholarEval's two defining moves) reliable.

API key (optional, never block). The literature-search skill owns the key convention: run <lit> keys --init, then have the user fill the printed keys.env themselves and never paste secrets into chat. Re-run <lit> keys to confirm presence and record the tier in loop.run.yaml (literature_tiers, presence only). A missing key just degrades to the keyless pool → WebSearch.

Initialise the sandbox once bindings are confirmed:

<sandbox_root>/
├── loop.run.yaml        ← resolved bindings + <intent> + grade_weights + literature_tiers
├── ledger.tsv           ← header only (see Ledger)
├── literature/.cache/   ← lit_search on-disk cache
└── iter1/proposal.md    ← the input proposal (the baseline)

The loop

<N> starts at 1; iteration 1 evaluates the unmodified proposal (the baseline grade — no revision before it). Re-evaluate fresh every iteration: the grade comes only from a new ScholarEval pass on the revised proposal, never carried over.

Copy this checklist and tick items off:

  • Evaluate — run roles/ScholarEval.md (spawn-or-degrade) on iter<N>/proposal.md with the <eval_scale> caps and <lit>; it writes iter<N>/scholareval.json (validates against schemas/scholareval.schema.json).
  • Grade — as the Judge (roles/Judge.md) apply rubrics/rubric.md to scholareval.json: evidence gate → 0-5 per axis → weighted grade → hard gates → pass + ranked fixes; write iter<N>/verdict.json (validates against schemas/verdict.schema.json).
  • Log — append one ledger.tsv row (see Ledger).
  • Stop check — verdict.pass == true, or N == <budget>, or grade flat for <patience> iterations → stop (see Stops).
  • Revise — run roles/Reviser.md (spawn-or-degrade) with verdict.json, scholareval.json, <intent>, <lit>, and a small revision search budget (≈<eval_scale> searches + a couple of reads); it applies one focused fix batch and writes iter<N+1>/proposal.md + iter<N+1>/revision_notes.md.
  • N = N + 1 and repeat.

Every cited snippet must come from a real retrieval that iteration — never fabricated. When a lit tool returns {"error","fallback"}, fall back to WebSearch/WebFetch; never invent a paper.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in free text:

iter	grade	pass	soundness	contribution	evidence_quality	top_fix	revision_summary
1	58.0	no	3	2	4	add MM-GBSA re-scoring	baseline (no revision)
2	71.0	no	4	3	4	add head-to-head vs [C2] pipeline	added re-scoring + scoped affinity claim
3	82.0	yes	4	4	4	-	reframed contribution around integration + new benchmark

The matching scholareval.json and verdict.json for each iteration live in iter<N>/. Report the best-grade iteration when stopping on budget/plateau, not necessarily the last. Leave ledger.tsv, iter*/, and literature/ untracked.

Constraints

  • Never fabricate citations or snippets — every evidence entry comes from a real <lit>/WebFetch result retrieved that iteration, verbatim; the evidence gate exists to catch fabrication.
  • The rubric is fixed — the Judge never edits rubrics/rubric.md, so the passing threshold stays meaningful across iterations.
  • Protect <intent> — the Reviser may strengthen but never replace the core research question or headline contribution, and never gut a central method just to lift the grade.
  • Grade through evidence, not prose — eloquence earns nothing; grounded support and surviving novelty earn the score.
  • One focused revision batch per iteration, so each grade move is attributable.
  • No installs — the sibling literature-search skill is stdlib-only; never print or commit API keys (keys.env stays gitignored at the project root). The sandbox is self-contained — no ../ escapes.

Stops

The loop stops on the first of:

  • Pass — verdict.pass == true. Report the final proposal (iter<N>/proposal.md), its grade, and the grade trajectory.
  • Budget — N == <budget>. Report the best-grade iteration as the deliverable.
  • Plateau — grade hasn't improved for <patience> consecutive iterations. Report the best iteration and the standing priority-1 fixes.

Always end with the deliverable proposal path, its grade and pass/fail, the grade trajectory from ledger.tsv, and — if it did not pass — the standing blockers between this proposal and the bar.

Research Question Loop

A sharpen → score → revise loop for the framing stage of research. The artifact is a small set of research questions; the feedback signal is how many clear the bar — each scored 0-5 on five fixed axes (Specific, Answerable, Novel, Feasible, Significant). You start from a vague topic, draft candidates, score each against the rubric (with a light novelty check against the literature), and rewrite the weakest axis of the promising ones until enough are strong.

A good research question is the hard part of research: too broad and it cannot be answered; too narrow and it does not matter; already settled and there is no point. The goal is a few excellent questions, not many mediocre ones — this loop drives toward the narrow band that is answerable, novel, and worth answering.

Scope & limitations

This loop produces and refines questions, grounded by a light novelty check (a few searches), not a full survey — for an exhaustive map use the literature-survey loop, and to turn a question into testable predictions use the hypothesis-generation loop. The novelty check needs web or literature access; without it (novelty_check: none), novelty is the loop's best judgment and must be labeled unverified.

When to use

Use when the user has a topic, area, or rough curiosity and wants it turned into concrete questions worth pursuing. Default: run the full draft→score→revise loop below. Escape hatch: if the user only wants candidates rated (no rewriting), score the set once and report the rubric breakdown. Not for grading a finished proposal, and not for generating hypotheses or experimental designs.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

bindingmeaningdefaulthow to infer
<topic>the area of interest (field, population, scope, what the user already cares about)—ask the user
<n_questions>how many strong questions to deliver3—
<pass_threshold>rubric score (0-100) a question must clear to count as strong75a solid question without demanding perfection
<novelty_check>how to check whether a question is already answered: lit | web | nonelit if the sibling skill is installed, else webprobe for the literature-search skill (below)
<report>output question set<sandbox_root>/questions.md—
<sandbox_root>where the ledger and report live./sandbox—
<budget>max iterations8—

Novelty toolchain (only for novelty_check: lit). Paper search goes through the sibling literature-search skill (<lit> = <lit_skill_dir>/tools/lit_search.py, with <lit_py> = python3 and <lit_skill_dir> its installed location, e.g. ~/.claude/skills/literature-search/); the relevant moves are <lit> search "<q>" (is a direct answer already published?) and <lit> snippet "<q>" (pinpoint the answering passage). Confirm <lit> --help works at setup; if the skill is absent, tell the user and either install it (copy the repo's loops/literature-search folder into ~/.claude/skills/) or degrade to web (host WebSearch/WebFetch) or none. Record the resolved choice in <novelty_check> so re-runs are non-interactive.

The loop

The rubric (a fresh Grader scores each question 0-5 per axis — see grading below):

Axis531
Specificone clear construct/relationship, well-scopeddirection clear, scope loosebroad/ambiguous topic, not a question
Answerablea concrete study/analysis could resolve it; the answer-shape is clearresolvable in principle, approach unclearnot empirically/analytically decidable
Novelopen per the novelty check; closest work citedpartly addressed; a real twist remainsalready answered (check found a direct answer)
Feasibledata/methods/access plausibly existfeasible with effortneeds unavailable data or impossible measurement
Significantanswering it changes understanding or practicea useful incrementmarginal even if answered

Grading — spawn a fresh Grader per iteration (spawn-or-degrade). Each iteration, spawn a freshly instantiated Grader subagent — separate from whoever drafted or revised the questions, so the score is independent and not self-graded — and give it each candidate plus its novelty evidence. It returns the five raw 0-5 per-axis points (no weights). On Claude Code spawn it as a real Agent; otherwise adopt the Grader role inline in a clean pass. The orchestrator sums to a raw score out of 25, then converts to the 0-100 score used everywhere:

total = 100 × raw / 25 (e.g. raw 20/25 → total 80).

A question is strong when total ≥ <pass_threshold> and no axis scored 1 (a single fatal axis sinks it regardless of total).

Copy this checklist and tick items off:

  • Iteration 0 — restate <topic> and what is interesting about it; draft 3-5 candidate questions spanning different angles (mechanism, comparison, condition/boundary, application). Record nothing as strong yet.
  • Gather novelty evidence per candidate via <novelty_check> (<lit> search/snippet, or WebSearch, or skip).
  • Score — spawn a fresh Grader with each candidate + its evidence; it returns raw 0-5 per axis; convert raw/25 → total/100.
  • Diagnose each promising question's lowest axis — the one thing keeping it from strong.
  • Revise that one axis (one focused move per question); drop a fatally-flawed question revision cannot save; add a fresh candidate if short.
  • Append a ledger row; stop when <n_questions> clear the bar, or at <budget>.

Iteration 0 — frame & draft. Restate <topic> and what is interesting about it; draft 3-5 candidate questions spanning different angles. Record nothing as strong yet.

Then, until stop (<n_questions> strong, or <budget>):

  1. Gather novelty evidence. Run the <novelty_check> for each candidate's core: <lit> search/snippet (or WebSearch). If a direct answer exists, note the closest answered work; if only related work exists, note the open part. Each <lit> call prints JSON; on failure it prints {"error","fallback"} and exits non-zero — then fall back to WebSearch/WebFetch.
  2. Score. Spawn a fresh Grader (spawn-or-degrade) and hand it each candidate plus its evidence; it returns the raw per-axis points. Convert raw/25 → total/100.
  3. Diagnose & revise. Find each promising question's lowest axis and make one focused move on it: narrow an over-broad question to a specific population/condition; operationalize an unanswerable one into a measurable comparison; pivot an already-answered one toward the part the check showed is still open; raise significance by tying it to a decision or a contested claim. Drop questions with a fatal axis that revision cannot save; add a fresh candidate if you are short.
  4. Log one ledger row and continue until <n_questions> are strong.

On stop, write <report>: each strong question with its rubric scores, the novelty note (closest answered work / the open part), why it is answerable (the study-shape that would resolve it), and why it matters — plus any runners-up and the axis that held them back.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header:

iter	question	total	weakest_axis	revision

Example:

iter	question	total	weakest_axis	revision
0	how does sleep affect learning	35	specific	drafted; far too broad
1	does sleep timing affect retention	62	answerable	operationalized: spaced-review vs sleep-matched review, 1-week retention
2	does post-learning sleep within 3h beat delayed sleep for procedural retention	86	-	strong (novel per check: tested for declarative not procedural)

Report the best outcome — the strong questions and their scores — not necessarily the last iteration's set.

Constraints

  • A few strong questions beat many weak ones — do not pad <report> with questions that do not clear the bar; report them as runners-up with the blocking axis instead.
  • Novelty is checked, not assumed — when <novelty_check> is lit/web, actually search, cite the closest answered work, and never claim novelty the check contradicts. When none, label novelty unverified.
  • Grade with a fresh Grader scoring raw 0-5 per axis (no weights); the orchestrator converts raw/25 → 100 and never lets the drafter/reviser grade its own questions, so the score stays honest.
  • One focused revision per question per iteration, targeting its weakest axis, so improvement is attributable and questions converge rather than thrash.
  • Keep the rubric and threshold fixed so "strong" means the same thing throughout.
  • The sandbox is self-contained — no ../ escapes. Do not pause the loop to ask whether to continue; run until <n_questions> clear the bar or <budget> is hit.

Scientific Figure Loop

The artifact is a scientific figure (the rendered image + the plot.py that produces it). Each iteration generates → critiques+grades: a generator authors a rendering script and renders the figure to land the frozen <goals> message; an adversarial critic grades it 0-100 against the fixed rubrics/rubric.md and decides pass; the generator then revises against the critic's concrete findings. The loop runs until the grade clears <pass_threshold> or the budget is hit. All work happens on copies inside a sandbox; the user's data is copied in read-only and never edited.

The cast (all in this folder):

  • roles/generator.md — drafts/revises plot.py, renders figure.png by running <render_cmd>, optionally grounds domain content via <lit>; writes generation_notes.md.
  • roles/critic.md — the adversarial grader: re-derives each rubric axis independently, spot-checks the figure's numbers against the data, optionally lit-checks domain completeness, and emits schemas/critique.schema.json (the grade + pass + executable findings).
  • rubrics/rubric.md — the fixed grading rubric (the critic never edits it).
  • schemas/critique.schema.json — the one validated output.

Spawn-or-degrade. On Claude Code, spawn the generator then the critic as real Agent subagents (sequential — the critic needs the generator's figure); otherwise adopt each role inline. You are the orchestrator.

Why the critic grades itself (the honesty problem)

The critic both critiques and grades, which under loop-termination pressure invites inflation and a generator that games the rubric. roles/critic.md + rubrics/rubric.md counter this: the critic (1) applies a fixed rubric it never edits, (2) re-derives each axis from the rendered figure + data + frozen <goals> rather than echoing the generator, (3) recomputes a sample of the figure's numbers itself instead of trusting "it's fixed", (4) holds a fixed, anchored bar with no credit for effort or elapsed iterations, and (5) applies hard gates (a figure value that contradicts the data, a misleading axis, or fabricated data presented as real fails the figure regardless of the average). The generator optimizes the concrete findings; the critic grades holistically against the frozen goal — so "address every finding" does not mechanically buy a pass. Because the two are separate agents, the critic never just rubber-stamps the generator's intent.

When to use

Use when scientific data (or a prompt describing it) exists and the user wants a polished figure pushed past a quality bar with adversarial critique and a graded rubric. Default: run the full generate→critique loop below. Escape hatch: if the user only wants one figure + a critique (no iterating), run one generate + critic pass and stop. Not for writing a paper or doing the analysis, and not for retouching an already-final image.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value — including the distilled <goals>, whether a journal spec applies, and the live/degraded literature tier — before creating any other files.

bindingmeaningdefaulthow to infer
<brief>the prompt describing the figure to create + the message/claim it must communicate (and, if data exists, what the data represents)—the user's request; if pasted as prose, save to <sandbox_root>/brief.md
<data_paths>data file(s) the figure visualizes (CSV/TSV/parquet/JSON…); empty → an illustrative/schematic figure (the integrity axis then checks internal consistency, not data fidelity)—scan the working dir near the request; may be null
<goals>the figure's communication objective(s), 1-3 bullets — frozen; the critic grades against these and the generator may never abandon them—distill from <brief> at setup, confirm with the user in one line
<render_cmd>command/interpreter that runs the plot script the generator writes (it appends iter<N>/plot.py), in the user's env — the skill ships no plotting deps, the same contract as scientific-writer's <plot_command>python3pyproject.toml/.venv/README; e.g. uv run python or a venv python
<style>optional aesthetic/style guide or a named target journal/venue — when a journal is named, its figure spec is fetched at setup (see below) and both roles conform to / grade against it—ask the user; check <brief> for a venue
<pass_threshold>overall_score (0-100) the critic must reach (and no hard gate) to stop85a polished, publication-ready figure without demanding perfection
<budget>max iterations6—
<patience>stop after this many consecutive no-improvement iterations2—
<sandbox_root>where the plot scripts, figures, critiques, and the ledger live./sandbox—

The domain axis is not a binding — the critic auto-detects whether the figure makes an external domain claim (a named pathway, gene set, canonical benchmark, taxonomy, mechanism, or a literature- established number) and activates the domain axis itself; no user toggle.

Literature toolchain (optional, S2 + arXiv only). Domain grounding goes through the sibling literature-search skill — resolve <lit_skill_dir> (it installs as a sibling, e.g. ~/.claude/skills/literature-search/), <lit_py> = python3, and <lit> = <lit_skill_dir>/tools/lit_search.py; append --cache-dir <sandbox_root>/literature/.cache after a subcommand to reuse the cache. Use only the keyless S2 + arXiv core (<lit> search default --source s2, snippet, cite, fulltext); do not use --source openalex|both, ask, or bgpt. Confirm <lit> --help works at setup; if the skill is absent, degrade all retrieval to WebSearch/WebFetch. Record the tier (presence only) in loop.run.yaml.

Reuse what you've already pulled — don't re-query every iteration. Every retrieval is cached under --cache-dir <sandbox_root>/literature/.cache, and each role appends the facts it establishes (claim → number/element → source) to <sandbox_root>/literature/sources.md. Both roles consult that record (and the cache) first and only fetch papers/snippets not already on hand; a value a prior iteration already verified is re-checked by re-reading its recorded source, not by re-searching from scratch. The point of the literature step is correctness, not call volume — once a paper is pulled, work from it.

Journal style sheets (separate path, via web search). When <style> names a journal/venue, fetch its figure guidelines once at setup via WebSearch/WebFetch → <sandbox_root>/style/journal_spec.md (column width in mm, minimum font size, fonts, line weights, color mode, panel-label convention, file requirements). Both roles read this single cached spec — the generator conforms, the critic anchors its aesthetic/clarity axes to it — so the two never grade against divergent specs. This is distinct from <lit>: web search finds the journal's style spec; <lit> (S2/arXiv) checks domain content.

Environment. The generator renders figures by running <render_cmd> in the user's own environment — that code needs third-party deps (matplotlib, pandas, …), so the skill ships none and never installs them; it shells out to <render_cmd> and reads the rendered figure.png. PNG is rendered so the critic can view the image (an SVG would be read as XML). The deliverable is figure.png plus its plot.py — the reproducible source the user re-renders to any vector format. Any helper code the skill writes stays stdlib-only.

Initialise the sandbox once bindings are confirmed (copy the data in read-only; never edit originals):

<sandbox_root>/
├── loop.run.yaml        ← resolved bindings + <goals> + literature_tiers
├── brief.md             ← <brief> (if pasted as prose)
├── ledger.tsv           ← header only (see Ledger)
├── data/                ← read-only COPY of <data_paths>   (omit if no data)
├── style/journal_spec.md ← fetched journal figure spec     (omit if no journal named)
├── literature/.cache/   ← lit_search on-disk cache
└── iter1/               ← created by the generator
    ├── plot.py
    ├── figure.png
    ├── generation_notes.md
    └── critique.json

The loop

<N> starts at 1. Unlike loops that grade an existing baseline, the generator runs first every iteration (there is no input figure to critique) — iteration 1 drafts from scratch, iterations 2+ revise. Re-grade fresh every iteration: the score comes only from a new critique of the current figure, never carried over. Surface-only changes won't move it.

Copy this checklist and tick items off:

  • Generate — spawn generator (roles/generator.md) with <brief>, <data_paths>, <goals>, <style> (+ style/journal_spec.md), <render_cmd>, <lit>, and — on iter 2+ — iter<N-1>/critique.json. It writes/edits iter<N>/plot.py, runs <render_cmd> iter<N>/plot.py inside the sandbox to render iter<N>/figure.png, grounds any domain content via <lit>, and writes iter<N>/generation_notes.md.
  • Critique + grade — spawn one fresh critic (roles/critic.md) over iter<N>/figure.png + the data + <goals>, applying rubrics/rubric.md: it re-derives each axis 1-5 independently, spot-checks the figure's numbers against the data, optionally lit-checks domain completeness, computes overall_score = 100 × Σscore / (5 × n_axes), applies hard gates → pass, and writes iter<N>/critique.json (validates against schemas/critique.schema.json).
  • Log — append one ledger.tsv row (see Ledger).
  • Stop check — critique.pass == true, or N == <budget>, or overall_score flat for <patience> iterations → stop (see Stops).
  • N = N + 1 and repeat (back to Generate, which now revises against the critique).

A critique looks like (abridged; full shape in schemas/critique.schema.json):

{"iteration": 2, "summary": "Needs revision: honest now, but the MAPK panel omits ERK and the y-axis lacks units.",
 "axes": {"message": {"score": 4, "justification": "Up-regulation reads clearly."},
          "aesthetic": {"score": 3, "justification": "Palette not colorblind-safe (red/green)."},
          "clarity": {"score": 4, "justification": "Y-axis missing units."},
          "integrity": {"score": 4, "justification": "Bar heights match data/levels.csv."},
          "domain": {"score": 3, "justification": "MAPK cascade missing ERK node."}},
 "overall_score": 72.0, "pass": false, "gate_failures": [],
 "spotchecks": [{"target": "group-B bar = 2.4", "method": "recomputed from data/levels.csv", "result": "confirmed"}],
 "findings": [{"urgency": "must_fix", "action_type": "add", "area": "domain:incomplete",
   "finding": "MAPK cascade panel omits ERK1/2 downstream of MEK.", "proposed_action": "Add ERK node + MEK→ERK edge.",
   "target_artifact": "iter2/plot.py", "evidence": "lit snippet: canonical MAPK = RAF→MEK→ERK"}]}

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in free text:

iter	overall_score	pass	message	aesthetic	clarity	integrity	domain	top_fix	generation_summary
1	52.0	no	3	2	2	4	-	label axes + fix palette	baseline draft
2	74.0	no	4	3	4	4	3	add missing MAPK nodes (lit)	relabeled; colorblind palette; +ERK/MEK
3	88.0	yes	5	4	5	5	4	-	rebalanced panels; legend off-data

Use - in the domain column when the domain axis is inactive (n_axes=4). The per-iteration critique.json and generation_notes.md live in iter<N>/. Report the best-scoring iteration when stopping on budget/plateau, not necessarily the last. Leave the sandbox untracked.

Constraints

  • Never edit or run anything outside <sandbox_root> — data is copied in read-only at setup; the generator's plot.py and <render_cmd> run from the sandbox; no ../ escapes.
  • Never fabricate data, numbers, or domain elements. A figure presented as real data must render from <data_paths>; with no data, the figure must read as clearly illustrative/schematic, not a fake data plot. Domain content (genes, nodes, baselines, reported numbers) added from <lit> comes from a real retrieval that iteration, never invented.
  • The grading bar is fixed and reproducible — the critic never relaxes a rubric anchor to let the loop finish; a confirmed hard gate fails the figure regardless of the average.
  • Protect <goals> — the generator makes the same message prettier and clearer; it never drops or distorts the intended message to chase a higher score.
  • One coherent revision batch per iteration, blocks/gates first, so score moves are attributable.
  • No installs — the skill ships no plotting deps; <render_cmd> runs in the user's env, helper code is stdlib-only; literature is the keyless S2 + arXiv core only. Never print or commit API keys (keys.env stays gitignored).

Stops

The loop stops on the first of:

  • Pass — critique.pass == true. Report the deliverable (iter<N>/figure.png + plot.py), the score, and the trajectory.
  • Budget — N == <budget>. Report the best-scoring iteration as the deliverable.
  • Plateau — overall_score flat for <patience> iterations. Report the best iteration + the standing gate_failures/must_fix blockers.

Always end with the deliverable (iter<N>/ path), its overall_score and pass/fail, the per-axis scores, the score trajectory from ledger.tsv, and — if it did not pass — the standing blockers (gate_failures + open must_fix) between the figure and the bar.

Scientific Writer Loop

The artifact is a piece of scientific writing (draft + its dataset + figures + optional code). Each iteration critiques → grades → revises: five specialist judges produce concrete findings, an independent peer_reviewer turns the paper into a graded 0-100 score on the same axes, and a scientific_writer fixes the prose, figures, and code — running the user's <plot_command> to regenerate figures. The loop runs until the score clears <pass_threshold> or the budget is hit. All work happens on copies inside a sandbox; the user's originals are never touched.

The cast (all in this folder):

  • roles/figures_judge.md, roles/scientific_judge.md, roles/style_judge.md, roles/formatting_judge.md, roles/code_reviewer.md — the five critics; each emits the shared schemas/finding.schema.json.
  • roles/peer_reviewer.md — the summative grader (its own honesty rules); emits schemas/peer_review.schema.json and decides pass.
  • roles/scientific_writer.md — the reviser; fixes code → regenerates figures → updates prose.
  • schemas/finding.schema.json, schemas/peer_review.schema.json — the two validated outputs.

Spawn-or-degrade. On Claude Code, spawn the active judges as real Agent subagents in parallel, then one fresh peer_reviewer, then the scientific_writer; otherwise adopt each role inline. You are the orchestrator.

Why the grader is built this way (the honesty problem)

The peer_reviewer grades on the same axes the judges critique — which invites echoing, inflation under loop-termination pressure, and a writer that games the rubric. roles/peer_reviewer.md counters this: it (1) grades independently, re-deriving each axis from the paper + dataset before reading the critiques, (2) verifies a sample of numbers/citations itself rather than trusting "it's fixed", (3) must surface issues the judges missed, (4) holds a fixed, anchored, reproducible bar with no credit for effort or elapsed iterations, (5) applies hard gates (a confirmed block fails the paper regardless of the average), and (6) runs a substance check against surface compliance. The writer optimizes the judges' concrete findings; the grader judges holistically — so "address every finding" does not mechanically buy a pass.

When to use

Use when a written scientific draft exists and the user wants it pushed past a quality bar with multi-judge critique and an independent peer-review grade. Default: run the full critique→grade→revise loop below. Escape hatch: if the user only wants the critique (no rewriting), run one round of judges

  • peer_reviewer and stop. Not for writing a paper from a blank page, and not for a standalone literature survey.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value — including which axes are active (4 vs 5) and the live/degraded literature tier — before creating any other files.

bindingmeaningdefaulthow to infer
<draft_path>the draft to improve (markdown/text/tex)—scan the working dir for a likely draft
<dataset_paths>data file(s) the claims/figures derive from — used to verify numbers—scan for data files near the draft
<figures_dir>directory of figure images the draft references—scan near the draft; may be null
<code_paths>source files that produce plots/results; empty → code axis dropped (n_axes=4, no code_reviewer)—scan for plotting/analysis scripts
<plot_command>command that regenerates figures/results from the code; null → figures/code are edit-only (flagged "needs regeneration")—pyproject.toml/.venv/README; e.g. python3 code/make_figures.py
<citation_style>the single style the formatting_judge enforces (APA|MLA)APAask the user
<target_venue>optional venue whose style/length norms apply—ask the user
<length_limits>optional abstract/paper word-count targets—ask the user
<intent>the paper's core finding/contribution, 1-2 sentences — frozen; the writer may never change it—read the draft, extract it, confirm with the user
<pass_threshold>overall_score (0-100) the peer_reviewer must reach (and no hard gate) to stop85a solid paper without demanding perfection
<budget>max iterations6—
<patience>stop after this many consecutive no-improvement iterations2—
<sandbox_root>where working copies, critiques, reviews, and the ledger live./sandbox—

Literature toolchain. Citation grounding (writer) and verification (scientific_judge, peer_reviewer) go through the sibling literature-search skill — resolve <lit_skill_dir> (it installs as a sibling, e.g. ~/.claude/skills/literature-search/), <lit_py> = python3, and <lit> = <lit_skill_dir>/tools/lit_search.py; append --cache-dir <sandbox_root>/literature/.cache after a subcommand to reuse the cache. Confirm <lit> --help works at setup; if the skill is absent, tell the user and either install it (copy the repo's loops/literature-search folder into ~/.claude/skills/) or degrade all retrieval to WebSearch/WebFetch. The keyless S2 + arXiv core works with no setup; a free S2_API_KEY makes snippet/cite reliable — run <lit> keys --init, have the user fill the printed keys.env themselves, and never paste secrets into chat. Record the tier (presence only) in loop.run.yaml.

Environment. The writer regenerates figures by running the user's <plot_command> in the user's own environment — that code may need third-party deps (matplotlib, pandas, …), so the skill ships none and never installs them; it shells out to the user's command and reads the regenerated outputs. Helper code the skill writes stays stdlib-only. tools/lit_search.py (in the sibling skill) is stdlib-only too.

Initialise the sandbox once bindings are confirmed (copy in the originals; never edit them in place):

<sandbox_root>/
├── loop.run.yaml        ← resolved bindings + <intent> + literature_tiers
├── ledger.tsv           ← header only (see Ledger)
├── literature/.cache/   ← lit_search on-disk cache
└── iter1/
    ├── draft.md         ← COPY of <draft_path>
    ├── figures/         ← COPY of <figures_dir>
    └── code/            ← COPY of <code_paths>   (omit if no code)

The loop

<N> starts at 1; iteration 1 critiques and grades the unmodified draft (the baseline — no revision before it). Re-grade fresh every iteration: the score comes only from a new peer review of the revised paper, never carried over. Surface-only changes won't move it.

Copy this checklist and tick items off:

  • Critique — spawn the active judges in parallel (spawn-or-degrade), each over the iter<N>/ working copies + dataset; each writes iter<N>/critiques/<reviewer>.json (validates against schemas/finding.schema.json). Skip code_reviewer when no code.
  • Grade — spawn one fresh peer_reviewer (roles/peer_reviewer.md): it grades independently (own read first, then reconcile with the critiques; spot-checks numbers/citations; surfaces missed issues), writes iter<N>/peer_review.json (validates against schemas/peer_review.schema.json): 1-5 per active axis, overall_score = 100 × Σscore / (5 × n_axes), hard gates → pass.
  • Log — append one ledger.tsv row (see Ledger).
  • Stop check — peer_review.pass == true, or N == <budget>, or overall_score flat for <patience> iterations → stop (see Stops).
  • Revise — spawn scientific_writer (roles/scientific_writer.md) with the critiques, the peer review, <lit>, <plot_command>, <citation_style>, and <intent>. It fixes block/gate items first, fixes code → regenerates figures (running the copied <plot_command> inside the sandbox) → updates prose, grounds new citations via <lit>, and writes iter<N+1>/{draft.md,figures/,code/} + iter<N+1>/revision_notes.md.
  • N = N + 1 and repeat.

A judge finding and a peer_review look like (abridged; full shapes in schemas/):

{"reviewer": "code_reviewer", "iteration": 1, "overall": "block",
 "summary": "Block: make_figures.py sorts the two columns independently, fabricating r=0.98 (true r≈0.62).",
 "findings": [{"urgency": "must_fix", "action_type": "replace", "area": "make_figures:broken-pairing",
   "finding": "xs=sorted(study); ys=sorted(score) destroys per-row pairing; r inflated to 0.98 (real ≈0.62).",
   "proposed_action": "Correlate the paired arrays; re-render Fig 1; update r everywhere.",
   "target_artifact": "iter1/code/make_figures.py", "evidence": "code/make_figures.py:~70; paired r=0.62"}]}
{"iteration": 1,
 "axes": {"figures": {"score": 1, "justification": "Fig 1 plots bug-sorted data; numbers != data.", "blocking_issues": ["fig r=0.98 != paired r=0.62"]},
          "scientific": {"score": 1, "justification": "Causal claim from one correlation; r unreproducible."},
          "style": {"score": 2, "justification": "Promotional, AI-flavored prose."},
          "formatting": {"score": 2, "justification": "Mixed APA/MLA; Results before Methods."},
          "code": {"score": 1, "justification": "Independent column sort fabricates r=0.98."}},
 "overall_score": 28.0, "pass": false,
 "gate_failures": ["make_figures.py sort bug makes r=0.98 an artifact (true 0.62)"],
 "issues_judges_missed": ["Methods omits the test used and n."],
 "spotchecks": [{"target": "paired Pearson r", "method": "recomputed from data", "result": "refuted"}],
 "substance_check": "Baseline iteration — nothing revised yet."}

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in free text:

iter	overall_score	pass	figures	scientific	style	formatting	code	top_fix	revision_summary
1	28.0	no	1	1	2	2	1	fix make_figures sort bug	baseline (no revision)
2	61.0	no	3	3	3	3	4	report honest r, de-causalize	fixed bug+claims; relabeled fig1
3	88.0	yes	4	5	4	4	5	-	unified APA; trimmed abstract; balanced sleep claim

Use - in the code column when the code axis is absent. The per-iteration critiques/*.json, peer_review.json, and revision_notes.md live in iter<N>/. Report the best-scoring iteration when stopping on budget/plateau, not necessarily the last. Leave the sandbox untracked.

Constraints

  • Never edit or run anything outside <sandbox_root> — read the originals once at setup, copy them in, and work only on the copies; the <plot_command> is copied in and run from the sandbox.
  • Never fabricate data, numbers, figures, or citations — new citations come from real <lit> / WebFetch retrievals, quoted verbatim; on {"error","fallback"}, fall back to WebSearch/WebFetch.
  • The grading bar is fixed and reproducible — the peer_reviewer never relaxes anchors to let the loop finish; a confirmed hard gate fails the paper regardless of the average.
  • Protect <intent> — the writer strengthens the same finding; it never changes the core result or deletes a real finding to dodge a critique, because removing a result to raise a score is gaming.
  • One coherent revision batch per iteration, blocks/gates first, so score moves are attributable.
  • No installs — the skill ships no deps; <plot_command> runs in the user's env, helper code is stdlib-only. Never print or commit API keys (keys.env stays gitignored).

Stops

The loop stops on the first of:

  • Pass — peer_review.pass == true. Report the deliverable (iter<N>/ artifacts), the score, and the trajectory.
  • Budget — N == <budget>. Report the best-scoring iteration as the deliverable.
  • Plateau — overall_score flat for <patience> iterations. Report the best iteration + the standing gate_failures/must_fix blockers.

Always end with the deliverable (iter<N>/ path), its overall_score and pass/fail, the per-axis scores, the score trajectory from ledger.tsv, and — if it did not pass — the standing blockers (gate_failures + open must_fix) between the paper and the bar.

SWE Loop

The execute stage of the prompt → plan → execute → debug pipeline: it turns a coding prompt into working, tested, well-organised code. It runs the plan-loop to get an executable tasks.json, then walks the tasks in dependency order. Each task is driven by two isolated subagents — an Engineer that writes the code and a QA that owns the tests and the quality bar — looping until the gate holds. The feedback signal is two-part, like the repo's other evaluator loops: an objective gate (the task's tests pass, the full regression suite stays green, and tools/quality_check.py reports no threshold violation) and a qualitative gate (QA's score against rubrics/quality-rubric.md — simplicity, readability, comment hygiene, organisation, style-match). One task is kept per outer step; one change is proposed per inner round, so every delta is attributable.

When to use

Use to implement a prompt in a real repository when you want the work decomposed, built, and tested rather than written in one shot — and when "done" means a downstream engineer would accept it: tests green and the code clean. Default to executing the plan-loop's tasks.json task by task; the escape hatch is a genuine plan defect, which becomes an open_question for the human rather than an improvised detour. Simpler is better — a change that adds complexity to pass a test will be sent back by QA. Not for planning alone (plan-loop) or for minimising a metric on a finished artifact (optimize-loop).

Setup

Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm before building anything.

bindingmeaningdefaulthow to infer
<prompt>the coding task to build — a file path or inline text—the user's request
<repo>the project to build in; the Engineer edits it.the repo being worked on
<tasks_file>the plan-loop output the loop executes (validated by plan-loop's validate_plan.py)<sandbox_root>/tasks.jsonproduced by plan-loop in Phase 0
<test_command>how the project's tests run—tasks.json.environment.test_command; else infer from the repo (pytest, npm test, …)
<sandbox_root>where the plan, iteration artifacts, and ledger live./sandbox—
<iter_strategy>branches (a git commit per kept task) or snapshots (folder copies)branchesbranches needs a clean repo
<pr_mode>how kept tasks become PRs (see Pull requests): stacked (one PR per task, each stacked on the prior — default), single (one PR for the whole run), or none (local commits, no PR)stackedsingle for a small change or no stacked-PR tooling; none when offline / no remote
<base_branch>the branch PRs ultimately targetmainthe repo's default branch (git symbolic-ref refs/remotes/origin/HEAD)
<task_budget>max Engineer⇄QA rounds per task6—
<patience>stop a task after N rounds with no improvement2—
<quality_thresholds>hard limits for quality_check.py: max_comment_block, max_comment_line_len, max_func_loc, max_nesting, max_file_loc8 / 100 / 60 / 4 / 400tighten/loosen to the repo's norms

<skill_dir> is this skill's installed folder; substitute the real path in loop.run.yaml. The objective quality gate each round is:

python3 <skill_dir>/tools/quality_check.py <changed_files> \
  --max-comment-block 8 --max-comment-line-len 100 --max-func-loc 60 --max-nesting 4 --max-file-loc 400

The loop

Phase 0 — Plan (once)

Copy this checklist and tick items off.

  • Get the plan. If <tasks_file> exists and plan-loop's validate_plan.py passes, load it, confirm the objective + task count in one line, and skip to execution. Otherwise run the plan-loop skill on <prompt> grounded in <repo> to produce <tasks_file> (+ plan.md), and confirm the plan with the user before building.
  • Baseline. Run <test_command> and record the set of currently-passing tests — this is the regression baseline every later task must preserve. Log the baseline ledger row.
  • PR preflight (skip if <pr_mode> = none). Confirm a remote exists and gh is authed (gh auth status); resolve <base_branch>; pick a run tag from today's date and set <run_branch> = swe/<tag>. Never push to or open PRs against a branch you don't own; never force-push.

Per task (outer loop — tasks in tasks.json.order)

For each task T in dependency order, never starting one before its depends_on are kept:

  • Checkpoint. Ensure a clean tree at a known commit (the revert point for T). Branch model by <pr_mode>: stacked → cut T's branch <run_branch>/<T> off the previous kept task's branch (or <base_branch> for the first task), so each task stacks on its predecessor (valid because order is topological — every depends_on is already in the ancestor chain). single/none → stay on the one work branch <run_branch> (cut off <base_branch> at the first task). In snapshots mode also copy <repo> into <sandbox_root>/iter/<T>/before/.
Inner loop (Engineer ⇄ QA — until pass or <task_budget>/<patience>)
  • Engineer (spawn-or-degrade, roles/engineer.md). Give it T, the repo, and — on refine rounds — the previous verdict.json. It reads the neighbouring code, implements T editing source only, and returns change.json (schemas/change.schema.json).
  • QA (spawn-or-degrade, roles/qa.md). Give it T, the Engineer's change, the repo, and rubrics/quality-rubric.md. It authors any missing tests for T's acceptance_criteria (or grades quality only if already covered, editing tests only), runs <test_command> over the full suite (T's tests + every prior task's), runs quality_check.py, scores the rubric, and returns verdict.json (schemas/verdict.schema.json).
  • Log a ledger row for the round.
  • Keep or refine. If verdict.pass → keep: commit T (branches) / promote the working copy (snapshots). Then open/refresh T's PR per <pr_mode> (see Pull requests): stacked → push <run_branch>/<T> and open a PR with base = the previous task's branch (or <base_branch> for the first task); single → just commit (the one PR is opened at the end); none → nothing. The regression baseline now includes T's tests; break to the next task. Else feed verdict.fixes to the Engineer and refine. If the verdict doesn't improve for <patience> rounds or rounds reach <task_budget>, stop the task: revert T, log it blocked, record an open_question, and halt — do not build later tasks on an unfinished one (and do not open its PR).

The loop ends when every task in order is kept (the prompt is implemented and tested) or a task blocks. On end, if <pr_mode> = single, push <run_branch> and open the one PR (base <base_branch>) covering all tasks. The deliverable is the built repo plus plan.md, tasks.json, the ledger, and the PR(s).

Pull requests (per task)

<pr_mode> controls how kept tasks reach the remote. A PR is opened only for a kept task (a blocked task gets none) — it is just the delivery wrapper around an already-passing task, so the disjoint-scope gate is unchanged. Requires gh authed + a remote; on a dirty tree or missing remote, fall back to none and tell the user. Never force-push a pushed task branch.

  • stacked (default) — one PR per task. Each task branch <run_branch>/<T> is cut off the previous kept task's branch and its PR targets that predecessor, so the stack mirrors tasks.json order:
    <base_branch> ◄─ PR T1 (swe/<tag>/T1) ◄─ PR T2 (base = T1) ◄─ PR T3 (base = T2) ◄─ …
    
    On keep: git push -u origin <run_branch>/<T> then gh pr create --base <parent_branch> --head <run_branch>/<T> --title "<T.id>: <T.title>" --body "<what it delivers; Depends on #<parent PR>; tests N/N; quality minK>". Review/merge bottom-up; after a base PR merges, GitHub auto-retargets the child onto <base_branch> (or rebase the stack). Each PR stays small and independently reviewable — the point of stacking.
  • single — one big PR. All tasks commit to <run_branch>; at loop end, one gh pr create --base <base_branch> covering the whole run. Use when reviewers want a single review or stacked tooling isn't available.
  • none — local only. Commit per task, no push/PR (the original behaviour) — offline work, or when the human opens PRs themselves.

Log each PR url in the ledger change note.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in free text. Columns iter task phase tests quality pass note:

iter	task	phase	tests	quality	pass	note
0	-	baseline	34/34	-	-	clean baseline; 34 passing tests
1	T1	engineer	-	-	-	implement config loader src/config.py
1	T1	qa	36/36	min4	yes	authored 2 tests; rubric all >=4
2	T2	engineer	-	-	-	add retry to client.py
2	T2	qa	37/38	min3	no	test_retry fails; backoff off-by-one
3	T2	qa	38/38	min2	no	tests green but readability=2: nested 5 deep
4	T2	qa	38/38	min4	yes	flattened; kept

tests is passed/total over the full suite; quality is the minimum axis score (min4 = lowest axis is 4); phase ∈ {baseline, engineer, qa, keep, blocked}. Report the final state at the task that was kept or blocked.

Constraints

  • Disjoint write scopes. The Engineer edits only source files; QA edits only test files. This is the spine of the loop: the producer can't weaken the tests it's judged by, and the critic can't patch the source to make a test pass. A change that crosses scopes is a hard gate failure.
  • One task at a time, in dependency order, and never build on a task that isn't kept — an unfinished dependency makes every later result untrustworthy.
  • The gate is non-negotiable. A task is kept only when its tests pass, the full regression suite stays green, and the quality rubric holds. Tests green with ugly code is not done; clean code with a failing test is not done.
  • Tests are strengthen-only and bound to the plan. QA's tests must prove the task's acceptance_criteria; never weaken, skip, or delete a passing test to get a change through.
  • Don't edit the plan to dodge work. tasks.json is the contract; a real defect in it is an open_question for the human, not a quietly rewritten task.
  • Run autonomously to completion or a genuine blocker; don't pause to ask "should I continue?".

Roles

  • roles/engineer.md — the code producer. Spawn-or-degrade: a real isolated subagent on Claude Code (the Agent/Task tool), else inline. Edits source, returns schemas/change.schema.json.
  • roles/qa.md — the test author and quality critic. Spawn-or-degrade likewise. Edits tests, runs the suite and tools/quality_check.py, grades rubrics/quality-rubric.md, returns schemas/verdict.schema.json.

Stops

This loop runs autonomously until every task is built and kept, or a task blocks after <task_budget> rounds. On a block, stop and report the task, the standing fixes, and the open_question — surfacing a real obstacle beats forcing a change that passes the letter of the tests but not their intent. If a task keeps failing the same way, have the Engineer pivot to a different approach rather than refining a dead end.

Tabular Cleanup Loop

A single agent that takes a messy data dump (<artifact>) to the cleanest defensible state, no human in the loop once running. The objective is a checklist, not a score: the agent infers a data contract, compiles it into deterministic binary checks (each reports a violation count, never a weighted float), then each iteration profiles the table, picks the worst open check, applies one pandas transform to resolve it, and keeps it only if that check's violations strictly drop with no collateral damage. Every accepted transform appends to a replayable pipeline.py; every attempt logs to the ledger. The work decomposes into structure (parse correctly, one tidy table, sane types) → contract synthesis (turn every observed anomaly into a check) → the fix loop. Contract synthesis is where quality is won or lost: an issue the profiler notices but never compiles into a check (classically, many spellings of one category) silently survives — a green checklist over dirty data. Checks read the stored value, so canonicalization is real work the loop must do, not a check-time trick.

When to use

Use this to autonomously clean a messy table to an inferred, confirmed contract where every defect is a deterministic check the loop must drive to zero or to an honest residual. Default to strict contract inference (a lenient contract that lets dirty data go "all green" fast is the primary failure mode); the only human checkpoint is confirming the contract at setup, after which the loop runs to a stop condition. Not for open-ended discovery over an already-clean dataset, diagnosing one known anomaly, or checking an external claim against sources — those are analytical loops; this one rewrites the data to match a contract.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files. The contract (below) is the one decision the user must actively approve — infer it, then get explicit sign-off; everything after is autonomous.

bindingmeaningdefaulthow to infer
<artifact>messy table to clean; READ-ONLY (v0 is a copy)—scan the working dir for a data file; detect <format> + delimiter/encoding/header/quote from the extension
<contract>inferred per-column contract + cross-column rules + guardrails; the sole authority for "correct"—profile the raw artifact, then synthesize (see below) and confirm
<retention_floor>cumulative unique_rows_kept / unique_rows_in_v0 must stay ≥ this (denominator is v0 deduped)0.95—
<protected_columns>columns that must survive; dropping one needs explicit allowanceall (*)—
<impute_cap>max fraction of a column's cells that may be imputed (synthetic substitute inserted)0.20—
<analysis_cmd>interpreter that runs profiling/transform code in the user's env (has pandas)python3pyproject.toml/.venv/uv in the working dir
<sandbox_root>where tcl/ (versions, transforms, profiles, ledger, pipeline) lives./sandbox—
<gate> / <budget>backstop only: iterations/tokens/time and its capiterations / 30—

Profiling and transform code run in the user's environment via <analysis_cmd>, so they may use pandas. Keep helper code stdlib-first: probe heavy imports with try/except ImportError and degrade, or offer a consented uv pip install "pandas==<ver>" — never assume it is present.

Contract synthesis (the make-or-break step — do not skip)

Profile the raw artifact, then in two parts turn what you observe into the contract:

Part 1 — Structure. Confirm it parsed correctly (delimiter/encoding/header), is a single tidy table (one variable per column, one value per cell, one observation per row — no merged cells, stacked sub-tables, or multi-value cells), and that names/dtypes are sane. A mis-parsed table makes every column check meaningless; structural defects become transforms/checks too.

Part 2 — Per-column contract + rules + guardrails. Per column, determine its semantic type and canonical form, then emit: dtype, nullable?, key?, range (numeric min/max), regex (format), canonical categories + a merge_map of variants → canonical (repair guidance, not a check-time substitution), severity (default high for keys/required columns, else normal), and an optional outlier method. Add cross-column rules the data evidences (e.g. start ≤ end). Then the guardrails above (retention_floor, protected_columns, impute_cap). Three rules keep this trustworthy: commitment — every observed anomaly compiles to a check or is explicitly waived with a reason (several spellings of a value MUST get a categories check, never stay free text); canonical form is the stored value — declaring a canonical set creates the open violations the loop clears by rewriting; strictness bias — when unsure, add the stricter check (over-strictness is cheap to undo; a wrong check is worse than a missing one, so never guess column meaning the data doesn't evidence).

Confirm the inferred contract with the user (Claude Code: present as a compact table via AskUserQuestion; other: print as YAML and ask to confirm/amend). After sign-off, write the contract + the compiled checklist into <sandbox_root>/tcl/schema.yaml; the contract is then fixed for the run. Full contract shape: examples/run.example.yaml.

The checklist

Each contract rule compiles to one binary check that reports a violation count and a state. There is no weighted float and no epsilon — that single fact removes all denominator ambiguity.

check id (pattern)dimensioncounts violations where…
<col>.requiredcompletenessa nullable:false cell is null
<col>.typevaliditya non-null cell isn't parseable as the contract dtype
<col>.rangevaliditya non-null numeric cell is outside [min,max]
<col>.regexvaliditya non-null cell fails the format regex
<col>.categoriesconsistencya non-null stored value isn't a canonical category
<rule_id> (cross-column)consistencya row violates a cross-column rule
<col>.keyuniquenessa declared-key value is duplicated
rows.uniqueuniquenessa row is an exact duplicate
<col>.outlier (opt-in)plausibilitya numeric cell is a statistical outlier (only if a method is declared)

Check states: pass (0 violations) · open (violations remain, still attackable) · residual (violations remain but can't be fixed within guardrails without regressing another check — an accepted, reported dead end). Nulls are a violation only for .required — the .type/.range/.regex checks ignore nulls. Priority among open checks: highest severity first, then most violations. The headline checks_passing% = checks_in_pass / total_checks is for report.md and status lines only — it never drives keep/revert. Honest limit: the checklist measures well-formed & self-consistent, not true accuracy (is "John Smith" the correct name?) — outlier/range checks are the plausibility proxy.

The loop

Let <best> be the current accepted version (starts at v0, an exact copy of <artifact>). Copy this checklist and tick items off, iterating on <best> until a stop condition fires:

  • Profile <best> — run deterministic profiling code → profiles/profile-vN.json (schema below): per-column stats + every check's violation count + state + retention + headline checks_passing%; write a 4–8 line human summary.
  • Pick the target & propose ONE transform — among open checks, pick by priority (severity high, then most violations). State the check id, its violation count, the repair strategy, and the expected effect. Never propose a transform you can't tie to a specific open check.
  • Write the transform as a pure transform(df) -> df pandas function, deterministic, touching only what the target check requires (it must run standalone in pipeline.py later).
  • Apply → candidate — load <best>, apply, write versions/vN+1.<ext>. On error, fix once; if still broken, log status=crash and discard (don't advance <best>).
  • Re-profile the candidate — re-run the checklist on vN+1 for new violation counts + retention.
  • Keep or revert (pure checklist logic, no epsilon) — accept iff the target check's violations strictly decreased AND no other check's violations increased AND no guardrail tripped. Else revert. If that was the last guardrail-safe strategy for the target, mark it residual.
  • Log + persist — append one ledger.tsv row. If kept: copy the function to transforms/tNN_<slug>.py and append its call to pipeline.py in order.
  • Check stops — if none fired, go to the next iteration on the new <best>.

Least-destructive principle (the loop's bias): prefer repair over removal. Try, and take the first that is guardrail-safe and regression-free: (a) repair (parse/standardize/ canonicalize the value), (b) impute (within <impute_cap>, flagged synthetic — nulling an unrecoverable value in a nullable:true column is repair, not imputation, and is uncapped), (c) remove (drop rows/cols, within retention + protected floors). Removal is a last resort. A check becomes residual only when all three genuinely fail — e.g. a malformed value in a nullable:false column where repair can't recover it, dropping breaches retention, and nulling would regress that column's .required check. An honest residual is correct; never null a required field or invent a value just to clear a check.

Stops

Because checks are deterministic counts, "done" is exact — no epsilon, no plateau heuristic. This is the key difference from the autoresearch loops: it is designed to terminate. Stop on any:

  1. All green — every check is pass. The clean, successful exit.
  2. All-residual — zero open checks remain (every check is pass or residual); the remaining violations are provably unfixable within the guardrails. The natural exit.
  3. Budget — <gate>/<budget> reached. Backstop only.

Before marking the last open check residual (which triggers stop #2), confirm you actually tried all three strategies (repair → impute → remove) — don't declare it unfixable just because the first regressed another check or hit a floor.

On stop: set cleaned.<ext> to <best>, write report.md, and print the final summary (headline checks_passing%, pass/residual counts, which stop fired).

Outputs & formats

The run produces three deliverables:

  1. cleaned.<ext> — the best version (copy of <best>).
  2. pipeline.py — a standalone, replayable script: read raw <artifact> → apply each accepted transform in order → write cleaned. Deterministic and idempotent; re-running it on the raw dump reproduces cleaned.<ext> exactly. The audit-grade artifact.
  3. ledger.tsv + report.md — the full audit trail and a before/after summary.

<sandbox_root>/tcl/ layout: schema.yaml (bindings + contract + compiled checklist), versions/ (v0.<ext> is the READ-ONLY copy of the raw artifact; new vN written per candidate), transforms/ (one file per accepted transform), profiles/, ledger.tsv, pipeline.py, report.md.

ledger.tsv — tab-separated, append-only, one row per attempt, never commas in free text. One row per schemas/ledger.schema.json; status ∈ {keep,revert,residual,crash}:

iter	transform_id	target_check	dimension	viol_before	viol_after	regressions	retention	status	rows_affected	cells_affected	summary
1	t01_drop_dupes	rows.unique	uniqueness	7	0	0	1.000	keep	7	0	remove 7 exact duplicate rows
2	t02_canon_status	status.categories	consistency	142	0	0	1.000	keep	0	142	rewrite variants to canonical set
3	-	contact.regex	validity	3	3	-	0.94	residual	-	-	repair impossible; drop breaches retention; null regresses contact.required

profiles/profile-vN.json — per schemas/profile.schema.json (per-column stats + every check's count & state + retention + headline). Compact instance:

{"version":"v2","rows":980,"cols":7,"checks_total":11,"checks_passing":9,
 "checks_passing_pct":0.82,"total_violations":3,"retention":0.98,
 "checks":[{"id":"status.categories","dimension":"consistency","scope":"status",
            "severity":"normal","violations":0,"state":"pass"}],
 "columns":[{"name":"status","dtype":"object","contract_dtype":"category",
             "pct_null":0.0,"n_unique":5}],
 "summary":"status canonicalized; one regex check residual on contact."}

report.md — v0-vs-final headline checks_passing%, the full checklist with start/end violation counts, the confirmed contract, the ordered accepted transforms (the pipeline), the residual set (with why), what was dropped/imputed, and which stop fired.

Constraints

  • The raw <artifact> is never modified — all work is in <sandbox_root>/tcl/; v0 is a copy, and the loop only writes new vN versions, because the raw dump is the replay ground truth.
  • Never add rows, fabricate a key/index, or invent values silently — imputation is allowed but flagged synthetic, logged, and capped at <impute_cap> per column.
  • Respect the guardrails — a candidate breaching the retention floor, dropping a protected column, or exceeding the impute cap is reverted no matter how many checks it cleared, because a checklist is otherwise gameable by deletion.
  • Every check is a deterministic count, not LLM judgement; the confirmed contract is the sole authority for "correct" — do not silently change the contract or compiled checklist mid-run.
  • Keep/revert is pure checklist logic — keep iff the target check improves with no regression and no guardrail breach. No epsilon, no weighted score driving the decision.
  • One transform per iteration, a pure pandas function tied to one open check, so each violation delta is attributable; no blind multi-step rewrites.
  • A check unfixable within the guardrails becomes residual and is reported — never forced shut by fabricating or nulling required data.
  • Keep pipeline.py faithful — exactly the accepted transforms in order; re-running it on the raw dump must yield cleaned.<ext>.
  • Do not pause to ask whether to continue while open checks remain — setup is the only human checkpoint; the loop runs to a stop condition. Do not install packages the env lacks beyond a consented pinned install, and do not commit tcl/ to git.

Tournament Autoresearch Loop

An ML autoresearch loop whose single "form a hypothesis" step is replaced by an idea tournament. The artifact is an experiment ledger; the feedback signal is the realized <metric> delta of the change that won the tournament. Each iteration <n> ResearchAgents propose competing architecture changes, a Judge critiques and ranks them, the proposers refine, and the Judge selects one change to run. The Judge is the orchestrator and self-calibrates: it scores its predictions against realized results, so it learns which kinds of ideas actually pay off. The experiment mechanics (snapshot → run → mandatory analysis → keep/revert) match the sibling ml-autoresearch loop.

When to use

Use this for open-ended ML experimentation where competing ideas should be vetted before compute is spent and the picker should improve over time. You are the Judge: adopt roles/Judge.md and spawn the proposers with roles/ResearchAgent.md. Default to <n> competing proposers with one refine round; widen <n> or add rounds when ideas are converging too fast. Not for running a single pre-decided experiment, and not for analysis-only exploration over a dataset — for one uncompeted hypothesis per iteration use the sibling ml-autoresearch loop.

The cast and files (all in this folder):

  • roles/Judge.md — your behavior: critique, rank, decide, self-calibrate.
  • roles/ResearchAgent.md — the proposer role, spawned <n> times each round.
  • rubrics/rubric.md — the scoring criteria (shipped defaults; copied to a working copy at setup).
  • schemas/idea.schema.json — what a proposer returns (one proposed change).
  • schemas/verdict.schema.json — what the Judge records per idea (scores, rank, decision).

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

Record <host> (claude-code or other) once — it also decides spawn-or-degrade: on Claude Code spawn real Agent subagents for the proposers, all in one turn; otherwise adopt the ResearchAgent role inline, one proposal at a time.

bindingmeaningdefaulthow to infer
<metric>scalar metric to optimize—infer from code/README; ask direction
<metric_direction>minimize or maximize—infer from the metric's meaning
<run_cmd> / <entrypoint>command that runs one experiment end to end—pyproject.toml/.venv/README
<editable_files>model/config/training files the loop may edit; never the eval harness—scan for the model + training script
<sandbox_root>where snapshots + ledgers live./sandbox—
<iter_strategy>branches or snapshotssnapshotsgit present → offer branches
<gate> / <budget>time (minutes) or epochs, plus the capepochs / —infer epoch arg from the script
<n>proposers competing each round3recommend 3 — competition without a crowd
<refine_rounds>propose→critique→refine rounds before the Judge decides1—

If <iter_strategy> is branches, create git checkout -b autoresearch/<run_tag> (must not exist). If <gate> is time, write <sandbox_root>/run_with_timeout.sh (timeout $(( <budget> * 60 )) <entrypoint> "$@") and hard-kill at 2 × <budget> min; if epochs, cap the epoch count in an editable file.

Initialize the sandbox (after confirmation):

<sandbox_root>/
├── loop.run.yaml       ← resolved bindings (written now)
├── results.tsv         ← experiment ledger, header only
├── calibration.tsv     ← Judge predicted-vs-realized ledger, header only
├── judge_lessons.md    ← append-only Judge lessons (header only)
├── rubric.active.md    ← copy of rubrics/rubric.md; the Judge self-refines THIS, never the shipped one
└── iter1/              ← created at loop start

Copy rubrics/rubric.md → <sandbox_root>/rubric.active.md. Write the headers (see Ledger).

The loop

<run_log> = the file capturing training output for an iteration (default <sandbox_root>/iter<N>/run.log). Everything in <editable_files> is fair game; code must run and finish within <budget>. Simplicity criterion: equal metric but simpler code is a keep. The tournament yields exactly one change per iteration. Iteration 1 is the unmodified baseline — skip the tournament; just run + analyse to seed the first analysis summary.

Copy this checklist and tick items off, looping until interrupted:

  • State. branches: git log --oneline -5. snapshots: confirm iter<N>/ is new.
  • Tournament (iter 2+) — run it as the Judge (roles/Judge.md): propose (spawn <n> ResearchAgents) → critique & score against rubric.active.md → refine <refine_rounds>× → select the single top-ranked change.
  • Snapshot/commit, then apply the winning change.
  • Analysis plan → iter<N>/analysis/plan.md: deliverables table covering the winner's prediction plus useful steps from losing ideas; ≥1 row on a not-yet-measured dimension.
  • Run (redirect, never tee) → read metric.
  • Analyse — execute every plan.md row → iter<N>/results/; check the winner's prediction; write a 3–8 bullet summary ending in the empirical anchor for next round.
  • Log the results.tsv row and the realized delta + hit to calibration.tsv.
  • Keep or revert (simplicity criterion before logging discard).
  • Self-calibrate (roles/Judge.md): update judge_lessons.md, refine rubric.active.md, update the hit-rate.

In detail, each iteration:

  1. State. branches: git log --oneline -5. snapshots: confirm iter<N>/ is new.
  2. Tournament (iter 2+). As the Judge (roles/Judge.md): spawn <n> ResearchAgents (spawn-or-degrade by <host>) with roles/ResearchAgent.md, the latest analysis summary, and schemas/idea.schema.json. Critique & score each against rubric.active.md — gate (reject ideas with no testable prediction, unscored) → pointwise 0–5 per axis as the learning signal → de-biased pairwise to rank (compare each pair in both orders, keep only consistent verdicts). Refine <refine_rounds>×, re-score, then select the single rank == 1 change (no merging). Write ideas to iter<N>/ideas/, one verdict per idea to iter<N>/verdicts/, and log the Judge's predicted outcome for the winner to calibration.tsv.
  3. Snapshot/commit, then apply the winning change. snapshots: copy <editable_files> → iter<N>/code_snapshot/, copy loop.run.yaml → iter<N>/, apply the change. branches: apply, git commit -am "<idea_id>: <short description>".
  4. Analysis plan → iter<N>/analysis/plan.md: a deliverables table including the winner's prediction and any useful analysis steps from the losing ideas. ≥1 row must cover a not-yet-measured dimension.
  5. Run (redirect to <run_log>, never tee) → read the metric (grep '^<metric>:' <run_log>; on empty, tail -n 50, one trivial fix, else log crash).
  6. Analyse — mandatory, real artifacts. Execute every plan.md row → files in iter<N>/results/; verify none missing; interpret; check the winner's prediction; write a 3–8 bullet analysis summary ending in the empirical anchor for the next round.
  7. Log. Append the results.tsv row (0.000000 on crash). Append the realized delta + hit to calibration.tsv against the prediction.
  8. Keep or revert. Improved → keep, update best. Equal/worse/crash → discard/crash (branches git reset --hard HEAD~1; snapshots restore from code_snapshot/). Apply the simplicity criterion before logging discard.
  9. Self-calibrate (roles/Judge.md): update judge_lessons.md, refine rubric.active.md (weights + anchors, bounded, from realized outcomes), update the selection hit-rate.
  10. Go to step 1.

Never stop. Once running, do not pause to ask "should I continue?" — the loop runs until manually interrupted. If ideas run dry: push proposal diversity, mine results.tsv/calibration.tsv for under-explored directions, go deeper on analysis.

Ledger

Three append-only files under <sandbox_root>, all tab-separated, never commas in free text.

results.tsv — the experiment ledger (same format as ml-autoresearch). Header:

iter	<metric>	status	analysis_summary	description

status ∈ {keep, discard, crash}. Example:

iter	val_acc	status	analysis_summary	description
1	0.6320	keep	baseline; grad norms even, no pathologies	baseline
2	0.6890	keep	layer-2 activations near-saturated; BN helped	iter2-a1: add BatchNorm after conv2

calibration.tsv — the Judge's track record (predicted vs realized). Header:

iter	idea_id	grounding	impact	feasibility	pred_direction	pred_magnitude	confidence	realized_delta	hit

Example:

iter	idea_id	grounding	impact	feasibility	pred_direction	pred_magnitude	confidence	realized_delta	hit
2	iter2-a1	5	4	4	improve	+2%	high	+0.057	1

judge_lessons.md — append-only prose, 1–3 bullets per iteration: which axis tracked gains, what kind of idea was over/under-rated, and the reason for each rubric.active.md refinement. Example:

## iter 2
- grounding tracked the gain (BN tied to the dead-unit finding hit +0.057, as predicted).
- bumped grounding weight 0.40 → 0.45; tightened the impact anchor (impact=5 picks over-promised).

Report the best iteration (highest keep metric), not necessarily the last, plus the running selection hit-rate. Leave results.tsv, calibration.tsv, judge_lessons.md, rubric.active.md, and iter*/ untracked.

Constraints

  • Only edit files in <editable_files> — confirm before every edit, because everything else (especially the eval harness) is read-only ground truth defining <metric>.
  • Exactly one change per iteration (the tournament winner) — no merging ideas — so each metric delta is attributable to one change.
  • An idea with no testable prediction is rejected before scoring; the rank is decided by de-biased pairwise comparison, never by the pointwise scores (which only feed calibration).
  • The Judge edits only rubric.active.md (the working copy), never the shipped rubrics/rubric.md.
  • Always redirect training output to <run_log>; never tee (it floods your context).
  • Do not install packages or add dependencies the project lacks; helper code stays stdlib-only.
  • Do not modify the evaluation harness, and do not pause the loop to ask for direction.
  • The sandbox must be self-contained — no ../ escapes.

Sponsors

Details

Marketplace
gaasher/Agent-Loop-Skills
Category
Community