
autoresearch
Autonomous experiment loop for optimizing any measurable target
- version
- 0.3.3
- GitHub stars
- 13
Install
/plugin install autoresearch@llm-pluginsRun it inside Claude Code after adding the marketplace with /plugin marketplace add pbdeuchler/llm-plugins.
Contents
Discovered from sourceSkills 5
Autoresearch
Autonomous experiment loop: try ideas, keep what works, discard what doesn't, never stop.
Architecture: The main thread is a lightweight loop controller. Each experiment iteration runs in an experiment-runner subagent to keep the main context clean and unbounded.
Sub-Skills (Reference)
These skills contain detailed protocols. The experiment-runner subagent follows them directly. The main thread references them only when needed for setup or recovery:
autoresearch:confidence-scoring— MAD-based confidence computation and interpretation.autoresearch:experiment-git-ops— Git commit/revert patterns with protected files.autoresearch:metric-extraction— METRIC line parsing, unit inference, tracking.autoresearch:session-persistence— JSONL logging, session init/resume, segment tracking.
Phase 1: Setup (Main Thread)
Run once at session start. This is the only phase where the main thread does heavy file work.
- Gather parameters. Ask (or infer): Goal, Command, Metric (+ direction), Files in scope, Constraints.
- Branch.
git checkout -b autoresearch/<goal>-<YYYY-MM-DD> - Read source files. Understand the workload deeply before writing anything.
- Write
autoresearch.md. The heart of the session (see template below). - Write
autoresearch.sh. Benchmark script outputtingMETRIC name=valuelines. - Write
autoresearch.checks.sh(only if constraints require correctness validation). - Commit all autoresearch files.
- Write config header to
autoresearch.jsonl:{ "type": "config", "name": "<session>", "metricName": "<name>", "metricUnit": "<unit>", "bestDirection": "<lower|higher>" } - Record start timestamp:
date +%s— store for duration limit checks. - Run baseline via the first subagent dispatch (see Phase 2). The baseline description should be "Baseline measurement".
autoresearch.md Template
# Autoresearch: <goal>
## Objective
<Specific description of what we're optimizing and the workload.>
## Metrics
- **Primary**: <name> (<unit>, lower/higher is better) — the optimization target
- **Secondary**: <name>, <name>, ... — independent tradeoff monitors
## How to Run
`./autoresearch.sh` — outputs `METRIC name=number` lines.
## Files in Scope
<Every file the agent may modify, with a brief note on what it does.>
## Off Limits
<What must NOT be touched.>
## Constraints
<Hard rules: tests must pass, no new deps, etc.>
## What's Been Tried
<Update as experiments accumulate — key wins, dead ends, architectural insights.>
autoresearch.sh
Use a generic subagent to create this in order to prevent polluting the main context.
Bash script (set -euo pipefail) that pre-checks fast, runs the benchmark, and outputs structured METRIC name=value lines. For fast noisy benchmarks (<5s), run multiple times and report median.
autoresearch.config.json (optional)
{
"workingDir": "/path/to/project",
"maxIterations": 50,
"maxDurationMinutes": 120
}
Phase 2: Loop (Main Thread as Controller)
The main thread is a strategy controller. It decides what to try, dispatches a subagent to execute it, and processes the result. The main thread NEVER modifies source files or runs benchmarks directly.
CONTINUE LOOPING UNTIL EITHER USER INTERRUPT OR TIME/ITERATION LIMIT. Never ask "should I continue?"
Loop Iteration
┌─────────────────────────────────────────────┐
│ MAIN THREAD (controller) │
│ │
│ 1. Check stop conditions │
│ 2. Read recent state (JSONL tail + ideas) │
│ 3. Decide hypothesis for next experiment │
│ 4. Dispatch experiment-runner subagent │
│ 5. Parse result block from subagent │
│ 6. Update strategy based on result │
│ 7. Every 5 runs: update autoresearch.md │
│ 8. Go to 1 │
└─────────────────────────────────────────────┘
│
▼ (dispatch)
┌─────────────────────────────────────────────┐
│ SUBAGENT: experiment-runner │
│ │
│ - Reads files, implements changes │
│ - Runs benchmark + checks │
│ - Evaluates metrics, computes confidence │
│ - Logs to JSONL │
│ - Commits or reverts git state │
│ - Returns structured result block │
└─────────────────────────────────────────────┘
Step 1: Check Stop Conditions
Before each iteration:
- If
maxIterationsis set and reached → graceful shutdown. - Run
date +%s, compare to start timestamp. IfmaxDurationMinutesexceeded → graceful shutdown.
Step 2: Read Recent State
Read the last 10 lines of autoresearch.jsonl to understand recent results. Also check autoresearch.ideas.md for queued ideas. This is lightweight — do NOT re-read the entire file each iteration.
Step 3: Decide Hypothesis
Based on accumulated results, decide what to try next. This is where the main thread's strategic value lives:
- After a
keep: Build on the improvement. What's the next bottleneck? - After a
discard: Try a structurally different approach. Don't thrash on the same idea. - After a
crash: Fix if trivial, otherwise skip and try something else. - After 3+ consecutive discards: Step back. Re-read source files. Think about what the CPU/runtime is actually doing.
- Confidence < 1.0×: Recent "improvements" may be noise. Try larger, more impactful changes.
- Ideas backlog: Pull from
autoresearch.ideas.mdwhen you need fresh directions.
Step 4: Dispatch Subagent
Spawn autoresearch:experiment-runner with a prompt containing all context needed for one iteration:
- Run number: {N}
- Working directory: {working_dir}
- Primary metric: {metric_name} ({unit}, {direction} is better)
- Baseline: {baseline_value}
- Best so far: {best_value} (run #{best_run})
- Current segment: {segment}
Hypothesis
{What to try and why — be specific about which files to change and what changes to make.}
Files in Scope
{Copy from autoresearch.md — every file the agent may modify.}
Off Limits
{Copy from autoresearch.md.}
Constraints
{Copy from autoresearch.md.}
Recent History (last 5 runs)
{Formatted summary: run#, status, metric, description, asi — from JSONL tail.}
Keep the prompt concise. The subagent doesn't need the full session history — just enough to execute one iteration well.
Step 5: Parse Result
The subagent ends with a result block:
status: keep|discard|crash|checks_failed
metric: <value>
confidence: <score_or_null>
commit: <hash_or_empty>
description: <what was tried>
asi: <what was learned>
secondary: key1=val1 key2=val2
Parse these fields to update your running state.
Step 6: Update Strategy
Print a one-line summary to the user:
Run #5: keep | total_µs: 14,600 (-3.8%) | confidence: 2.3× | "Inline hot loop"
Adjust your internal strategy:
- Track consecutive discards/crashes.
- Note patterns in ASI across runs.
- Add deferred ideas to
autoresearch.ideas.md.
Step 7: Periodic Maintenance
Every 5 runs, update autoresearch.md:
- Refresh "What's Been Tried" with key wins, dead ends, and architectural insights.
- Commit the updated file.
This ensures a fresh agent (or context recovery) has current state.
Phase 3: Graceful Shutdown (Main Thread)
When a stop condition is reached:
- Wait for current subagent to finish — don't abandon a running experiment.
- Update
autoresearch.mdwith final state, total iterations, best result, promising unexplored ideas. - Commit all autoresearch files.
- Print summary:
- Iterations completed
- Best result (metric value + description + commit hash)
- Elapsed time
- Reason for stopping
Resuming a Session
When autoresearch.md exists (resume scenario):
- Read
autoresearch.mdfor session context. - Read
autoresearch.jsonlto reconstruct state (seesession-persistenceskill):- Find current segment, baseline, best kept, run count.
- Read
autoresearch.ideas.mdfor queued ideas. - Check git log for recent commits.
- Record new start timestamp for duration tracking.
- Resume looping from Phase 2.
Strategy Guidelines
- Primary metric is king. Improved → keep. Worse/equal → discard. Secondary metrics are informational.
- Simpler is better. Removing code for equal perf = keep. Ugly complexity for tiny gain = probably discard.
- Don't thrash. Same idea failing repeatedly → try something structurally different.
- Think longer when stuck. Re-read source files, study profiling data, reason about what the workload actually does. The best ideas come from deep understanding, not random variation.
- Annotate heavily. The main thread accumulates ASI across runs — this is your institutional memory. Use it to avoid repeating dead ends.
User Messages During Experiments
If the user sends a message while a subagent is running, wait for the subagent to finish, then incorporate their feedback into the next hypothesis.
Confidence Scoring
Determines whether an observed improvement is real or within measurement noise using Median Absolute Deviation (MAD).
When to Compute
- After 3+ experiment runs in the current segment (including baseline).
- Skip if fewer than 3 runs with positive metric values — report
confidence: null.
Algorithm
Given all metric values in the current segment (positive values only):
- Sorted median: Sort values, take middle element (or average of two middle elements for even count).
- MAD: For each value, compute
|value - median|. Take the sorted median of those absolute deviations. - Baseline: The metric value of the first experiment in the current segment.
- Best kept: The best
keep-status metric value (respecting optimization direction). - Delta:
|best_kept - baseline| - Confidence:
delta / MAD
Edge Cases
- If MAD = 0 (all values identical): return
null— no measurable noise to compare against. - If no
keepresults exist yet: returnnull. - If best kept equals baseline: return
null— no improvement to score.
Interpreting the Score
The confidence score is a multiple of the session's noise floor:
| Score | Meaning | Action |
|---|---|---|
| ≥ 2.0× | Improvement likely real | Safe to trust |
| 1.0×–2.0× | Marginal — could be noise | Consider re-running to confirm |
| < 1.0× | Within noise floor | Treat as no improvement |
How to Apply
When logging an experiment result to autoresearch.jsonl:
- Collect all positive metric values from the current segment.
- If count < 3, set
"confidence": nullin the JSONL record. - Otherwise, compute MAD and confidence as above.
- Record the numeric confidence value in the JSONL entry.
- When deciding
keepvsdiscard: the confidence score is advisory. It never auto-discards. But flag improvements below 1.0× in your ASI notes as "within noise — may not be real."
Example
Runs: [15200, 15400, 14800, 15100, 14600]
Median: 15100
Deviations: [100, 300, 300, 0, 500] → sorted: [0, 100, 300, 300, 500]
MAD: 300
Baseline: 15200 (first run)
Best kept: 14600
Delta: |14600 - 15200| = 600
Confidence: 600 / 300 = 2.0× ← improvement is real
Experiment Git Operations
Manages git state during the autoresearch loop. Successful experiments are committed with structured metadata. Failed experiments are reverted while preserving autoresearch state files.
Protected Files
These files are never reverted, regardless of experiment outcome:
autoresearch.jsonlautoresearch.mdautoresearch.ideas.mdautoresearch.shautoresearch.checks.sh
On keep — Commit
When an experiment improves the primary metric:
- Stage all changes:
git add -A - Check if there are staged changes:
git diff --cached --quiet(exit code 1 = changes exist) - If changes exist, commit with structured message:
<description>
Result: {"status":"keep","<metric_name>":<value>,<secondary_metrics>}
The Result: trailer is a JSON object containing:
status: always"keep"- Primary metric: key is the metric name, value is the numeric result
- Secondary metrics: each as key-value pairs
Example commit message:
Inline hot loop in render phase
Result: {"status":"keep","total_µs":14600,"compile_µs":4100,"render_µs":9500}
- If no changes to commit (pure measurement variation), skip the commit but still log to JSONL.
On discard, crash, or checks_failed — Revert
When an experiment fails or doesn't improve:
- Stage protected files first (so they survive the revert):
git add autoresearch.jsonl autoresearch.md autoresearch.ideas.md autoresearch.sh autoresearch.checks.sh 2>/dev/null || true
- Stash protected files, revert everything, restore them:
git stash push -- autoresearch.jsonl autoresearch.md autoresearch.ideas.md autoresearch.sh autoresearch.checks.sh 2>/dev/null || true
git checkout -- .
git clean -fd 2>/dev/null
git stash pop 2>/dev/null || true
This restores the working tree to the last committed state while keeping all autoresearch state files intact.
Branch Management
- New session: Create branch
autoresearch/<goal>-<YYYY-MM-DD>from current HEAD. - Resume: Stay on the existing autoresearch branch.
- Graceful shutdown: Commit all autoresearch files before stopping.
Why This Matters
Discarded and crashed experiments are completely erased from the working tree. The only surviving record is the description and ASI in autoresearch.jsonl. This is why heavy annotation of failures is critical — without it, future iterations will re-discover the same dead ends.
Metric Extraction
Parses structured output from autoresearch.sh to extract primary and secondary metrics.
Output Format
Each metric is a single line matching:
METRIC <name>=<value>
- Name: word characters (
a-z,A-Z,0-9,_), dots (.), orµ. Examples:total_µs,compile_ms,cache.hits - Value: any token parseable as a finite number.
NaN,Infinity, and non-numeric values are silently ignored. - One metric per line. Lines not starting with
METRICare ignored (but may contain useful diagnostics). - If a name appears multiple times, the last occurrence wins.
Primary vs Secondary
- Primary metric: The one whose name matches what was declared at session init (the optimization target). This is what determines
keepvsdiscard. - Secondary metrics: All other
METRIClines. Tracked for tradeoff monitoring but don't affect keep/discard decisions.
If the primary metric is missing from output, treat the run as a crash — the benchmark didn't produce the expected data.
Unit Inference
Infer units from metric name suffixes for display and context:
| Suffix | Unit |
|---|---|
µs | µs (microseconds) |
_ms | ms (milliseconds) |
_s or _sec | s (seconds) |
_kb | kb (kilobytes) |
_mb | mb (megabytes) |
| (none matched) | (unitless) |
Units are informational — they don't affect computation.
Tracking Across Runs
Maintain a list of known secondary metrics discovered across the session. When a new metric name appears in output that hasn't been seen before, register it with its inferred unit. This allows consistent reporting even when scripts evolve during the loop.
Recording in JSONL
When logging an experiment, record metrics as:
{
"metric": 14600,
"metrics": {
"compile_µs": 4100,
"render_µs": 9500,
"cache.hits": 42
}
}
metric: the primary metric's numeric value (top-level for easy querying)metrics: object of all secondary metric name→value pairs
Designing Informative Output
The autoresearch.sh script should output whatever helps the agent make better decisions:
- Phase timings when the workload has distinct stages
- Error counts or categories when correctness can fail in different ways
- Memory/cache diagnostics when relevant to the optimization
- Domain-specific signals that help localize regressions
The script can be updated during the loop as you learn what signal matters. Add instrumentation when you need more data to decide where to focus next.
Session Persistence
All experiment data is persisted to autoresearch.jsonl in JSONL format (one JSON object per line). This file survives across discards, crashes, and context resets — it is the canonical record of everything tried.
JSONL Record Types
Config Header
Written once at session init (and again on re-initialization when the optimization target changes):
{"type":"config","name":"<session name>","metricName":"<primary metric>","metricUnit":"<unit>","bestDirection":"<lower|higher>"}
| Field | Type | Description |
|---|---|---|
type | "config" | Distinguishes from experiment records |
name | string | Human-readable session name |
metricName | string | Primary metric name (must match METRIC output) |
metricUnit | string | Display unit (inferred or explicit) |
bestDirection | "lower" or "higher" | Optimization direction |
Experiment Record
Written after every experiment (keep, discard, crash, or checks_failed):
{"run":5,"commit":"a1b2c3d","metric":14600,"metrics":{"compile_µs":4100},"status":"keep","description":"Inline hot loop","timestamp":1699564800000,"segment":0,"confidence":2.3,"asi":{"hypothesis":"inlining reduces call overhead"}}
| Field | Type | Description |
|---|---|---|
run | number | Sequential experiment number (1-indexed) |
commit | string | Git short hash (7 chars) for keep; empty string for others |
metric | number | Primary metric value (0 for crashes) |
metrics | object | Secondary metric name→value pairs |
status | string | "keep", "discard", "crash", or "checks_failed" |
description | string | What was tried this run |
timestamp | number | Milliseconds since epoch (date +%s000 in bash) |
segment | number | Current segment index (0-indexed) |
confidence | number or null | Confidence score (null if < 3 runs) |
asi | object or null | Agent-supplied intelligence (free-form key-value) |
Segments
A segment groups experiments under a single baseline and config. Segments increment when init_experiment is called again (e.g., when the optimization target changes mid-session).
- Segment 0: experiments from initial setup
- Segment 1+: experiments after re-initialization
Confidence scoring and baseline comparisons only consider experiments within the current segment.
Session Recovery
When resuming (context reset, crash, or explicit resume):
- Read
autoresearch.jsonlline by line. - Parse config headers: Each
"type":"config"line starts a new segment. ExtractmetricName,metricUnit,bestDirection. - Parse experiment records: Reconstruct the results list with all fields. Track which segment each belongs to.
- Rebuild secondary metrics: Collect all unique keys from
metricsobjects across the current segment. - Find baseline: First experiment record in the current segment.
- Find best kept: Best
keep-status metric in the current segment (respecting direction). - Compute run count: Total experiment records (to set the next
runnumber). - Resume looping from where the session left off.
Also read autoresearch.md and autoresearch.ideas.md for context on what was tried and what ideas remain.
Writing Records
When logging an experiment:
- Compute all fields (including confidence via the confidence-scoring skill).
- Serialize as a single JSON line (no pretty-printing, no trailing newline between records).
- Append to
autoresearch.jsonl— never overwrite the file. - The JSONL file is the only record that survives discarded/crashed experiments. Every run must be logged before any git revert happens.
Reporting
After logging, print a one-line summary:
Run #5: keep | total_µs: 14,600 (-3.8%) | confidence: 2.3× | "Inline hot loop"
Components:
- Run number and status
- Primary metric with delta % vs baseline
- Confidence score (if available)
- Description (truncated if long)
After every 5 runs or on shutdown, print an expanded summary showing all runs in the current segment with their status, metrics, and descriptions.
Commands 1
Autoresearch
Duration: $1 minutes (optional - no limit if omitted)
Setup
-
If
$1is provided, write or updateautoresearch.config.jsonin the current directory:{ "maxDurationMinutes": $1 }If the file already exists, merge
maxDurationMinutesinto it (preserve existing fields likeworkingDirandmaxIterations). -
If
autoresearch.mdalready exists in the current directory (orworkingDirif configured), this is a resume. Announce: "Resuming existing autoresearch session with a {$1}-minute time limit." and skip to step 3.If no
autoresearch.mdexists, this is a new session. Announce: "Starting new autoresearch session." and continue to step 3. -
Invoke the skill:
Skill: autoresearch:autoresearch-createThe skill handles everything from here: gathering the optimization target (for new sessions), reading existing state (for resumes), and running the loop.
Agents 1
Experiment Runner
You execute a single autoresearch experiment iteration. You receive a hypothesis to test, implement it, run the benchmark, evaluate the result, handle git state, and report back.
Input Contract
Your prompt contains:
- Run number: sequential experiment index
- Hypothesis: what to try and why
- Baseline metric: the value to beat
- Primary metric name, unit, direction: what you're optimizing
- Files in scope: what you may modify (with notes)
- Off limits: what you must NOT touch
- Constraints: hard rules (tests must pass, etc.)
- Recent history: summary of last few runs (what worked, what didn't)
- Working directory: where all files live
Workflow
1. Read Context
Read the files relevant to your hypothesis. Understand what you're changing and why before writing anything.
2. Implement Changes
Make the code changes described in your hypothesis. Stay within the files in scope. Follow the constraints.
3. Run Benchmark
cd <working_dir> && ./autoresearch.sh
Parse output for METRIC <name>=<value> lines:
- Extract the primary metric (must match the declared metric name)
- Extract all secondary metrics
- If the primary metric is missing from output, this is a crash
4. Run Checks (if applicable)
If autoresearch.checks.sh exists and the benchmark succeeded:
cd <working_dir> && timeout 300 ./autoresearch.checks.sh
If checks fail → status is checks_failed.
5. Evaluate
Compare primary metric to baseline, respecting optimization direction:
- Improved →
keep - Worse or equal →
discard - Benchmark didn't produce primary metric →
crash - Checks failed →
checks_failed
Compute confidence score if 3+ runs exist (read autoresearch.jsonl to get prior values):
- Collect all positive primary metric values from current segment
- Sorted median of values → median
- Sorted median of
|value - median|→ MAD - If MAD = 0 or no kept results or best kept = baseline → confidence = null
- Otherwise → confidence =
|best_kept - baseline| / MAD
6. Log to JSONL
Append one JSON line to autoresearch.jsonl:
{"run":<N>,"commit":"<hash_or_empty>","metric":<value>,"metrics":{<secondaries>},"status":"<status>","description":"<what was tried>","timestamp":<epoch_ms>,"segment":<seg>,"confidence":<score_or_null>,"asi":{<learnings>}}
Write the JSONL record before any git revert — it must survive.
7. Git State
On keep:
git add -A
git diff --cached --quiet || git commit -m "<description>
Result: {\"status\":\"keep\",\"<metric_name>\":<value>,<secondaries>}"
On discard, crash, or checks_failed:
# Preserve autoresearch files, revert everything else
git stash push -- autoresearch.jsonl autoresearch.md autoresearch.ideas.md autoresearch.sh autoresearch.checks.sh 2>/dev/null || true
git checkout -- .
git clean -fd 2>/dev/null
git stash pop 2>/dev/null || true
Output Contract
You MUST end your response with a fenced result block in exactly this format:
status: keep|discard|crash|checks_failed
metric: <primary_metric_value>
confidence: <score_or_null>
commit: <short_hash_or_empty>
description: <one-line summary of what was tried>
asi: <one-line summary of what was learned — not what was done>
secondary: <key1>=<val1> <key2>=<val2> ...
This block is how the orchestrator reads your result. It must be the last thing in your response.
Rules
- Do not ask questions. You have everything you need. If something is ambiguous, make a reasonable choice and note it in ASI.
- Do not skip the benchmark. Every iteration must run
autoresearch.shand produce a metric. - Do not skip JSONL logging. The log must be written before any git revert.
- Do not modify files outside scope. Respect the "off limits" list.
- Annotate failures heavily in ASI. Reverted code is gone — ASI is the only surviving record of what was tried and why it failed.
- Keep it fast. You are one iteration of many. Don't over-engineer or over-analyze. Implement, measure, report.
Sponsors
Details
- Marketplace
- pbdeuchler/llm-plugins
- Source
- plugins/autoresearch
- Author
- PBD
- Category
- Community