dinosn avatar

raptor-loop-hunt

Autonomous looping multi-altitude vulnerability hunt (generate -> judge -> verify).

version
1.0.1
GitHub stars
278

Install

/plugin install raptor-loop-hunt@raptor

Run it inside Claude Code after adding the marketplace with /plugin marketplace add dinosn/raptor-loop-hunt.

Contents

Discovered from source

Skills 1

RAPTOR Auto-Research Vuln Hunt

A natural-language agent loop for security research. A generic "find bugs" prompt lets the model fall back to its defaults — single pass, mid-level altitude, converge fast, summarize, stop. Those defaults are wrong for vuln hunting. This skill replaces them with an explicit search procedure: traverse every altitude, generate then adversarially verify from raw, run isolated parallel reasoners, and keep a persistent ledger so each loop is net-new coverage instead of rediscovery.

The prompt is the program. In an agentic system the model's "algorithm" is whatever you tell it to be. This skill specifies that algorithm. Follow the structure; the quality comes from the structure, not from any single clever instruction.

When to use

Use for any "go deep / find everything / audit this thoroughly" security request against a codebase. Don't use for a quick triage, a single known-CVE reproduction, or a one-file sanity check — those want /scan or /understand --hunt directly. This is the heavy, looping, high-coverage mode.

The core loop

Run this as a loop, not a one-shot. Each round:

  1. Pick an altitude and a slice you have not exhausted (see traversal below).
  2. Generate candidate findings on that slice with an independent reasoner working from raw source (no prior summaries in its context — summaries cause anchoring).
  3. Judge each candidate with a separate reasoner, also from raw, prompted to refute.
  4. Live-verify survivors against the actual code before they count (see guardrails).
  5. Record everything tried and everything found in the ledger.
  6. Vary the approach next round — new altitude, new bug-class lens, new slice. Novelty is mandatory; repeating a round wastes budget.
  7. Check the stop condition. Loop until dry, not literally forever.

Multi-altitude traversal (the coverage guarantee)

Different bug classes live at different zoom levels. A single-pass scan implicitly picks one altitude and is structurally blind to the others. Cover all four, in order, and revisit:

  • Whole project — architecture, trust boundaries, auth model, data flows across modules. Catches the broken-object-level-authz / IDOR / missing-authz class that dominates real web audits, plus deserialization sinks, SSRF, and design-level bypasses.
  • File by file — each file's responsibilities, its inputs, its exported surface.
  • Functionality by functionality — each feature end to end (upload, import, export, templating, auth, admin actions). Trace source → sink for every bug class, not just the headline one. A mapped-but-untraced entry point is an uncovered entry point.
  • Function by function — parsing, memory, encoding, length math, crypto comparisons, format strings, integer handling. Line-level bugs only surface here. This is the altitude with the weakest mechanical support — the Semgrep anchors above are seeded from whole-project trust boundaries, so nothing systematically walks the interior functions no entry-point cell anchors. RAPTOR's /audit can be used as an experimental candidate source over that interior: it works from its own checklist-derived gap set and, for functions that reach LLM review, forms a hypothesis and may invoke an applicable analyser. Neither a hypothesis nor a sweep is guaranteed per function — triage and prefilter can short-circuit to clean first. Wire it in the generator seat only, and read the whole contract in "Mapping to RAPTOR's machinery" before using any of its output.

Track which (altitude × slice) cells you've covered. The point of "start with the whole, then file by file, then functionality, then function" is that you sweep the whole grid, not one band.

Component inventory first — the completeness gate (MANDATORY)

Before round 1, enumerate the entire project as a flat list of components — every top-level module, package, service, transport, and deployable unit — not just the subsystems that look interesting. For a multi-module build, list every module directory (ls modules/, every Maven pom.xml, every top-level source package). This inventory is the denominator for coverage.

Then maintain a coverage matrix: every component maps to a hunt cell — now, or a named, logged later round. The hunt is not "done", and the report must not read as done, until every component is either covered or explicitly listed as omitted with a reason (out of scope, build-time-only tool, generated code, third-party vendored copy). Silent omission is the exact failure this gate prevents: slicing by hot-spots and quietly skipping whole modules (transports, databinding, the newer/less-audited modules) is how a shallow audit masquerades as a complete one — especially on a mature project that has had many issues over the years, where the unexamined module is often where the next bug lives.

Rules:

  • The per-round slices must be drawn from the full inventory; you may prioritise, but every component not yet assigned to a cell is recorded as UNCOVERED in TRIED.md — never absent from it. The ledger's component list must equal the project's component list.
  • A component is the unit of accountability. "I audited the interesting parts" is not a complete assessment. When the codebase is large, scale the number of rounds to cover all of it rather than narrowing the inventory to what is convenient.
  • Re-run the enumeration when the target changes (new clone, new release) — modules get added (e.g. a new protocol bridge, an OpenAPI/REST surface) and a stale inventory silently drops them.

The entrypoint manifest is the coverage SPINE (mechanical, not a hand list). A component list is too coarse — a route can fall between hand-picked cell anchors and be silently skipped (the miss: GET /users/:id/calendar-heatmap was never read because one cell listed six other controllers and another opened the file at the wrong route). Before scheduling cells, deterministically extract every externally-reachable entrypoint — controller routes (method, path, class+method decorators, inherited auth defaults, handler span), framework filesystem routes (e.g. SvelteKit +page.ts / +page.server.ts loads & actions), and statically-enumerable RPC/event handlers — each with a stable entry_id. A cell may claim coverage of an entrypoint only by recording its exact entry_id + handler span + effective audience/auth + primary callee + the boundary-scout check-ids run; opening one line in a controller does not cover the controller, and a wildcard "all routes covered" receipt is forbidden. Round closure fails when manifest_entry_ids − covered − approved_exceptions is non-empty — a deterministic diff, not a model "completeness critic" re-reading its own work. Keep hand-picked non-route anchors (repositories, background workers, parser/process sinks, state transitions) — routes are the spine, not the whole skeleton.

Deterministic front-load — cheap ground truth before the LLM loop (Round 0)

Deterministic tools are fast, hallucination-free, and refusal-free. Run them before the first LLM round and don't spend inference rediscovering what they already know:

  • Cross-run Knowledge Base (kb/) — a MONOTONIC-SCRUTINY signal: it can only ever make you hunt MORE. If this target was hunted before, a durable KB sits beside the ledger ($KB/kb.json). It stores no coverage and no "this is safe" signal; it can only raise scrutiny. Load it into the planner context only, AFTER you have freshly enumerated the whole inventory this run (the completeness gate below) into inv.txt: scripts/raptor-loop-kb load --kb "$KB" --target "" --inventory inv.txt
    • Priority, never coverage. priority_order puts confirmed-dirty (a prior confirmed/corrected finding) first, then prior-rejection components (recheck), then everything else. Every current_state is uncovered and nothing is ever deprioritized — historical work NEVER counts as current coverage and never pushes a component out of scope. Draw this run's slices from the fresh inventory; the KB only changes the order.
    • Rejections are recheck ANNOTATIONS, never an exclusion. Each annotations[] entry says "previously rejected for X — recheck X and ALL delivery vectors (path/query/body/cookie/header/enc)"; a stale one (tree changed since) reads "FULLY OPEN, recheck from scratch." The candidate still runs the full generate → judge → live-verify chain from raw — the annotation only tells you where to look harder.
    • Round 0 is never skipped. The KB seeds /sca, inventory enumeration, prior-art recon, and mapping — it does not replace them. New CVEs, new lockfiles, new advisories, and new modules are seen every run.
    • Isolation (MANDATORY). The payload stays in the planner/orchestrator context. Never feed a prior rejection or summary into the independent generator, judge, or live-verifier — they reason from raw source
      • current scope. Injecting history re-creates the anchoring "Generate → judge, both from raw" prevents.
  • Known-CVE deps (/sca). SBOM + dependency-CVE audit is deterministic and cheap. Run it first, log the hits, and exclude those packages from the LLM hunt scope — the model's budget is for the bespoke bugs a scanner can't find, not for re-deriving a public CVE in a pinned dep.
  • Native-target reachability ground truth (binary-oracle). For C/C++/Rust/Go targets with a locally-built debug binary, the binary-oracle is deterministic reachability: it joins the source inventory to the binary via DWARF + nm and marks each function symbol_present / inlined / folded (survived compilation) or absent (compiler/linker removed it). It is auto-detected and on by default in /agentic, /codeql and /audit — pass --binary <path> for an explicit build, --binary-auto for a louder auto-detect, --target-kind auto|library|hybrid|application (default auto). An absent verdict hard-suppresses the finding before the LLM ever sees it (/agentic and /codeql log it to suppressions.jsonl; /audit suppresses without logging, so that path has no audit trail). A symbol_present / inlined verdict refutes a later "that's dead code" kill — but note it proves survival in that binary, not reachability; "no caller" is a separate claim needing --binary-edges or a source call graph.
    • --no-binary-oracle is /codeql and /audit only. /agentic never registered it (its own argparse block declares --binary, --binary-auto, --binary-edges and nothing else), so the flag is silently ignored there (worse: /agentic's own code reads and recommends the flag it never registered) — /agentic's escape hatch is --allow-unreachable, which bypasses the reachability suppression chokepoint wholesale, not just this oracle. It does not stop binary auto-detection or inventory enrichment: the verdicts are still computed and still annotate the inventory.
    • This is build-specific compilation-survival evidence, not source ground truth. --binary is validated as "is a file" and nothing binds it to the current commit, dirty tree or build config, so a stale or partial binary can hard-suppress live source findings. Treat a verdict as evidence about that build; when the binary's provenance is not pinned to the audited tree, do not let it suppress.
    • /audit suppresses on MIXED-TIER evidence — prefer /audit --no-binary-oracle. The canonical reachability path refuses to suppress when any contributing binary is below full-DWARF tier, but /audit's extraction keeps an absent verdict when merely one binary has full DWARF, then drops path, line and tier and keys by bare function name. That defeats both the symbol_only-never-demotes rule and the two-signal gate.
    • The oracle is not the only pre-LLM kill. module_aborts and lexical_dead are also suppression-eligible structural witnesses, and /agentic deterministically marks test fixtures clean and skips the LLM entirely, with no CLI switch to disable it. /audit's small-function batch path has its own gate: a function hitting _dead_code_reason is committed dormant with evidence_tool="reachability:dead_code" without any LLM review at all. "Binary absent is the one permitted hard-suppressor" is this methodology's policy, not the framework's behaviour — import every fixture/structural suppression as an open candidate unless a loop-owned receipt backs it.
  • CodeQL on C/C++ is BUILDLESS by default — its negatives are scope-limited. RAPTOR now creates C/C++ databases with codeql database create --build-mode=none, so the untrusted repo's build scripts never execute; detected build commands are ignored unless traced extraction was explicitly selected, and a CodeQL CLI older than 2.16 fails the create rather than silently falling back to a traced build. Traced extraction is opt-in — --traced-build, or an explicit per-language --build-command — and is deliberately independent of --trust-repo: trusting the repo buys you no traced build, and a traced run still refuses on unsafe CodeQL pack config. The cost is coverage: build-generated headers are invisible. RAPTOR surfaces this, but weakly — the logged number is a regex count of extractor-diagnostic lines mentioning unresolved includes, it collapses to zero on a parse failure, zero prints a generic warning with no number, the summary runs only after a fresh successful create (a cached database returns before it), and the count is never persisted into database metadata. Consequence for this loop: a "CodeQL found nothing here" receipt on a C/C++ target must record whether extraction was buildless or traced and, when buildless, retain the original creation log (or an independently captured extractor-diagnostic inventory). A zero or a missing count is not proof that headers resolved. Without that provenance the sweep is partial coverage presented as complete — the same fraud class as a capped rule sweep. Other languages are unaffected.
  • /scan can silently lose registry rule packs. The Semgrep preflight opens a 3-second TCP probe (to the configured HTTPS proxy's first hop when one is set, else semgrep.dev:443); on failure it logs a warning, drops every uncached registry pack, and scans on with what remains — exiting 0. A /scan receipt must carry the resolved pack list, the applicable-rule count, and any "dropping uncached registry pack(s)" warning. A clean exit after pack-dropping is not a full sweep.
  • Project trust markers change later runs — announced, but easy to miss. /project trust config|build|dynamic (libexec/raptor-project-manager trust <marker>) persists an operator assertion that RAPTOR applies to every subsequent run: config → --trust-repo and build → --traced-build for /agentic and /codeql; dynamic → dynamic validation for /audit. Direct /scan does not consume them (only flags forwarded from /agentic or /codeql carry that state). Precedence is explicit-negative > explicit-positive > marker > off, and RAPTOR prints a one-line banner whenever a marker actually affected the run — so this is visible, not silent, provided you read it. Record the effective flags and the active marker set per round in TRIED.md: flipping a marker mid-engagement re-characterises the provenance of every receipt that follows it.
  • Target's OWN known vulns + upstream fixes (prior-art recon) — MANDATORY, not optional. Pull the TARGET application's history, not just its dependencies, BEFORE the LLM loop:
    • Its CVE/GHSA record — OSV (POST https://api.osv.dev/v1/query {"package":{"name":..,"ecosystem":..}}), NVD (keywordSearch=<name>), gh api repos/<o>/<r>/security-advisories. Every past CVE names a vulnerable sink class + file; treat each as a post-fix variant re-audit lead (is the fix complete? does a sibling path / different delivery vector bypass it? is the sink still reachable?). It also calibrates severity — if the vendor/CVE process treated an authenticated/admin-area bug of class X as CVE-worthy before, don't dismiss your class-X finding as "admin-only, informational."
    • Upstream open + recently-merged security PRs and recent security commits — gh pr list --state open, gh search, git log <last-release>..HEAD -- <hot files>, patch-diff <vuln-tag>..<fixed-tag>. An open or queued fix is a vendor-acknowledged live bug in the current release: seed it as a known-real finding, and NEVER reject a candidate that an upstream PR is actively fixing (this is exactly how a real finding gets wrongly killed — the vendor was patching it while the audit rejected it).
    • Public PoC/exploit search — gh search repos/code <name>, WebSearch <name> exploit/PoC. This is the standing "search existing PoCs / CVE-breadth" discipline applied to a FROM-SCRATCH AUDIT, not only to known-CVE reproduction. Skipping it means re-deriving — or wrongly rejecting — bugs the vendor already flagged. Log what you checked in TRIED.md as methodology evidence.
  • STRIDE template (/threat-model build). One pass over the /understand --map recon that yields trust boundaries, entry points, and a per-boundary STRIDE classification. Its output is a reusable template that seeds every later round's bug-class lenses — it tells the generators where privilege changes and which STRIDE class to hunt at each boundary. Recon and the threat model are infrastructure for the loop, not outputs — never report them as findings.
  • Semgrep anchors seeded from the trust boundaries. Turn each threat-model trust boundary into a Semgrep pattern and run those rules alongside the generators every round. Semgrep gives fast deterministic anchors at exactly the boundaries the threat model flagged; the LLM reasons about the semantics, data flow, and exploitability pattern-matching can't reach. The union of Semgrep hits and LLM candidates is the candidate pool (high recall, high noise — by design). A Semgrep hit is a lead, not a finding: it enters the same finding contract and the same from-raw judge as any LLM candidate. Keep Semgrep in the generator (recall) seat only — a pattern match never stands in for the judge.
    • Standing anchor — check-view ≠ use-view. Beyond the per-boundary rules, always seed one anchor for the policy–execution interpretation differential class (references/vuln-class-discovery.md): a security-relevant boolean derived from equalsIgnoreCase / regionMatches(true,…) / startsWith / indexOf(x)==0 / a decode-then-match on a name/URI/id that guards a privileged branch. It is a high-recall/high-noise candidate generator, never a detector: promote a hit only when a judge can exhibit a witness x the control interpretation misses but the action interpretation accepts, and suppress it when the identifier is not attacker-influenced or both consumers share one canonicalization. (Both our own scheduler-alias false-negative and the CVE-2026-45505 RCE reduce to this class — and note grep alone under-matches regionMatches(true and over-matches benign equalsIgnoreCase, so treat it as an AST/taint-aware anchor, not a text scan.)

Per-class discovery method (load when a bug-class lens is active)

The altitude traversal says where to look; references/vuln-class-discovery.md says how to find and confirm each vulnerability class, generically. For every bug-class lens a round runs, load that file's matching section and fill its five slots against the target — source class, sink class (by mechanism), violated invariant (the finding's root cause), enumeration strategy (derive the complete sink set from the inventory), confirmation oracle. Stack-agnostic by design; concrete signatures stay in the deterministic layer (Semgrep, /sca).

Generate → judge, both from raw

Separating generation from verification is the single biggest quality lever:

  • The judge kills false positives — independent skeptic, prompted to refute. It defaults to "not a bug" only when the defect itself is uncertain (the code path isn't real, the entry isn't attacker-controlled, the sink isn't dangerous). It does not default to "not a bug" because some unobserved layer might mitigate it — see below.
  • The act of re-reading from raw to verify surfaces new findings the generator missed.
  • "From raw" is load-bearing. A judge that reads the generator's summary rubber-stamps it. A judge that reads the source re-examines it. Always feed the judge the code, not the claim.

What counts as a valid refutation (and what doesn't). The judge may only kill or downgrade a finding for a reason it can see in the artifacts:

  • VALID: the cited code doesn't do what the claim says; the entry point isn't attacker-reachable; a mitigating check is present in code you can read; it's designed behavior under the established trust model.
  • INVALID: "a well-built server probably enforces this," "the framework likely handles it," "presumably there's authz upstream," "the server surely re-hashes/re-validates." Assuming an unseen layer is secure is not refutation — it is the single most common false-negative in this methodology. The whole reason for the test is that the unseen layer might be broken. Don't assume the unseen layer is secure.
  • INVALID (reachability / gating): "that component isn't built/loaded," "it needs a non-default config," "it's behind a flag," "the trigger needs an impractical number of iterations" — asserted without checking. A reachability or gating claim kills a finding, so it carries the finding's own proof burden: verify it against the observable build + default config (the build flag, whether a default deployment exposes the entry point, the real trigger path) before it counts as refutation. A capability that ships and is enabled by default is in scope even when it's packaged as an optional-looking "module" or "plugin." Don't assume the gate is closed. For native (C/C++/Rust/Go) targets, the binary-oracle mechanizes exactly this check: an absent verdict is an observed dead-code reject (the function isn't in the shipped binary), while symbol_present / inlined / folded refutes a "compiled away / dead code" kill — so a compiled-away claim is only a valid refutation when the oracle (or an equivalent DWARF/nm check on the actual build) confirms it, never on assertion.

When the only barrier between a finding and exploitation is a layer you cannot observe (server code you don't have, a runtime you can't run, a config out of scope), the correct verdict is not "rejected/low" — it is needs-live-validation (see the disposition below). Preserve the worst-case severity and emit the exact, safe test that would confirm it.

For higher-stakes hunts, use a small panel (e.g. 3 skeptics with distinct lenses — correctness, reachability/exploitability, does-it-actually-reproduce) and require a majority to keep a finding. A panel may move a finding to needs-live-validation, but a finding is only rejected when a majority can point to an observed reason it cannot be a bug.

Concurrency and isolation

Run N independent reasoners (default N=10) with full isolation — each works from raw, blind to the others' conclusions. This is an ensemble: isolation preserves diversity, which is the coverage. Drop isolation and they converge on the first finding (groupthink). Use a pipeline so generation, judging, and verification stream rather than barrier-block.

The per-anchor boundary scout + lossless leads (the recall-miss and false-clear fixes)

A cell scoped to ONE headline bug-class lens is structurally blind to a different class living at the same code. This is the dominant real-world recall failure — measured, on a re-hunt that used stronger models than the run it was compared against: an SQLi-lens cell read searchStatistics and missed the cross-tenant authz omission there; an XSS-lens cell read the license route and missed the CSRF state-changing-GET; an authz-gate cell proved a shared-link "correctly gated" and never looked at the response mapper leaking emails; an IDOR cell read tag.addAssets and mistook Permission.AssetShare (owner OR partner) for an owner-only gate. Every one of those was read and still didn't surface. Two mechanisms fix it — neither is "run five full hunts per anchor":

1. A short, isolated boundary scout after the primary lens. For each externally-reachable or security-sensitive anchor, run ONE extra reasoner that is blind to the primary generator's verdict and coverage prose (feed it the anchors and the shared cards below, never "authz is soundly implemented" — otherwise it rationalizes the same omission). It evaluates only the boundary checks that apply, emitting a structured observation or a source-backed N/A for each:

  • ENTRY — the real audience/auth decorator (incl. inherited class defaults); any state-changing GET / page-load / load() that a link click triggers (CSRF / forced-action). (F-17, F-25)
  • SUBJECT — bind the actor to every path/body id, session id, and target resource; expand a permission's semantics, never trust its name (AssetShare = owner-or-partner-or-shared-link, resolved to its predicate spans). (F-20, F-24)
  • SECURITY STATE — visibility (Locked/Hidden), elevation/PIN, revocation, lease/session renewal, stale memberships. (F-34, F-02/F-08)
  • OUTPUT — follow the final serializer/mapper graph and enumerate the sensitive fields it emits against every route audience (unauth / shared-link / user / partner / admin). (F-18, F-19)
  • DANGEROUS SINK — for any interpreter/parser/process, resolve the data channel, invocation mode/options, interpreter grammar/metacommands, and execution identity. (F-22 psql \!) Reuse capability/mapper cards across cells so this is cheap: one card per Permission.* (its real predicate + accepted principals), per response mapper (fields × audiences), per visibility/elevation check, per process sink (argv/stdin/identity). Cost is ~+20–35% generator work, not 5×.

2. Leads are lossless — a concrete defect cannot die in prose. Cell output is typed: a hypothesis (a question — "does statistics have the same visibility bug?") may stay a lead until traced, but a defect_observation — a reachable operation + a specific missing/mismatched binding, filter, cleanup, elevation, or policy condition, with source spans for both the operation and the control/asymmetry — automatically opens or joins a candidate and runs the full generate→judge→cross-vendor chain. A defect_observation may never be marked safe / latent / intended / defense-in-depth in a coverage_note; coverage notes are non-dispositive (they carry coverage-receipt references, not security verdicts). This is the exact gap behind the three false-clears: the HLS sessionId-not-bound-to-caller (F-24), the "duplicate groups are always single-owner" invariant (F-07), and the "mapUser email is intended baseline" dismissal (F-18) were all present in the run's prose and never became candidates, so no gate ever fired. Two of those are now typed rejection receipts the ledger enforces: an invariant kill must inventory every writer to the invariant-bearing field (incl. bulk/import/restore/deser — the F-07 miss was the unlisted PUT /assets {duplicateId} writer) plus a counterexample attempt; an intended-behavior kill must cite a real policy artifact — "another endpoint already exposes it" is a second bug, not intent, and a contrary privacy control (publicUsers) is evidence against. Dedup observations by root span/resource/control so one defect is one candidate, not two chains.

3. Bounded security-differential cards feed the scout. Several misses were near-sibling contradictions a compact mechanical diff puts straight in front of the reasoner: search() vs getStatistics() visibility predicates (F-42), searchStatistics omitting visibility its siblings pass (F-23), bulk updateAll cleanup vs the singular update (F-02/F-08), a download path lacking the elevation its sibling timeline/search paths enforce (F-34). Generate security_contrast cards — singular-vs-bulk on a resource, sibling query methods differing in owner/visibility/deleted/elevation/ session/permission predicates, omitted security args at sibling callsites, all writers to an invariant field — restricted to the same resource/operation family (AST/call-fingerprint + LSP references; batch, never top-K truncate), and attach them to the scout. Don't launch a separate full generator.

Discovery is severity-neutral. "No LOW-padding" is a reporting rule, not a generation filter: a concrete reachable disclosure/control defect enters the candidate pipeline regardless of tentative severity — severity is assigned after evidence. Pre-judging externalDomain as "by-design public" suppressed a credential-bearing-URL leak (F-33) before it could be evaluated. The concrete-defect threshold (not a severity floor) is what controls candidate volume.

Invalid-cell lint. A cell whose only output is a placeholder/stub candidate (fields literal "test", evidence file:"a" line 1, no source spans) is an invalid execution, not a dry cell — it fails closure and reruns (the ledger's add refuses it). A "dry" verdict requires source-backed coverage receipts.

The attempt ledger (what makes "loop forever" productive)

This is the part most people omit, and it's what turns an infinite loop from waste into a converging search. Maintain two files in the run's output directory:

  • TRIED.md — every (altitude, slice, bug-class, approach) attempted this engagement, with the outcome (nothing / lead / confirmed / refuted). Before each round, read it; never repeat a cell within the current engagement. TRIED.md is engagement-scoped — it is NOT the cross-engagement coverage authority. On a new engagement (fresh clone / new release / changed tree — a changed target identity) start a new TRIED.md: every component begins UNCOVERED regardless of what any prior engagement covered. Cross-engagement history enters only as the KB's monotonic priority/recheck signal, never as coverage.
  • FINDINGS.md (or EXISTING-FINDINGS.md) — confirmed findings, used as an exclusion set so the loop doesn't re-surface what's already logged.

Dedup new candidates against FINDINGS.md and against the judge-rejected set for the current engagement only — this is what stops a rejected finding reappearing every round within a run so the loop converges. This dedup is strictly engagement-scoped; it is NOT a cross-engagement exclusion. A candidate is suppressed only when it is the same candidate instance already adjudicated in this engagement, against the same tree identity. Across engagements — or the moment the tree identity changes — confirmed/rejected history is not an exclusion set: it becomes a KB recheck annotation and the resurfaced candidate re-enters the full generate → judge → live-verify chain from raw. A changed-code variant that merely shares an old signature must never be dropped on that signature. The ledger gives "try something new every time" a reference point — but that reference point resets at the engagement boundary.

Cross-run Knowledge Base — a monotonic-scrutiny layer above the ledger

TRIED.md / FINDINGS.md are the engagement-scoped ledger (above). The Knowledge Base (kb/) is the durable cross-engagement layer, and it obeys one rule: it can only ever RAISE scrutiny. It stores no coverage and no "safe" signal, does no deprioritization, and never excludes a candidate. It emits exactly two things: confirmed-dirty components (hunt first) and prior-rejection recheck annotations. Because every signal only adds effort, the KB cannot semantically suppress, cover, or clear a candidate: given the freshly enumerated inventory is accepted in full (the helper fails closed on any inventory cap/invalid entry) and the completeness gate is honoured, a maximally poisoned or stale KB can only change hunt order and add recheck work — the worst case is wasted budget re-hunting a real component, never a skipped one.

Only structured, locally-recomputed facts steer it. Component identity from a fresh enumeration (a name not in this run's inventory is ignored); source identity + drift from local digests; finding disposition from typed, schema-validated records the orchestrator writes; evidence spans resolved under the canonical target root. Trajectory final_summary / "gave-up" / tool-error prose is attacker-influenced in origin, so it is display-only telemetry and can never become a rejection, a priority, or any durable state.

Where it lives (RAPTOR-owned, OUTSIDE the scanned tree): <project_output_dir>/kb/ (or out/kb/<target_path_id>/). The helper refuses a symlinked KB path, a KB inside the target, or an unowned/corrupt store; writes are atomic + locked + fsynced; the inbox is race-free (all appends take the lock; synthesize rotates it aside before folding). Schema + learnings.jsonl grammar: references/kb-schema.md.

Enable trajectory capture. Pass the run's output directory to every /understand and /cve-diff call — the FLAG is what sets the location, and the flag name differs per command: /understand --out "$OUTPUT_DIR", /cve-diff --output-dir "$OUTPUT_DIR" (--out is not a /cve-diff flag and is rejected). Exporting RAPTOR_TRAJECTORY_DIR yourself is a no-op: both libexecs assign os.environ["RAPTOR_TRAJECTORY_DIR"] from their own resolved output dir, overwriting whatever you exported. But the flag only sets the location — it does not guarantee a trajectory.

A trajectory is written only by an agent tool-use loop, and /understand reaches one on a minority of its routes. /cve-diff opts its agent loop in under --output-dir. /understand is mode-routed (dispatch: skill): source-tree --map, --hunt, --trace, --teach and --study run in-session (you are the LLM) and write lifecycle and result artifacts — variants.json, context-map.json — but no trajectory; the libexec rejects source-tree --map and refuses --hunt / --trace without --model. Even on the libexec path only the LLM hunt/trace loop persists: with --hunt-tool auto (the default) a C/C++ target with spatch on PATH selects the Coccinelle backend, which makes one rule-writing model call and no tool-use loop; binary --map sets the env var but never runs one either. So for a hunt trajectory pass --model <name> --hunt-tool llm, and keep the Coccinelle rule + match output as a mechanical sweep artifact when auto picks that backend instead. Verify by existence: no trajectories/<run_id>/trajectory.json means no reflect input for that round — never assume the record landed. (/agentic does not persist trajectories either; for that stage reflect has no input and the steering records come from adjudication.)

Stop condition — loop until dry, not literally forever

"Continue no matter what" is the right attitude but the wrong literal rule — it burns budget on diminishing returns. Concretely: stop after K consecutive rounds (default K=2) that surface nothing new across all remaining uncovered cells. If a round is dry, increment the counter and switch to an under-explored altitude or bug class before giving up. Report what was left uncovered rather than implying the grid was exhausted.

What counts as a finding — the bar, and how to rate it

The loop generates candidates aggressively. This is the bar a candidate must clear before it counts. The loop has no schema validator, so enforce this contract yourself.

The finding contract. A reportable finding states, every time:

  • Root cause, templated: "<function> in <file> does not <missing check>, allowing <consequence>." Name the function and file where the defect actually lives.
  • A trace that starts at an attacker-controlled entry point and ends at a dangerous sink. If you can't name both endpoints from the map, the trace is incomplete — that's a lead, not a finding. Check this invariant by hand on every finding (the first hop is an entry point, the last is a sink).
  • A concrete attack: who the attacker is, the exact input / request / action sequence, and the observable result. "An attacker could theoretically…" is not a finding.
  • Severity as likelihood × impact (below) and confidence with the reason you scored it.

Severity = likelihood × impact. Rate on both axes; don't inflate:

  • Critical — unauth RCE, full data dump, admin takeover without credentials.
  • High — authed RCE, SQLi with exfiltration, stored XSS firing for all users, auth bypass, or an explicit role/permission boundary completely defeated for a consequential action.
  • Medium — XSS needing specific conditions, CSRF with real state change, secret/credential disclosure, logic bypass confined to the attacker's own data or needing uncommon conditions.
  • Low — non-secret info disclosure, DoS needing sustained effort, hardening gaps.
  • If you can't describe the concrete damage, the severity is lower than you think.

Two-axis rating when evidence is partial. When you can see only part of the system (client code without the server, one service of many, no running instance), rate two numbers — never one collapsed number:

  • Confirmed severity — what you can prove from the artifacts in hand.
  • Potential severity — the worst case if the unobserved assumption turns out insecure.

Carry the higher one into triage as needs-live-validation. Never collapse potential severity to Low just because the confirming layer is out of view. A client-side trace showing an unauthenticated password-reset request with a client-controlled target id and a client-controlled "skip the email check" flag is a potential critical the moment the path is real — even though the server isn't in scope. That is a validate-now item, not a low hardening note. (This is not hypothetical: that exact finding was downgraded to "low/server-dependent" by an over-eager judge and later confirmed as live, unauthenticated mass account takeover.)

Rate the sink, not the symptom — trace consequence before any final Low. When a finding's claimed impact depends on attacker-influenced data or control reaching a security-sensitive operation (command execution, unsafe query construction, unsafe deserialization, template eval, arbitrary file access, or restore/import content being interpreted as code), record the entry→operation trace and rate the operation's consequence, preconditions, barriers, and execution privilege — never the entry symptom alone. A finding whose disposition is written from the symptom ("missing @Authenticated", "unauth endpoint", "missing check") without following the invoked operation is a lead, not a rated finding: do not finalize it Low / hardening / not-security while the downstream consequence or an alternate ingress is unresolved — record it as a lead with a potential-impact hypothesis in a needs-trace/needs-live state. Operation-first, even for an "auth" finding: when you flag an endpoint as unauth/missing-auth, inspect what it invokes — if that is a restore/import/eval/subprocess or a query builder, trace it before rating. (Two honest boundaries: a protected operation whose impact is already evident supports an authorization finding without tracing to a separate sink; and ordinary parameterized SQL, safe deserialization, or routine file access is not automatically a dangerous-sink finding — the trigger is attacker-influence reaching the operation with plausible High/Critical impact, not the mere presence of the API.)

Disposition — every survivor gets exactly one:

  • confirmed — proven exploitable from the artifacts or a run. Rate confirmed severity.
  • needs-live-validation — the defect pattern is real and reachable in what you can see, but the final proof depends on an unobserved layer. Required fields: the exact minimal, safe validation step (the request/command/test, plus the expected vulnerable-vs-safe response) and the potential severity. A first-class outcome, not a soft reject.
  • corrected — real but mis-scoped/mis-rated; give the accurate version. Use this to fix observed errors, not to downgrade for unseen mitigations.
  • rejected — a majority can cite an observed reason it is not a bug (wrong code, unreachable, mitigation present in the code, designed behavior). "Probably handled elsewhere" is not such a reason.

Disposition receipts — the reasoner proposes, the orchestrator certifies

A disposition written by the same generator/judge that reasoned the finding is a claim, not proof. The weak-tier seats this loop fans out to (Sonnet judges, uncensored local generators) will state a discipline in prose and then skip it — so a disposition is not a sentence a reasoner emits, it is a state transition the orchestrator refuses until an evidence receipt exists. Division of labour: the reasoner proposes the disposition and the search recipe; the tool harness executes and emits the observation; the orchestrator (this trusted session) owns the transition and populates the fields a model could otherwise fabricate. A cross_vendor verdict or an oracle result is filled from an actual dispatched job, never from the reasoner's own text — a self-attributed cross_vendor=…:UPHOLD line only makes the fraud easier to format. Enforce this at the moment of the disposition, not at end-of-run: the KB already validates the rejection bundle at synthesize (_classify_outcome), which is too late — a real bug rejected in round 3 is gone long before round 20's fold runs.

Each terminal disposition carries a typed receipt, stored in a run-dir sidecar (an event log), not dumped into the human report — the report renders a one-line reference (C-123 confirmed [R-91]); verbose receipt blocks in the report are their own artifact-fatigue failure. The three:

  • confirmation receipt — oracle_class, the replay command / fixture id, the input hash, a machine predicate for the expected signal, and the observed-artifact hashes, emitted by the verifier harness. "It crashed / looks exploitable" cannot fill the predicate. A bare crash may confirm a reproducible DoS (its own oracle class), but it does not fill the predicate for memory-corruption exploitability — that wants the sanitizer class + a stack fingerprint. Oracle classes are open, not the three-item list a first draft reaches for: sanitizer, differential, authorization, state-change/integrity, disclosure, execution-marker, dos, timing, crypto-failure, parser-differential, policy-bypass, race-invariant.
  • rejection receipt — a reason-specific counter-hypothesis (the concrete proposition that kills the finding); a per-(vector × transform) result — an execution receipt or a rationale'd N/A, because a bare list of vectors proves no test ran, and enc is a transform dimension across vectors, not a vector; build/config digests wherever the kill is a reachability claim; and an orchestrator-dispatched cross-vendor verdict that is neither overturn nor inconclusive. Miss any and the transition fails — the candidate stays open / needs-live-validation, never silently rejected. (Rejection is not a severity — do not "downgrade" an incomplete one.)
  • material-downgrade receipt — a severity downgrade is a partial rejection, so it carries the same burden. A material downgrade is DERIVED, never a label you pick: a finding whose adopted potential high-water-mark is High/Critical landing at an effective-final Low/hardening/ not-security (via corrected, or extinguished as duplicate/out-of-scope without a preserving canonical link) — regardless of the transition name, and the high-water-mark is retained so a High→Medium→Low stair-step still trips it. It requires: the severe hypothesis being negated; a consequence/impact-cap trace (what the sink actually does / why the ceiling is Low); a bounded inventory of the sink's statically-discoverable distinct trigger paths, each with path-scoped evidence (a failed test on one ingress is evidence for that path only; a path left needs-live/needs-review is not cleared); gating digests; and an orchestrator-dispatched cross-vendor verdict of concur_downgrade (an overturn/inconclusive/uphold does not permit it). This is the gate the F-22 restore-RCE downgrade escaped: a High→Low re-rate recorded as a corrected slid past a reject-only gate. Critical→High and High→Medium are ordinary corrections (not material); a High-potential finding kept open as needs-live with a Low observed result on one path is fine — it is alive, not downgraded. Enforced by raptor-loop-ledger transition; the closure gate re-flags any severe finding sitting at effective-Low with no such receipt (anti-laundering).
  • dirty-sweep receipt — the enumeration recipe (the commands / semantic queries / call-graph root), the discovered-site set with each site's own disposition, the subsystem (de)serialization loader identified and checked, and a re-runnable site-set hash. deser_loader=yes with no denominator is worthless; the judge re-runs the recipe and diffs the resulting site set.

PENDING actions — the follow-up that must not vanish. A confirmation event auto-enqueues its required follow-up as an orchestrator-owned action: a confirmed memory-safety bug → a DIRTY_SWEEP on its file; a server-dependent finding → a live_validation; an incomplete rejection → a complete_rejection_bundle; an incomplete material downgrade → a complete_material_downgrade_bundle; and a severe-sink hypothesis (any candidate whose potential ≥ High names a sink) → a sink-keyed enumerate_trigger_paths (shared by every candidate reaching that sink, so it enumerates the sink's callers/routes/dispatchers/loaders once — this is what stops a re-sweep from silently omitting the files that actually reach the sink). A file cannot be marked closed, and the report cannot render, while its action is open. This is deliberately event-driven, not memory-driven: fable-method's evals show a forced artifact transfers when it annotates an action in hand and fails when it asks a model to notice an absence — so the sweep is enqueued by the confirmation, never left for a later "did I forget to sweep?" pass (which is exactly the pass that doesn't transfer).

Bias note: this methodology is tuned against false positives. A false negative — dismissing a real, high-impact, server-dependent finding as "theoretical/low" — is just as much a failure, and is the easier mistake to make once you are in refute mode. Hold both error types in view.

Comparable-baseline triage. For each candidate ask: what mainstream software is comparable, does it carry this same pattern, and has it ever been exploited there? Same pattern + exploited elsewhere → stronger finding. Same pattern + never exploited in years → understand why before reporting. Use the baseline to focus effort, never to auto-dismiss.

Anti-patterns — these turn a hunt into noise:

Anti-patternWhy it's wrong
Rating a defense-in-depth gap as a real finding — High, or Medium "to be safe"If another layer you can observe already blocks the attack, the missing layer is a hardening note / Low at most. E.g. a cookie missing secure/sameSite when CSRF tokens already stop CSRF and HSTS + TLS-redirect already stop downgrade is Low/hardening. But the blocking layer must be visible in the artifacts — "a server probably validates this" is not an observed mitigation (see the false-negative row below).
Treating designed behavior as a bugIf the trust model says admins are fully trusted, admin-does-admin-thing isn't a finding. Establish the trust model first.
Using "potential" / "theoretical" as a synonym for "dismissed"Two different cases — don't conflate. (a) Theoretical because you haven't finished reasoning over code you do have → finish it; if inert, drop it. (b) Unconfirmable because the deciding layer is out of scope (server you don't have, runtime you can't run) → that is not "theoretical/low," it is needs-live-validation with worst-case severity and an exact safe test.
Padding with LOWsThree real MEDIUMs beat ten LOWs. Volume is not thoroughness.
Exploits built on assumed parser/runtime behaviorThe most convincing false positives. If the exploit depends on how a parser/runtime treats input, cite the spec or test it — don't reason from intuition.
Listing every OWASP/checklist deviationA checklist is not a bug list; every real application makes tradeoffs.
Dismissing a finding because an unseen layer "probably" handles itThe defining false-negative of refutation. "A real server would enforce authz / verify the OTP / re-hash the password" is an assumption, not evidence. If you cannot see the enforcing layer, the verdict is needs-live-validation, not rejected — the test exists because that layer may be broken. (Canonical miss: an unauthenticated UpdateUserPassword that trusts a client-supplied user id + a client-supplied "skip email check" boolean, dismissed as "the server surely binds the reset" — it did not.)
Killing a finding (or a whole subsystem) on an unverified gating / reachability claimThe mirror of the row above. "That component isn't built/loaded," "needs a non-default config," "the trigger needs an impractical number of ops" are finding-killers, so they carry the finding's own proof burden — verify against the observable build + default config before dismissing. A capability enabled by default is not "gated" just because it's packaged as an optional-looking module/plugin.

Fraud classes the closure gate hunts (security-specialized)

The anti-patterns above are reasoning errors. These are dishonesty / costume-rigor signals — a report claiming work that the receipts and logs do not back. The closure gate (see Reporting) and the cross-vendor judge actively search for them; they are not just formatting checks. Each has a mechanical tell:

Fraud classThe tell (what the gate re-derives)
Fabricated PoC outputThe claimed observation has no matching artifact hash / harness run in the event log.
Stale PoCThe confirmation receipt's build_digest / commit ≠ the report's target identity — it reproduced against a different build.
Crash inflated to RCEoracle_class=dos (bare crash) but the finding is rated as memory-corruption exploitability with no sanitizer-class receipt.
Unexecuted vector listed as testedA delivery_vectors_tested entry with no per-vector execution receipt behind it.
Rationale-free N/AA vector marked N/A with no route/sink reason for why it cannot reach the sink.
Self-attributed cross-vendorcross_vendor_result present but no orchestrator-dispatched review job produced it.
Read-derived coverageA component marked covered from file reads alone, with no logged hunt action (generate→judge→verify) against it.
Grep-only sweepA DIRTY_SWEEP whose enumeration recipe is a text grep where the class demands a semantic / call-graph enumeration.
Server-dependent finding collapsedAn auth/IDOR/reset/deser finding marked rejected/Low when the deciding layer was out of view (must be needs-live-validation).
Severe finding downgraded to Low without the bundleA High/Critical-potential finding re-rated to Low (via corrected/duplicate), or its severe sink not swept, on a single-ingress test — no material-downgrade receipt (trigger-path inventory + concur_downgrade). The F-22 restore-RCE failure: gated one entry, missed the sink + the second entry.
Assumed-default config"Default deployment exposes this" with no observed config/build evidence.
Dedup across distinct entriesOne sanitizer finding used to cover several distinct attacker entry points reaching the same sink.
Dropped pending/inconclusiveThe report omits candidates still open / needs-live-validation, reading as done while work is outstanding.
Skip counted as reviewA cell marked covered off an /audit record whose status=clean carries evidence_tool=triage or prefilter — no rule ran, no loop-owned hunt receipt. Triage can fire before the source context is built; prefilter is a bounded heuristic, not a hypothesis-directed hunt. The tell is that pairing in .audit-log.jsonl with no receipt joined to it.
Capped or completeness-unknown sweep sold as completeA rule sweep used as the discovered-site denominator when completeness is not positively established — capped=true, or complete≠true, or total_matches absent, or no re-runnable site-set hash. A hit count landing exactly on a known cap is a mandatory re-run trigger, not proof on its own; and absence of a cap flag proves nothing, because the /audit adapter drops it and replayed rules never set it. Fail closed.
Tacit workaround sold as reproductionThe author needed an undocumented retry, dependency, privilege/config change, remembered path, disabled control, or manual repair, but no intervention + resolution receipt exists.
Author-coached handoff sold as owner-readyThe PoC works only while its author supplies missing steps, or the latest handoff predates a newly discovered intervention. delivery_readiness is recomputed and cannot be ready.

Guardrails (where this methodology bites back)

These are hard-won failure modes. Bake them in:


View the full raptor-loop-hunt on GitHub

Sponsors

Details

Marketplace
dinosn/raptor-loop-hunt
Category
Community