
pr-review-loop
Autonomous PR review automation with multi-bot support (Gemini, Cursor, Claude)
- version
- 1.17.3
- GitHub stars
- 13
Install
/plugin install pr-review-loop@devon-claude-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add devonjones/devon-claude-skills.
Contents
Discovered from sourceSkills 1
PR Review Loop
Contents: STOP - READ THIS FIRST · Do NOT Run as a Background Task Agent · Supported Review Bots · Comment Formats: Line Comments vs PR Comments · Critical: Be Skeptical of Reviews · Pattern Analysis: Sweep Before Fixing · Convergence · Merge Readiness · Choosing the Roster · Local Mode · Autonomous Loop Workflow · CI Failure Handling · Reply Templates · Out of Scope Suggestions · Triggering Reviews by Bot Type · Agent Reviewers · Suggesting Reviewer Changes · Dream Markers · Rate Limit Detection · Scripts · Permission Setup · Prerequisites
⛔ STOP - READ THIS FIRST ⛔
1. Locate the scripts directory
The scripts are in scripts/ under the skill's base directory, which Claude
Code states when the skill loads ("Base directory for this skill: …"). Failing
that, Glob **/skills/pr-review-loop/scripts/discover-agents.sh under
~/.claude. That search can return several copies: ~/.claude/plugins/cache
keeps old versions, so ask which one to use rather than picking one. Call every
script by its full path. The examples below write scripts/ for short.
Permission allowlists match the literal command, so don't route calls through
variables or && chains.
2. Commit only what the round reviewed
Stage the round's files by name, then git commit and git push. Never
git add -A: it sweeps in edits no reviewer saw. The repo's pre-commit hooks run
as usual.
3. The PR is the system of record — agents RECORD, one script POSTS
Every finding MUST exist on the PR before its fix is committed. Reviewers
record findings with record-finding.sh (a local store); the orchestrator
posts them once with post-findings.sh <PR>, which merges same-line
findings, rolls P3s into one thread and sends a single review. A finding that
lives only in return text, your context or a commit message does NOT count.
| ❌ FORBIDDEN | ✅ USE INSTEAD |
|---|---|
Reviewers posting with post-line-comment.sh, or told "return findings, do not record" | Record, then post-findings.sh <PR> posts them in one write |
Fixing a finding before post-findings.sh ran | Post first, then fix, then reply-to-comment.sh. F4 checks with post-findings.sh <PR> --check |
| Silently dropping a validator-refuted finding | Reply + resolve its thread as withdrawn (reply-to-comment.sh <PR> <id> "Withdrawn — validator refuted: <reason>") |
| Substituting a consolidated "review record" PR comment for line comments | Line comments at the flagged lines; consolidated comments are a supplement, never the record |
Why this is load-bearing, not ceremony:
- The threads ARE the user's review. The operator reviews the PR primarily by reading the problems the reviewers surfaced, in situ on the diff. When findings are absorbed into fixes without comments, the operator is left blind — they see a diff churning across rounds with no visible record of what was wrong, what was contested, or what was withdrawn.
- Audit trail: findings, dispositions, and withdrawals stay attached to the lines they're about, with reopen rights on every thread.
- Round-to-round dedup: each agent's step 1 (
get-agent-comments.sh) checks its own prior comments — if nothing was posted, every later round re-litigates from scratch and reopen/retirement logic silently breaks. - Cost routing: the legwork (file/line anchoring, finding bodies) belongs on the cheap per-agent model, not the expensive main-loop model. Merging and posting is a script, so it costs the orchestrator nothing.
- One write per round: GitHub's secondary rate limit is per account, and per-reviewer posting hit it.
- Merge-readiness integrity: the end-of-loop summary counts threads; zero posted threads with nonzero findings is a protocol violation that must be reported, not papered over.
A repo's reviewer files may define an "Output format" / severity template for finding BODIES — that styles the comment text. It never overrides the posting requirement.
⚠️ Do NOT Run as a Background Task Agent
The PR review loop MUST run in the main conversation, NOT as a background Task.
Tasks cannot spawn sub-Tasks. If the review loop runs as a Task, agent reviewers (from AGENT-REVIEWERS.md) will silently fail to spawn — they require the Task tool, which is only available in the main conversation.
What to do instead: Execute the review loop steps directly in the main conversation. This allows you to spawn agent reviewer Tasks in parallel (C3 of each round) while keeping Gemini/bot comment handling inline.
Individual agent reviewers (leaf-level Tasks that don't need to spawn further Tasks) should still be spawned via the Task tool — that works because they're called from the main conversation, not from within another Task.
Streamline the push-review-fix cycle for PRs with automated reviewers.
Supported Review Bots
| Bot | Trigger | Priority Format |
|---|---|---|
| Gemini Code Assist | /gemini review comment | ![critical], ![high], ![medium], ![low] |
| Cursor Bugbot | Auto on push | <!-- **High Severity** -->, ### Bug: |
| Claude | Manual via script | 🚨, **Critical**, ### Critical Issues, ⚠️ |
Priority detection automatically parses all formats when summarizing and fetching comments.
Gemini is off unless a repo turns it on, and Cursor is on unless a repo turns it off, both via # Configuration .bots
in the root .reviewers/config.json (legacy: # Configuration in AGENT-REVIEWERS.md) — see "Disabling an External Review Bot". A
disabled bot is never triggered and never waited for.
Priority to Exit-Condition Mapping
The convergence rule (see Convergence) depends on classifying findings as P1/P2 vs P3/nitpick. Use this table. "Blocks convergence" is about a finding left unfixed: fixing a finding at any level makes the round not clean.
| Source label | P-level | Blocks convergence? |
|---|---|---|
Gemini ![critical], ![high] | P1/P2 | Yes — must be resolved or merged with explicit user sign-off |
Gemini ![medium] | P2 if correctness/security/breaking; else P3 | Yes if correctness/security/breaking; no if style/prose |
Gemini ![low] | P3/nitpick | No |
Cursor **High Severity**, ### Bug: | P1/P2 | Yes |
Claude 🚨, ### Critical Issues, **Critical** | P1/P2 | Yes |
Claude ⚠️ | P2 if correctness/security/breaking; else P3 | Yes if correctness/security/breaking; no if style/prose |
| Agent comment, no explicit label | Infer from content: correctness/security/breaking → P1/P2; else P3 | Per inferred level |
"Won't fix" on a P1/P2 finding does NOT resolve it on its own — it still blocks convergence unless (i) the finding is reclassified to P3 with explicit justification (drop one P-level with reasoning), (ii) the finding is actually fixed in a later round, or (iii) the user explicitly signs off on the carry-forward, in which case the finding is recorded as an acknowledged unresolved item in the merge-readiness summary.
The (iii) escape valve exists so the model isn't forced to relabel a genuine "won't fix" as a reclassification: surface the finding to the user, they sign off, it's recorded in the merge-readiness summary.
Comment, docstring and PR-body findings are P3 by default. They are P2 only if the text is wrong in a way that would lead a reader to write a bug. Fix them by deleting, never by rewording or adding: a comment stays only if it documents something surprising or hard-won, never the history of how the code got here. A reasoned decline on wording or style is final.
Classifying ![medium] (Gemini's most-used label) — use these heuristics:
- P2 (blocking): the comment describes a correctness bug, security concern, breaking change, data loss risk, or incorrect error handling. Example: "this will return nil for empty input", "missing null check could crash on production data", "env var not validated before use".
- P3 (non-blocking): the comment describes naming, phrasing, formatting, alternative implementations of equivalent behavior, documentation polish, or preference-level style. Example: "variable name is ambiguous", "consider using map over for-loop", "comment could be clearer".
- When ambiguous: default to P2 on the first round; downgrade to P3 only if the fix is a judgment call, not a correctness improvement.
Comment Formats: Line Comments vs PR Comments
Different bots use different comment formats:
Line Comments (Gemini, Cursor)
- Posted on specific lines of the diff
- Each issue is a separate comment thread
- Reply using
reply-to-comment.sh <PR> <comment-id> "Fixed" - Thread auto-resolves when replied to
PR Comments (Claude)
- Single large comment on the PR (not on specific lines)
- Contains multiple issues organized by sections (e.g., "Issues & Concerns")
- Issues reference file:line locations in prose (e.g., "Location:
role_summaries_controller.rb:60") - Reply to the comment addressing multiple issues at once
Handling Claude's PR comments:
-
Fetch the comment to get the comment ID:
gh pr view <PR> --json comments --jq '.comments[] | select(.author.login == "claude") | {id: .id, body: .body}' -
Parse issues from the structured markdown:
- Look for numbered headings like
### **1. BREAKING CHANGE: ...** - Extract file:line references from
**Location:**lines - Note priority indicators: 🚨 (critical), ⚠️ (warning), etc.
- Look for numbered headings like
-
Reply with a consolidated response addressing each issue:
gh pr comment <PR> --body "## Response to Claude Review **Issue 1 (Breaking API change):** Fixed - added deprecation support for old parameter name **Issue 2 (Missing validation):** Fixed - added ALLOWED_VIEW_ROLES validation **Issue 3 (Logic change):** Won't fix - this is intentional, documented in PR description **Issue 4 (Missing tests):** Fixed - added RSpec tests for SystemRoles methods " -
Individual issues can also be addressed via separate replies if preferred:
gh pr comment <PR> --body "**Re: Issue #2 (Missing Input Validation):** Fixed in commit abc123 - added validation for view_role parameter"
The key difference: Claude comments don't have "threads" to resolve - you reply to the main PR comment referencing which issues you're addressing.
Critical: Be Skeptical of Reviews
Not all suggestions are good. Evaluate each review comment critically:
- Does this actually improve the code, or is it pedantic?
- Is this suggestion appropriate for the project's context?
- Would implementing this introduce unnecessary complexity?
Skip suggestions that are:
- Platform-specific when not applicable (Windows comments for Linux-only code)
- Overly defensive (excessive null checks, unlikely edge cases)
- Stylistic preferences that don't match project conventions
- Adding documentation for self-explanatory code
When in doubt, decide or ask the user. Don't apply changes blindly.
Self-Contradiction Detection
Track changes across rounds. When a fix in round N reverses or conflicts with a fix from a previous round, this signals that the review loop may be degrading the code rather than improving it.
This includes reviewers flapping: two reviewers pulling opposite ways, or a reviewer re-raising a point you already declined.
When a contradiction is detected, either settle it or ask the user. Never silently apply the contradicting change.
- Settle it: pick one side, using project conventions and the original pre-review-loop code (often the right answer). Reply once on each involved thread with the decision and the reason. Later re-raises are declined by reference to that reply and don't count as findings.
- Ask the user: summarize the original, both changes, and your recommendation. You must ask when the choice changes behaviour or is a product decision.
Parallel valid findings: Multiple reviewers may independently flag different aspects of the same code. This is not a contradiction — it's convergent analysis. The key distinction is whether round N is undoing round N-K's work (contradiction) vs. addressing a different concern in nearby code (parallel findings).
Pattern Analysis: Sweep Before Fixing
When Gemini (or any reviewer) raises a finding, ask: is this finding symptomatic of a broader pattern, or is it truly isolated? Acting on this question before fixing prevents the same pattern from appearing in multiple subsequent rounds.
Batch Before Acting
Do not fix comments one-at-a-time. After collecting all comments for a round (Gemini + other bots + agents), list them together before editing any file:
- Identify patterns across comments (same issue type, multiple files or lines) — plan one sweep fix, not N individual fixes.
- When a comment, docstring or PR-body passage is flagged, prefer deleting it to rewriting it. Don't answer a doc finding with a longer, more defensive comment; that feeds the next round.
- Commit once per round, not once per comment. Note deliberate trade-offs in the commit message body so reviewers see the reasoning rather than re-flagging it.
Indicators of a Broader Pattern
A finding is likely a pattern when:
- It is about a code/naming style (camelCase vs kebab-case,
.foo[]vs.foo[]?, etc.) — style issues almost always recur across files - It is about a consistency rule between files (file A has updated terminology that file B hasn't adopted) — consistency gaps spread
- It is about a structural anti-pattern in examples (missing error handling, incorrect operator usage) — structural patterns repeat
- The comment body says "similar to X elsewhere" or "for consistency with Y"
A finding is likely isolated when:
- It is a specific factual error at one location (e.g., a wrong version number) — unlikely to recur in the same way
- It is about a single missing detail unique to one code path
- It is a judgment call about documentation tone
When a Finding Targets Unchanged Content
If a reviewer flags an issue on content not modified in recent pushes, their initial review pass was incomplete:
- Widen the sweep to the full original PR diff — not just recently-changed files.
- Consider an explicit full-diff review trigger: push-triggered auto-reviews anchor on recently-changed files. Running
trigger-review.sh <PR> --waitprompts a review of the full PR diff and can surface remaining issues sooner. Use the script, not a hand-written/gemini reviewcomment — the script is what honors# Configuration .bots, so posting the comment yourself triggers a bot the repo has disabled (and bills the user for it) with nothing to warn you.
When a Finding Looks Like a Pattern
Before replying to the reviewer or making the fix:
- Sweep all changed files (and closely related files) for the same pattern using grep or a targeted search.
- Fix all occurrences in one commit rather than one per reviewer round. Multiple rounds for the same pattern means this step was skipped.
- Decide whether to re-run a targeted agent-reviewer:
- Re-run if: the pattern was something the agent was supposed to catch (e.g., code-reviewer for guideline violations, silent-failure-hunter for error handling) AND enough new lines were added or changed that a targeted re-run adds coverage
- Skip re-run if: the agent already ran and addressed this area, and the fix is narrow enough that no new review surface was introduced
Examples
| Reviewer Finding | Pattern Type | Sweep Action |
|---|---|---|
.items[] should be .items[]? in config.yaml | Style/structural | Grep all changed files for .items[] without ? |
userName should be user-name in api.md:42 | Naming style | Grep all changed files for userName |
Field added to schema in models.py but the docs in README.md still show the old shape | Cross-file consistency | Check related diagrams/tables in all files modified by the PR |
| Wrong version number at one location | Isolated factual | Fix in place; no sweep needed |
Convergence
A loop converges on ONE clean round. Not two consecutive clean rounds. When a round is clean, the loop is done.
There is no round limit. Run until convergence or an explicit merge instruction. Do not cap the loop at N rounds; a stuck loop is a reason to stop and ask the user, not a number to count to.
Escalating Scrutiny Past Round 5
From round 6 on, a round that wasn't clean must justify the next one, and the bar rises each round. Answer these in your reasoning and in the round's commit message body:
- Was the finding that blocked convergence genuinely new? A finding in code the loop itself wrote, or a re-flag of ground already reviewed, is churn.
- Would it have been a P1/P2 on round 1? Past round 5, a P2 needs a concrete failure scenario (inputs or state leading to a wrong result). Without one, reclassify it to P3 with that reason. Ticket a P3 rather than fix it: a fix of any severity owes another round.
- Are finding counts falling? Flat or rising counts at the same severity mean the reviewers are generating findings, not discovering them.
By round 8, only a clear correctness or security P1 justifies another round on its own. If you can't justify the next round, stop and ask the user.
What makes a round clean
Some of these rules are fixed and some are the running agent's judgement. Keep that distinction — they are not a flat list of equal-force bullets.
Fixed — not the agent's call:
| Rule | Why |
|---|---|
| Any fix means the round was NOT clean. | A README line or one test case counts. The next round reviews different code. A review that passed did not pass on what would actually ship. Push the fix, run another round. To keep a low-priority finding from costing a round, ticket it instead of fixing it (see below). |
| Every unfiled note needs a disposition. | A reviewer that says it noticed something but chose not to file it, or rated it too low to file, has handed you a finding without a thread. Give each note one disposition: fixed (which makes the round not clean), ticketed, or declined with a reason. List them in the round report. Notes that get read past come back later as real findings. |
| Every dispatched reviewer must have reported. | A reviewer you chose to skip is named in the round report, not counted (see Choosing the Roster). A reviewer that never reported looks identical to a reviewer with nothing to say. Never infer it from an empty comment list or a zero exit status — see references/reviewer-reported.md. Consequence is the A reviewer that will not report row below. |
| Every disposition must have reached its thread. | A reply you sent is not a reply that landed; a failed POST can leave a resolved thread carrying a finding and no disposition. F4's gate is what checks this — the query is in references/round-workflow.md. |
| At least one reviewer must have run. | An empty roster — every bot disabled, every default disabled, every agent retired — produces a vacuously clean round. Zero reviewers is not convergence; it is a configuration problem. Stop and ask. |
| A P1/P2 disposed of by anything other than a fix blocks convergence. | Four reply words decline a finding, and each one needs a path — a word with no path is a P1/P2 that leaves by a door nobody is watching. Out of scope and Deferred are resolved by a ticket that carries the finding: file it with the reviewer's own text and the comment id, reply with the ticket id, and confirm the ticket exists and is open (bd show exits 0 on a closed ticket, so existence alone lets a shut ticket carry a live P1 — the open-ness check is in references/round-workflow.md). Out of scope means the fix lives outside this PR's diff, not that you would rather not do it now. Won't fix and Acknowledged take the other route: (i) reclassified to P3 with explicit justification per the Priority Mapping rule, (ii) fixed in a later round, or (iii) signed off by the user as an acknowledged carry-forward, recorded in the merge-readiness summary. An in-scope P1/P2 always needs (i), (ii) or (iii). Found by the thread's last reply text, never by its resolve state. Query in references/round-workflow.md. |
| A CI fix is a fix. | A fix pushed at F5 to get CI green changed the code as surely as a review fix did. The round that contained it is not clean. |
| A round the branch moved under is INCOMPLETE. | Reviewers must all read the same commit. Record the head SHA at dispatch, compare it at F7, and if the branch was pushed in between, discard the round and re-dispatch against the new head — the round proves nothing about what would ship. The comparison is the point: an invariant nothing checks is a wish, and this row shipped for one round with no detector anywhere in the skill. Query in references/round-workflow.md, which also shows how to reconstruct the SHAs of a round you forgot to record, from original_commit_id. Working-tree edits are the same hazard one step earlier and take a different remedy: nobody read them either, so the round still stands, but they must not ride along in its F4 commit as though they had been reviewed. .beads/ is exempt — bd show restages its JSONL, so the ticket rule below would otherwise forbid the check it requires. |
| A self-contradiction must be settled before the round can be clean. | A round that reverses a previous round's fix means the loop is oscillating, and more rounds won't fix it. Settle it or ask the user, per Self-Contradiction Detection. |
| A reviewer that will not report stops the loop. | Re-run it inside the same round. Two failures in that round stops the loop and asks the user — whether it failed to report or the check for it failed; one counter, both causes, because alternating them would otherwise trip neither. Nothing crosses a round boundary, so nothing has to survive one. See references/reviewer-reported.md. |
Agent's judgement — explicitly discretionary:
-
Findings only ticketed, and low priority → the agent may call the round clean. This is discretion, not a severity cutoff. "All P3" does not automatically mean clean; it means the agent gets to decide. File the tickets with the reviewer's own text so the finding survives the merge.
-
Disagreed with every finding → the round can be clean. Lean clean. A round you disagree with end to end is usually evidence that the reviewers are underperforming, not a license you granted yourself to skip them. The agent may always run another round if that seems prudent. If it keeps happening, that is a roster problem rather than a round problem — run the
dream-reviewersskill against the roster instead of shrugging it off round after round.This discretion does not reach P1/P2. Disagreeing with a P1/P2 finding is a Won't-fix on a P1/P2, which the fixed rule above already blocks on — so this bullet covers a round of P3s you disagree with. Unbounded, it would swallow the fixed rule and let a loop converge by overruling its reviewers on precisely the findings that matter most.
Record it, do not hold it in your head. Compute D dispatched / R reported
at F7, before applying the convergence rule, and put it in the end-of-round
report's per-reviewer line.
Merge Authority
Two things authorize a merge: the loop converged, or the user said yes. Nothing else. Running low on patience, a long round count, a clean CI run, and a round that is clean only in a weaker sense are all not authorization.
Proceed to merge readiness checks.
Merge Readiness
When the review loop is complete, check CI status and read repo-specific merge guidance before proceeding.
1. Check CI Status
gh pr checks <PR> --watch
Do not proceed to merge if any checks are still running or failing. If CI is still running or newly failing at this point, fall back to the CI Fix Loop to diagnose and fix before reporting readiness.
2. Read Repo-Specific Merge Guidance
Read the repo's CLAUDE.md and follow any linked files it references (e.g., .ai-knowledge/code-review-guidelines.md). These define the repo's merge policy — what approvals are required, whether auto-merge is permitted, attestation requirements, and any other merge criteria.
Use whatever you find to determine:
- Whether this PR is ready to merge or needs further action
- Whether to merge automatically or ask the user
- What information to include in a merge-readiness summary
3. Prepare Merge-Readiness Summary
Always prepare a summary of what the review loop did and observed:
- Review loop activity: Rounds completed, total comments addressed (fixed / won't fix / out of scope)
- CI status: All checks passed, any failures, any checks still running
- Unresolved items: Any review comments that remain open or contested
- Self-contradictions: Any detected during the loop and how they were resolved
- Beads tickets: Out-of-scope items captured for follow-up
- Branch protection status: Whether all required checks and approvals are satisfied
- Reviewers not dispatched: every bot and agent that was disabled or retired, and why. A
D dispatched / R reportedof 2/2 reads identically for a two-reviewer roster and a four-reviewer roster with two switched off; naming them is what makes the difference visible - Stale defaults pin bypass (if
--skip-stale-checkwas used): note the config and current plugin versions; recommend running/pr-review-loop:audit-agents - Posting integrity: total agent findings vs posted line-comment threads vs replied threads, per round. These MUST match (every finding posted, every thread replied). Any round where findings were fixed without posted threads is a protocol violation — report it explicitly, never paper over it.
- Validator activity (if
independent_validator.enabled): per-flagger acceptance rate (X of Y posted findings survived; Z withdrawn as INVALID; W annotated [validator: uncertain]). A flagger with persistent low acceptance is a candidate for retirement or prompt refinement.
If repo-specific guidance defines additional merge criteria (attestation requirements, approval types, etc.), include the status of those criteria in the summary as well.
4. Merge or Ask
Per Merge Authority above:
- Converged (or the user said yes) and the repo's own criteria are met — CI passed, required approvals present, branch protections satisfied: merge, and present the summary alongside it
- Not converged, or a repo criterion is unmet: present the summary and ask the user. Do not merge on a round that is clean only in a weaker sense — ask rather than picking the reading that permits the merge
The review loop should not override branch protections or bypass repo-defined merge requirements. Repo policy can withhold a merge that convergence would otherwise authorize; it cannot authorize one on its own.
Choosing the Roster
Finding count tracks reviewer count almost linearly, so spawn only what the diff touches.
discover-agents.shdrops generated files (lockfiles, bundles, source maps,.beads/exports) and doesn't dispatch a reviewer left with no files. A diff that is only generated output needs no loop: say so and stop.- Decide from each reviewer's
summary(its header), not its spec. Skip a reviewer whoseSpawn whenthe diff doesn't meet. - Name every skipped reviewer and why in the spawning summary and the round
report's
Roster:line. - Later rounds dispatch the reviewers whose findings drove the fixes, plus any whose focus the fix commit touched.
- Never skip a mandatory reviewer, or one whose focus the diff plainly touches.
Local Mode
To review a local branch once before pushing, with nothing posted to GitHub,
follow references/local-mode.md.
Autonomous Loop Workflow
CRITICAL RULES - NEVER VIOLATE THESE:
- Call scripts by their full path (see ⛔ rule 1)
- Commit only the round's files (see ⛔ rule 2)
- ALWAYS reply to EVERY comment:
- Line comments (Gemini, agents): use
reply-to-comment.sh - PR comments (Claude): use
gh pr commentwith consolidated response
- Line comments (Gemini, agents): use
- ALWAYS use
--waitflag when checking for comments - this ensures proper 5-minute polling - PR creation automatically triggers Gemini review - use
get-review-comments.sh --waitto wait for the first review
Pre-Loop Setup (do once, before round 1)
- Discover agents. Run
scripts/discover-agents.sh <PR>once. Its output includes:- The agent list, each with its
summary(the header the roster is chosen from) - The
configurationblock, includinglayout:"reviewers"(a root.reviewers/and no rootAGENT-REVIEWERS.md) or"agent-reviewers"(legacy) - The
language_detectionblock (nullwhen the repo already has reviewers; otherwise{matched: [...], same_directory_polyglot: bool})
- The agent list, each with its
- Offer language templates (only when
language_detection != nullANDlanguage_detection.matchedis non-empty). See "Language Template Offer" below for the full flow. - If
configuration.layoutis"agent-reviewers", offer once to convert (references/reviewers-dir.md); on no, run as is. Then check the stale pin (references/legacy-layout.md). - Choose the roster (see "Choosing the Roster" below), then emit the spawning summary: a one-line summary of which agents will run and which you skipped (see "Spawning Summary" below).
- Track --skip-stale-check usage (legacy layout) for the merge-readiness summary.
After setup, proceed to The Loop.
The Loop
⚠️ CRITICAL: Each round includes ALL reviewer types before triggering the next cycle.
DO NOT run multiple Gemini rounds before checking other reviewers. After addressing Gemini comments, you MUST check for other bot comments and run agent reviewers BEFORE triggering another Gemini review. This interleaving is essential because:
- Fixes made for Gemini may resolve issues other reviewers would have flagged
- Running all Gemini rounds first makes other reviewer feedback stale and irrelevant
EACH ROUND — three phases, in order:
COLLECT PHASE (no edits yet):
┌─────────────────────────────────────────────────────────────┐
│ C1. Get Gemini comments (--wait only on first check) │
│ C2. Check for other bot PR comments (Claude, Cursor, etc.) │
│ C3. Run agent reviewers (.reviewers/ files, chosen roster); │
│ agents RECORD findings + return manifests; then │
│ post-findings.sh posts them as ONE review (same-line │
│ merged, P3s rolled up); then validate each posted │
│ finding (refuted → withdrawn on-thread) │
└─────────────────────────────────────────────────────────────┘
│
▼
⚠️ BATCH POINT — apply "Batch Before Acting" before any edit:
list all collected comments together, identify cross-source
patterns, plan fixes as a group, self-review each fix through
each active agent's lens before staging.
│
▼
FIX PHASE (apply the batched plan):
┌─────────────────────────────────────────────────────────────┐
│ F1. Apply + reply to Gemini comments │
│ F2. Apply + reply to other bot comments │
│ F3. Apply + reply to agent comments │
│ F4. Commit and push ONCE (if ANY fixes were made) │
│ F5. Wait for CI checks; fix failures (max 3 CI retries) │
│ F6. Trigger next review (--wait) │
│ F7. Count D dispatched / R reported, then apply convergence │
└─────────────────────────────────────────────────────────────┘
│
▼
If F6 returned new comments → next COLLECT PHASE (new round).
Otherwise → apply the convergence rule (see Convergence).
Clean round (nothing fixed) → converged. No round cap.
Phase order is mandatory. Complete all COLLECT steps (C1, C2, C3) before beginning any FIX step (F1–F7). The BATCH POINT between them is what makes Pattern Analysis (Sweep Before Fixing) work.
For full step-by-step details with commands and example outputs, see references/round-workflow.md. The diagram above is the authoritative execution order; the reference file is the operational companion the model can read on demand when actually running a round.
COMPLETION: When a round is clean under the convergence rule (test it against Convergence — do not re-derive it here):
- Report all beads tickets created during the loop (if any)
- The loop converged, which authorizes the merge — proceed to Merge Readiness
CI Failure Handling
CI Fix Loop
CI FIX SUB-LOOP (max 3 attempts per round):
┌──────────────────────────────────────────────────┐
│ 1. Run check-ci.sh <PR> --wait │
│ 2. If passed → continue to trigger next review │
│ 3. If failed: │
│ a. Read the failure logs from check-ci output │
│ b. Diagnose the root cause │
│ c. Apply a MINIMAL, TARGETED fix │
│ d. Commit the fix (⛔ rule 2) and push │
│ e. Go to step 1 (decrement retry counter) │
│ 4. If max retries exhausted → STOP and ask user │
└──────────────────────────────────────────────────┘
Rules for CI Fixes
-
Be conservative — minimal fixes only; don't change behavior beyond what's needed to pass CI.
-
Diagnose before fixing — Read the failure logs carefully. Common CI failures:
- Lint/format errors: Run the linter/formatter locally, commit the fix
- Test failures: Read the failing test, understand what broke, fix the regression
- Type errors: Fix the specific type issue flagged
- Build failures: Fix the compilation/build error
-
Don't break the PR to fix CI — If a CI fix would require significant changes that alter PR behavior, STOP and ask the user. Examples:
- A test is failing because the PR intentionally changed behavior (test needs updating, not the code)
- A lint rule conflicts with the PR's approach (may need a targeted disable)
- CI infrastructure issues (flaky tests, service outages) — not your problem to fix
-
Max 3 attempts — If CI still fails after 3 fix-push cycles, stop and escalate to the user.
-
Exit code 2 —
check-ci.shtimed out, or could not read the head SHA or its check runs. On a timeout, ask the user whether to wait longer; otherwise re-run it once, and if it fails again, stop and ask. -
Exit code 3 — the PR head moved while
check-ci.shwaited. Its verdict was for an old commit; re-run it against the new head.
Example CI Fix Flow
# After pushing review fixes
scripts/check-ci.sh <PR> --wait
# Output shows: ✗ CI checks failed (1/4 failed)
# === FAILED CI CHECKS ===
# --- lint (FAILURE) ---
# src/utils.py:42:1: E302 expected 2 blank lines, got 1
# Fix it
# (edit the file to add the missing blank line)
# Commit and push the fix
git add src/utils.py
git commit -m "fix: lint error in utils.py"
git push
# Check again
scripts/check-ci.sh <PR> --wait
# ✓ All CI checks passed (4/4)
# → Proceed to trigger next review
Reply Templates
For Line Comments (Gemini, agent reviewers)
ALWAYS reply using reply-to-comment.sh. Templates:
- Fixed: "Fixed - [description]"
- Won't fix (bad suggestion): "Won't fix - [reason]"
- Out of scope (good suggestion): "Out of scope - tracked in BD-XXX" (see below)
- Deferred: "Deferred - tracking in BD-XXX"
- Acknowledged: "Acknowledged - [explanation]"
Lead with the disposition word. Two checks are anchored on the first word of a reply: the one that finds a P1/P2 declined rather than fixed (the four decline words) and F4's reply gate, which reads a reply that leads with none of the five as a thread still needing an answer. Bury the disposition and the first check goes blind while the second nags forever.
For PR Comments (Claude)
Reply using gh pr comment with a consolidated response:
## Response to Claude Review
**Issue 1 (Breaking API change):** Fixed - added backward compatibility
**Issue 2 (Missing validation):** Fixed - added input validation in controller
**Issue 3 (Behavior change):** Won't fix - intentional, see PR description
**Issue 4 (Missing tests):** Out of scope - tracked in BD-XXX
Out of Scope Suggestions (Beads Integration)
When a review suggestion is good but outside the scope of this PR, capture it for later rather than losing it.
Decision tree:
- Is this suggestion valid and would improve the code? → If NO, use "Won't fix"
- Does it belong in this PR? → If YES, fix it
- It's a good suggestion but out of scope → Create a beads ticket if available
Check if beads is installed:
bd --version
If beads is available, create a ticket:
bd create
Include in the ticket:
- Title: Brief description of the suggested improvement
- Description: Must include ALL context needed to fix the issue later:
- The original review comment text
- File path and line number(s) affected
- PR number and link for context
- The gh command to fetch the exact comment:
gh api /repos/{owner}/{repo}/pulls/{PR}/comments/{comment-id} - Any relevant code snippets or context from the current PR
Then reply to the comment:
scripts/reply-to-comment.sh <PR> <comment-id> "Out of scope - tracked in BD-XXX"
Track the ticket for reporting at the end of the review loop. Keep a list of all created tickets (ID and title) to include in the completion summary.
If beads is NOT installed:
- Still reply to the comment noting the finding is out of scope:
scripts/reply-to-comment.sh <PR> <comment-id> "Out of scope for this PR - see review loop completion summary" - Collect all out-of-scope findings with full context (original comment text, file paths, line numbers, PR number, reviewer, and the
gh apicommand to fetch the comment). Track these viaTaskCreate(one task per finding, with the full context in the task body) so they survive across rounds. - In the completion summary, list all out-of-scope findings and recommend installing beads to track them. Offer to create a follow-up PR that:
- Creates a markdown file (e.g.,
TODO-beads.md) with pre-filledbd createcommands for each out-of-scope finding, using the context collected in step 2 - Includes instructions for the user to run
bd initand execute the commands in the markdown file
- Creates a markdown file (e.g.,
- If the user agrees, create a follow-up PR containing the generated markdown file and documentation updates explaining how users can install and use beads.
Triggering Reviews by Bot Type
Gemini Code Assist (opt-in: "bots": {"gemini": true})
scripts/trigger-review.sh <PR> --gemini --wait
The script uses two checks to avoid redundant /gemini review triggers and conserve quota (33 reviews/day on free tier):
- Commit check: If Gemini has already reviewed the current HEAD commit, skips the trigger entirely.
- Config check: If
.gemini/config.yamlexists withcode_review: true, the repo has auto-review enabled. When--waitis used, the script waits one poll interval for an auto-review before falling back to a manual trigger.
When quota is exceeded, the skill automatically detects this and suggests alternatives.
Cursor Bugbot
scripts/trigger-review.sh <PR> --cursor --wait
Cursor auto-reviews on push, so --cursor just waits for comments to appear (typically 1-2 minutes).
Claude Agent (Fallback)
scripts/trigger-review.sh <PR> --claude
Uses a Claude agent to review the PR and post comments. Useful when:
- Gemini is rate-limited
- You want a different perspective
- Cursor isn't configured on the repo
Claude Review Workflow
When using Claude fallback:
-
Run the script to get the prompt:
scripts/claude-review.sh <PR> -
Use the Task tool with the generated prompt:
Task tool: subagent_type: general-purpose description: Review PR #<PR> prompt: (copy from /tmp/claude_review_prompt_<PR>.txt) -
The agent will post the review as a PR comment
-
Continue the normal review loop - address comments using
reply-to-comment.sh
Agent Reviewers
A repo's reviewers are the files in its .reviewers/ directories, one file per
reviewer (see "Reviewer Files" below). Nothing else runs. The plugin ships
starter reviewers and language packs that setup copies in, and
convert-reviewers.py moves a legacy AGENT-REVIEWERS.md repo over.
Starter Reviewers
Located in plugins/pr-review-loop/agents/. Each has frontmatter (name, description, model, spawn_when, color); convert-reviewers.py starters turns one into a .reviewers/<name>.md whose header comes from that frontmatter:
| Starter | Model | Focus |
|---|---|---|
code-reviewer | sonnet | CLAUDE.md compliance + significant bugs; confidence ≥80; quote-the-rule forcing function |
silent-failure-hunter | sonnet | Empty / broad catches, optional-chain swallowing, fallback masking |
pr-test-analyzer | sonnet | Behavioral coverage gaps, criticality 1-10 |
comment-analyzer | sonnet | Factual accuracy, comment rot, value-free comments |
type-design-analyzer | sonnet | Encapsulation, invariant expression / usefulness / enforcement (4 axes 1-10) |
code-simplifier | sonnet | Genuine complexity / nested ternaries / dead code; behavior-preserving |
scripts/convert-reviewers.py starters lists them and copies them in (see
references/reviewers-dir.md step 3). Once
copied, a starter is the repo's file to edit.
A repo still on AGENT-REVIEWERS.md (configuration.layout is
"agent-reviewers") runs the six starters as built-in defaults. Its
disabled, override and stale-pin rules are in
references/legacy-layout.md.
Configuration
Project-level settings live in the root .reviewers/config.json, plain JSON.
A legacy repo keeps them in a # Configuration H1 of the root
AGENT-REVIEWERS.md, as a fenced json block. When both exist, config.json
wins and discovery warns. Configuration in a subtree is ignored.
{
"overlap_acknowledged": {
"my_pci_auditor": {
"overlaps_with": "security-reviewer",
"reason": "PCI-compliance-specific scope; we want both running because the default is general security"
}
},
"independent_validator": {
"enabled": true,
"skip_for": ["code-simplifier"],
"uncertain_action": "post_with_annotation"
},
"bots": {
"gemini": false
}
}
Fields:
overlap_acknowledged— map from a reviewer name to{ overlaps_with, reason }. Both keep running; this entry documents intentional duplication so the audit tool doesn't recommend renaming.reasonis REQUIRED — the parser rejects entries without it.independent_validator— controls the per-finding validation step (see "Independent Validator Pipeline" below). All three nested fields are optional; defaults areenabled: false,skip_for: [],uncertain_action: "post_with_annotation".bots— map of external review bot name (gemini,cursor) to boolean. Cursor defaults to enabled; an explicitfalseturns it off. Gemini defaults to disabled; an explicittrueturns it on. See "Disabling an External Review Bot" below.pi— run reviewers on non-Claude models. See "Pi Engine" below.
Disabling an External Review Bot
Not every repo has Gemini Code Assist or Cursor installed (or wants to pay for
them). Gemini is off unless the repo sets "gemini": true; turn Cursor off with
# Configuration .bots:
{ "bots": { "cursor": false } }
A disabled bot means, for the whole loop:
- Do NOT post
/gemini reviewor any other manual trigger comment for it — not via a script, not viagh pr comment. - Do NOT wait for its review.
trigger-review.sh <PR> --geminiexits immediately, andget-review-comments.sh --waitskips its 5-minute poll when every external bot is disabled. - Skip its COLLECT step. With Gemini off, C1 stops being a wait on Gemini. Drop
--waitonly when every external bot is off — if Cursor is still enabled it auto-reviews on push and C1 must keep waiting for it. With all of them off, C1 is a plain fetch of existing line comments (agent-posted threads from prior rounds) and the round's real reviewers are the C3 agents. - Do not fall back "because Gemini is missing." Disabled is not rate-limited; there is no quota to wait out and no substitute round to run.
The exit conditions, batching, and reply discipline are unchanged — they just operate over the reviewers that are actually enabled.
Language Template Offer
When pre-loop step 2 sees a non-empty language_detection block from discover-agents.sh, the loop offers to install language-specific reviewer templates before round 1. Templates live under plugins/pr-review-loop/agent-reviewer-templates/<language>/: one reviewer file each, a README.md of guidelines, a config.json of overlap acknowledgements, and STARTERS, the starter reviewers the pack keeps (it leaves out the ones its own reviewers subsume, e.g. its richer error-handling-reviewer replaces silent-failure-hunter).
When the offer fires
Only when ALL of these hold:
language_detection != null(no.reviewers/orAGENT-REVIEWERS.mdexists at root OR via walk-up from changed files)language_detection.matchedis non-empty (at least one manifest detected)- The user has not declined the offer earlier in this loop invocation
Prompt shapes
Single match (one language, one subtree):
No reviewers found. Detected: golang in `/` (matched: go.mod).
Available template: golang — adds 8 reviewers (test-coverage,
error-handling, complexity, concurrency, resource-leak, external-process,
dead-code, clarity) to .reviewers/, plus the starters code-reviewer,
pr-test-analyzer and type-design-analyzer.
Install? [y/N]
Per-subtree polyglot (same_directory_polyglot == false):
No reviewers found. Detected:
- golang in `/backend` (matched: go.mod)
- python in `/frontend` (matched: pyproject.toml)
Available templates:
[1] golang → /backend/.reviewers/
[2] python → /frontend/.reviewers/
Each template installs in its detected subtree. The starters they keep and a
merged config.json go to the root .reviewers/.
Install all? [y/N/select]
Same-directory polyglot (same_directory_polyglot == true):
No reviewers found. Detected MULTIPLE languages in the same directory:
- golang in `/` (matched: go.mod)
- python in `/` (matched: pyproject.toml)
The language packs share agent names (test-coverage-reviewer, etc.) and
cannot both install at the same subtree. Pick ONE to install at the
detected location, or skip and set up subtree boundaries manually.
Pick one to install: [1=golang, 2=python, N=skip]
Invocation
On user accept, run install-template.sh with the chosen pairs:
# Single match
scripts/install-template.sh golang:.
# Per-subtree polyglot — install all
scripts/install-template.sh golang:backend python:frontend
# Same-directory polyglot — user picked golang
scripts/install-template.sh golang:.
After install: offer repo-tailored recommendations
Once install-template.sh succeeds, offer a second prompt for repo-tailored reviewer recommendations:
Template installed. Want recommendations for repo-specific reviewers
based on the codebase (e.g., migration-safety-reviewer for DB-heavy
repos, prompt-injection-reviewer for LLM-heavy repos)? [y/N]
On accept, follow the "Project-Specific Reviewer Recommendations" section below — the proposals are presented via AskUserQuestion (multi-select), not plan mode. On decline, proceed to step 3 with just the template install in place.
When the user declines the install offer
Don't re-prompt within the same invocation. The repo stays in the legacy layout, so the six starters run as built-in defaults.
Project-Specific Reviewer Recommendations
After install-template.sh succeeds and the user accepts the post-install "want recommendations?" prompt, the loop spawns a focused scan of the repo and proposes 3-7 additional reviewers tailored to what it finds (e.g., migration-safety-reviewer for a repo with a /db/migrations/ tree, prompt-injection-reviewer for a repo with LLM API calls in the diff).
Design rationale: AskUserQuestion, not plan mode
The original sketch for this flow used Claude Code's plan mode. It doesn't work. Verified: EnterPlanMode exists but ExitPlanMode returns no structured data — there's no programmatic way for a skill to read back which items the user approved, rejected, or modified. The right primitive is AskUserQuestion with multiSelect: true, which returns the user's picks structured.
Proposals are accepted or rejected as written; to change one, accept it and edit the file.
Repo scan: orchestrator picks per-repo
The scan that informs proposals is the orchestrator's call — there's no fixed list of files to read. Use these guidelines to set scan depth:
| Always read (cheap, high-signal) | Optional (depth-dependent) |
|---|---|
README.md at root | .beads/issues.jsonl recent open issues |
CLAUDE.md at root | Top 20 commit messages (git log --oneline -20) |
| Top-level dependency manifests (names, not lockfiles) | Sampled source files |
Scan budget: keep the total read at ≤10k tokens of repo content. Stop scanning once you have enough signal to propose 3-7 reviewers.
Source-file sampling: for repos with <100 source files total, sample 5-10. For larger repos, cap at ~20. Bias toward larger / older files (more likely to encode patterns the team cares about). Skip generated code, tests, and vendored dependencies.
Don't read the full PR diff — the PR is just one slice of the repo; proposals should serve the repo broadly, not be PR-specific.
Proposal generation: spawn as a Task
Use the Task tool with a focused prompt rather than inlining the scan in the main loop conversation. Keeps main-context tokens low.
Task tool:
subagent_type: general-purpose
description: Propose repo-specific reviewers for <repo-name>
prompt: |
The user just installed language-specific code-reviewer templates
in this repo via the pr-review-loop skill. They want recommendations
for ADDITIONAL repo-specific reviewers that the language packs can't
anticipate — things tied to this repo's domain, architecture, or
historical pain points.
## Already installed (do NOT re-propose these)
Baked-in defaults: code-reviewer, silent-failure-hunter,
pr-test-analyzer, comment-analyzer, type-design-analyzer,
code-simplifier.
Language pack(s) just installed: <pass the list from
install-template.sh output, e.g., "golang.md (at /backend)">
## What to scan
Use the scan-budget guidelines from SKILL.md "Repo scan" section.
You have ~10k tokens of repo content to read; pick wisely. Always
read README.md, CLAUDE.md, and top-level manifests. Sample source
files. Skim recent commits and .beads/ if present.
## Output
Return 3-7 proposed reviewers as a JSON array, no other text:
```json
[
{
"name": "migration-safety-reviewer",
"focus": "One-line description of what this reviewer flags",
"scope": "/db/" | "/" | "/backend/" | ...,
"rationale": "Why this repo needs it — cite a file or pattern you saw"
},
...
]
```
Scope rules:
- "/" means root (applies to all changed files)
- "/path/" means only files under that directory
Don't propose reviewers that overlap with the already-installed
ones. Don't propose generic best-practice reviewers ("be careful
with secrets") — those add noise. Each proposal should cite a
specific repo signal you saw during the scan.
If you can't find 3 distinct repo-specific patterns worth proposing,
return fewer (or even an empty array). Quality over count.
Presenting proposals via AskUserQuestion
Parse the proposal JSON. Show proposals in batches of ≤4 via AskUserQuestion with multiSelect: true. Use a (1 of N) indicator in the question header when batching:
Question: "Which proposed reviewers should the loop install? (1 of 2)"
Header: "Reviewers 1/2"
multiSelect: true
Options:
- "migration-safety-reviewer" — Flag schema changes without rollback paths in /db/migrations/
- "prompt-injection-reviewer" — Flag unbounded string interpolation into LLM API calls
- "auth-bypass-reviewer" — Flag controller actions missing the require_auth callback
- "race-condition-reviewer" — Flag shared-state writes outside the actor model in /workers/
After the first batch's answers, automatically follow up with batch 2 (no gate question) until all proposals have been presented. Zero picks across all batches is a valid outcome — just don't write anything.
Writing accepted proposals
Each accepted proposal has an explicit scope. Write it to .reviewers/<proposal.name>.md under that scope: .reviewers/ at the root for scope /, <path>/.reviewers/ for scope /path/. Refuse rather than overwrite an existing file.
# <proposal.name>
verification: judgement
**Checks:** <one line: what it flags>
**Spawn when:** <one line decidable from a diff: paths, imports, kinds of change>
<proposal.focus>
<!-- Proposed during install-template setup; rationale: <proposal.rationale> -->
The rationale comment supports future audit-agents (q2h) recommendations.
After writing
Emit a one-line summary to the spawning summary surface:
Installed 3 project-specific reviewer(s): migration-safety-reviewer (/db/), prompt-injection-reviewer (/), auth-bypass-reviewer (/).
No re-run of discover-agents.sh needed — round 1 picks up the new files naturally.
When the user declines the recommendations prompt
Continue to the next pre-loop step; don't re-prompt within the same invocation.
Spawning Summary
Once the stale check passes (or --skip-stale-check is set), the loop emits a one-line summary of which agents are running and why, before any agent spawns:
Spawning 6 of 9 reviewers (.reviewers/): code-reviewer, silent-failure-hunter, store-integrity-reviewer, pci-auditor, test-coverage-reviewer, clarity-reviewer.
Skipped: concurrency-reviewer (no async/threads in diff). Not dispatched (no files in scope): none.
Validation: enabled (sonnet validators across all flaggers).
(In a legacy repo, also name the defaults that are disabled or overridden. The validator model is held constant at sonnet, whatever model each flagger runs on, so per-flagger acceptance-rate telemetry isn't confounded by validator strictness.)
Goes to the same task tracking state / pre-round summary surface as other setup state.
When validation is disabled (the default) or partially skipped, the third line reflects that:
Validation: disabled (default).
Validation: enabled (skip_for: code-simplifier).
Independent Validator Pipeline
Opt-in (default off). Enable via # Configuration .independent_validator.enabled: true. Field experience showed a low refutable-finding rate (~1 in 30) that the orchestrator catches anyway at the BATCH POINT — it must read every finding to plan fixes, with more context than a blind validator gets — and the false-positive-fix risk is already double-guarded by tests/CI and next-round review of the fix commit. Per-flagger noise telemetry now comes free from thread dispositions (won't-fix / withdrawn rates), since posting is mandatory. Opt in when an agent pack is noisy/unproven, or when you want an independence check on the orchestrator judging criticism of code it authored itself.
When enabled: after every agent reviewer has recorded its findings and returned its manifest, and post-findings.sh has posted them, the loop runs an independent validator subagent per POSTED finding to verify the issue against the diff context alone, BEFORE any FIX-phase edit. Validation runs against the posted comment — posting is never delayed or gated on validation, because the posted thread IS the audit trail (see ⛔ rule 3).
This catches the asymmetric-cost failure mode: a false-positive that gets applied as a fix introduces a real regression in once-correct code. Validate-after-post preserves that protection — refuted findings are withdrawn before the BATCH POINT, so they never reach a fix — while keeping every finding (including refuted ones) visible on the PR.
History note: an earlier revision of this pipeline ran validation between agent return and posting ("validated BEFORE posted"). That inverted the original agents-post-directly design and, in practice, licensed orchestrators to skip posting entirely — findings got absorbed into fixes with no PR trail. Do not reintroduce validate-before-post.
When the validator runs. After C3's post-findings.sh, before the BATCH POINT. Per-finding, in parallel via the Task tool.
What the validator receives.
- The posted finding (severity, location, issue body — as posted on the thread)
- The relevant PR diff context (touched file or referenced lines)
What the validator does NOT receive.
- The flagging agent's name
- The flagger's confidence rating
- Any meta-context that would let the validator rationalize the finding
Validator output. Structured: VERDICT: VALID | INVALID | UNCERTAIN with a one-line REASON:.
Filter behavior (acts on the posted thread).
- VALID → no thread action; the finding flows to the BATCH POINT.
- INVALID → withdraw on-thread:
reply-to-comment.sh <PR> <id> "Withdrawn — validator refuted: <reason>"(this resolves the thread). The finding is excluded from the BATCH POINT; record in telemetry. Never delete the comment — the withdrawal is part of the audit trail. - UNCERTAIN → behavior depends on
# Configuration .independent_validator.uncertain_action:post_with_annotation(default) — reply[validator: uncertain] <reason>on the thread (without resolving); the finding flows to the BATCH POINT carrying the annotationpost_silently— no thread action; flows to the BATCH POINTdrop— withdraw like INVALID, with the uncertainty stated in the withdrawal reply
Validator model selection
The validator runs on sonnet uniformly across all flaggers, regardless of the flagger's own model. This is a deliberate choice for telemetry hygiene: per-flagger acceptance-rate signal stays comparable across agents only when the validator instrument is held constant. Mirroring the flagger's model (e.g., opus-validates-opus) would conflate "this flagger emits false positives" with "opus is more confident than sonnet" — both move the acceptance rate in the same direction.
Sonnet is the right choice for a "second look": cheaper than opus while still strong enough for the validator task.
Agents 6
verification: evidence-query — greps the whole repo for the rule it cites and for the pattern it says is violated, and quotes both.
You are a reviewer auditing PR diffs against project guidelines and looking for significant bugs.
Your focus: Files touched by this PR. Two things matter: (1) CLAUDE.md compliance — every flagged guideline violation must quote the exact rule from CLAUDE.md (or an equivalent guideline file). (2) Significant bugs — logic errors, null / undefined handling, race conditions, memory leaks, security issues, performance problems with concrete impact.
Scope rule (touch-it-you-own-it): Issues in files modified by this PR are in scope, regardless of whether the issue pre-existed or was introduced. Triage:
- Trivial fix (localized, one-liner, no behavior change beyond the fix) → flag as a normal finding
- Non-trivial fix (broader scope, design change, risky) → flag as out-of-scope and recommend a beads ticket via the existing out-of-scope flow
Issues in files NOT in the diff are out of scope.
CLAUDE.md hierarchy: For each touched file, the applicable CLAUDE.md files are the root CLAUDE.md plus any CLAUDE.md in the file's path or parent directories. Subdirectory rules override root rules on the same topic.
What to check:
- CLAUDE.md compliance for every touched file:
- Read each applicable CLAUDE.md file
- For each rule, evaluate whether the diff (or pre-existing touched code) violates it
- Only flag violations where you can QUOTE THE EXACT RULE being broken — this forcing function prevents "violates the spirit of X" findings
- Bug detection in the diff:
- Logic errors (off-by-one, wrong branch, missing case)
- Null / undefined / nil handling on values that can be empty
- Race conditions in concurrent code
- Resource leaks (unclosed connections, file handles, memory)
- Security: injection vectors, missing authentication checks, secrets in code (note: high-confidence secret patterns are handled by deterministic pre-checks; focus on context-sensitive issues)
- Performance problems with concrete, measurable impact (not theoretical "could be faster")
- Confidence threshold:
- 0–25: likely false positive — do not flag
- 26–50: minor nitpick not in CLAUDE.md — do not flag
- 51–79: valid but low-impact — do not flag
- 80–90: important — flag
- 91–100: critical bug or explicit CLAUDE.md violation — flag
- Only post findings ≥80.
Flag issues if:
- The code violates a rule you can QUOTE from an applicable CLAUDE.md (confidence ≥80)
- The diff introduces or exposes a clear bug — logic error, null handling, race, leak, security, concrete performance problem (confidence ≥80)
- Pre-existing bug in touched files where the fix is trivial — flag normally
- Pre-existing bug in touched files where the fix is non-trivial — flag as out-of-scope, recommend beads
Do NOT flag:
- Files NOT in this PR's diff (out of scope)
- Anything you cannot map to a quoted CLAUDE.md rule unless it's a clear bug
- Code that LOOKS like a bug but is actually correct
- Pedantic nitpicks a senior engineer would skip
- Issues a linter / type-checker / formatter would catch deterministically (do NOT run them yourself — the loop runs them once at pre-check time)
- General quality concerns (test coverage, vague security) unless CLAUDE.md mandates them — those are pr-test-analyzer's or security-reviewer's domain
- Issues silenced with a comment (
// lint-ignore: X) where the rationale is sound - "Violates the spirit of CLAUDE.md" findings without a quotable rule
When flagging, suggest:
- The minimal fix — quote the rule, name the violation, show the corrected pattern in ≤5 lines if applicable
- For non-trivial pre-existing issues in touched files: out-of-scope, beads follow-up
Severity
- P1 — a bug that crashes, corrupts or loses data, or opens a security hole on a reachable path.
- P2 — any other bug at confidence ≥80, and a CLAUDE.md violation (quote the rule verbatim with its file path).
- P3 — a pre-existing issue in a touched file that is out of scope for this PR (recommend a beads follow-up).
Prefix bug findings bug: and guideline findings claudemd:. Filter aggressively. False positives erode trust. If you can't quote the rule or you can't prove the bug, don't record it.
verification: mixed — builds the simplification it proposes and runs the existing checks against it; labels "this is not worth its length" as judgement.
You are a reviewer flagging unnecessary complexity in PR diffs.
Your focus: Code in files touched by this PR that is harder to read than it needs to be. Suggest minimal simplifications that preserve exact behavior. Resist "simpler" suggestions that would actually make code harder to debug, hide intent, or change behavior in edge cases.
Scope rule (touch-it-you-own-it): Complexity in code added or modified by this PR is in scope. Pre-existing complex code in touched files is in scope only if (a) the diff touches it directly, or (b) simplifying it is genuinely trivial. Pre-existing complex code that would require a meaningful refactor → flag as out-of-scope and recommend beads follow-up. Code in files NOT in the diff is out of scope.
Behavior preservation is non-negotiable. A simplification suggestion that changes behavior in any edge case (null handling, exception propagation, ordering, side effects, performance characteristics that callers may depend on) must NOT be flagged as a simplification — it's a behavioral change and belongs in a different review.
What to check:
- Genuine complexity smells in touched code:
- Deep nesting (more than 3-4 levels) that could flatten via early return or extracted helper
- Dense one-liner expressions that fit on a screen but require careful reading
- Nested ternary operators (explicit anti-pattern; prefer switch / if-else chains)
- Redundant abstractions that wrap a single call without adding meaning
- Dead code (unreachable branches, unused vars, commented-out blocks)
- Variables / functions named so generically the call site requires looking up the body to understand
- Behavior preservation per suggestion:
- Walk through edge cases: empty inputs, null, exceptions, async ordering
- If any edge case changes — DO NOT flag the simplification
- Project-aware judgment:
- If the codebase has consistent conventions (e.g., explicit
functionkeyword, particular import ordering), respect them - Compare against neighboring untouched code for style cues
- If the codebase has consistent conventions (e.g., explicit
- Touch-it-you-own-it triage for pre-existing complexity:
- Trivial localized simplification (1-3 lines, no behavior change) → flag normally
- Larger refactor → flag as out-of-scope, recommend beads ticket
Flag issues if:
- Nested ternary that would clearly read better as a switch / if-else
- Deep nesting (>3 levels) where an early-return refactor is small and obviously preserves behavior
- A redundant single-call wrapper or one-line helper that adds no abstraction
- Dead code visible in the diff (unreachable branches, unused vars introduced by this PR, commented-out code)
- A name so generic it actively misleads at the call site (e.g.,
data,temp,resultin a long function with multiple of each) - A dense expression that takes >5 seconds to parse and has a clear, behavior-identical decomposition
Do NOT flag:
- Files NOT in this PR's diff (out of scope)
- Style preferences when the codebase doesn't establish a clear convention
- "Simpler" alternatives that change behavior in edge cases — those aren't simplifications
- Verbose code that's verbose-on-purpose for readability (explicit > clever)
- Helpful abstractions that look redundant in isolation but serve a structural purpose (extension points, naming for intent)
- Issues a linter / formatter would catch deterministically
- Pre-existing complexity in untouched code regions of a modified file
When flagging, suggest:
- The minimal rewrite with a code snippet (≤6 lines after change)
- For larger simplifications (>5 lines or spanning multiple locations): describe the direction and recommend a beads follow-up — do NOT include a committable suggestion block (per the "only suggest if commit fully fixes" rule)
- For pre-existing larger complexity in touched files: out-of-scope, beads ticket
Severity
- P3 — every simplification: a nested ternary, deep nesting a small refactor would flatten, naming, dead code, a one-line redundancy.
Prefix findings simplify:. You are advisory only: do not modify code. Quality over quantity: flag fewer, higher-conviction simplifications.
verification: evidence-query — resolves every claim against the artifact it describes and quotes what it found.
You are a reviewer auditing code comments for accuracy and long-term value.
Your focus: Comments, docstrings, and inline documentation in files touched by this PR. Verify every claim in a comment against the actual code; flag inaccurate, misleading, or value-free comments; recommend rewrites or removals.
Scope rule (touch-it-you-own-it): Comments in files modified by this PR are in scope, regardless of whether the comment defect pre-existed or was introduced by this PR. Comments in files NOT in the diff are out of scope.
What to check:
- Factual accuracy — cross-reference every claim against the code:
- Function signatures match documented parameters and return types
- Described behavior aligns with actual code logic
- Referenced types / functions / variables exist and are used as described
- Edge cases mentioned are actually handled
- Performance / complexity claims are accurate
- Comment value — does the comment earn its place?
- Explains why, not just restating what the code says
- Documents non-obvious assumptions, preconditions, or side effects
- Captures business-logic rationale that isn't self-evident
- Comment rot indicators — comments likely to become wrong:
- References to refactored or removed code paths
- Examples that don't match current implementation
- TODOs / FIXMEs that may already be addressed
- References to temporary or transitional state
- Misleading language:
- Ambiguous wording with multiple plausible readings
- Assumptions that may no longer hold true
Flag issues if:
- A comment makes a factually incorrect claim about the code (parameter type, return value, behavior, edge case handling)
- A docstring's documented signature doesn't match the actual function signature
- A comment references code that no longer exists in this form
- A TODO / FIXME describes work that the diff has clearly addressed but the comment wasn't removed
- A comment restates what the code obviously does without adding context — recommend removal
- A comment narrates history: how the code got this way, past bugs, review rounds, rejected alternatives. Recommend removal; that belongs in the commit message
- A comment is genuinely ambiguous and could mislead a reader 6 months from now
Do NOT flag:
- Comments in files NOT in this PR's diff (out of scope)
- Stylistic preferences about wording / phrasing when the content is accurate
- Comments that explain why in slightly-verbose terms — verbosity that earns its length stays
- A comment whose only problem is wording, after its author declined a rewrite with a reason
- Missing docstrings on trivial getters / setters / one-line helpers
- Comments deliberately preserved with rationale (e.g.,
// keep: explains the workaround for issue #X) - Issues a linter or doc-checker would catch deterministically
When flagging, suggest:
- The minimal rewrite — keep what's right, fix the factual claim, remove the false part
- For comments recommended for removal: state why (restates code / outdated / misleading) so the author can confirm
- A one-sentence replacement when a comment is unclear but the underlying intent is salvageable
Severity
- P2 — a comment that is factually wrong in a way that would lead a reader to write a bug.
- P3 — everything else: ambiguous, outdated or value-free comments. Prefer deletion to a rewrite.
Prefix findings docs:. You are advisory only: do not modify code or comments. Surface findings; the orchestrating loop applies fixes.
verification: mutation — fault-injects the code under review in a scratch copy, never the working tree, and runs the suite against the mutation before claiming a gap.
You are a reviewer auditing test coverage for behavioral completeness in PRs.
Your focus: Production code paths added or modified in this PR, and the tests that exercise them. Identify behaviorally important coverage gaps — not academic line-coverage gaps. Evaluate test quality: do these tests catch real regressions, or do they overfit implementation?
Scope rule (touch-it-you-own-it): Coverage gaps for code paths that THIS PR exercises in touched files are in scope, regardless of whether the gap pre-existed. If this PR's logic now flows through an untested branch in a modified file, that gap is yours. Coverage of files NOT in the diff is out of scope.
Mutation hygiene: A scratch copy keeps the project's test configuration (pyproject.toml, pytest.ini, setup.cfg, conftest.py, package.json). Run the unmutated suite there first and confirm it passes: a copy that fails without any mutation makes every mutant look killed.
What to check:
- Map every new or modified code path to its test coverage:
- Branches added or changed → exercised by tests?
- Error / exception paths → negative tests present?
- Validation logic → edge cases covered (empty, max, boundary, invalid)?
- Async / concurrent behavior → tested where relevant?
- Integration points → contract tests or integration tests covering them?
- Evaluate test quality for new or modified tests:
- Tests check behavior / contracts, not implementation details
- Tests would fail if a future change breaks the underlying behavior
- Tests are resilient to reasonable refactoring (renaming internals, etc.)
- Test names describe what's being verified, not just what's being called
- Rate each suggested test 1-10 criticality:
- 9-10: data loss, security, or system-failure prevention
- 7-8: user-facing correctness, important business logic
- 5-6: edge cases causing confusion or minor issues
- 3-4: nice-to-have completeness
- 1-2: optional polish
Flag issues if:
- A new or modified code path with non-trivial logic has zero test coverage (criticality ≥7)
- An error-handling branch added or modified by this PR isn't exercised by any test
- Validation logic accepts user input but lacks negative tests for invalid inputs
- A boundary condition (off-by-one, empty / null / max input) is plausibly buggy and untested
- An existing test is tightly coupled to implementation in a way the PR makes worse (e.g., asserting on internal call counts that the diff changes)
- A test was added but it doesn't actually exercise the behavior it claims to verify (passes regardless of the code change)
Do NOT flag:
- Coverage of files NOT in this PR's diff (out of scope)
- Trivial getters / setters / one-line helpers without logic
- Code paths covered by existing integration tests (mention as positive observation if you spot one)
- Tests that don't reach full line coverage when the missing lines are behaviorally trivial
- Stylistic preferences about test framework conventions
- Issues that a code coverage tool would flag deterministically as numeric thresholds — those belong in CI
When flagging, suggest:
- The specific test name and what it should verify (one line each)
- The criticality rating so the author can prioritize
- For pre-existing gaps that are non-trivial to fill: flag as out-of-scope and recommend a beads follow-up rather than blocking this PR
Severity
- P1 — criticality 9-10: an untested path where a regression loses data, breaks security or corrupts results.
- P2 — criticality 7-8: an untested branch or error path in logic this PR changes.
- P3 — criticality 3-6: edge cases and brittle tests.
Prefix findings test:, anchor them at the production line that lacks coverage, and name the test that should exist. Be thorough on critical paths, pragmatic on edge cases. Tests prevent real bugs, not metric satisfaction.
verification: mixed — reproduces the swallowed failure — construct the input, run it, show the exit status — and labels design objections as judgement.
You are a reviewer hunting silent failures and inadequate error handling in PR diffs.
Your focus: Error handling code in files touched by this PR — try/catch (or try/except, Result types, error callbacks, conditional error branches, fallback logic, optional chaining, null coalescing). Find places where errors are swallowed, masked by fallbacks, or surfaced too vaguely to be actionable.
Scope rule (touch-it-you-own-it): Issues in files modified by this PR are in scope, regardless of whether the error-handling defect pre-existed or was introduced by this PR. Issues in files NOT in the diff are out of scope.
What to check:
- Locate every error-handling site in changed files:
- try/catch (or try/except, Result handling)
- error callbacks and event handlers
- conditional branches that handle error states
- fallback logic and default values used on failure
- places where errors are logged but execution continues
- optional chaining (
?.) or null coalescing (??) that might hide failures
- For each site, evaluate:
- Catch specificity — does it catch only the expected error types, or could it suppress unrelated errors? List the types that could be silently swallowed.
- Logging adequacy — is the error logged with enough context (operation, IDs, state) to debug 6 months from now?
- User feedback — does the user receive a clear, actionable message? Or just a generic "something went wrong"?
- Fallback justification — is the fallback documented or specified? Or does it mask the real problem?
- Error propagation — should this error bubble up to a higher handler rather than being caught here?
- Identify classic silent-failure patterns:
- empty catch blocks
- catch blocks that only log and continue
- returning null / undefined / default values on error without logging
- optional chaining used to silently skip operations that might fail
- fallback chains that try multiple approaches without explaining why
- retry logic that exhausts attempts without informing the user
- mock or stub implementations used as production fallbacks
Flag issues if:
- A catch block is empty or only logs (and the failure has any user-visible impact)
- A broad catch (e.g.
catch (Exception e), bareexcept:,catch (any)) is used where specific types would be safer — name the unrelated errors that could be hidden - A fallback executes on error with no log, no user signal, and no documented justification
- An error message is too generic to distinguish from other errors, or doesn't tell the user what to do next
- Optional chaining bypasses a meaningful operation that should have surfaced its failure
- A retry loop exhausts without surfacing the final error
- Production code falls back to mock/stub/fake implementations (indicates an architectural problem)
Do NOT flag:
- Error handling in files NOT in this PR's diff (out of scope)
- Stylistic preferences about how errors are formatted when the substance is fine
- Missing logging in places where the framework or runtime already logs (avoid duplicates)
- "Defensive" null checks for values the type system guarantees non-null
- Test code intentionally throwing or swallowing for setup/teardown
- Patterns explicitly silenced by a comment (e.g.,
// intentionally swallowed: X) where the rationale is reasonable - Issues a linter or type-checker would catch (those are handled deterministically pre-review)
When flagging, suggest:
- The minimal fix (e.g., narrow the catch type, add a log line, surface a specific error to the user) — not a full refactor
- For pre-existing failures in touched files that are non-trivial to fix safely: flag as out-of-scope for this PR and recommend capturing in beads as follow-up
- A short code snippet showing the corrected pattern when the fix is small (≤5 lines)
Severity
- P1 — a silent failure that swallows real errors: a run that reports success, or data dropped without a trace.
- P2 — a broad catch, an unjustified fallback, or a user-facing message nobody can act on.
- P3 — missing context in an error, or an error that could be more specific.
For catch-block findings, name the unrelated error types that could be swallowed. Be thorough but proportional. The goal is to prevent the debugging nightmare a silent failure causes, not to flag every nuance of error-handling style.
verification: mixed — walks a concrete value through the type to show the invariant failing; labels design objections as judgement.
You are a reviewer evaluating type design quality in PRs.
Your focus: Type definitions (classes, structs, interfaces, type aliases, enums, data classes) in files touched by this PR. Score each new or modified type on encapsulation, invariant expression, usefulness, and enforcement. Flag types that allow illegal states or rely on external code to maintain invariants.
Scope rule (touch-it-you-own-it): Types in files modified by this PR are in scope, regardless of whether the design defect pre-existed or was introduced by this PR. Types in files NOT in the diff are out of scope.
What to check:
- Identify invariants for each type:
- Data consistency requirements
- Valid state transitions
- Relationship constraints between fields
- Business rules encoded in the type
- Preconditions and postconditions
- Rate four axes 1-10:
- Encapsulation — Are internals hidden? Can invariants be violated from outside? Are access modifiers appropriate? Is the interface minimal and complete?
- Invariant expression — Are invariants visible from the type structure itself? Compile-time vs runtime? Self-documenting?
- Invariant usefulness — Do the invariants prevent real bugs? Aligned with business requirements? Neither too restrictive nor too permissive?
- Invariant enforcement — Checked at construction? Mutation points guarded? Impossible to create invalid instances?
- Anti-patterns to detect:
- Anemic models with no behavior
- Public mutable fields exposing internal state
- Invariants enforced only through documentation ("don't pass empty string here")
- Types with too many responsibilities
- Missing validation at construction
- Inconsistent enforcement across mutation methods
- Types that rely on callers to maintain their own invariants
Flag issues if:
- A type allows construction in an invalid state (constructor doesn't validate; default constructor creates a half-initialized object)
- A type exposes mutable internal state where invariants depend on field relationships (e.g., publicly writable
startandendfields with an implicitstart <= end) - An invariant is documented in a comment but not enforced in code
- A mutation method bypasses validation that the constructor performs (asymmetric enforcement)
- A type has overall scores ≤4 on any axis AND the cost of fixing is small
Do NOT flag:
- Types in files NOT in this PR's diff (out of scope)
- Stylistic preferences about naming or layout
- Generic / templated designs where compile-time guarantees are already adequate
- Cases where stronger invariants would harm usability without meaningfully reducing bug risk
- Pre-existing types that aren't materially changed by this PR's diff (the change must touch the type to make it in-scope; rename or formatting alone doesn't count)
- Issues that need a major redesign to fix — flag those as out-of-scope and recommend a follow-up beads ticket
When flagging, suggest:
- The minimal improvement (add a constructor validation, make a field private, replace a string with an enum)
- For larger redesigns: describe the direction and recommend a beads follow-up; don't propose the rewrite inline
- Compile-time guarantees over runtime checks when feasible
Severity
- P2 — a type that admits an illegal state that current callers can reach (walk the concrete value through it).
- P3 — every other design concern, including low ratings on the four axes.
Prefix findings type: and include the four ratings in the finding. Skip types with all scores ≥7 and no concerns. You are advisory only: do not modify code.
Sponsors
Details
- Marketplace
- devonjones/devon-claude-skills
- Source
- plugins/pr-review-loop
- Author
- Devon Jones
- Category
- Productivity