
codex-peer-review
Peer validation system using OpenAI Codex CLI. Two AI perspectives catch more issues than one.
- version
- 2.1.1
- GitHub stars
- 34
Install
/plugin install codex-peer-review@agent-peer-review-marketplaceRun it inside Claude Code after adding the marketplace with /plugin marketplace add jcputney/agent-peer-review.
Contents
Discovered from sourceSkills 1
Codex Peer Review
Symmetric two-AI peer review using OpenAI Codex CLI. Both AIs review independently, then debate per-issue with terminal states until convergence — not a one-shot validation.
Core principle: Asymmetric "validate my answer" loops anchor on the proposer's framing. Symmetric blind passes catch ~2x as many issues because each AI examines the work without priming. The debate phase then resolves conflicts deterministically via per-issue terminal states.
This is the single source of truth for the peer review protocol. The agent file is a thin dispatcher that loads this skill.
Reference Files
Read these with the exact paths below when needed — never search the filesystem for them:
${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/discussion-protocol.md— debate mechanics (round-by-round prompts)${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/escalation-criteria.md— when to skip the debate and go to external research${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/common-mistakes.md— anti-patterns and recovery
Modes
| Mode | Command | When |
|---|---|---|
| blind-debate (default) | /codex-peer-review | Symmetric blind pass + structured debate. Best signal. |
| classic (deprecated) | /codex-peer-review --mode classic | Old single-pass validation. Cheap, weaker signal. Will be removed. |
Auto-trigger (proactive validation before presenting plans/designs/reviews) always uses blind-debate.
Codex CLI Compatibility
Tested against codex-cli 0.136.0. Hard requirements:
codex execfor ALL machine-readable output (codex reviewexposes only--base— no--json,-o, or--output-schema— so it cannot drive parseable workflows)jq(fail fast if missing — required for session ID extraction and JSONL parsing)- A configured
peer-reviewprofile file at~/.codex/peer-review.config.toml, layered via--profile peer-review(see "Codex Profile Setup" below)
Schema is enforced via prompt template, parsed with jq — not via --output-schema. (codex exec does expose --output-schema <FILE>, but the prompt-template + jq path is version-stable and avoids depending on its behavior.)
Codex Profile Setup
Model selection lives in per-profile files under ~/.codex/, not in this plugin. Codex CLI (profile-v2) resolves --profile <name> by layering ~/.codex/<name>.config.toml on top of the base ~/.codex/config.toml. This keeps CLI flags out of prompts and lets users tune without editing the plugin or touching config.toml.
Run this once per machine (the plugin's init flow does this automatically if the files are missing):
cat > ~/.codex/peer-review.config.toml <<'EOF'
model = "gpt-5.4"
model_reasoning_effort = "high"
EOF
cat > ~/.codex/peer-review-summarizer.config.toml <<'EOF'
model = "gpt-5.4-mini"
model_reasoning_effort = "low"
EOF
The plugin invokes Codex as codex exec --profile peer-review ... — the file is layered automatically. Never hardcode -m gpt-5.x-... in agent prompts.
gpt-5.4-mini is the durable cheap-workhorse choice for the summarizer profile.
Legacy note: Pre-0.12 Codex used
[profiles.peer-review]tables insideconfig.toml. Those are inert under profile-v2 — remove them by hand if you upgraded from an older plugin version. The plugin never writes to or editsconfig.toml.
The Workflow
digraph blind_debate {
rankdir=TB;
node [shape=box];
start [label="User invokes /codex-peer-review\nOR Claude is about to present\na plan/design/review" shape=ellipse];
blind [label="ROUND 0: Blind pass\nClaude review ‖ Codex review\n(neither sees the other)"];
canon [label="Canonicalize issues\nid = sha1(file + normalized claim)\nMerge duplicates\nDrop severity:style"];
states [label="All issues start in state: proposed"];
debate [label="ROUND N (N≥1): Per-issue debate\nEach side responds: concede/defend/dismiss\nNew issues allowed (also start as proposed)"];
transition [label="Transition states based on responses"];
converged [label="All issues in terminal state?\n(accepted/rejected/merged/escalated/deferred)" shape=diamond];
cap [label="Round count >= cap?" shape=diamond];
extend [label="Ask user: extend cap?" shape=diamond];
synth [label="Verdict synthesis\nCritical / Important / Contested / Dismissed" shape=ellipse];
start -> blind;
blind -> canon;
canon -> states;
states -> debate;
debate -> transition;
transition -> converged;
converged -> synth [label="yes"];
converged -> cap [label="no"];
cap -> debate [label="no"];
cap -> extend [label="yes"];
extend -> debate [label="yes"];
extend -> synth [label="no — escalate remaining"];
}
Default cap: 3 rounds total (1 blind + 2 debate). The peer review found that 5 (the dg default) is too expensive for serious code review and risks rationalization loops. Allow extension only when new evidence appears in the final round.
Round 0: Blind Pass
Both AIs review the same scope with the same prompt, neither seeing the other's work.
Scope determination
| User input | Scope |
|---|---|
/codex-peer-review (no args) | Use AskUserQuestion to select: changes vs branch / uncommitted / specific commit |
/codex-peer-review --base X | git diff X...HEAD |
/codex-peer-review --uncommitted | Staged + unstaged + untracked |
/codex-peer-review --commit SHA | Single commit |
/codex-peer-review <question> | Question text — no diff, validate the answer |
| Auto-trigger from Claude's plan | Plan text + affected files |
Never guess the base branch. Always ask via AskUserQuestion if not specified.
The blind-pass prompt template
Both Claude and Codex receive this exact template (variables filled in):
You are performing an independent code review. Another AI is reviewing the same
work in parallel. You will not see their findings until after this pass.
## Scope
{scope_description}
## Files / Diff
{files_or_diff}
## Review lenses (apply BOTH)
1. CRITIC LENS: For each issue you raise, you MUST provide ONE of:
- A concrete exploit path or attack scenario
- A failing test case (input + expected vs actual)
- A specific failure mode (e.g., "concurrent writes to map at handler.go:42 will panic under load")
Vague concerns ("could be improved", "might be fragile") are REJECTED.
2. DEFENDER LENS: Before raising an issue, check whether:
- An existing test covers it
- A codebase invariant or convention makes it impossible
- It is intentional per a comment, ADR, or commit message
If yes, do not raise it.
## Output format (strict)
Emit a single fenced code block tagged `findings` containing JSONL — one finding per line:
```findings
{"id":"<sha1(file+claim)>","file":"path:line","severity":"critical|high|medium|low|style","claim":"<one sentence>","evidence":"<exploit/test/failure mode>","category":"security|correctness|performance|maintainability|style"}
{"id":"...","file":"...","severity":"...","claim":"...","evidence":"...","category":"..."}
If you find no issues, emit an empty findings block.
After the block, write a 2-3 sentence summary of the work's overall quality and your confidence level.
### Codex invocation
```bash
codex exec --profile peer-review --sandbox read-only \
-o /tmp/codex_round0.txt \
--json 2>&1 | tee /tmp/codex_round0.jsonl <<'EOF'
[blind-pass prompt above]
EOF
-o writes the final assistant message only — that is what we parse for the findings block. The JSONL stream is for progress polling and session ID extraction.
Issue canonicalization
After both sides emit findings, normalize and merge:
# Concatenate both findings blocks
jq -s '.' /tmp/claude_findings.json /tmp/codex_findings.json > /tmp/all_findings.json
# Merge by id; if both AIs reported the same id, mark source="both"
jq '
group_by(.id)
| map({
id: .[0].id,
file: .[0].file,
severity: .[0].severity,
claim: .[0].claim,
evidence: (map(.evidence) | unique | join(" || ")),
category: .[0].category,
source: (if length == 2 then "both" else .[0]._source end),
status: "proposed"
})
' /tmp/all_findings.json > /tmp/canonical_issues.json
Issue IDs are content-hashed, not positional. The same finding from both AIs collapses to one row with source: "both" (a strong signal — these usually become accepted immediately).
Drop severity:style from the debate. Style issues never converge through debate; they're resolved by project conventions. Surface them in the final report as a separate "style notes" section, not in the verdict.
Rounds 1+: Per-Issue Debate
Each side sees:
- The full canonical issue table with current states
- The previous round's response (if any)
Each side emits per-issue stances for every non-terminal issue, plus any new issues (which start as proposed).
State machine
┌──────────┐
│ proposed │ ◄── new in this round
└────┬─────┘
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌──────────┐ ┌────────────┐
│accepted │ │ rejected │ │ escalated │
└─────────┘ └──────────┘ └─────┬──────┘
▲ ▲ │
│ │ ▼
│ │ (next round)
│ │
┌────┴────┐ ┌────┴────┐
│ merged │ │deferred │
└─────────┘ └─────────┘
Transition rules
For each non-terminal issue, after both sides respond in round N:
| Claude stance | Codex stance | New state |
|---|---|---|
| concede | concede | rejected (both withdrew) |
| defend | defend | escalated (carry to next round) |
| dismiss | dismiss | rejected |
| concede / dismiss | accept / push | accepted (one side conceded the other's claim) |
| accept / push | concede / dismiss | accepted |
| proposed (new evidence merges with existing) | — | merged into target id |
Terminal states: accepted, rejected, merged, deferred. Issues in these states drop out of the debate.
escalated issues survive to round N+1, but require new evidence to remain alive past round 2. An issue defended only by re-asserting the same point in round 3 auto-transitions to deferred (presented to user as "contested, both held position").
Convergence
converged = (no issues in state ∈ {proposed, escalated})
This is derived, not declared. No top-level converged: true flag from the model — that just invites premature claims of consensus.
Round prompt template (Round 1+)
You are continuing the peer review debate. The other AI has emitted findings.
You must respond to each non-terminal issue with a stance.
## Canonical issue table (current state)
{table_of_issues_with_states}
## The other AI's round N response
{other_ai_response}
## Your task
1. For each non-terminal issue, emit ONE stance line:
```stances
{"id":"<id>","stance":"concede|defend|dismiss|accept","reasoning":"<one sentence>","new_evidence":"<optional — required to keep defend alive past round 2>"}
concede= you no longer believe this issue is realdefend= you maintain this issue (must include new_evidence in round 3+)dismiss= you believe the OTHER AI's claim is wrongaccept= you accept the OTHER AI's claim that you originally raised was wrong (use when conceding YOUR OWN issue under their dismiss)
-
After the stances block, you may emit a
findingsblock with NEW issues only. New issues must follow all CRITIC LENS rules (concrete evidence required). -
Apply both lenses (critic and defender) as before.
### Session resume (optimization, not correctness)
```bash
# Round 1 — extract session ID from JSONL
SESSION_ID=$(jq -r 'select(.type=="thread.started") | .thread_id' /tmp/codex_round0.jsonl | head -1)
# Round 2 — try to resume
if [ -n "$SESSION_ID" ]; then
codex exec --profile peer-review --sandbox read-only resume "$SESSION_ID" \
-o /tmp/codex_round1.txt --json 2>&1 | tee /tmp/codex_round1.jsonl <<'EOF'
[round prompt]
EOF
fi
# If resume fails (session store error, missing ID), fall back to fresh exec with full canonical table re-injected
Session resume is best-effort. Each round prompt re-injects the canonical issue table, so a session-store error degrades latency but not correctness.
Verdict Synthesis
After convergence (or cap exhausted), categorize all issues by terminal state:
| Verdict | Source states | Meaning |
|---|---|---|
| Critical | accepted AND severity ∈ {critical, high} | Real bugs, ship as required fixes |
| Important | accepted AND severity = medium | Strong recommendation |
| Contested | escalated (cap hit) OR deferred | Both held positions — present both views, user decides |
| Dismissed | rejected | Raised but withdrawn — informational only |
| Style notes | severity = style | Bypassed debate entirely — informational |
Output format
## Peer Review Result — {scope}
**Mode:** blind-debate
**Rounds:** N (converged | cap reached)
**Issues canonical:** X total, Y from both AIs, Z unique to one side
### Critical
- `file:line` — {claim}
- Evidence: {evidence}
- Source: both | claude | codex
### Important
- `file:line` — {claim}
- Evidence: {evidence}
### Contested (both AIs held position)
- `file:line` — {claim}
- Claude's view: {summary}
- Codex's view: {summary}
- Recommendation: {how user should decide}
### Dismissed (raised but withdrawn)
- `file:line` — {claim} — {why withdrawn}
### Style notes
- `file:line` — {note}
### Process notes
- {anything notable about the debate, e.g., "Codex raised 3 security issues Claude missed"}
- {immediate-escalation triggers fired, if any}
Immediate Escalation (Skip Debate)
Some classes of finding skip the per-issue debate and go straight to external research arbitration:
- Security: auth bypass, injection, secrets exposure, crypto misuse
- Architecture: fundamental design conflicts (e.g., one side says "this whole module should be event-driven")
- Breaking changes: backward compat disputes
- Order-of-magnitude perf: "this is O(n²) in a hot path" vs "fine"
If either AI flags a security concern in the blind pass, escalate that issue immediately to external research. The peer review skill is agnostic about which research tool to use — pick the best one available (web search, an MCP research tool, vendor docs, etc.). See ${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/escalation-criteria.md.
Subagent Dispatch (CRITICAL)
Always run via the codex-peer-reviewer subagent. The main conversation must never see raw Codex output, JSONL streams, or per-round transcripts — only the final synthesized verdict.
The agent file (agents/codex-peer-reviewer.md) is a thin dispatcher that loads this skill and runs the protocol.
Prerequisites
# Required
command -v codex >/dev/null || { echo "ERROR: install codex CLI: npm i -g @openai/codex"; exit 1; }
command -v jq >/dev/null || { echo "ERROR: install jq: brew install jq"; exit 1; }
# Verify peer-review profile file exists
[ -f ~/.codex/peer-review.config.toml ] || {
echo "ERROR: missing ~/.codex/peer-review.config.toml. Run: codex-peer-review init"
exit 1
}
# Verify auth
codex login --check 2>/dev/null || echo "WARNING: run 'codex login'"
Quick reference
| Scenario | Action |
|---|---|
| About to present design/plan | Auto-trigger blind-debate, no args |
| About to present code review | Auto-trigger blind-debate, scope = the diff |
| User asks broad question | blind-debate with question as scope, no diff |
User runs /codex-peer-review | Ask for scope, run blind-debate |
User runs /codex-peer-review --mode classic | Single-pass validation (deprecated) |
| Security issue surfaces | Immediate escalation, skip per-issue debate |
| Cap hit, issues still escalated | Mark as Contested, present both views |
Key changes from prior versions:
- ❌
codex review --jsonremoved (codex reviewexposes only--base; usecodex exec) - ❌ Hardcoded
gpt-5.3-codex-sparkremoved (use Codex profiles instead) - ❌ Asymmetric "Claude proposes, Codex validates" removed (now symmetric blind pass)
- ❌ Top-level
converged: boolremoved (now derived from per-issue states) - ✅ Schema enforced via prompt template, parsed with jq (NOT
--output-schema, which is unstable) - ✅ Session resume is best-effort optimization, not correctness-critical
Commands 1
Codex Peer Review Command
You have been explicitly asked to run peer review validation using OpenAI Codex CLI.
Parse Arguments
/codex-peer-review → ask for scope, run blind-debate (default)
/codex-peer-review --base <branch> → blind-debate vs branch
/codex-peer-review --uncommitted → blind-debate of staged/unstaged/untracked
/codex-peer-review --commit <SHA> → blind-debate of one commit
/codex-peer-review --mode classic [...] → legacy single-pass validation (deprecated)
/codex-peer-review --mode blind-debate [...] → explicit default
/codex-peer-review init → write peer-review profile files to ~/.codex/
/codex-peer-review <question> → blind-debate of an answer to a broad question
Modes:
blind-debate(default): symmetric blind pass + per-issue debate. Best signal. See thecodex-peer-reviewskill for full protocol.classic(deprecated): old behavior — Claude proposes, Codex validates, max 2 rounds. Cheaper but weaker. Will be removed in a future version. When this mode is invoked, warn the user once.
init subcommand
Codex CLI uses profile-v2 files: --profile <name> layers ~/.codex/<name>.config.toml on top of the base ~/.codex/config.toml. Profiles are NOT [profiles.*] tables inside config.toml (that was the pre-0.12 scheme). The plugin writes one standalone file per profile and never edits config.toml — that file holds unrelated user settings.
If the user runs /codex-peer-review init, do not dispatch the agent. Instead:
-
Check if the profile files already exist:
if [ -f ~/.codex/peer-review.config.toml ] && [ -f ~/.codex/peer-review-summarizer.config.toml ]; then echo "Profile files already exist. Nothing to do." exit 0 fi -
Write the profile files:
mkdir -p ~/.codex cat > ~/.codex/peer-review.config.toml <<'EOF' model = "gpt-5.4" model_reasoning_effort = "high" EOF cat > ~/.codex/peer-review-summarizer.config.toml <<'EOF' model = "gpt-5.4-mini" model_reasoning_effort = "low" EOF -
Confirm to the user:
Wrote ~/.codex/peer-review.config.toml and ~/.codex/peer-review-summarizer.config.toml You can now run /codex-peer-review.If a legacy
[profiles.peer-review]block still exists in~/.codex/config.tomlfrom an older plugin version, mention that it is now inert and can be removed by hand — but do NOT editconfig.tomlyourself. -
Stop. Do not dispatch the agent.
Execute (non-init)
IMPORTANT: All peer review work MUST run as a subagent (codex-peer-review:codex-peer-reviewer) to keep the main context clean.
Step 1: Pre-flight checks (in the main context)
command -v codex >/dev/null || { echo "ERROR: install codex CLI: npm i -g @openai/codex"; exit 1; }
command -v jq >/dev/null || { echo "ERROR: install jq: brew install jq"; exit 1; }
[ -f ~/.codex/peer-review.config.toml ] || {
echo "ERROR: missing ~/.codex/peer-review.config.toml (Codex profile-v2 file)."
echo "Run: /codex-peer-review init"
exit 1
}
If any check fails, surface the error and stop. Do not dispatch.
Step 2: Gather scope
If the user did not provide explicit scope flags or a question, use AskUserQuestion:
question: "What would you like to review?"
header: "Review type"
options:
- label: "Changes vs branch"
description: "Compare current changes against a base branch"
- label: "Uncommitted changes"
description: "Review staged, unstaged, and untracked changes"
- label: "Specific commit"
description: "Review changes from a specific commit (will ask for SHA)"
multiSelect: false
If "Changes vs branch" is selected, ask for the base branch (do not auto-detect):
question: "Which branch should I compare against?"
header: "Base branch"
options:
- label: "main"
- label: "develop"
- label: "master"
multiSelect: false
If "Specific commit" is selected, ask for the SHA via free-form prompt.
Step 3: Dispatch to the agent
Use Task tool:
subagent_type: "codex-peer-review:codex-peer-reviewer"
prompt: |
Run peer review.
Mode: blind-debate | classic (default: blind-debate)
Type: code-review | design | architecture | question
Scope:
[for code-review] base=<branch> | uncommitted | commit=<sha>
[for design/arch/question] inline content
Focus area: [optional, e.g. "security in auth flow"]
Follow the codex-peer-review skill protocol exactly.
Return only the synthesized verdict in the format documented in the skill.
Step 4: Present the verdict
The agent returns a structured verdict (see the skill's "Output format" section). Present it to the user verbatim — do not re-summarize or re-classify.
Mode warnings
If the user passes --mode classic:
Note:
--mode classicruns the legacy single-pass validation. The defaultblind-debatemode catches significantly more issues. Classic mode is deprecated and will be removed.
Handling edge cases
- No changes to review: if
git diff <base>...HEADis empty, tell the user and exit before dispatching. - No question and no scope: prompt via
AskUserQuestion(above). - Codex auth missing: the agent's pre-flight will catch this and surface
codex login.
Agents 1
Codex Peer Reviewer Agent
You are a thin dispatcher. The full peer review protocol lives in the codex-peer-review skill — load it and follow it. This file is intentionally short to prevent drift between agent and skill.
Your job
- Load the protocol. If the
codex-peer-reviewskill content is not already in your context, Read${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/SKILL.md(it is the single source of truth for the protocol). Never search the filesystem for it — use this exact path. - Run the protocol as documented in the skill.
- Return only the synthesized verdict to the main conversation. Never return raw Codex JSONL, per-round transcripts, or progress chatter.
Mandatory contract
- Run in your own context. The main conversation must never see Codex output. Summaries only.
- Use the Codex profile, not hardcoded models. All Codex invocations must use
--profile peer-review(or--profile peer-review-summarizerfor cheap summarization). If the~/.codex/peer-review.config.tomlprofile file is missing, surface the init instructions from the skill and stop. - Never create, edit, or delete
~/.codex/config.tomlor any~/.codex/*.config.tomlprofile file. Profile setup is theinitcommand's job. If a profile file is missing, tell the user to run/codex-peer-review initand stop — do NOT "fix" the config yourself. (Doing so caused a config-clobbering loop in earlier versions.) - Never use
codex review --jsonorcodex review -o.codex reviewexposes only--base(verified on 0.136.0). Usecodex execfor everything that needs structured/streamed output. - Never use
--output-schema. Schema is enforced via the prompt templates in the skill, parsed withjq. (codex exec --output-schemaexists in 0.136.0 but the plugin does not depend on it.) - Require
jq. Fail fast if missing — do not fall back to grep parsing.
Input you will receive
One of:
- Code review request with scope (branch / commit / uncommitted)
- Plan or design to validate (auto-trigger from main Claude before presenting)
- Architecture recommendation to cross-check
- Broad technical question Claude is about to answer
The dispatching prompt should tell you which mode (blind-debate default, or classic for legacy single-pass). If unspecified, default to blind-debate.
Progress reporting
Create a TaskCreate at the start so the user sees a spinner. Update activeForm as you progress through the protocol's phases:
"Verifying Codex CLI and profile..."— prerequisites"Round 0: blind pass (Claude + Codex in parallel)..."— symmetric review"Canonicalizing N issues..."— merge step"Round 1: per-issue debate..."— first debate round"Round 2: per-issue debate..."— second debate round"Synthesizing verdict..."— final synthesis
Mark the task completed when you return the verdict.
Output
Return exactly the format documented in the codex-peer-review skill ("Output format" section). Do not improvise — the main conversation expects that exact structure for downstream processing.
Reference
Everything else — the prompts, the state machine, the convergence rule, the verdict categorization, the lens prompts, the escalation criteria — lives in the skill:
${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/SKILL.md— main protocol${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/discussion-protocol.md— debate mechanics${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/escalation-criteria.md— when to escalate${CLAUDE_PLUGIN_ROOT}/skills/codex-peer-review/common-mistakes.md— anti-patterns
If you find yourself improvising protocol logic in this file, stop and add it to the skill instead. This file is a dispatcher, not a manual.
Sponsors
Details
- Marketplace
- jcputney/agent-peer-review
- Source
- plugins/codex-peer-review
- Author
- jcputney
- Category
- Code Quality