Content Refinement Agent
This is the fifth stage of the PaperOrchestra pipeline, where an initial draft gets iteratively refined through simulated peer review. It runs about three rounds of review-then-revise, checking each iteration against a six-axis rubric (scientific depth, technical execution, logical flow, writing clarity, evidence presentation, academic style). The score delta determines whether to accept the revision or revert to the previous snapshot, and the loop halts on revert or when no actionable weaknesses remain. The paper claims refinement alone added 19-22% to acceptance rates in their experiments. You get snapshots of every iteration, a worklog of decisions, and gates that check for hallucinated claims against your experimental log. Costs around five to seven LLM calls per run, mostly on the reviewer and revision prompts.
- GitHub stars
- 572
- last updated
- Jun 2, 2026
Install
$ npx -y skills add ar9av/paperorchestra --skill content-refinement-agent --agent claude-codeInstalls into .claude/skills of the current project.
Files
Content Refinement Agent (Step 5)
Faithful implementation of the Content Refinement Agent from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §4 Step 5, App. F.1 pp. 49–51).
Cost: ~5–7 LLM calls (App. B), typically ~3 refinement iterations, each consisting of one reviewer call and one revision call.
The paper highlights this step as one of the largest contributors to overall quality: refinement alone accounts for +19% (CVPR) and +22% (ICLR) absolute acceptance-rate improvement (Fig. 4). Get this step right.
Inputs
workspace/drafts/paper.tex— output of Step 4workspace/inputs/conference_guidelines.mdworkspace/inputs/experimental_log.md— used as ground truth for the hallucination checkworkspace/citation_pool.json/workspace/refs.bib— the allowed bibliography
Outputs
workspace/refinement/iter1/,iter2/,iter3/— per-iteration snapshots containingpaper.tex,paper.pdf,review.json,score.jsonworkspace/refinement/worklog.json— append-only history of decisionsworkspace/final/paper.texandworkspace/final/paper.pdf— copy of the best accepted snapshot
The refinement loop
Sponsors
Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthly
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any language
Agent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/ar9av/paperorchestra
- Categories
- DevOps & CI/CD, Git & Pull Requests, Code Review & Quality, AI & Agent Building, Data Science & ML, Automation & Workflows, Sales & Marketing
- First seen
- Jun 11, 2026
