Paper Autoraters
Four LLM-as-judge metrics from the PaperOrchestra paper (arXiv:2604.05018) for scoring research paper quality. You get Citation F1 with P0/P1 partitioning, a six-axis literature review scorer with anti-inflation caps, and two side-by-side comparators (full paper and lit review only). The literature review rubric is strict by design: default range is 45-70, anything above 85 needs strong evidence on all axes, and there are hard penalties for overclaiming or citation dumping. Useful if you're building paper-writing agents and want the same benchmarks the paper used to compare against AI-Scientist-v2, or if you just want a systematic way to score drafts without handwaving.
- GitHub stars
- 572
- last updated
- Jun 2, 2026
Install
$ npx -y skills add ar9av/paperorchestra --skill paper-autoraters --agent claude-codeInstalls into .claude/skills of the current project.
Files
Paper Autoraters (App. F.3)
Faithful implementation of the four LLM-as-judge autoraters used in PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).
These are the metrics the paper uses to demonstrate that PaperOrchestra beats single-agent and AI-Scientist-v2 baselines. Use them to:
- Score a generated paper against a ground-truth paper.
- Compare two paper-writing pipelines side-by-side.
- Validate your own host-agent execution of the paper-orchestra pipeline.
The four autoraters
Sponsors
Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthly
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any language
Agent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/ar9av/paperorchestra
- Categories
- Git & Pull Requests, Code Review & Quality, Data Science & ML
- First seen
- Jun 11, 2026
