agent-regression-evals

alirezarezvani/claude-skills

Prove an agent, prompt, model or tool change did not make the agent worse. Three deterministic stdlib scripts: a trajectory checker that grades tool-call paths (strict / unordered / subset / superset match modes, argument matching, forbidden tools that block in every mode, call ceilings); a flake detector that classifies each case as stable or flaky from repeated trials and reports pass@k and pass^k with Wilson intervals; and a paired regression gate that averages trials per case, tests per-case differences with a t-based CI and an exact one-sided McNemar test, and blocks the merge on a significant drop, a critical-case regression, a dropped case, or a quarantine entry without owner and expiry. Ships cs-eval-gatekeeper agent, /cs-agent-evals and /cs-eval-gate commands, 3 references, and a CI job example.

version

2.14.1

GitHub stars

28.8K

Sponsors

SnugThe clipboard manager you and your agent can use, safelyAgent CaffeineClose your lid, come back to finished workWaxBlock anything with supervision, without erasing your iPhoneOpen StepsSponsoredFree and open-source skills that make AI coding agents easier to understand, verify, and control.