Vally Eval

If you're building agent skills and need proper integration testing, this handles the full lifecycle of Vally eval suites. It'll help you author eval.yaml files, validate they follow conventions like required tags for type and tier, run them with custom executors that support early termination and screenshot capture, and even re-grade existing trajectories when you're tuning grader configs. It's designed for the azure-skills plugin ecosystem where test results feed into monitoring dashboards. The custom executor piece is crucial here since it extends Vally with features the base tool doesn't support yet. Worth noting you can't use the completed grader if you specify early termination conditions, which makes sense but could trip you up.

Install to Claude Code

npx -y skills add microsoft/github-copilot-for-azure --skill vally-eval --agent claude-code