Vally Eval

It'll help you author eval.yaml files, validate they follow conventions like required tags for type and tier, run them with custom executors that support early termination and screenshot capture, and even re-grade existing trajectories when you're tuning grader configs. It's designed for the azure-skills plugin ecosystem where test results feed into monitoring dashboards. The custom executor piece is crucial here since it extends Vally with features the base tool doesn't support yet. Worth noting you can't use the completed grader if you specify early termination conditions, which makes sense but could trip you up.