Dataset Evaluation

This validates your dataset formatting before you kick off a SageMaker fine-tuning job, saving you from the annoying failures that happen when your data schema doesn't match what SFT, DPO, or RLVR expects. It auto-detects file format, checks schema compliance against your chosen model and technique, and tells you straight up if your data is ready or broken. Worth noting that it handles both training and evaluation datasets differently, which matters because they have distinct format requirements. The workflow is smart enough to pull model and strategy from conversation context if you've already set those up, otherwise it'll prompt you to select them first. One solid feature is that it won't unnecessarily copy or download S3 data, it just validates in place.

Install to Claude Code

npx -y skills add awslabs/agent-plugins --skill dataset-evaluation --agent claude-code