Hyperpod Nccl
awslabs/agent-pluginsOfficial
When your distributed training job hangs at AllReduce or you see EFA TCP fallback warnings, this is the skill to reach for. It runs a read-only diagnostic script that probes your HyperPod cluster (EKS or Slurm) via AWS APIs, kubectl, and SSM, then prints every issue with a direct pointer to the fix: security group missing self-references, NCCL version mismatches across pods, memlock limits, NetworkPolicy blocks, GPU OOM versus container OOM. The workflow is strict about not running destructive commands without your approval, which matters when training state is on the line. It won't help with single-node hardware faults or cluster creation failures, but for multi-node NCCL and rendezvous problems it cuts straight to root cause.
- GitHub stars
- 783
- last updated
- Jun 11, 2026
Install
$ npx -y skills add awslabs/agent-plugins --skill hyperpod-nccl --agent claude-codeInstalls into .claude/skills of the current project.
Files
HyperPod NCCL Debugger
Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation.
Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). scripts/nccl-diagnose.sh reads state via AWS APIs, kubectl, and SSM, then prints each issue as [FAIL] ... → references/<file>.md § <section>. Read-only.
Signal sourcing: list-cluster-events carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does not carry NCCL timeouts, GPU XID/ECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. "No events" on a training-time NCCL issue is expected, not a clean bill of health.
Workflow
Sponsors
Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthly
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any language
Agent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/awslabs/agent-plugins
- Categories
- AI & Agent Building, Release Management, Debugging
- License
- Apache-2.0
- First seen
- Jun 11, 2026
