HyperPod Cluster Debugger
Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer to run it. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes).
Before any state-changing CLI: ask if it's IaC-managed. HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).
scripts/diagnose-cluster.sh is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as [FAIL] ... → references/<file>.md § <section>.
Workflow
- Collect HyperPod cluster name (not EKS name), region, exact error string.
- Run
scripts/diagnose-cluster.sh (or --validate for pre-create).
- For every
[FAIL] line, Read the referenced section.
- Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval.
- Re-run the diagnostic to confirm.
Step 1: Run diagnostics
# Diagnose an existing cluster:
bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>
# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,
# optionally S3 lifecycle scripts and per-AZ capacity:
bash scripts/diagnose-cluster.sh --validate --region <REGION> \
--sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
[--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge]
Pass --instance-type when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).
Tags: [PASS] · [FAIL] (counted, has → references/... pointer) · [WARN] · [INFO]. Priorities: P0 blocks operation · P1 degraded · P2 informational.
Step 2: Match signal → section
Error messages / events:
A: EFA Health Checks
SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 / ECR / SageMaker / SSM / STS / CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: cluster-diagnostics-detail.md § A.
B: Capacity & AZ
Instance type unavailable in the requested AZ. Verify with describe-instance-type-offerings, then change AZ, use Flexible Training Plans, or request ODCR. Full: § B · strategy: capacity-planning.md.
C: Lifecycle Scripts
Script failed or timed out during provisioning. Read CloudWatch under /aws/sagemaker/Clusters/<name>/<id> — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: § C · layout: lifecycle-scripts.md.
D: EKS Access / kubectl
IAM identity not in EKS access entries. Verify with sts get-caller-identity, create an access entry with admin policy, update kubeconfig. Full: § D.
E: Cluster Provisioning
InService without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck Creating/Updating/Deleting: check CFN nested stacks (§ H), IAM, capacity, events; if stuck Deleting check VPC ENI dependencies. Full: § E.
F: SSM Connectivity
Target is not connected: use sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID> format (not raw EC2 ID), install session-manager-plugin, confirm node Running. Check IAM + VPC endpoints on timeouts. Full: § F.
G: Node Replacement
Auto-repair: confirm NodeRecovery=Automatic, check Health Monitoring Agent (HMA) logs + node labels / Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via UpdateClusterSoftware at least once and cannot target a Slurm controller node. Full: § G.
H: CloudFormation Errors
Embedded stack failed hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: describe-stack-events --query 'StackEvents[?ResourceStatus==\CREATE_FAILED`]'`. Also covers SLR creation failures and permission-boundary denials. Full: § H · catalog: cloudformation-errors.md.
I: Utilities
Map Slurm node names (ip-10-x-y-z) to HyperPod instance IDs via list-cluster-nodes or on-node /opt/ml/config/resource_config.json. Full: § I.
J: AMI & Cluster Updates
UpdateClusterSoftware fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: § J.
K: Dangling Nodes & Cleanup
Nodes in kubectl get nodes but not in list-cluster-nodes (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: § K.
L: Autoscaler Compatibility
Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: § L.
Prerequisites
aws CLI v2.13+ authenticated to the cluster's accountjq, python3, bash 4.2+kubectl authenticated to the EKS cluster (EKS checks skipped if absent)session-manager-plugin (Slurm controller health checks only)
IAM policy: references/iam-permissions.md.
Defaults
- Region — required: pass
--region or set $AWS_DEFAULT_REGION.
- Mode —
--cluster <NAME> (diagnose) or --validate (pre-create).
- Event window — up to 500 most recent events (5 × 100, paginated).
- Colors — auto-disabled on non-TTY;
--no-color to force off.
Error handling
| Failure | Script | Tell the customer |
|---|
aws sts get-caller-identity fails | Exit 1 | "Fix AWS credentials and rerun." |
| Cluster not found | Exit 1 after listing region's clusters | "Confirm HyperPod cluster name (not EKS) and region." |
sagemaker:* / ec2:* / eks:* / logs:* denied | Warn, add Missing IAM permission for <API>, continue | "Grant the listed IAM action and rerun." |
kubectl absent or unauthenticated | Skip EKS checks (access entries, add-ons, aws-auth, nodes) | "Install/authenticate kubectl." |
session-manager-plugin absent (Slurm) | Skip Slurm controller probe | "Install session-manager-plugin." |
| SSM throttled / times out (180s) | Retry with backoff; warn and continue if still failing | "Rerun later — script is idempotent." |
| CloudWatch log group not found | Skip CloudWatch check | "CloudWatch not configured on this cluster." |
Exit codes: 0 no critical failures · 1 one or more critical failures (cluster not found, fatal prerequisite missing, or any [FAIL] in diagnose or --validate mode). [WARN] lines do not affect the exit code.
Skill delegation
| Need | Use |
|---|
| Shell on nodes | hyperpod-ssm |
| Version comparison across nodes | hyperpod-version-checker |
Escalate to AWS Support
Escalate when:
- EFA health checks fail despite correct SG rules.
- Capacity errors persist despite a valid Flexible Training Plan / ODCR.
- Node replacement fails repeatedly without clear events / log signal.
- Cluster stuck in a non-terminal state (
Creating, Updating, or a post-maintenance rollback state) for an extended period.
- CloudFormation root-cause is an internal service error.
Before opening the case
Run these commands and attach the output. Goal: AWS Support has everything at case open.
# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION>
# 2. Full cluster-level diagnostic bundle
bash scripts/diagnose-cluster.sh --cluster <CLUSTER> --region <REGION> > diag.txt
# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report skill)
# See skills/hyperpod-issue-report/SKILL.md for the exact invocation.
Include in the case
- Cluster name + ARN (or
ClusterId suffix) and AWS region
ClusterStatus + FailureMessage from describe-cluster- Timestamp window (UTC start / end) of the failure
- Exact error strings observed (copy verbatim from events / logs / console)
- Affected instance IDs /
NodeLogicalIds / instance group names
diag.txt from step 2 above- S3 URI of the
hyperpod-issue-report bundle from step 3