nccl-all-reduce-bw training gate is a fixed absolute fabric-specific busbw value applied to SKU-agnostic recipes → false-fails EKS/H100 small SKUs

[yuanchen8911]

Summary

The nccl-all-reduce-bw training gate is a fixed absolute bus-bandwidth (busbw) value, but the training recipes it lives in are SKU-agnostic (matched only on service + accelerator, no node shape or fabric class). On a node whose fabric differs from the one the floor was calibrated against, the gate false-fails a healthy run.

This is a sibling of #1254 (the inference throughput gate), but the fix is different: inference throughput is ~count-linear, so it gets per-GPU normalization. NCCL busbw is topology/transport-class dependent and is not count-linear — per-GPU scaling must not be applied. The correct invariant is fabric/transport class, not GPU count.

Evidence

Flat absolute compare, no scaling. validators/performance/nccl_all_reduce_bw_constraint.go:257:

passed := bandwidth >= (threshold * 0.9)

The parsed busbw is compared directly against the recipe value with 10% tolerance — no GPU-count or topology scaling.

The clean active threshold false-fail today is H100 on EKS small SKUs. The check only runs for (variant, service, accelerator) tuples present in supportedNCCLCombinations (nccl_all_reduce_bw_constraint.go:151); the lookup is exact-service with no Any fallback (:183). So the active default-variant gates are just:

Overlay	constraint	value	status
h100-eks-training	nccl-all-reduce-bw	>= 300	active (EKS/H100 supported)
h100-gke-cos-training	nccl-all-reduce-bw	>= 250	active (GKE/H100 supported)
h100-aks-training	nccl-all-reduce-bw	>= 100	declared but skips (AKS not in matrix)
h200-eks-training	nccl-all-reduce-bw	>= 300	declared but skips (EKS/H200 not in matrix)
b200-gke-cos-training	nccl-all-reduce-bw	>= 100	declared but skips (GKE/B200 not in matrix)
gb200-eks-training	nccl-all-reduce-bw-net / -nvls	>= 40 / >= 500	active (EKS/GB200 NET+NVLS supported) — reference pattern

So the clean active threshold false-fail is concretely EKS/H100 p5.4xlarge × 2: a supported tuple, the gate runs, and a healthy 1-GPU-per-node run measures network-class busbw well under >= 300. GKE/H100 small SKUs expose the same SKU-agnostic recipe problem but along a broader axis: the GKE runtime is hardcoded to a3-megagpu-8g (TCPXO/FastRak, cloud.google.com/gke-accelerator=nvidia-h100-mega-80gb worker selector at nccl_all_reduce_bw_constraint.go:836, and GPU-NIC-network discovery at :563), so an a3-highgpu-1g × 2 cluster likely fails earlier on runtime/scheduling/template assumptions rather than reaching the busbw threshold. The AKS/H200/B200-default rows are declared-but-inactive gates — alignment/config concerns to fix before enabling support, not current runtime false-fails.

The gate runs on 2-node × 1-GPU clusters. It skips only when fewer than 2 GPU nodes are present (:217); a 2-node, 1-GPU-per-node cluster on a supported service/accelerator runs the full gate.

The thresholds are fixed absolute topology/fabric-specific values, not GPU-count-scaled. The 300/250 H100 numbers were calibrated on 8-GPU high-bandwidth node templates with NVLink/NVSwitch plus multi-NIC transport; a 1-GPU-per-node configuration measures a materially different path (pure inter-node NIC, network class, tens of GB/s) that cannot meet such a floor.

Sub-8-GPU / different-fabric H100 nodes exist on every provider: EKS p5.4xlarge (1), GKE a3-highgpu-1g/2g/4g, Azure NC40ads (1) / NC80adis (2, PCIe-NVL). SKU-agnostic recipes will legitimately land on them.

Network performance: why busbw is fabric-class-bound, not GPU-count-bound

busbw is defined as algbw × 2(n-1)/n — deliberately constructed to report per-link bandwidth, not aggregate. For a multi-node all-reduce the bottleneck is the inter-node NIC path, and on a balanced fabric each GPU has its own fixed-bandwidth NIC (e.g., 1× 400 Gbps EFA per H100 on p5). So 8-GPU → 8 NICs, 4-GPU → 4, 2-GPU → 2, but per-link bandwidth is identical, and busbw normalizes to that per-link figure. A single floor may be valid within a measured equivalent fabric class — it is not a guarantee (measured bandwidth still depends on rank count, NCCL channel selection, NIC mapping, and fabric), but within one balanced fabric busbw lands in the same band regardless of GPUs/node.

What actually breaks it: fabric class, not count

In the real H100 catalog, the smaller-GPU shapes usually aren't "the same node with fewer GPUs" — they're a weaker fabric tier:

• Azure NCads (NC40/NC80) is PCIe H100 NVL with no InfiniBand; the 8-GPU ND family is SXM + IB.
• GCP a3-highgpu (1/2/4/8g) has less per-GPU network than a3-megagpu-8g.
• AWS p5.4xlarge (1 GPU) vs p5.48xlarge differ in per-GPU NIC provisioning.

That's why the calibrated numbers diverge (EKS SXM 300 vs AKS NCads 100) — the gap is per-GPU fabric bandwidth, not GPU count. A single target is portable across 2/4/8-GPU nodes only if they are a measured-equivalent balanced RDMA fabric; it is not portable if the 2-GPU shape is a cheaper, RDMA-less SKU.

One caveat even within a fabric

At small total rank counts (2 nodes × 2 GPU = 4 ranks), the 2(n-1)/n factor and fewer NCCL channels make measured busbw run a bit lower than at 16 ranks. Set the floor with ~10–15% headroom below the large-node value — the existing 10% tolerance roughly absorbs it, but don't calibrate the floor right at the 8-GPU measurement.

Proposed fix: make NCCL gates fabric/transport-class aware, not GPU-count scaled

• Keep nccl-all-reduce-bw as the legacy absolute gate for fixed-shape recipes (unchanged semantics — same backward-compat stance as inference-perf throughput floor is a fixed absolute full-node value applied to SKU-agnostic recipes → false-fails smaller node shapes #1254).
• Do not infer NVLS from gpuCountPerNode. nccl-all-reduce-bw-nvls is MNNVL across an NVL72 IMEX domain (recipes/validators/catalog.yaml:219), a GB200 NVL72-class transport — GPU count alone does not imply it.
• Add or extend named variants only where the validator actually has runtime templates and transport assertions for that (service, accelerator, fabric) — i.e. extend supportedNCCLCombinations (nccl_all_reduce_bw_constraint.go:151) and ship the matching testdata template, rather than declaring a variant a recipe can't run.
• For SKU-agnostic recipes, pick one of:
  • (a) set a conservative weakest-in-scope fabric floor and document it as a coarse liveness gate (simple, portable, weaker on premium fabric), or
  • (b) introduce a recipe criteria dimension for fabric / SKU class and carry measured per-class thresholds (correct, more work, requires per-class hardware to calibrate).
• The NET vs NVLS variants then separate "inter-node fabric healthy" from "NVLS/MNNVL fabric healthy where that variant is implemented" — and keep the 1-GPU case coherent (NET-only).

Do not auto-skip gpuCountPerNode == 1. A 2-node × 1-GPU all-reduce is still a useful signal: it proves distributed training can launch and the inter-node path carries traffic. Treat it as a NET / network-class baseline and name/document it honestly as such — not full-node bandwidth validation.

Implementation sketch

1. Floor/variant by actual supported fabric class, not by GPU count. Keep the legacy nccl-all-reduce-bw absolute gate. Where per-fabric coverage is wanted, gate on the NET/NVLS variants only for (service, accelerator) combinations the validator supports.
2. Align the declared-but-skipping overlays (h100-aks, h200-eks, b200-gke): either add real support (matrix entry + testdata template + transport assertion) with a measured per-fabric floor, or drop the declared constraint until support exists, so a skipped gate isn't mistaken for a passing one.
3. SKU-agnostic floors: apply option (a) or (b) above; if (a), set the H100 EKS/GKE floors to a weakest-in-scope value that a small SKU can still meet, and document the coarse-liveness intent.
4. Update nccl unit tests for the chosen approach (fabric-class selection / weakest-in-scope floors / 1-GPU NET-only path).

Why the inference fix does NOT transfer

For #1254, inference throughput is ~count-linear at fixed concurrency-per-GPU, so a per-GPU floor × benchmarked GPU count is correct. busbw is not count-linear — multiplying a busbw threshold by GPU count would bake in a hardware coincidence (NIC count ≈ GPU count on today's SKUs) and is wrong in general. The correct invariant for NCCL is per-fabric / per-transport, which busbw already normalizes for count.

Scope / related

• Sibling of inference-perf throughput floor is a fixed absolute full-node value applied to SKU-agnostic recipes → false-fails smaller node shapes #1254 (inference throughput). Same root cause — SKU-agnostic recipes vs node-shape/fabric-fixed perf gates — but distinct fix (fabric/transport-class floors here; per-GPU normalization there).
• Clean active threshold false-fail today: EKS/H100 p5.4xlarge × 2. GKE/H100 small SKUs hit the same SKU-agnostic recipe problem but current runtime assumptions (a3-megagpu-8g hardcoding) may fail before threshold evaluation. gb200-eks already uses the NET/NVLS variant pattern and is the reference. h100-aks / h200-eks / b200-gke declare thresholds that currently skip — align before enabling.

References (fabric-class point)

• AWS EC2 accelerated computing: https://aws.amazon.com/ec2/instance-types/accelerated-computing/
• AWS EFA / multi-network-card instance types: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html
• GCP GPU machine types: https://cloud.google.com/compute/docs/gpus/

[yuanchen8911]

Related: #442 (Add AKS H100 NCCL All-Reduce performance runtime).

Distinct axis from this issue:

• feat: Add AKS H100 NCCL All-Reduce performance runtime #442 is a coverage gap — the NCCL validator has no AKS TrainingRuntime template and AKS isn't in the support matrix, so nccl-all-reduce-bw currently skips on AKS. feat: Add AKS H100 NCCL All-Reduce performance runtime #442 makes it run.
• This issue (nccl-all-reduce-bw training gate is a fixed absolute fabric-specific busbw value applied to SKU-agnostic recipes → false-fails EKS/H100 small SKUs #1256) is the threshold model — making the busbw floor fabric/transport-class aware instead of a fixed absolute.

They intersect: #442 assumes 8-GPU ND H100 v5 + IB, so when it lands its threshold should use the fabric-aware floor proposed here rather than a fixed absolute value.

[beanscg]

Bean Labs took a public-source pass on this issue.

The narrowest useful next step looks like: Suggest a narrow regression-test plan tied to the named source/test files and an acceptance check; do not claim implementation.

Public evidence checked:

• issue state OPEN; repo public; assignees=0; comments=1
• title: nccl-all-reduce-bw training gate is a fixed absolute fabric-specific busbw value applied to SKU-agnostic recipes → false-fails EKS/H100 small SKUs
• license: Apache-2.0

A good acceptance check would be a small regression or smoke test that fails before the change and passes after it.

No PR opened from this pass.