inference-perf on GB300 needs a CUDA 13 (sm_103) vLLM runtime

[yuanchen8911]

Summary

inference-perf cannot produce a meaningful GB300 measurement with the currently pinned Dynamo worker image, because that image is CUDA 12 and GB300 (Blackwell Ultra) is sm_103. The check is deferred on gb300-eks-ubuntu-inference-dynamo until an sm_103-capable (CUDA 13) vLLM / Dynamo runtime ships.

Evidence

Measured 2026-08-25 on nhensley-gb300 (2× p6e-gb300r.36xlarge, Qwen/Qwen3-8B, 4 GPU workers, concurrency 1024, 8192 requests) — the same configuration as the GB200 baseline:

Metric	GB200 reference	GB300 measured	Gate
throughput	~65,952 tok/s	20,769 tok/s	>= 50000
TTFT p99	~1,240 ms	33,320 ms	<= 2000

GB300 should exceed GB200. A 3× throughput drop paired with a 27× latency blowup is not a hardware characteristic — it is the signature of serving from JIT-compiled kernels.

Confirmed from the image config of nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 (linux/arm64):

CUDA_VERSION=12.9.1
NV_LIBNCCL_PACKAGE=libnccl2=2.27.3-1+cuda12.9

Mechanism

Plain CUDA kernels forward-compat onto a newer arch via PTX JIT, but runtime kernel compilers (Triton/LLVM) need explicit arch support. Without sm_103 codegen the workers still serve — they just compile kernels on demand, which is what destroys first-token latency while leaving throughput merely degraded.

This is the same class of gap already documented for VR200 / sm_107a, where the toolchain does not recognize the arch at all and the decode workers crash-loop outright rather than degrade. GB300 is the milder variant because CUDA 12.9 partially handles sm_103.

Why the thresholds were not simply retuned

Setting floors from 20,769 tok/s / 33,320 ms would encode a degraded configuration as the GB300 baseline. It would turn the PR green while permanently masking this gap, and a correct CUDA 13 image would then exceed the floor by ~3×, leaving the gate unable to detect a regression in either direction.

Done when

1. An sm_103-capable (CUDA 13) vLLM / Dynamo runtime image is available and pinned.
2. inference-perf is re-enabled on gb300-eks-ubuntu-inference-dynamo.
3. Throughput and TTFT floors are measured on GB300 and set from that run.

Notes

• Deployment (4/4) and conformance (10/10) pass on this leaf and continue to gate it; only the performance phase is deferred.
• cacheWorkerImage in validators/performance/model_cache.go is pinned to the same image and is kept in sync with testdata/inference/dynamo-deployment.yaml, so both move together when the runtime is bumped.

Related: #2381 (GB300 EKS overlays), #2382 (the PR that deferred the check).

[yuanchen8911]

Correction to the "Done when" framing above: this is not blocked on upstream. CUDA 13 images already exist and ship linux/arm64.

Tag	CUDA	arm64
1.2.1 (currently pinned)	12.9.1	yes
1.3.1	13.0.2	yes
1.4.0	13.0.2	yes
1.4.1	13.0.2	yes

There is also an explicit 1.3.0-dev.1-cuda13 tag, but the stable 1.3.1/1.4.x line is already CUDA 13.0.2, so no dev tag is needed.

Why this is its own PR rather than part of #2382

The image is not a GB300 setting. It is pinned in validators/performance/model_cache.go (cacheWorkerImage) and validators/performance/testdata/inference/dynamo-deployment.yaml, kept in sync by TestCacheWorkerImageMatchesTemplate, and it drives every Dynamo leaf:

b200-gke-cos-inference-dynamo       gb200-eks-ubuntu-inference-dynamo
b200-gke-cos-inference              gb200-oke-ubuntu-inference-dynamo
h100-aks-ubuntu-inference-dynamo    h100-eks-ubuntu-inference-dynamo
h100-gke-cos-inference-dynamo       rtx-pro-6000-eks-ubuntu-inference-dynamo
gb300-eks-ubuntu-inference-dynamo

Each of the eight non-GB300 leaves carries inference-throughput and inference-ttft-p99 floors that were measured on 1.2.1. A major runtime bump can move those numbers, so the change needs re-baselining across H100 (AKS/EKS/GKE), B200 (GKE), GB200 (EKS/OKE), and RTX PRO 6000 (EKS) — not just GB300.

It also touches tests/manifests/dynamo-vllm-smoke-test.yaml, and interacts with the UAT cold-pull timeout budgets in uat-aws.yaml / uat-gcp.yaml / uat-azure.yaml, which already note that the ~12 GB pull strains the readiness budget. 1.4.1 is likely larger.

Suggested scope

1. Bump cacheWorkerImage and the deployment template together (the drift guard enforces this).
2. Re-measure and reset throughput/TTFT floors for each affected leaf on its own hardware.
3. Re-check the UAT cold-pull budgets against the new image size.
4. Re-enable inference-perf on gb300-eks-ubuntu-inference-dynamo and set its floors from a GB300 run — the two-line change this issue was originally filed for.

Step 4 is the only part specific to GB300; steps 1-3 are catalog-wide and are what keep this out of #2382.

[yuanchen8911]

Closing — the premise of this issue was wrong, and the real cause has been found and fixed.

What this issue claimed

That GB300's inference-perf failure (20,769 tok/s, 33,320 ms TTFT p99 against gates of >= 50,000 and <= 2,000) was caused by the pinned Dynamo worker image being CUDA 12.9 while GB300 is sm_103, and that it was blocked pending a CUDA 13 runtime.

Why that was wrong

Tested directly by rebuilding the worker on vllm-runtime:1.2.1-cuda13 (CUDA 13.0.2, same Dynamo version, so no operator/worker skew) and re-running on the same cluster:

Image	throughput	TTFT p99
1.2.1 (CUDA 12.9)	20,769 tok/s	33,320 ms
1.2.1-cuda13 (CUDA 13.0.2)	21,100 tok/s	33,362 ms

Within 1.6% on throughput and 0.1% on latency. CUDA is not the variable.

Actual cause

The benchmark was running past GB300's saturation knee. The compiled default is 256 concurrency/GPU, tuned for GB200 and EKS H100. GB300's knee sits lower in concurrency because each GPU is faster and the single Dynamo frontend replica saturates sooner — the per-replica frontend ceiling documented in ai-dynamo/dynamo#9466 (measured on B300 with a -cuda13 image, so also independent of CUDA version) and the knee volatility described in docs/contributor/inference-perf-fluctuation.md.

At a sub-knee 128 concurrency/GPU, on the unchanged CUDA 12.9 image:

84,788 tok/s   TTFT p99 677 ms   PASS

That beats the GB200 reference (65,952 tok/s / 1,240 ms) by 29% on throughput at roughly half the latency. GB300 was never slow.

Disposition

• The fix is a per-accelerator inference-concurrency-per-gpu: "128" constraint on gb300-eks-ubuntu-inference-dynamo, with inference-perf enabled and floors set from measurement. Landing in feat(recipes): add GB300 EKS training and inference overlays #2382.
• The CUDA 13 runtime bump has no measured benefit and is folded into Upgrade gpu-operator to v26.7.0 and the DRA driver to 0.5.0 #2383 (catalog-wide component version audit) as routine currency maintenance: Upgrade gpu-operator to v26.7.0 and the DRA driver to 0.5.0 #2383 (comment)
• One genuine upstream gap is recorded there too: neither 1.2.1 nor 1.2.1-cuda13 ships sm_103 cubins, so GB300 runs via PTX JIT forward-compat from sm_100. It works and still outperforms GB200, but there may be headroom once upstream builds sm_103 kernels. Not actionable in this repo.