feat(recipes): OKE RDMA fabric wiring (L40S RoCE + GB200 IB)

[atif1996]

Summary

Upstreams the OKE RDMA fabric: network-operator with manifest-supplied NicClusterPolicies on both OKE training chains (L40S SR-IOV/RoCE, GB200 IB), the GB200 kernel-module-params wiring, and the nccl-all-reduce-bw-net performance gate with its embedded TrainingRuntime — closing the gb200-oke-training "NET/RDMA intentionally left out until OCI-specific pod RDMA exposure is verified" carve-out.

Motivation / Context

The exposure the carve-out was waiting for exists and is production-validated: a BM.GPU.GB200.4 NVL72 rack (IB east-west over rdma0-3, rdmaSharedDevicePlugin, both NCCL variants passing) and a BM.GPU.L40S.4 RoCE cluster (SR-IOV VFs via device IDs 101a/101e, nv-ipam + multus). This PR ports that configuration from the downstream data repo, de-specialized: hardcoded scheduling taxonomy replaced by the bundler's system-node scheduling injection, downstream-only tolerations and resource aliases dropped.

Stacked on the OKE gpuStack profile PR — the fabric is deliberately present in every profile value (orthogonal to driver/plugin ownership); retargets when the base merges.

Fixes: #2345
Related: #2344, #1716

Type of Change

• New feature (non-breaking change that adds functionality)

Component(s) Affected

• Recipe engine / data (pkg/recipe)
• Validator (pkg/validator)
• Docs/examples (docs/, examples/)

Implementation Notes

• No ofedDriver anywhere, in any configuration: OCI nodes carry host MOFED in every image (Oracle and BYO alike). The device-plugin corollary (MOFED_ENABLED=false) landed in the profile PR.
• One resource name across fabrics: both plugins advertise nvidia.com/mlnxnics, so workloads request RDMA uniformly on OKE.
• deployCR: false + manifest CR: the manifest is the only place the OCI VF selectors / IB ifNames can be expressed; the CR carries the standard post-install Helm hook annotations (deployment-ordering test enforces them).
• GPUDirect via DMA-BUF, peermem off: on L40S every GPU↔NIC PCIe path is NODE/SYS (peermem inert, measured); on GB200 peermem's chroot modprobe fails against the -64k Grace kernel. GB200 needs NVreg_GrdmaPciTopoCheckOverride=1 (kernel-module-params ConfigMap, already embedded — this PR adds the wiring).
• NET gate: supportedNCCLCombinations[variantNET] gains oke/gb200 with testdata/gb200/oke/runtime-net.yaml (NVLS/MNNVL forced off; transport confirmed from the NCCL banner). Constraint >= 40 matches gb200-eks-training.
• Incompatibility with Oracle's opt-in NvidiaNetworkOperator add-on documented on the componentRefs (two lifecycle managers, one release).

Testing

make qualify   # green

• L40S and GB200 bundles render the operator + -post NicClusterPolicy wrapper; ofedDriver absent from both rendered CRs; GB200 bundle carries the kernel-module-params ConfigMap.
• TestOKEPerformanceGoalsFollowTrainingInferencePattern updated for the NET check on all three gb200 training leaves; TestNCCLCombinationSupported OKE NET row flipped to covered; parity goldens + BOM regenerated (four NicClusterPolicy image digest exemptions — triplet CRD schema, same as AKS).

Risk Assessment

• Medium — Touches multiple components or has broader impact

Rollout notes: Additive for existing OKE users without RDMA-capable pools (the NicClusterPolicy DaemonSets simply schedule nowhere without matching NICs). Clusters running Oracle's NvidiaNetworkOperator add-on must disable it before deploying these bundles.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)

[github-actions]

🌿 Preview your docs: https://nvidia-preview-feat-oke-fabric-wiring.docs.buildwithfern.com/aicr

[github-actions]

Recipe evidence check

Other affected recipes without evidence yet: 4

These recipes are affected by this PR but carry no committed evidence pointer, so there is
nothing to verify. This is expected — evidence is hardware-gated and added over time.

• gb200-oke-training
• gb200-oke-ubuntu-training-kubeflow
• gb200-oke-ubuntu-training
• l40s-oke-training

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: ee933145-e8f5-451e-8f8f-bb58170f2a56

📥 Commits

Reviewing files that changed from the base of the PR and between fe764f3 and c5b54ae.

📒 Files selected for processing (14)

• docs/user/container-images.md
• pkg/bundler/testdata/stock_render_golden.yaml
• pkg/recipe/performance_goals_oke_test.go
• pkg/recipe/testdata/catalog_parity_golden.yaml
• recipes/components/network-operator/manifests/nic-cluster-policy-oke-gb200.yaml
• recipes/components/network-operator/manifests/nic-cluster-policy-oke-l40s.yaml
• recipes/components/network-operator/values-oke-gb200.yaml
• recipes/components/network-operator/values-oke-l40s.yaml
• recipes/manifest_images_test.go
• recipes/overlays/gb200-oke-training.yaml
• recipes/overlays/l40s-oke-training.yaml
• validators/performance/nccl_all_reduce_bw_constraint.go
• validators/performance/nccl_benchmark_profile_test.go
• validators/performance/testdata/gb200/oke/runtime-net.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Walkthrough

Walkthrough

The OKE GB200 and L40S training recipes now configure Network Operator networking for InfiniBand or RoCE RDMA devices. GB200 GPU Operator settings enable the required kernel-module configuration. GB200 OKE validation now runs NCCL NET and NVLS checks. A GB200 OKE NET runtime and benchmark mapping were added. Image inventories, digest exemptions, and generated recipe digests were updated.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to c5b54

This PR adds OKE RDMA fabric configuration and related performance validation; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: mchmarny

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The L40S-specific network-operator manifests, values, and overlay are not covered by the only provided linked issue, which targets GB200 OKE.	Link an issue that covers the L40S OKE scope or move the L40S changes into a separate pull request.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the OKE RDMA fabric wiring for both L40S RoCE and GB200 InfiniBand.
Description check	✅ Passed	The description directly explains the RDMA configurations, validation changes, motivation, testing, and rollout considerations.
Linked Issues check	✅ Passed	The PR satisfies the GB200 OKE requirements in [#2345], including RDMA exposure, DMA-BUF wiring, no OFED driver, and the NET bandwidth gate.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/oke-fabric-wiring

---

_{Comment @coderabbitai help to get the list of available commands.}