feat(recipe): add NVKE GB300 Ubuntu training recipe (shared RDMA)

docs/index.yml

@@ -97,6 +97,8 @@ navigation:
         path: integrator/k0s-h200-setup.md
       - page: NVKE GB200 Setup
         path: integrator/nvke-gb200-setup.md
+      - page: NVKE GB300 Setup
+        path: integrator/nvke-gb300-setup.md
       - page: EKS Dynamo Networking
         path: integrator/eks-dynamo-networking.md
       - page: GKE TCPXO Networking

docs/integrator/index.md

@@ -31,6 +31,7 @@ This section is for integrators who:
 | [Nebius B300 Inference](nebius-b300.md) | MK8s reference configuration, operator ownership, shared RDMA and qualification limits |
 | [k0s H200 Setup](k0s-h200-setup.md) | k0s setup for the H200 training Preview coordinate: host-provided driver posture, k0s's bundled containerd wiring, StorageClass note, and known gaps |
 | [NVKE GB200 Setup](nvke-gb200-setup.md) | NVKE (Forge/NICo) GB200 training Preview coordinate: inbox-driver posture (no DOCA-OFED, DMA-BUF GPUDirect), SR-IOV InfiniBand rails with IPoIB via nv-ipam, RDMA ULP module loading via nodewright, fabric P_Key prerequisite, and known gaps |
+| [NVKE GB300 Setup](nvke-gb300-setup.md) | NVKE (Forge/NICo) GB300 training Preview coordinate: the bare-metal GB300 shape on the inbox driver (no DOCA-OFED, DMA-BUF GPUDirect), shared-HCA RoCE over ConnectX-8 rails with per-rail macvlan attachments via nv-ipam, RDMA ULP module loading via nodewright, and known gaps |
 | [Talos Integration](talos-integration.md) | Running AICR on Talos Linux |
 | [OpenShift Deployment](openshift.md) | OpenShift/OCP-specific Helm and OLM integration and two-phase operator deployment |
 | [Recipe Development](recipe-development.md) | Creating and modifying recipe metadata for custom environments |

docs/integrator/nvke-gb300-setup.md

@@ -0,0 +1,165 @@
+# NVKE GB300 Setup
+
+The `gb300-nvke-ubuntu-training` recipe targets NVKE (NVIDIA Kubernetes Engine,
+formerly NKE): bare-metal GB300 trays provisioned by Forge/NICo, running
+Ubuntu 24.04 with NVIDIA's `-nvidia-64k` kernel. It is a Preview coordinate.
+The reference tray is a Supermicro `ARS-121GL-NB3-LCC-FG001` (Firmus CBR01):
+4x GB300, 144 Grace cores, two dual ConnectX-8 SuperNIC mezzanines for the
+east-west fabric and a BlueField-3 for north-south.
+
+It is the bare-metal `gb300-generic-ubuntu-training` component shape (GPU
+Operator with an operator-installed driver, nvidia-tuned GB300 tuning through
+nodewright, NFD, Network Operator with shared-HCA RDMA) with the fabric
+changed: no `ofedDriver` (the host keeps its inbox driver), RoCE over
+per-rail macvlan attachments instead of InfiniBand, and the `rdma-modules`
+NodeWright from the GB200 leaf in its RoCE form. The NVKE platform prerequisites are the same as for GB200; see
+[NVKE GB200 Setup](nvke-gb200-setup.md) for the `nico://` providerID mapping,
+the control-plane label caveat, and the BlueField-3 VF cap on pod count.
+
+## Cluster prerequisites
+
+- Kubernetes 1.34 or newer (DRA `resource.k8s.io/v1` for ComputeDomains).
+- Nodes carry `providerID: nico://<uuid>`, which `aicr snapshot` maps to
+  `service: nvke`.
+- The node image has no DOCA-OFED: `mlx5_core`, `mlx5_ib`, `ib_core` and
+  `ib_uverbs` load from the kernel tree and userspace is Ubuntu's
+  `libibverbs1`. The recipe runs on that inbox stack and never installs
+  OFED (see below). An image that ships DOCA-OFED also works.
+- `rdma_ucm` is not loaded by the image (`/dev/infiniband/rdma_cm` is
+  missing) and nothing loads it without OFED; the recipe's `rdma-modules`
+  component does (see below). A modules-load.d entry for it in the image
+  makes that component a no-op.
+- The primary CNI is OVN-Kubernetes on the BlueField-3 (`15b3:a2dc`, in
+  zero-trust host-restricted mode), bound to the host's inbox `mlx5_core`.
+  The OVN pod network is `192.168.0.0/16`; the recipe's rail subnets stay
+  out of it.
+- East-west fabric: four ConnectX-8 rails per tray (`mlx5_0`, `mlx5_2`,
+  `mlx5_4`, `mlx5_6` at `0000/0002/0010/0012:03:00.0`; each NIC's second plane
+  is the `.1` function and is not a separate rail), Ethernet link type, 400G.
+  The rails must be linked up with the host PF MTU set as the fabric expects;
+  on the reference tray they came up with no carrier and MTU 1500, which blocks
+  any data-plane validation until the switch side is configured.
+- ConnectX-8 firmware exposes one VF per PF (`sriov_totalvfs` 1). The recipe
+  therefore uses shared-HCA RDMA; see below before trying SR-IOV.
+- Whether the fabric is Spectrum-X is a switch-side fact the recipe cannot
+  detect. If it is, the Network Operator needs the Spectrum-X profile
+  (`l8k generate --spectrum-x RA2.x`), which this recipe does not carry.
+- On a tenant whose only node is a tainted control plane that is also the
+  GPU node, pass the control-plane toleration at bundle time so the GPU
+  Operator and nodewright agree with the fabric DaemonSets, which tolerate it
+  already:
+
+```bash
+aicr bundle --recipe recipe.yaml \
+  --accelerated-node-selector feature.node.kubernetes.io/pci-15b3.present=true \
+  --accelerated-node-toleration node-role.kubernetes.io/control-plane=:NoSchedule \
+  --output bundle
+```
+
+## Never install DOCA-OFED on these nodes
+
+The Network Operator in this recipe ships **no `ofedDriver`**. The host runs
+the inbox `mlx5_core` and OVN-Kubernetes rides on it through the BlueField-3.
+A DOCA driver container unloads `mlx5_core` to replace it, and the node's
+primary address disappears with it; a reboot restores the inbox driver and
+the operator applies the DOCA driver again. This was observed on Firmus GB300
+clusters with recipes that carried an `ofedDriver` block. A tenant whose only
+node is the control plane also cannot be drained for a driver rollout. Do not
+add an `ofedDriver` back for NVKE, and do not combine this recipe with
+another Network Operator installation.
+
+GPUDirect RDMA runs over DMA-BUF: the recipe sets no `driver.rdma` value
+(no `nvidia-peermem`, which has no build for the 64k-page Grace kernel).
+
+## Shared-HCA RoCE with per-rail macvlan
+
+The Network Operator chart deploys only the operator. The NicClusterPolicy
+carries `nvIpam` and `rdmaSharedDevicePlugin`, which exposes one pooled
+resource per rail, `rdma/rdma_shared_resource_rail_0` to `_3`, each selected
+by the rail's PCI address (`deviceID 1023`, link type `ether`; the BlueField-3
+is excluded). The `macvlan-rdma-networks` component (manifest-only, ordered
+after `network-operator` because the nv-ipam CRDs appear only once the
+NicClusterPolicy is reconciled) supplies per rail one nv-ipam `IPPool`
+(`10.255.<4n>.0/22`) and one `MacvlanNetwork` in bridge mode over the rail's
+PF netdev, with `mtu: 0` so the attachment inherits the PF's MTU. The
+NetworkAttachmentDefinitions land in `default`. A pod takes one RDMA
+allocation and one attachment per rail:
+
+```yaml
+metadata:
+  annotations:
+    k8s.v1.cni.cncf.io/networks: default/macvlan-network-rail-0,default/macvlan-network-rail-1,default/macvlan-network-rail-2,default/macvlan-network-rail-3
+spec:
+  containers:
+    - resources:
+        limits:
+          rdma/rdma_shared_resource_rail_0: 1
+          rdma/rdma_shared_resource_rail_1: 1
+          rdma/rdma_shared_resource_rail_2: 1
+          rdma/rdma_shared_resource_rail_3: 1
+```
+
+This is the shape k8s-launch-kit generates for
+`--deployment-type rdma_shared --fabric ethernet --multirail` on this
+topology, minus its `secondaryNetwork.multus`/`cniPlugins` (preinstalled on
+NVKE), its `nfd.enabled: true` (the recipe's own NFD labels the nodes), and
+its `NicConfigurationTemplate` (NIC firmware is a host prerequisite on NVKE;
+the trays reject operator-driven firmware resets). `MacvlanNetwork.master`
+names the PF netdev, so the manifest is specific to the reference tray's
+kernel interface names; a different board needs a copy with its own names.
+
+## rdma-modules and node tainting
+
+`rdma-modules` is a manifest-only component carrying one `NodeWright` CR that
+loads `rdma_ucm` and `ib_umad` and writes
+`/etc/modules-load.d/aicr-rdma-ulp.conf` on every `pci-15b3.present` node.
+This is the RoCE form of the GB200 leaf's manifest: no `ib_ipoib`, since
+pods get a macvlan netdev rather than an IPoIB one, and no control-plane
+exclusion, since on an NVKE GB300 tenant the control plane can be the only
+GPU tray. Without it the rdma-shared device plugin finds no
+`/dev/infiniband/rdma_cm`, advertises no rail resource, and the deployment
+gate's expected-resources check never passes. The package has no interrupt
+(no cordon, drain or reboot). `runtimeRequired: true` with
+`autoTaintNewNodes` lets nodewright taint GPU nodes until the modules are
+loaded. Disable the taint with `--set rdmamodules:autoTaintNewNodes=false`,
+or the whole component with `--set rdmamodules:enabled=false` on images that
+already load `rdma_ucm`.
+
+Why not SR-IOV like the NVKE GB200 leaf: `l8k discover` defaults this tray to
+SR-IOV with `numVfs: 8`, but the firmware exposes one VF per PF and nothing in
+the recipe may rewrite firmware. Raise `NUM_OF_VFS` on the host first if you
+need VFs; then the GB200 leaf's `sriov-ib-networks` component is the model,
+with `linkType: ETH` and `SriovNetwork` in place of `SriovIBNetwork`.
+
+## Generate and validate
+
+```bash
+aicr snapshot --node-selector kubernetes.io/hostname='<gpu-node>' --privileged \
+  --output snapshot.yaml
+aicr recipe --snapshot snapshot.yaml --intent training --output recipe.yaml
+aicr bundle --recipe recipe.yaml --output bundle
+aicr validate --recipe bundle/recipe.yaml
+```
+
+The `nccl-all-reduce-bw-nvls` floor is copied from
+`gb300-generic-ubuntu-training` until an NVKE GB300 baseline is measured.
+Record the measured peak in qualification evidence before re-flooring.
+
+## Known gaps
+
+- The reference tray already ran a GPU Operator (driver 580.173.02), NFD and
+  the NVIDIA DRA driver before this recipe existed. Who installs the GPU
+  stack on an NVKE tenant, the platform or the operator, decides whether this
+  recipe's `gpu-operator`, `nfd` and `nvidia-dra-driver-gpu` components take
+  over an existing install or must be disabled; not yet confirmed.
+- Multus is assumed preinstalled, as on NVKE GB200; not yet confirmed on a
+  GB300 tenant. If absent, the macvlan attachments need `secondaryNetwork`
+  enabled in the NicClusterPolicy.
+- The inbox-driver and missing-`rdma_ucm` findings come from one node on one
+  image build (`nke-node-non-fde-ubuntu-24.04.5-arm64`, checked 2026-10-23).
+  A later NVKE image may load the module or ship DOCA-OFED; either makes
+  `rdma-modules` a no-op rather than a conflict.
+- No NCCL evidence on an NVKE GB300 cluster yet; the east-west rails on the
+  reference tray are not linked up.
+- No published evidence yet.
+- The KWOK profile models scheduling only.

docs/integrator/recipe-development.md

@@ -447,6 +447,7 @@ land later; none is implied by Preview alone.
 | `rke2 / vr200 / ubuntu / inference / dynamo` | [RKE2 VR200 Setup](rke2-vr200-setup.md) | [validation.aicr.run/#/rke2/vr200-ubuntu/inference-dynamo](https://validation.aicr.run/#/rke2/vr200-ubuntu/inference-dynamo) |
 | `k0s / h200 / ubuntu / training` | [k0s H200 Setup](k0s-h200-setup.md) | [validation.aicr.run/#/k0s/h200-ubuntu/training](https://validation.aicr.run/#/k0s/h200-ubuntu/training) |
 | `nvke / gb200 / ubuntu / training` | [NVKE GB200 Setup](nvke-gb200-setup.md) | not yet published |
+| `nvke / gb300 / ubuntu / training` | [NVKE GB300 Setup](nvke-gb300-setup.md) | not yet published |
 
 The platform-neutral `inference` row is the base the Dynamo leaf inherits from; it exists so that resolving `rke2/vr200/ubuntu/inference` **without** `--platform` resolves to the VR200-safe overlay rather than falling through to the generic `rke2-inference` base.
 

docs/user/cli-reference.md

@@ -531,7 +531,7 @@ node shape the resolved recipe targets. See
 
 > **`--service k0s` is Preview**, covering the single `k0s / h200 / ubuntu / training` coordinate. See [k0s H200 Setup](../integrator/k0s-h200-setup.md) for its prerequisites and known gaps.
 
-> **`--service nvke` is Preview**, covering the single `nvke / gb200 / ubuntu / training` coordinate. See [NVKE GB200 Setup](../integrator/nvke-gb200-setup.md) for its prerequisites and known gaps.
+> **`--service nvke` is Preview**, covering the `nvke / gb200 / ubuntu / training` and `nvke / gb300 / ubuntu / training` coordinates. See [NVKE GB200 Setup](../integrator/nvke-gb200-setup.md) and [NVKE GB300 Setup](../integrator/nvke-gb300-setup.md) for their prerequisites and known gaps.
 
 **Examples:**
 ```shell

docs/user/container-images.md

@@ -19,7 +19,7 @@ A machine-readable **CycloneDX 1.6 JSON** companion to this page is produced by
 <!-- BEGIN AICR-BOM -->
 ## Summary
 
-- Components: **56**
+- Components: **57**
 - Unique images: **140**
 - Distinct registries: **12**
 
@@ -57,6 +57,7 @@ _Rendering fidelity:_ `catalog-parity: charts are rendered with the shared recip
 | kube-prometheus-stack | helm | prometheus-community/kube-prometheus-stack | 84.4.0 | 8 |
 | kubeflow-trainer | helm | kubeflow-trainer | 2.2.0 | 4 |
 | kueue | helm | kueue | 0.19.3 | 1 |
+| macvlan-rdma-networks | manifest | — | — | 0 |
 | mariadb-operator | helm | mariadb-operator | 26.6.0 | 1 |
 | mariadb-operator-crds | helm | mariadb-operator-crds | 26.6.0 | 0 |
 | network-operator | helm | nvidia/network-operator | 26.7.0 | 15 |
@@ -257,6 +258,10 @@ _No images extracted._
 
 - `registry.k8s.io/kueue/kueue:v0.19.3`
 
+### macvlan-rdma-networks
+
+_No images extracted._
+
 ### mariadb-operator
 
 - `ghcr.io/mariadb-operator/mariadb-operator:26.6.0`

docs/user/recipe-health.md

@@ -42,8 +42,8 @@ The deep-link is the current Evidence rendering. It is distinct from — and coe
 {/* BEGIN AICR-HEALTH */}
 ## Summary
 
-- Recipes: **71**
-- Pass: **71** · Warn: **0** · Fail: **0** · Unknown: **0**
+- Recipes: **72**
+- Pass: **72** · Warn: **0** · Fail: **0** · Unknown: **0**
 
 ## Recipes
 
@@ -110,6 +110,7 @@ The deep-link is the current Evidence rendering. It is distinct from — and coe
 | rtx-pro-6000-lke-ubuntu-training-kubeflow | lke | rtx-pro-6000 | ubuntu | training | kubeflow | pass | R:0 D:4 P:0 C:10 | pending |
 | b300-mk8s-ubuntu-inference-dynamo | mk8s | b300 | ubuntu | inference | dynamo | pass | R:0 D:4 P:1 C:3 | pending |
 | gb200-nvke-ubuntu-training | nvke | gb200 | ubuntu | training | — | pass | R:0 D:4 P:2 C:5 | pending |
+| gb300-nvke-ubuntu-training | nvke | gb300 | ubuntu | training | — | pass | R:0 D:4 P:1 C:5 | pending |
 | ocp-inference-nim | ocp | — | — | inference | nim | pass | R:0 D:3 P:0 C:11 | pending |
 | ocp-training | ocp | — | — | training | — | pass | R:0 D:3 P:0 C:1 | pending |
 | a100-oke-ubuntu-training-kubeflow | oke | a100 | ubuntu | training | kubeflow | pass | R:0 D:4 P:0 C:10 | pending |

kwok/profiles/nvke/accelerated-gb300.yaml

@@ -0,0 +1,44 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# NVKE GB300 tray as observed on Firmus firmus-tenant8 (Supermicro
+# ARS-121GL-NB3-LCC-FG001): 4 GB300 per tray, 144 Grace arm64 cores, Ubuntu
+# 24.04, driver 580.173.02 (matches the gpu-operator component default).
+# Memory is carried over from the NVKE GB200 tray profile pending an observed
+# allocatable; nothing schedules against it at this magnitude.
+apiVersion: aicr.run/v1beta1
+kind: KWOKNodeProfile
+metadata:
+  name: accelerated-gb300
+  labels:
+    provider: nvke
+    nodeType: accelerated
+    accelerator: gb300
+
+spec:
+  instanceType: gb300-nvl-tray
+  arch: arm64
+  osImage: Ubuntu 24.04
+
+  resources:
+    cpu: "144"
+    memory: 938Gi
+    storage: 800Gi
+
+  gpu:
+    product: NVIDIA-GB300-288GB-HBM3e
+    count: 4
+    memory: 294912  # MiB per GPU (288GB)
+    driver: "580.173.02"
+    driverMajor: "580"
+    driverMinor: "173"