Spectrum-X Kubernetes Quick Start

On this page

Choose a deployment mode
Step 1: Prepare the Worker Nodes
Step 2: Install the Helm Chart
Step 3: Apply NicClusterPolicy
Step 4: Name the Rail Interfaces
Step 5: Apply the Profile ConfigMap
Step 6: Configure the SuperNICs
Step 7: Allocate Rail Subnets
Step 8: Define the Rail Pool
Step 9: Deploy a Test Pod

Note

These walkthroughs use the RA2.4 configuration workflow on Network Operator v26.10.0-rc.7. They require the matching NIC Configuration Operator CRDs, including platformType. RA3.1 is released alongside RA2.4 for Vera Rubin (VR); these non-VR examples remain RA2.4. For VR, select version: RA3.1 and platformType: vr with the matching profile and dual-NIC rail layout described in Launch Kit Spectrum-X Profile. For the RA-to-release mapping and other RAs, see NVIDIA Spectrum-X. For supported platforms (operating systems, Kubernetes distributions, NIC hardware), see Platform Support.

The tabs cover Single-Plane, Hardware Multiplane (hwplb), and Software Multiplane (swplb) configuration. Use the support matrix for the validation status of your hardware and release. For background on the operators and CNIs each walkthrough relies on, see Architecture and Components.

Before configuring the NICs, apply one Spectrum-X profile ConfigMap. Unlike 26.7.x, spectrumXOptimized.version selects the architecture version and does not name the ConfigMap — see Spectrum-X NIC Configuration.

Tip

The configuration in each walkthrough can be automated end-to-end with NVIDIA Kubernetes Launch Kit — see Configuration Assistance with Kubernetes Launch Kit.

Note

To allocate rail VFs through Dynamic Resource Allocation and pair each VF with the GPU behind the same PCIe root, add one ResourceClaimTemplate per rail after completing a walkthrough — see DRA SR-IOV Driver.

Once the cluster is up, Verify and Troubleshoot walks through checking each layer.

The walkthrough installs Network Operator via Helm, applies the Spectrum-X CRDs, and deploys a test pod, adapted to the multiplane mode and NIC family you choose below. The examples show two rails to keep the manifests short; repeat the naming, NIC selection, pools, rail topology, and Pod resource requests for every east-west rail in your platform.

Choose a deployment mode

Mode	`multiplaneMode`	NICs	GPU platforms	Use when
Single-Plane	`none`	BlueField-3 SuperNIC, ConnectX-7 NIC, ConnectX-8 SuperNIC	H100/H200/B200 (BlueField-3 SuperNIC), GB200 (ConnectX-7 NIC)	One PF per rail. Simplest setup. ConnectX-8 SuperNIC also supports single-plane configuration.
Hardware Multiplane	`hwplb`	ConnectX-8 SuperNIC	B300, GB300	Plane Load Balancing happens in the NIC hardware and the planes are hidden from the workload. The recommended mode on multiplane platforms. Set `numberOfPlanes: 2` (Dual-Plane) or `4` (Quad-Plane).
Software Multiplane	`swplb`	ConnectX-8 SuperNIC	B300, GB300	Software Plane Load Balancing across planes, with each plane exposed to the workload as its own interface. Set `numberOfPlanes: 2` (Dual-Plane) or `4` (Quad-Plane).

Select your mode below. The tabs on this page are linked — choosing here switches every mode-specific step that follows.

Single-Plane

This walkthrough deploys a Single-Plane Spectrum-X cluster on Kubernetes: one PF per rail, one CIDRPool per rail, one network per rail. Used on HGX H100/H200/B200 platforms (BlueField-3 SuperNIC, nicType: a2dc) and GB200 NVL72 platforms (ConnectX-7 NIC, nicType: 1021). ConnectX-8 SuperNIC (nicType: 1023) also supports single-plane configuration — useful if you want a single-plane setup on B300/GB300 hardware. The configuration uses Spectrum-X RA 2.4 with multiplaneMode: none and numberOfPlanes: 1. The example below uses BlueField-3 SuperNIC; change nicType for other NICs. Replace TODO_* values with your cluster-specific values before applying.

Hardware Multiplane

This walkthrough deploys a Hardware Multiplane Spectrum-X cluster on Kubernetes using ConnectX-8 SuperNICs (nicType: 1023). NIC LAG and Hardware Plane Load Balancing (hwplb) handle per-plane fan-out at the hardware layer, so each rail still uses a single CIDRPool while exposing multiple per-plane PFs. Used on B300 and GB300 platforms — set numberOfPlanes: 2 for Dual-Plane or numberOfPlanes: 4 for Quad-Plane. The configuration uses Spectrum-X RA 2.4 with multiplaneMode: hwplb. The example below uses numberOfPlanes: 2. Replace TODO_* values with your cluster-specific values before applying.

Software Multiplane

This walkthrough deploys a Software Multiplane Spectrum-X cluster on Kubernetes using ConnectX-8 SuperNICs (nicType: 1023). Each SuperNIC is split into multiple PFs, each assigned to a separate plane, and the software stack performs Software Plane Load Balancing (swplb) across them. Used on B300 and GB300 platforms — set numberOfPlanes: 2 for Dual-Plane or numberOfPlanes: 4 for Quad-Plane. The configuration uses Spectrum-X RA 2.4 with multiplaneMode: swplb. The example below uses numberOfPlanes: 2. Replace TODO_* values with your cluster-specific values before applying.

Step 1: Prepare the Worker Nodes

Configure the switch fabric and prepare every worker node for the supplied RA2.4 profile before installing Network Operator. Install the matching DOCA-Host driver and OVS-DOCA packages, and reserve the hugepages needed by OVS-DOCA and, for Hardware Multiplane, DOCA xPlane. Follow Architecture and Components for ownership and sizing. This walkthrough uses host-installed drivers and omits the containerized ofedDriver configuration.

The following host settings take effect after a reboot:

Virtualization support. SR-IOV Virtual Functions require IOMMU and VF capability to be enabled in the platform firmware and on the kernel command line. Follow the host preparation procedure for your platform in the NVIDIA Spectrum-X deployment guide for your Reference Architecture.

RDMA subsystem namespace awareness. Linux network namespaces do not isolate RDMA devices by default — every RDMA device stays visible in every namespace and Pod. Switch the subsystem to namespace-exclusive mode:

echo "options ib_core netns_mode=0" >> /etc/modprobe.d/ib_core.conf

Reboot the node for both settings to take effect.

Warning

Without netns_mode=0, RDMA devices are not isolated per Pod: workloads see RDMA devices that belong to other Pods, and the per-rail isolation this walkthrough builds does not hold.

Step 2: Install the Helm Chart

Add the NVIDIA NGC Helm repository:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

Spectrum-X requires several non-default chart values. Create values.yaml:

sriovNetworkOperator:
  enabled: true

maintenanceOperator:
  enabled: true

sriov-network-operator:
  sriovOperatorConfig:
    deploy: true
    configDaemonNodeSelector:
      network.nvidia.com/operator.nic-configuration.wait: "false"
    featureGates:
      manageSoftwareBridges: true
    disablePlugins:
    - mellanox

Value	Why it is required
`featureGates.manageSoftwareBridges`	Lets the SR-IOV Network Operator manage the OVS software bridges that the Spectrum-X data path needs. Without it, the bridges are never created.
`disablePlugins: [mellanox]`	Hands SuperNIC firmware, NV config, and SR-IOV enablement to the NIC Configuration Operator. Leaving the SR-IOV Network Operator’s own Mellanox plugin enabled configures the same state twice and makes the configuration flap.
`configDaemonNodeSelector`	Makes the SR-IOV config daemon wait for the NIC Configuration Operator to finish per-node firmware and NIC configuration before it touches VFs.
`featureGates.dynamicResourceAllocation`	Not set in this walkthrough, which allocates VFs through the SR-IOV device plugin. Add it only for Dynamic Resource Allocation (DRA) workloads: enabling it replaces the device plugin with the DRA driver, so the resource requests in Step 9 no longer apply. See DRA SR-IOV Driver.

Install the Network Operator. Spectrum-X Operator and NIC Configuration Operator are deployed via the same chart and enabled later through NicClusterPolicy.

helm install network-operator nvidia/network-operator \
  -n nvidia-network-operator \
  --create-namespace \
  --version 26.10.0-rc.7 \
  -f values.yaml \
  --wait

Verify the installation:

kubectl -n nvidia-network-operator get pods

Step 3: Apply NicClusterPolicy

Enable the NIC Configuration Operator, NV-IPAM, Spectrum-X Operator (with XPlane), and the secondary network components. Save the following as nicclusterpolicy.yaml. Set storageClassName to an existing storage class that supports the firmware storage PVC; see NIC Firmware Configuration.

apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
spec:
  nicConfigurationOperator:
    operator:
      image: nic-configuration-operator
      repository: nvcr.io/nvstaging/mellanox
      version: network-operator-v26.10.0-rc.7
    configurationDaemon:
      image: nic-configuration-operator-daemon
      repository: nvcr.io/nvstaging/mellanox
      version: network-operator-v26.10.0-rc.7
    nicFirmwareStorage:
      create: true
      pvcName: nic-fw-storage-pvc
      storageClassName: nic-fw-storage-class
      availableStorageSize: 1Gi
    logLevel: info
  nvIpam:
    image: nvidia-k8s-ipam
    repository: nvcr.io/nvstaging/mellanox
    version: network-operator-v26.10.0-rc.7
    enableWebhook: false
  spectrumXOperator:
    image: spectrum-x-operator
    repository: nvcr.io/nvstaging/mellanox
    version: network-operator-v26.10.0-rc.7
    # xPlane is only used when multiplaneMode=hwplb (Hardware Multiplane).
    # The rail topology must also list every PF in the hardware-multiplane rail.
    xPlane:
      image: xplane
      # xPlane ships on the DOCA release cadence, so its repository and tag
      # differ from the other Network Operator components.
      repository: nvcr.io/nvidia/doca
      version: 3.5.0041
  secondaryNetwork:
    cniPlugins:
      image: plugins
      repository: nvcr.io/nvstaging/mellanox
      version: network-operator-v26.10.0-rc.7
    multus:
      image: multus-cni
      repository: nvcr.io/nvstaging/mellanox
      version: network-operator-v26.10.0-rc.7

kubectl apply -f nicclusterpolicy.yaml

Step 4: Name the Rail Interfaces

Create the NicInterfaceNameTemplate that maps PCI addresses to rails and gives each interface a predictable name. For the four-plane B300/GB300 layout, set pfsPerNic: 4 here and numberOfPlanes: 4 in Step 6. Then include all four PF names in each Hardware Multiplane rail in Step 8; for Software Multiplane, add the plane-2 and plane-3 pools, topology entries, and Pod attachments and resource requests in Steps 7–9.

Host PF RDMA names must differ from pod VF RDMA names. The Spectrum-X Operator names pod RDMA devices rdma_<railTopology.name>, for example, rdma_rail0. The examples use the roce_r prefix for host PFs to avoid name collisions.

Single-Plane

Map PCI addresses to rails and define interface naming. With single-plane configuration there is one PF per NIC, so pfsPerNic is 1 and %plane_id% is always 0. Replace TODO_PCI_* with the PCI addresses of the BlueField-3 SuperNICs on your nodes.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
  name: spectrum-x-interface-names
  namespace: nvidia-network-operator
spec:
  pfsPerNic: 1
  rdmaDevicePrefix: "roce_r%rail_id%"
  netDevicePrefix: "net_rail%rail_id%"
  railPciAddresses:
    - ["TODO_PCI_RAIL0_NIC0"]
    - ["TODO_PCI_RAIL1_NIC0"]

kubectl apply -f nicinterfacenametemplate.yaml

Hardware Multiplane

Map PCI addresses to rail/plane indices and define interface naming. List one base PCI address per physical NIC; pfsPerNic: 2 generates names for its two PFs. Replace TODO_PCI_* with the PCI addresses of the Spectrum-X NICs on your nodes.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
  name: spectrum-x-interface-names
  namespace: nvidia-network-operator
spec:
  pfsPerNic: 2
  rdmaDevicePrefix: "roce_r%rail_id%_plane%plane_id%"
  netDevicePrefix: "net_rail%rail_id%_plane%plane_id%"
  railPciAddresses:
    - ["TODO_PCI_RAIL0_NIC0"]
    - ["TODO_PCI_RAIL1_NIC0"]

kubectl apply -f nicinterfacenametemplate.yaml

Software Multiplane

Map PCI addresses to rail/plane indices and define interface naming. List one base PCI address per physical NIC; pfsPerNic: 2 generates names for its two PFs. Replace TODO_PCI_* with the PCI addresses of the Spectrum-X NICs on your nodes.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
  name: spectrum-x-interface-names
  namespace: nvidia-network-operator
spec:
  pfsPerNic: 2
  rdmaDevicePrefix: "roce_r%rail_id%_plane%plane_id%"
  netDevicePrefix: "net_rail%rail_id%_plane%plane_id%"
  railPciAddresses:
    - ["TODO_PCI_RAIL0_NIC0"]
    - ["TODO_PCI_RAIL1_NIC0"]

kubectl apply -f nicinterfacenametemplate.yaml

Step 5: Apply the Profile ConfigMap

Obtain the complete published Spectrum-X profile ConfigMap for your deployment, or contact your NVIDIA CPM. It must carry the network.nvidia.com/operator.nic-configuration.spectrum-x-profile label, data.format: dospcx-data.tar.gz/v1, and the compressed archive under binaryData.dospcx-data.tar.gz. Keep only one current-format profile selected cluster-wide.

kubectl apply -f spectrum-x-profile-configmap.yaml

The profile must support the architecture version, platform, mode, and NICs you select. Its name is independent of spectrumXOptimized.version. For the profile ConfigMap format and migration from 26.7.x, see Spectrum-X NIC Configuration.

Step 6: Configure the SuperNICs

Create the NicConfigurationTemplate that selects the architecture version, platform, and multiplane mode. Match platformType to the deployed GPU platform and the profile’s supported identifiers, rather than the NIC model. Use Platform configuration values to select the platform type, NIC ID, multiplane mode, and plane count for your worker nodes.

Single-Plane

Configure the NICs in single-plane mode. This example uses BlueField-3 SuperNIC on H100. For H200 or B200, change platformType to h200 or b200. For ConnectX-7 NIC on GB200, use nicType: "1021" and platformType: gb200.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
    nicType: "a2dc"  # BlueField-3 SuperNIC (HGX H100/H200/B200). Use "1021" for ConnectX-7 NIC (GB200) or "1023" for ConnectX-8 SuperNIC.
    pciAddresses:  # every east-west PF, all rails
      - "TODO_PCI_RAIL0_NIC0"
      - "TODO_PCI_RAIL1_NIC0"
  template:
    numVfs: 1
    linkType: Ethernet
    spectrumXOptimized:
      enabled: true
      version: "RA2.4"
      platformType: "h100"
      overlay: "none"
      multiplaneMode: "none"
      numberOfPlanes: 1
    pciPerformanceOptimized:
      enabled: true
      maxReadRequest: 4096

kubectl apply -f nicconfigurationtemplate.yaml

Note

List the east-west PFs explicitly in nicSelector.pciAddresses. On nodes that also carry a north-south DPU with the same device ID, selecting on nicType alone matches the north-south device as well and applies Spectrum-X configuration to the wrong NIC.

Hardware Multiplane

Configure the ConnectX-8 SuperNICs with hwplb. This example uses GB300; for B300 use platformType: b300. hwplb requires ConnectX-8 SuperNIC (nicType: 1023) or ConnectX-9 SuperNIC (nicType: 1025, tech preview); it is not supported on BlueField-3 SuperNIC or ConnectX-7 NIC. For Quad-Plane, set numberOfPlanes: 4.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
    nicType: "1023"  # ConnectX-8 SuperNIC (B300, GB300)
    pciAddresses:  # every east-west PF, all rails
      - "TODO_PCI_RAIL0_NIC0"
      - "TODO_PCI_RAIL0_NIC1"
      - "TODO_PCI_RAIL1_NIC0"
      - "TODO_PCI_RAIL1_NIC1"
  template:
    numVfs: 1
    linkType: Ethernet
    spectrumXOptimized:
      enabled: true
      version: "RA2.4"
      platformType: "gb300"
      overlay: "none"
      multiplaneMode: "hwplb"
      numberOfPlanes: 2
    pciPerformanceOptimized:
      enabled: true
      maxReadRequest: 4096

kubectl apply -f nicconfigurationtemplate.yaml

Note

List the east-west PFs explicitly in nicSelector.pciAddresses. On nodes that also carry a north-south DPU with the same device ID, selecting on nicType alone matches the north-south device as well and applies Spectrum-X configuration to the wrong NIC.

Software Multiplane

Configure the ConnectX-8 SuperNICs with swplb. This example uses GB300; for B300 use platformType: b300. For Quad-Plane, set numberOfPlanes: 4.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
    nicType: "1023"  # ConnectX-8 SuperNIC (B300, GB300)
    pciAddresses:  # every east-west PF, all rails
      - "TODO_PCI_RAIL0_NIC0"
      - "TODO_PCI_RAIL0_NIC1"
      - "TODO_PCI_RAIL1_NIC0"
      - "TODO_PCI_RAIL1_NIC1"
  template:
    numVfs: 1
    linkType: Ethernet
    spectrumXOptimized:
      enabled: true
      version: "RA2.4"
      platformType: "gb300"
      overlay: "none"
      multiplaneMode: "swplb"
      numberOfPlanes: 2
    pciPerformanceOptimized:
      enabled: true
      maxReadRequest: 4096

kubectl apply -f nicconfigurationtemplate.yaml

Note

List the east-west PFs explicitly in nicSelector.pciAddresses. On nodes that also carry a north-south DPU with the same device ID, selecting on nicType alone matches the north-south device as well and applies Spectrum-X configuration to the wrong NIC.

Step 7: Allocate Rail Subnets

Create the CIDRPool resources that NV-IPAM uses to assign rail addresses to Pods.

Each per-node /31 contains two addresses. With gatewayIndex: 0, index 0 is reserved for the gateway and index 1 is available for the pod. Do not exclude index 1, which would leave no address for pod allocation.

Single-Plane

In single-plane mode each rail is a single network, so one CIDRPool per rail is sufficient. Replace TODO_* with subnets that match your cluster’s east-west topology.

apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
  name: rail-0
  namespace: nvidia-network-operator
spec:
  cidr: TODO_RAIL0_CIDR             # e.g., 10.0.0.0/15
  gatewayIndex: 0
  perNodeNetworkPrefix: 31
  routes:
    - dst: TODO_RAIL0_SUBNET        # same as cidr
    - dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
  name: rail-1
  namespace: nvidia-network-operator
spec:
  cidr: TODO_RAIL1_CIDR
  gatewayIndex: 0
  perNodeNetworkPrefix: 31
  routes:
    - dst: TODO_RAIL1_SUBNET
    - dst: TODO_EAST_WEST_SUBNET

kubectl apply -f cidrpool.yaml

Hardware Multiplane

With hwplb, load balancing happens at the NIC layer, so each rail uses a single CIDRPool covering all of its planes. Replace TODO_* with subnets that match your cluster’s east-west topology.

apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
  name: rail-0
  namespace: nvidia-network-operator
spec:
  cidr: TODO_RAIL0_CIDR             # e.g., 10.0.0.0/15
  gatewayIndex: 0
  perNodeNetworkPrefix: 31
  routes:
    - dst: TODO_RAIL0_SUBNET        # same as cidr
    - dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
  name: rail-1
  namespace: nvidia-network-operator
spec:
  cidr: TODO_RAIL1_CIDR
  gatewayIndex: 0
  perNodeNetworkPrefix: 31
  routes:
    - dst: TODO_RAIL1_SUBNET
    - dst: TODO_EAST_WEST_SUBNET

kubectl apply -f cidrpool.yaml

Software Multiplane

With swplb, each rail is split into multiple planes, and each plane requires its own CIDRPool. Create one pool per (rail, plane) combination. Replace TODO_* with subnets that match your cluster’s east-west topology.

apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
  name: rail-0-plane-0
  namespace: nvidia-network-operator
spec:
  cidr: TODO_RAIL0_PLANE0_CIDR      # e.g., 10.0.0.0/15
  gatewayIndex: 0
  perNodeNetworkPrefix: 31
  routes:
    - dst: TODO_RAIL0_PLANE0_SUBNET # same as cidr
    - dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
  name: rail-0-plane-1
  namespace: nvidia-network-operator
spec:
  cidr: TODO_RAIL0_PLANE1_CIDR
  gatewayIndex: 0
  perNodeNetworkPrefix: 31
  routes:
    - dst: TODO_RAIL0_PLANE1_SUBNET
    - dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
  name: rail-1-plane-0
  namespace: nvidia-network-operator
spec:
  cidr: TODO_RAIL1_PLANE0_CIDR
  gatewayIndex: 0
  perNodeNetworkPrefix: 31
  routes:
    - dst: TODO_RAIL1_PLANE0_SUBNET
    - dst: TODO_EAST_WEST_SUBNET
---
apiVersion: nv-ipam.nvidia.com/v1alpha1
kind: CIDRPool
metadata:
  name: rail-1-plane-1
  namespace: nvidia-network-operator
spec:
  cidr: TODO_RAIL1_PLANE1_CIDR
  gatewayIndex: 0
  perNodeNetworkPrefix: 31
  routes:
    - dst: TODO_RAIL1_PLANE1_SUBNET
    - dst: TODO_EAST_WEST_SUBNET

kubectl apply -f cidrpool.yaml

Step 8: Define the Rail Pool

Create the SpectrumXRailPoolConfig that describes the rail topology and exposes each rail as a schedulable resource.

Single-Plane

One entry in railTopology per rail, selecting the single PF in that rail (matching NicInterfaceNameTemplate) and referencing the matching CIDRPool.

apiVersion: spectrumx.nvidia.com/v1alpha2
kind: SpectrumXRailPoolConfig
metadata:
  name: rails
  namespace: nvidia-network-operator
spec:
  draEnabled: false
  networkNamespace: default
  numVfs: 1
  railTopology:
    - name: rail0
      nicSelector:
        pfNames: ["net_rail0"]
      cidrPoolRef: rail-0
      mtu: 9216
    - name: rail1
      nicSelector:
        pfNames: ["net_rail1"]
      cidrPoolRef: rail-1
      mtu: 9216

kubectl apply -f spectrumxrailpoolconfig.yaml

Hardware Multiplane

With hwplb, railTopology has one entry per rail. Each entry lists all per-plane PF netdev names belonging to that rail and references a single per-rail CIDRPool.

apiVersion: spectrumx.nvidia.com/v1alpha2
kind: SpectrumXRailPoolConfig
metadata:
  name: rails
  namespace: nvidia-network-operator
spec:
  draEnabled: false
  networkNamespace: default
  numVfs: 1
  railTopology:
    - name: rail0
      nicSelector:
        pfNames: ["net_rail0_plane0", "net_rail0_plane1"]
      cidrPoolRef: rail-0
      mtu: 9216
    - name: rail1
      nicSelector:
        pfNames: ["net_rail1_plane0", "net_rail1_plane1"]
      cidrPoolRef: rail-1
      mtu: 9216

kubectl apply -f spectrumxrailpoolconfig.yaml

Software Multiplane

With swplb, railTopology has one entry per (rail, plane) combination. Each entry selects a single PF (netdev name from NicInterfaceNameTemplate) and references the matching per-plane CIDRPool.

apiVersion: spectrumx.nvidia.com/v1alpha2
kind: SpectrumXRailPoolConfig
metadata:
  name: rails
  namespace: nvidia-network-operator
spec:
  draEnabled: false
  networkNamespace: default
  numVfs: 1
  railTopology:
    - name: rail0p0
      nicSelector:
        pfNames: ["net_rail0_plane0"]
      cidrPoolRef: rail-0-plane-0
      mtu: 9216
    - name: rail0p1
      nicSelector:
        pfNames: ["net_rail0_plane1"]
      cidrPoolRef: rail-0-plane-1
      mtu: 9216
    - name: rail1p0
      nicSelector:
        pfNames: ["net_rail1_plane0"]
      cidrPoolRef: rail-1-plane-0
      mtu: 9216
    - name: rail1p1
      nicSelector:
        pfNames: ["net_rail1_plane1"]
      cidrPoolRef: rail-1-plane-1
      mtu: 9216

kubectl apply -f spectrumxrailpoolconfig.yaml

Step 9: Deploy a Test Pod

Single-Plane

Request one VF per rail. The network annotation references the rails created by SpectrumXRailPoolConfig.

apiVersion: v1
kind: Pod
metadata:
  name: spectrum-x-test
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: rail0,rail1
spec:
  containers:
    - name: spectrum-x-test
      image: nvcr.io/nvidia/doca/doca:3.3.0-full-rt-host
      command: ["/bin/bash", "-c", "sleep infinity"]
      securityContext:
        capabilities:
          add: ["IPC_LOCK", "NET_RAW"]
      resources:
        requests:
          nvidia.com/rail0: "1"
          nvidia.com/rail1: "1"
        limits:
          nvidia.com/rail0: "1"
          nvidia.com/rail1: "1"

kubectl apply -f pod.yaml
kubectl -n default exec -it spectrum-x-test -- rdma link

Hardware Multiplane

Request one VF per rail. The network annotation references the rails created by SpectrumXRailPoolConfig.

apiVersion: v1
kind: Pod
metadata:
  name: spectrum-x-test
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: rail0,rail1
spec:
  containers:
    - name: spectrum-x-test
      image: nvcr.io/nvidia/doca/doca:3.3.0-full-rt-host
      command: ["/bin/bash", "-c", "sleep infinity"]
      securityContext:
        capabilities:
          add: ["IPC_LOCK", "NET_RAW"]
      resources:
        requests:
          nvidia.com/rail0: "1"
          nvidia.com/rail1: "1"
        limits:
          nvidia.com/rail0: "1"
          nvidia.com/rail1: "1"

kubectl apply -f pod.yaml
kubectl -n default exec -it spectrum-x-test -- rdma link

Software Multiplane

Request one VF per (rail, plane) combination. The network annotation lists each rail-plane and the resource request matches the SR-IOV device-plugin resource the Spectrum-X Operator creates for each rail-plane.

apiVersion: v1
kind: Pod
metadata:
  name: spectrum-x-test
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: rail0p0,rail0p1,rail1p0,rail1p1
spec:
  containers:
    - name: spectrum-x-test
      image: nvcr.io/nvidia/doca/doca:3.3.0-full-rt-host
      command: ["/bin/bash", "-c", "sleep infinity"]
      securityContext:
        capabilities:
          add: ["IPC_LOCK", "NET_RAW"]
      resources:
        requests:
          nvidia.com/rail0p0: "1"
          nvidia.com/rail0p1: "1"
          nvidia.com/rail1p0: "1"
          nvidia.com/rail1p1: "1"
        limits:
          nvidia.com/rail0p0: "1"
          nvidia.com/rail0p1: "1"
          nvidia.com/rail1p0: "1"
          nvidia.com/rail1p1: "1"

kubectl apply -f pod.yaml
kubectl -n default exec -it spectrum-x-test -- rdma link