NVIDIA Spectrum-X NIC Configuration

On this page

NVIDIA NIC Configuration Operator in Network Operator 26.10 applies Spectrum-X settings from a profile ConfigMap. The spectrumXOptimized.version field selects the architecture version, and platformType selects the hardware platform. Neither field names the ConfigMap.

RA3.1 for Vera Rubin (VR) and RA2.4 for all other platforms are released together. Select version: RA3.1 with platformType: vr for VR; select version: RA2.4 with the corresponding platform type otherwise. Both use the doSPCX ConfigMap format below.

The examples on this page target the component versions shipped with Network Operator v26.10.0-rc.7.

For the Network Operator Spectrum-X RA support matrix and validated hardware, operating system, and Kubernetes combinations, see Platform Support.

Spectrum-X configuration applies to ConnectX-7 NIC, ConnectX-8 SuperNIC, ConnectX-9 SuperNIC, and BlueField-3 SuperNIC; Hardware Plane Load Balancing (hwplb) additionally requires ConnectX-8 SuperNIC or ConnectX-9 SuperNIC. Device IDs and the modes each NIC accepts are listed under NIC type constraints.

Note

ConnectX-9 SuperNIC support for Spectrum-X is Tech Preview. The NIC Configuration Operator API accepts ConnectX-9 SuperNIC and applies Spectrum-X settings to it. API acceptance does not establish a validated hardware/software combination. Select RA3.1 for VR. Use it for evaluation with a profile supplied for the requested platform and NIC combination; see Platform Support.

ConnectX-9 SuperNIC remains fully supported by Network Operator for general RoCE and InfiniBand workloads — see Platform Support.

Install and Configure the NIC Configuration Operator

To install the operator and for more information on the CRDs, see NIC Firmware Configuration and Configuration Details.

Apply the Spectrum-X Profile ConfigMap

Obtain the RA2.4 or RA3.1 doSPCX profile ConfigMap published for your deployment, or contact your NVIDIA CPM. Apply the complete ConfigMap as supplied, including its labels, annotations, and contents.

The following shows the required API format. Use the published ConfigMap rather than applying this placeholder example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: spectrum-x-profile
  namespace: nvidia-network-operator
  labels:
    network.nvidia.com/operator.nic-configuration.spectrum-x-profile: ""
data:
  format: dospcx-data.tar.gz/v1
binaryData:
  dospcx-data.tar.gz: <base64-encoded-archive>

kubectl apply -f spectrum-x-profile-configmap.yaml

Note

Network Operator 26.10 does not accept the legacy data.profile format used with RA2.3 on 26.7.x. Renaming a legacy ConfigMap does not convert it. Keep exactly one current-format profile selected cluster-wide.

Admission checks the template fields, but does not establish that the profile supports the architecture, platform, and NIC combination. Check NicDevice conditions as described in Verify and Troubleshoot.

Firmware Upgrade

If the firmware on the devices needs to be updated, extend the NicFirmwareSource CR with fields for ConnectX and BlueField firmware. Make sure to use the correct firmware for your devices.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareSource
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  # a list of firmware binaries zip archives from the Mellanox website, can point to any URL accessible from the cluster
  binUrlSources:
    - https://www.mellanox.com/downloads/firmware/fw-ConnectX8-rel-40_46_3048-900-9X85E-00NX-MC0_Ax-UEFI-14.39.14-FlexBoot-3.8.100.signed.bin.zip
  # a URL to the BlueField Bundle (BFB) file, can point to any URL accessible from the cluster
  bfbUrlSource:
    - https://example.com/bf-fwbundle-3.1.0-77_25.07-prod.bfb

Configure and apply the NicFirmwareTemplate CR:

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nicSelector:
    nicType: "a2dc" # BlueField-3 SuperNIC. Can also be "1021" for ConnectX-7, "1023" for ConnectX-8, or "1025" for ConnectX-9
    # partNumbers:
    #   - "MCX713106AEHEA_QP1"
  template:
    nicFirmwareSourceRef: spectrum-x-configuration
    updatePolicy: Update

Enable SPC-X Optimizations for Devices

Platform configuration values

PLATFORM_TYPE maps to spec.template.spectrumXOptimized.platformType, NIC_TYPE to spec.nicSelector.nicType, MP_MODE to spec.template.spectrumXOptimized.multiplaneMode, and NUM_PLANES to spec.template.spectrumXOptimized.numberOfPlanes.

PF and VF totals describe the east-west NICs on one worker node in the non-VR RA2.4 platform layouts below, excluding north-south NICs. For HGX H100 use h100; for HGX H200 use h200. Select the row matching your NIC and physical plane configuration.

Platform configuration by worker node
Platform type
(PLATFORM_TYPE)
NIC type ID
(NIC_TYPE)
Multi-plane mode
(MP_MODE)
Number of planes
(NUM_PLANES)

Total PFs per platform

Total VFs per platform

h100 / h200

a2dc (BlueField-3)

none

1

8

8

b200

a2dc (BlueField-3)

none

1

8

8

b200

1021 (ConnectX-7)

none

1

8

8

gb200

1021 (ConnectX-7)

none

1

4

4

rtx (RTX PRO 6000)

1023 (ConnectX-8)

none

1

8 (2 x 4)

8

b300

1023 (ConnectX-8)

hwplb

2

16 (2 x 8)

8

b300

1023 (ConnectX-8)

hwplb

4

32 (4 x 8)

8

b300

1023 (ConnectX-8)

swplb

2

16 (2 x 8)

16

b300

1023 (ConnectX-8)

swplb

4

32 (4 x 8)

32

gb300

1023 (ConnectX-8)

hwplb

2

8 (2 x 4)

4

gb300

1023 (ConnectX-8)

hwplb

4

16 (4 x 4)

4

gb300

1023 (ConnectX-8)

swplb

2

8 (2 x 4)

8

gb300

1023 (ConnectX-8)

swplb

4

16 (4 x 4)

16

For the RA2.4 layouts in this table, hwplb exposes one VF per rail regardless of the number of planes. In swplb, each PF exposes a VF, so the VF count increases with the number of planes. The totals above are platform-wide counts, not values for template.numVfs; the NIC configuration template still uses numVfs: 1.

Single-plane configuration

Set the architecture version and the platform identifier supported by the profile. Spectrum-X optimizations require linkType: Ethernet and numVfs: 1. This example selects a GB200 single-plane deployment:

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
    nicType: "1021" # ConnectX-7 NIC on GB200
  template:
    numVfs: 1
    linkType: Ethernet
    spectrumXOptimized:
      enabled: true
      version: "RA2.4"
      platformType: "gb200"
      overlay: "none" # For an L3 EVPN overlay, use "l3"

Multiplane configuration

Multiplane modes let a NIC drive multiple data planes on the same rail. Set multiplaneMode and numberOfPlanes alongside the architecture and platform fields.

Note

It is recommended to perform a NIC configuration reset before applying or switching between multiplane configurations to ensure a clean and consistent initial state. See Reset NIC Configuration to Default for details.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-multiplane-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
    nicType: "1023" # ConnectX-8 SuperNIC on GB300
  template:
    numVfs: 1
    linkType: Ethernet
    spectrumXOptimized:
      enabled: true
      version: "RA2.4"
      platformType: "gb300"
      overlay: "none"
      multiplaneMode: "hwplb" # Hardware Plane Load Balancing
      numberOfPlanes: 2

The profile must support the requested architecture, platform, mode, and NIC. platformType: rtx permits only multiplaneMode: none. The API requires swplb or hwplb for platformType: vr. For Vera Rubin, use version: RA3.1, platformType: vr, and ConnectX-9 SuperNIC (nicType: "1025"). The eight-plane HWPLB layout groups two NICs per logical rail, exposing a separate VF endpoint for each NIC. See Launch Kit Spectrum-X Profile for the rail grouping and generation inputs.

Multiplane modes

The following modes are accepted by the NIC Configuration Operator API. The supplied profile must also support the selected platform, NIC, and plane count; this table is not a hardware qualification matrix:

Mode

Description

Supported NICs

Planes

none

Single plane mode (no multiplane). This is the default.

BlueField-3 SuperNIC, ConnectX-7 NIC, ConnectX-8 SuperNIC, ConnectX-9 SuperNIC

1

hwplb

Hardware Plane Load Balancing. Uses hardware LAG resource allocation and NIC-level plane configuration to distribute packets across planes in the NIC hardware. The planes are hidden from the workload, which sees one interface per NIC endpoint: one per rail in the non-VR layouts above, or two per logical rail on VR. The recommended mode on multiplane platforms.

ConnectX-8 SuperNIC, ConnectX-9 SuperNIC

2, 4, 8 (subject to platform and profile support)

swplb

Software Plane Load Balancing. The NIC port is split into multiple PFs, each assigned to a separate data plane; the software stack distributes packets across planes. Each plane is exposed to the workload as its own interface.

BlueField-3 SuperNIC, ConnectX-8 SuperNIC, ConnectX-9 SuperNIC

2, 4, 8 (subject to platform and profile support)

Note

For new multiplane deployments, select hwplb when supported by the selected RA2.4 or RA3.1 profile and fabric. Use swplb when each plane must be visible to the workload as a separate interface, or on NICs that do not support hwplb. This is not an API default: the multiplaneMode field itself defaults to none, so multiplane deployments must set it explicitly.

NIC type constraints

NIC Type

Device ID

Supported Multiplane Modes

ConnectX-7 NIC

1021

none

ConnectX-8 SuperNIC

1023

none, swplb, hwplb

ConnectX-9 SuperNIC (tech preview)

1025

none, swplb, hwplb

BlueField-3 SuperNIC

a2dc

none, swplb

Warning

The hwplb multiplane mode requires ConnectX-8 SuperNIC (device ID 1023) or ConnectX-9 SuperNIC (device ID 1025). Attempting to configure hwplb on BlueField-3 SuperNIC or ConnectX-7 NIC is rejected by the API validation.

Configure custom interface names

The NicInterfaceNameTemplate CRD allows you to define custom naming patterns for RDMA and network device interfaces on Spectrum-X NICs. This is useful in multiplane and multi-rail deployments where predictable interface naming is required.

The operator deploys udev rules to the host to rename network and RDMA interfaces according to the specified naming template.

The template uses the following placeholders for device name construction:

  • %nic_id%: The index of the NIC in the flattened list of NICs

  • %plane_id%: The index of the plane of the specific NIC

  • %rail_id%: The index of the rail where the given NIC belongs to

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
  name: spectrum-x-interface-names
  namespace: nvidia-network-operator
spec:
  # Number of PFs per NIC, used to calculate the number of planes per NIC
  pfsPerNic: 2
  # Template for RDMA device names. Placeholders: %nic_id%, %plane_id%, %rail_id%
  rdmaDevicePrefix: "rdma_%nic_id%_%plane_id%_%rail_id%"
  # Template for net device names. Placeholders: %nic_id%, %plane_id%, %rail_id%
  netDevicePrefix: "net_%nic_id%_%plane_id%_%rail_id%"
  # PCI address to rail mapping. First dimension is rail index, second is NIC PCI addresses in the rail
  railPciAddresses:
    - ["0000:1a:00.0", "0000:2a:00.0"]
    - ["0000:3a:00.0", "0000:4a:00.0"]

The railPciAddresses field defines the PCI address to rail mapping. The first dimension is the rail index and the second dimension lists one PCI address per NicDevice in that rail. pfsPerNic determines how many PF rules are generated for each device; do not list each PF of the same device as another NIC.

Generated udev rules

The operator generates udev rules based on the template and writes them to the host. The rules are written to two separate files.

Example generated udev rules for one PF’s net device (/etc/udev/rules.d/10-nic-net-interface-naming.rules):

# Auto-generated by nic-configuration-operator
# Do not edit manually

SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="p[0-9]*", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"
SUBSYSTEM=="net", ACTION=="add", TEST!="phys_port_name", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"

The physical-port match prevents VF and SF representors from taking the PF name. The other rules cover hosts where phys_port_name is empty or absent.

Example generated udev rules for RDMA devices (/etc/udev/rules.d/10-nic-rdma-interface-naming.rules):

# Auto-generated by nic-configuration-operator
# Do not edit manually

ACTION=="add", KERNELS=="0000:1a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_0_0"
ACTION=="add", KERNELS=="0000:1a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_1_0"
ACTION=="add", KERNELS=="0000:3a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_0_1"
ACTION=="add", KERNELS=="0000:3a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_1_1"

Note

The actual generated names depend on the netDevicePrefix / rdmaDevicePrefix you set on the NicInterfaceNameTemplate — see the Quick-Start walkthroughs for the recommended rail / plane-based naming (for example, net_rail0_plane0).

Validation rules

The following validation rules are enforced by the API:

  • Spectrum-X optimizations require numVfs: 1 and linkType: Ethernet. When networkBay is configured, omit linkType because the system profile governs it.

  • Spectrum-X optimizations can only be enabled for ConnectX-7 NIC (nicType: 1021), ConnectX-8 SuperNIC (nicType: 1023), ConnectX-9 SuperNIC (nicType: 1025), or BlueField-3 SuperNIC (nicType: a2dc).

  • When Spectrum-X optimizations are enabled, roceOptimized must not be enabled (RoCE settings are included in the Spectrum-X configuration).

  • version and platformType are required for enabled Spectrum-X configuration. Their values must be supported by the supplied profile; version is not a ConfigMap reference.

  • multiplaneMode must be none, swplb, or hwplb; numberOfPlanes must be 1, 2, 4, or 8. API acceptance of a plane count does not establish that the supplied RA2.4 or RA3.1 profile supports it; the B300/GB300 walkthroughs use two or four planes.

  • When multiplaneMode is none, numberOfPlanes must be 1.

  • When multiplaneMode is not none, numberOfPlanes must not be 1.

  • The hwplb multiplane mode can only be enabled for ConnectX-8 SuperNIC (nicType: 1023) or ConnectX-9 SuperNIC (nicType: 1025).

rawNvConfig may be combined with Spectrum-X optimizations. Values in rawNvConfig override explicit template settings, which override Spectrum-X profile settings. Parameter names must not use index-range syntax such as NAME[0..3] — list each index explicitly.

ConnectX-9 Network Bay configuration can combine networkBay.conf with Spectrum-X. Select both ASICs of the card, omit linkType, and use a system profile supported by the firmware. Spectrum-X profile settings override the Network Bay system-profile defaults; explicit template settings and rawNvConfig override both. See Configuration Details for Network Bay pairing and validation requirements.