[TECH PREVIEW] NVIDIA Spectrum-X NIC Configuration

On this page

Install and Configure the NIC Configuration Operator
Provision the DOCA SPC-X CC Algorithm Package
- Firmware Upgrade
Enable SPC-X Optimizations for Devices
- RA2.1 configuration with multiplane support
  - Multiplane modes
  - NIC type constraints
- Configure custom interface names
  - Generated udev rules
Validation rules

NVIDIA NIC Configuration Operator offers NVIDIA Spectrum-X-specific NIC configuration for different versions of the Reference Architecture (RA1.3, RA2.0, and RA2.1). RA2.1 introduces multiplane mode support for enhanced network performance with multiple data planes.

Note

Currently, only ConnectX-8 (device ID 1023) and BlueField-3 SuperNIC (device ID a2dc) devices are supported for Spectrum-X configuration. Hardware Packet Load Balancing (hwplb) multiplane mode is only supported on ConnectX-8.

Install and Configure the NIC Configuration Operator

To install the operator and for more information on the CRDs, see NIC Firmware Configuration and Configuration Details.

Provision the DOCA SPC-X CC Algorithm Package

Note

For Spectrum-X RA2.1 and later, the DOCA SPC-X CC algorithm package is included in the operator image and does not need to be deployed separately. For RA2.0 and earlier, the package must be deployed manually using the example below.

To enable the DOCA SPC-X CC algorithm on NIC devices, the DOCA SPC-X CC .deb package for ubuntu 22.04 is required. This configuration step will be removed in the future, once the DOCA SPC-X CC algorithm is publicly available. To access the package, contact your NVIDIA CPM. The package should be available in the cluster and then its URL should be provided in the packageUrlSource field of the SpectrumXOperator CR.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareSource
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  # should point to the URL of the DOCA SPC-X CC .deb package for Ubuntu 22.04
  docaSpcXCCUrlSource: "https://example.com/doca-spcx-cc_3.1.0105-1_amd64.deb"

Firmware Upgrade

If the firmware on the devices needs to be updated, extend the NicFirmwareSource CR with fields for ConnectX and BlueField firmware. Make sure to use the correct firmware for your devices.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareSource
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  # should point to the URL of the DOCA SPC-X CC .deb package for Ubuntu 22.04
  docaSpcXCCUrlSource: "https://example.com/doca-spcx-cc_3.1.0105-1_amd64.deb"
  # a list of firmware binaries zip archives from the Mellanox website, can point to any URL accessible from the cluster
  binUrlSources:
    - https://www.mellanox.com/downloads/firmware/fw-ConnectX8-rel-40_46_3048-900-9X85E-00NX-MC0_Ax-UEFI-14.39.14-FlexBoot-3.8.100.signed.bin.zip
  # a URL to the BlueField Bundle (BFB) file, can point to any URL accessible from the cluster
  bfbUrlSource:
    - https://example.com/bf-fwbundle-3.1.0-77_25.07-prod.bfb

Configure and apply the NicFirmwareTemplate CR:

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nicSelector:
    nicType: "a2dc" # BlueField-3 SuperNIC, Can also be "1023" for ConnectX-8 or "1025" for ConnectX-9
    # partNumbers:
    #   - "MCX713106AEHEA_QP1"
  template:
    nicFirmwareSourceRef: spectrum-x-configuration
    updatePolicy: Update

Enable SPC-X Optimizations for Devices

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
      feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
      nicType: a2dc # BlueField-3 SuperNIC, Can also be "1023" for ConnectX-8 or "1025" for ConnectX-9
      # partNumbers:
      #   - "MCX713106AEHEA_QP1"
  template:
      numVfs: 1
      linkType: Ethernet
      spectrumXOptimized:
          enabled: true
          version: "RA2.0" # For Reference Architecture v1.3, use "RA1.3" value for this field.
          overlay: "none" # For L3 overlay, use "l3" value for this field.

RA2.1 configuration with multiplane support

Reference Architecture 2.1 introduces multiplane mode support, allowing NICs to be configured with multiple data planes for enhanced network performance.

Note

It is recommended to perform a NIC configuration reset before applying or switching between multiplane configurations to ensure a clean and consistent initial state. See Reset NIC Configuration to Default for details.

To enable multiplane support, set spectrumXOptimized.version to RA2.1 and configure the multiplaneMode and numberOfPlanes fields.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-multiplane-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
      feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
      nicType: "1023" # ConnectX-8. Use "1025" for ConnectX-9, or "a2dc" for BlueField-3 SuperNIC (hwplb not supported on BF3)
      # partNumbers:
      #   - "MCX713106AEHEA_QP1"
  template:
      numVfs: 1
      linkType: Ethernet
      spectrumXOptimized:
          enabled: true
          version: "RA2.1"
          overlay: "none"
          multiplaneMode: "hwplb" # Hardware Packet Load Balancing, ConnectX-8 only
          numberOfPlanes: 4

Multiplane modes

The following multiplane modes are available with RA2.1:

Mode	Description	Supported NICs	Planes
`none`	Single plane mode (no multiplane). This is the default.	ConnectX-8, BF3 SuperNIC	1
`swplb`	Software Packet Load Balancing. The NIC port is split into multiple PFs, each assigned to a separate data plane.	ConnectX-8, BF3 SuperNIC	2, 4
`hwplb`	Hardware Packet Load Balancing. Uses hardware LAG resource allocation and NIC-level plane configuration for load balancing across planes.	ConnectX-8 only	2, 4
`uniplane`	Uniplane mode. Each port is configured as a separate plane without plane-level load balancing.	ConnectX-8, BF3 SuperNIC	2

Note

Multiplane modes (swplb, hwplb, uniplane) are only supported with RA2.1. For RA1.3 and RA2.0, multiplaneMode must be none and numberOfPlanes must be 1.

NIC type constraints

NIC Type	Device ID	Supported Multiplane Modes
ConnectX-8	`1023`	`none`, `swplb`, `hwplb`, `uniplane`
BlueField-3 SuperNIC	`a2dc`	`none`, `swplb`, `uniplane`

Warning

The hwplb multiplane mode is only supported on ConnectX-8 (device ID 1023). Attempting to configure hwplb on a BlueField-3 SuperNIC will be rejected by the API validation.

Configure custom interface names

The NicInterfaceNameTemplate CRD allows you to define custom naming patterns for RDMA and network device interfaces on Spectrum-X NICs. This is useful in multiplane and multi-rail deployments where predictable interface naming is required.

The operator deploys udev rules to the host to rename network and RDMA interfaces according to the specified naming template.

The template uses the following placeholders for device name construction:

%nic_id%: The index of the NIC in the flattened list of NICs
%plane_id%: The index of the plane of the specific NIC
%rail_id%: The index of the rail where the given NIC belongs to

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
  name: spectrum-x-interface-names
  namespace: nvidia-network-operator
spec:
  # Number of PFs per NIC, used to calculate the number of planes per NIC
  pfsPerNic: 2
  # Template for RDMA device names. Placeholders: %nic_id%, %plane_id%, %rail_id%
  rdmaDevicePrefix: "rdma_%nic_id%_%plane_id%_%rail_id%"
  # Template for net device names. Placeholders: %nic_id%, %plane_id%, %rail_id%
  netDevicePrefix: "net_%nic_id%_%plane_id%_%rail_id%"
  # PCI address to rail mapping. First dimension is rail index, second is NIC PCI addresses in the rail
  railPciAddresses:
    - ["0000:1a:00.0", "0000:2a:00.0"]
    - ["0000:3a:00.0", "0000:4a:00.0"]

The railPciAddresses field defines the PCI address to rail mapping. The first dimension is the rail index and the second dimension is the list of PCI addresses of the NICs in that rail.

Generated udev rules

The operator generates udev rules based on the template and writes them to the host. The rules are written to two separate files.

Example generated udev rules for net devices (/etc/udev/rules.d/10-nic-net-interface-naming.rules):

# Auto-generated by nic-configuration-operator
# Do not edit manually

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:1a:00.1", NAME="net_0_1_0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:3a:00.0", NAME="net_1_0_1"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:3a:00.1", NAME="net_1_1_1"

Example generated udev rules for RDMA devices (/etc/udev/rules.d/10-nic-rdma-interface-naming.rules):

# Auto-generated by nic-configuration-operator
# Do not edit manually

ACTION=="add", KERNELS=="0000:1a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_0_0"
ACTION=="add", KERNELS=="0000:1a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_1_0"
ACTION=="add", KERNELS=="0000:3a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_0_1"
ACTION=="add", KERNELS=="0000:3a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_1_1"

Validation rules

The following validation rules are enforced by the API:

Spectrum-X optimizations can only be enabled when linkType is Ethernet and numVfs is 1.
Spectrum-X optimizations can only be enabled for ConnectX-8 (nicType: 1023) or BlueField-3 SuperNIC (nicType: a2dc).
When Spectrum-X optimizations are enabled, roceOptimized must not be enabled (RoCE settings are included in the Spectrum-X configuration).
When Spectrum-X optimizations are enabled, rawNvConfig must be empty.
When multiplaneMode is none, numberOfPlanes must be 1.
When multiplaneMode is not none, numberOfPlanes must not be 1.
When version is RA1.3 or RA2.0, multiplaneMode must be none and numberOfPlanes must be 1.
The hwplb multiplane mode can only be enabled for ConnectX-8 (nicType: 1023).