Verify and Troubleshoot Spectrum-X

On this page

Step 1: Check the profile ConfigMap
Step 2: Check the NIC configuration applied
Step 3: Check the rails are configured
Step 4: Check plane health (Hardware Multiplane)
Step 5: Check the workload sees its rails
Common failure modes
Further reading

Work through the checks in the order below: each layer depends on the one before it, so the first failing check is where to start debugging. The examples assume the Network Operator is installed in the nvidia-network-operator namespace.

Step 1: Check the profile ConfigMap

Confirm the profile ConfigMap carries the required label:

kubectl get configmap -A \
  -l network.nvidia.com/operator.nic-configuration.spectrum-x-profile

The label value is ignored. Exactly one current-format profile may be selected cluster-wide. Check that its contents match the published file, including data.format: dospcx-data.tar.gz/v1 and a non-empty binaryData.dospcx-data.tar.gz. A legacy data.profile body from 26.7.x is not supported by the current release.

spectrumXOptimized.version is the architecture version, not the ConfigMap name. Every enabled template must also specify platformType. Confirm the profile supports the requested version, platform, NIC, and mode. For configuration errors, inspect the NIC configuration daemon logs:

kubectl -n nvidia-network-operator logs \
  -l app.kubernetes.io/name=nic-configuration-daemon \
  --prefix --tail=200

Step 2: Check the NIC configuration applied

Each SuperNIC appears as a NicDevice. Check the conditions:

kubectl -n nvidia-network-operator get nicdevices
kubectl -n nvidia-network-operator get nicdevice <name> -o jsonpath='{.status.conditions}' | jq

Reasons you may see on the ConfigUpdateInProgress condition:

Reason	Meaning
`UpdateSuccessful`	Configuration applied. This is the target state.
`UpdateStarted`, `PendingReboot`, `PendingNodeMaintenance`	In progress. `PendingReboot` and `PendingNodeMaintenance` wait for a maintenance window — check the Maintenance Operator if they persist.
`PendingFirmwareUpdate`	Waiting on a firmware update, either on this device or on another device on the same node.
`PartiallyApplied`	Some parameters were skipped as unsupported on this device. The message lists them: configuration partially applied; the following spec params are not supported on this device and were skipped: …
`IncorrectSpec`, `SpecValidationFailed`	The template is invalid for this device — check `nicType`, `linkType`, `numVfs`, and the multiplane mode against Spectrum-X NIC Configuration.
`NonVolatileConfigUpdateFailed`, `RuntimeConfigUpdateFailed`, `FirmwareError`	The apply itself failed. Get the daemon Pod on the affected node with `kubectl -n nvidia-network-operator get pods -l app.kubernetes.io/name=nic-configuration-daemon --field-selector spec.nodeName=<node>`, then inspect that Pod’s logs.

Runtime validation reports all network interfaces have NO-CARRIER only when every checked interface on a NIC has no carrier. One working plane allows validation to proceed; this does not establish that all planes are healthy. Inspect the fabric and xPlane state for partial failures.

Step 3: Check the rails are configured

SpectrumXRailPoolConfig reports an aggregate status and a per-node breakdown:

kubectl -n nvidia-network-operator get spectrumxrailpoolconfig <name> \
  -o jsonpath='{.status.syncStatus}'

kubectl -n nvidia-network-operator get spectrumxrailpoolconfig <name> \
  -o jsonpath='{.status.nodeStates}' | jq

syncStatus is one of Unknown, InProgress, Failed, or Succeeded. When it is not Succeeded, nodeStates[] identifies which nodes are affected and carries a message for each.

Confirm the operator generated one OVSNetwork per railTopology entry. Traditional layouts have one entry per rail, or per rail-plane in swplb. VR HWPLB has eight entries for its four dual-NIC logical rails, one for each NIC endpoint. Hardware Multiplane additionally creates an SR-IOV policy with an -unused suffix for the remaining PFs:

kubectl -n nvidia-network-operator get ovsnetwork
kubectl -n nvidia-network-operator get sriovnetworknodepolicy

Step 4: Check plane health (Hardware Multiplane)

With hwplb the planes are hidden from the workload: each NIC endpoint presents one interface whether every plane is healthy or only one is. VR has two such endpoints per logical rail. A degraded plane is therefore invisible from inside the Pod. Inspect DOCA xPlane telemetry on each affected worker.

The xPlane DaemonSet runs as xplane-ds:

kubectl -n nvidia-network-operator get ds xplane-ds
kubectl -n nvidia-network-operator get pods -l app=xplane -o wide
kubectl -n nvidia-network-operator logs -l app=xplane

Select the xPlane Pod on the affected node, then query plane state with the bundled client:

kubectl -n nvidia-network-operator exec <xplane-pod> -- \
  doca-xplane-client get-status

kubectl -n nvidia-network-operator exec <xplane-pod> -- \
  doca-xplane-client get-planes-summary

get-status reports an overallState and counts local versus remote failures. When a plane is DEGRADED, narrow it down:

# Local failures — link down on this host
kubectl -n nvidia-network-operator exec <xplane-pod> -- \
  doca-xplane-client get-plane-failures-local --plane_id 0

# Remote failures — route withdrawn by the fabric
kubectl -n nvidia-network-operator exec <xplane-pod> -- \
  doca-xplane-client get-plane-failures-remote --plane_id 0

# Traffic counters, including diverted and dropped traffic
kubectl -n nvidia-network-operator exec <xplane-pod> -- \
  doca-xplane-client get-plane-traffic-summary --plane_id 0

Note

local failures are host-side (a down link on this node); remote failures are learned from fabric route-availability updates. Compare diverted and dropped counters to assess failover. For the complete command reference, see the DOCA 3.5 XPlane Service Guide.

Step 5: Check the workload sees its rails

Deploy the test Pod from your walkthrough and confirm one RDMA device per rail:

kubectl exec -it spectrum-x-test -- rdma link

On IPv6 rails the Spectrum-X Operator places each rail in its own VRF, named after the rail topology entry, so routes are not in the main table:

kubectl exec -it spectrum-x-test -- ip -6 route show vrf rail0

Important

RDMA-CM is not VRF-aware. Traffic tests that use RDMA-CM (for example ib_write_bw -R) must be run inside the VRF context:

ip vrf exec rail0 ib_write_bw -R ...

Workloads that need ip vrf exec require the IPC_LOCK, NET_ADMIN, DAC_OVERRIDE, and BPF capabilities.

Common failure modes

Symptom	Cause and fix
Template rejected for missing `platformType` or invalid `uniplane`	A 26.7-era template is being used with the current CRD. Add the supported platform identifier, set `version` to the architecture version, and select `none`, `swplb`, or `hwplb`.
Multiple profile ConfigMaps selected	Keep exactly one current-format profile selected cluster-wide. Remove the extra ConfigMap or its selector label so configuration can resume.
Invalid profile ConfigMap contents	Reapply the complete published profile, preserving `data.format` and `binaryData`. Renaming a legacy `data.profile` ConfigMap does not convert its contents to the current format.
Spectrum-X configuration fails for the selected platform	Check the profile supports the requested platform, version, NIC, and mode. Selected devices on a node must use consistent API field values. Check `NicDevice` conditions and the daemon logs.
`RuntimeConfigUpdateFailed` with `all network interfaces have NO-CARRIER`	Restore carrier on at least one NIC port and check cabling, switch ports, and the physical plane configuration. Runtime validation retries when connectivity returns.
Pod rail goes down when all its PFs lose carrier	Expected with `vfStateFollow: true` (the default). Active VF representors are restored when a PF recovers. Use xPlane telemetry to diagnose individual failed planes on Hardware Multiplane.
Configuration flaps between two states	The SR-IOV Network Operator’s `mellanox` plugin is still enabled and is configuring the same NIC state as the NIC Configuration Operator. Set `disablePlugins: [mellanox]` — see Architecture and Components.
No OVS bridge, VFs never attach	The `manageSoftwareBridges` feature gate is not set on the SR-IOV Network Operator, so it cannot create the per-PF bridge.
RDMA devices from other Pods are visible	RDMA subsystem namespace awareness is not enabled. Set `options ib_core netns_mode=0` and reboot the node.
OVS starts with uplinks missing after a reboot	Expected on Hardware Multiplane: the operator installs an `ExecStartPre` hook that removes xPlane uplinks whose PF has not yet entered switchdev mode. Check `/var/log/xplane-ovs-pre.log` — the ports are re-added once the PFs are ready.