NVIDIA Network Operator with Red Hat OpenShift

Use Launch Kit to configure NVIDIA networking for workloads on Red Hat OpenShift and verify inter-node connectivity. Install the OpenShift operators first; Launch Kit configures those installations and validates the resulting network.

Validated Profiles

The following standard Launch Kit profiles are validated for OpenShift. Choose one for your fabric and workload requirements. Pass --flavor ocp when using these profiles.

Profile	Networking use case
SR-IOV Ethernet RDMA	Dedicated virtual functions for Ethernet and RoCE workloads.
SR-IOV InfiniBand RDMA	Dedicated virtual functions for InfiniBand workloads.
Host-device RDMA	Direct access to a NIC from a workload.
MacVLAN with RDMA shared devices	Ethernet networking with shared RDMA device access.
IPoIB with RDMA shared devices	IP over InfiniBand networking with shared RDMA device access.

Consult Platform Support when selecting an OpenShift and Network Operator version combination. Record exact software versions, hardware, and test scope when validating your site. Spectrum-X has no OpenShift profile.

SR-IOV Ethernet Validation Example

A published Launch Kit validation run on two OpenShift 4.22.5 workers reported 9/9 resources ready and 6/6 connectivity tests passed: two ICMP, two RDMA ping, and two RDMA bandwidth tests. Strict validation used an 80 Gbps minimum bandwidth threshold and measured 84.7 and 83.9 Gbps. These measurements apply to that test setup; select thresholds appropriate to your hardware.

Prerequisites

• Follow NVIDIA Network Operator Deployment Guide with OpenShift to install the certified NVIDIA Network Operator and Node Feature Discovery, plus the Red Hat SR-IOV Network Operator for SR-IOV profiles.

• Install NVIDIA Maintenance Operator through OperatorHub/OLM in the namespace you will configure in Launch Kit. Follow Red Hat’s operator installation instructions. Confirm that the required operators have successful ClusterServiceVersions (CSVs) and their APIs are available before proceeding.

• Prepare at least two Ready workers with NVIDIA NICs and a configured fabric for the selected profile. The SR-IOV Ethernet example below requires SR-IOV-capable Ethernet NICs, a RoCE fabric, image access, and sufficient VF capacity. InfiniBand profiles require a subnet manager and appropriate partition membership.

• Confirm cluster permissions for discovery and deployment, and for temporary validation namespaces and security context constraints (SCCs). Validation also needs access to the network attachments and image pull secrets it uses.

• Install a pinned Launch Kit release. Set KUBECONFIG and agree on worker/NIC selection, IP ranges, MTU, routing, and maintenance limits as described in Partner-Validated Configurations.

Check the environment before making changes:

oc config current-context
oc get nodes -o wide
oc get csv -A
l8k version
l8k schema

Configure and Deploy

The procedure below uses SR-IOV Ethernet RDMA. For another validated profile, choose its fabric and deployment type from the Launch Kit profile guide during generation.

1. Prepare cluster-config.yaml using the seed configuration in the Launch Kit OpenShift walkthrough. Set flavor: ocp, the selected Network Operator release, actual operator namespaces, workload namespace, and intended worker names.

2. Discover hardware into a separate file, preserving your seed configuration:

   l8k discover --flavor ocp --kubeconfig "$KUBECONFIG" \
     --user-config ./cluster-config.yaml \
     --save-cluster-config ./discovered-cluster-config.yaml
   

   Review the discovered workers, NICs, rails, and profile. Set site-specific addresses, routing, maintenance limits, and GPUDirect validation settings before generation.

3. Generate the SR-IOV Ethernet deployment:

   l8k generate --flavor ocp \
     --user-config ./discovered-cluster-config.yaml \
     --fabric ethernet --deployment-type sriov \
     --save-deployment-files ./ocp-deployment
   

   Review the policies, namespaces, VF resources, and IP pools. Keep the complete directory, including .l8k. OpenShift uses the preinstalled operators and platform-managed Multus/CNI plugins; do not install the Kubernetes Helm bundle.

4. Preview the changes and deploy after resolving admission or configuration conflicts:

   l8k deploy --flavor ocp --kubeconfig "$KUBECONFIG" \
     --deployment-files ./ocp-deployment --dry-run
   l8k deploy --flavor ocp --kubeconfig "$KUBECONFIG" \
     --deployment-files ./ocp-deployment
   

Validate and Record Results

Follow Run and Assess Validation using --flavor ocp and --deployment-files ./ocp-deployment. Require results for both intended workers and all intended rails. OpenShift’s platform-managed Multus and CNI installer checks are reported as unverified; use platform health checks and the connectivity results to assess those prerequisites.

Launch Kit normally removes its temporary validation resources. Use --keep only when they are needed to investigate a failure. After testing, follow the OpenShift resource-removal procedure to remove the intended test deployment while preserving shared operators. l8k clean does not support OpenShift.

Use Contribute a Configuration to document your exact versions, hardware, thresholds, reports, and any additional application tests.

Getting Support

For OpenShift platform issues, use the Red Hat Customer Portal. For Network Operator issues, use Troubleshooting and the Network Operator issue tracker. Include the OpenShift and operator versions and a sanitized validation report.