SR-IOV InfiniBand

On this page

Use Case

Virtualized InfiniBand with hardware acceleration. Each pod receives a dedicated VF with isolated IB partitions and direct RDMA access. Common for large-scale HPC clusters and AI/ML training on InfiniBand fabrics.

Prerequisites

  • Cluster access and operator ownership configured (see Installation).

  • l8k binary on your PATH.

  • Mellanox NICs in InfiniBand mode on each node.

Discover

l8k discover --kubeconfig ~/.kube/config \
    --save-cluster-config ./cluster-config.yaml

Generate

l8k generate --user-config ./cluster-config.yaml \
    --fabric infiniband --deployment-type sriov --multirail \
    --save-deployment-files ./deployments

Deploy

l8k deploy --deployment-files ./deployments

Verify

Validate the generated bundle, then inspect the relevant resources:

l8k validate --deployment-files ./deployments --wait 10m

kubectl -n nvidia-network-operator get pods
kubectl get sriovnetworknodepolicy
kubectl get sriovibnetwork

See Also