IP over InfiniBand (RDMA Shared)

On this page

Use Case

InfiniBand fabric with IPoIB and a shared RDMA device plugin. Multiple pods on a node can share access to the same RDMA hardware. Suited for parallel I/O workloads, distributed storage, and data analytics where dedicated VFs are not required.

Prerequisites

  • Cluster access and operator ownership configured (see Installation).

  • l8k binary on your PATH.

  • Mellanox NICs in InfiniBand mode on each node.

Discover

l8k discover --kubeconfig ~/.kube/config \
    --save-cluster-config ./cluster-config.yaml

Generate

l8k generate --user-config ./cluster-config.yaml \
    --fabric infiniband --deployment-type rdma_shared --multirail \
    --save-deployment-files ./deployments

Deploy

l8k deploy --deployment-files ./deployments

Verify

Validate the generated bundle, then inspect the relevant resources:

l8k validate --deployment-files ./deployments --wait 10m

kubectl -n nvidia-network-operator get pods
kubectl get ipoibnetwork

See Also