Volcengine Kubernetes Engine (VKE) can use Remote Direct Memory Access (RDMA) resources through add-ons to eliminate bottlenecks of computing tasks brought by traditional network communication. This topic introduces how a VKE cluster uses RDMA resources provided by RDMA over Converged Ethernet (RoCE) high-performance compute (HPC) clusters.

Background information

RDMA overview

RDMA is a high-performance network protocol that reduces CPU usage and memory bandwidth bottlenecks, as well as improves bandwidth utilization. It comes with the following advantages:

Zero Copy: Applications can transfer data without involving the network software stack, allowing data to be directly sent to the buffer or directly received from the buffer without being copied to the network layer.

Kernel Bypass: RDMA provides a proprietary Verbs interface, allowing applications to transfer data directly from user space without switching context between kernel and user modes.

CPU Offload: Applications can access memory on remote hosts without consuming any CPU there. Remote memory will be read without any help from remote processes (or CPUs), and the content read from them will not be written into the remote CPU caches.

The High Performance Compute (HPC) cluster is a cluster computing environment used to handle large-scale computing tasks, usually composed of a large number of compute nodes interconnected by an RDMA high-speed network. It supports both RoCE and InfiniBand.

An HPC cluster can provide powerful computing capability, large memory, high-speed storage, and networks to meet the demands of compute-intensive, data-intensive, or combined workloads. In an HPC cluster, computing tasks are assigned to each compute node for parallel processing, fully utilizing the computing power of multiple nodes to accelerate computation and improve efficiency. For details, see HPC cluster overview.

RDMA add-on mode

In the RoCE network mode, the RDMA add-on supports the following modes. The following table describes different modes and their usage restrictions.

RDMA add-on mode	Mode description	Usage limits
shared	Create a shared virtual network interface for pods in the cluster by virtualizing the RDMA physical network interface.	Each RDMA physical network interface can only provide one shared virtual network interface. The RDMA NIC mask needs to be less than 30, such as /29 and /28.
shared-multi	Create multiple shared virtual network interfaces for pods as needed in the cluster by virtualizing the RDMA physical network interface.	We recommend that you do not use this mode with GPUs. The RDMA NIC mask needs to be less than 30, such as /29 and /28.
shared-host	All RDMA NICs in the host are plugged into containers as IPVLAN sub-interfaces and provided to pods in the container cluster. Description [Beta - apply for trial]: This feature is currently in beta. To use it, submit an application.	Katalyst topology scheduling is not supported. The RDMA NIC mask needs to be less than 30, such as /29 and /28. To use the shared-host mode, the add-on version needs to meet the following requirements: Upgrade the rdma-device-plugin add-on to v3.5.1-vke.12 or later version. Upgrade the vpc-cni add-on to v1.8.9 or later version.
exclusive	[Not recommended] RDMA physical NICs are directly assigned to pods in the container cluster. This method applies to scenarios where physical devices cannot be virtualized or partitioned.	This mode cannot be used in the vePFS scenario. Performance degradation occurs when some models use this mode. We do not recommend that you use this mode if no forcible dependency exists, such as the scenario where a model uses the BF3 mini NIC. New models introduced after 2026 are not guaranteed to support this mode.

Usage instructions

The following document introduces the officially proven practices. To achieve the expected results and comply with the Volcengine Kubernetes Engine (VKE) Limits, you need to strictly follow the procedures (or operate on the recommended resources) in this document. If you want to use another solution, consult your account manager.

Usage limits

A pod cannot use mGPU and RDMA add-ons simultaneously.

RDMA devices can be used only in clusters with the VPC-CNI network model, and the vpc-cni add-on must be upgraded to v1.7.1 or later version.

The Kubernetes version of the cluster must meet the following requirements: If the cluster version is outdated, upgrade it first. For details, see Upgrading clusters.

Other later Kubernetes versions released by VKE

Kubernetes v1.24: v1.24.15-vke.12 or later

Kubernetes v1.20: v1.20.15-vke.18 or later

For the methods and the restrictions on using RDMA topology-aware scheduling, see RDMA topology-aware scheduling.

If a NCCL network error occurs in an environment running RDMA, please refer to FAQ for solutions.

Procedure: Create resources (recommended)

We recommend that you use RDMA resources when you create a VKE cluster and node pool. The detailed steps are as follows.

Step 1: Create an HPC cluster

Log in to the HPC cluster console.

Click Create HPC cluster and finish configurations based on the parameter descriptions in the table below.

Parameter	Description
Region and AZ	Intranets within different regions are completely isolated. It is recommended to select a region as close as possible to where your business is located to reduce network latency and improve access speed. AZ (Availability Zones) refers to areas isolated from each other in terms of electricity and network. In this case, resources such as node subnets, instance types, and HPC clusters are required to be in a single availability zone. For example: Availability zone C.
Name	Enter a name for HPC cluster.
Cluster properties	Select a property for HPC cluster. Each property has its own limits. RoCE is selected in this scenario. RoCE (RDMA over Converged Ethernet) is a network communication protocol designed based on Ethernet technology to realize Remote Direct Memory Access. It features high performance and high cost-efficiency, suitable for data exchange among instances in an HPC cluster. InfiniBand is a network communication protocol designed based on dedicated network hardware to implement Remote Direct Memory Access, featuring high throughput and low latency, and suitable for data exchange among instances in an HPC cluster. Cluster property needs to be selected according to the instances. tip HPC clusters of infiniBand can only work with ecs.hpcpni3.45xlarge, which is one of the High Performance Computing Optimized instances. Apart from that one, all the remaining instances can be candidates for RoCE-powered HPC clusters. InfiniBand clusters can only add or expand infiniBand-powered instances. Similarly, RoCE clusters apply the same logic.
Description	Provide descriptions or comments for the HPC cluster.

Step 2: Create a cluster and prepare resources

Log in to the VKE console.

In the left navigation pane, click Cluster. Click Create cluster and specify the required parameters.

The table below contains instructions as well as descriptions of key parameters in this step. Please follow the instructions in the table and configure the rest as needed.

Parameter	Description
Network model	Select a Container Network Interface (CNI) for the cluster. In this case, VPC-CNI is required. Flannel: an independent underlay network model based on which a high-performance network for clusters can be built along with the global routing capabilities of VPCs. VPC-CNI: an underlay network model built based on Elastic Network Interfaces (ENI) on VPC, providing high network performance.

On the Node pool configuration (optional) page, you need to finish key configurations, including settings of a new node pool, node labels, and Kubelet custom parameters. Click Next: Addon configuration when you finish all necessary settings on this page.

The table below contains instructions as well as descriptions of key parameters in this step. Please follow the instructions in the table and configure the rest as needed.

Parameter	Description
Node pool management	Select Create now.
ECS node source	Select Create node in this case.
Availability zone	Availability Zones (AZs) are physically isolated infrastructure units within a single region, featuring independent power and networking systems. AZs communicate with each other over internal networks. Different regions have their own availability zones. See Regions and Availability Zones for details. In this case, resources such as node subnets, instance types, and HPC clusters are required to be in a single availability zone. For example: Availability zone C.
Node subnet	Select subnets available in the list for the nodes.
Instance type	Instance type defines resources such as CPU, memory, GPU, RDMA of a node. In this case, instance types enabled with RDMA device are required, e.g., High Performance Computing Optimized instances.
High-performance computing cluster	A cluster computing environment used to handle large-scale computing tasks, typically consisting of a large number of worker nodes interconnected through a high-speed RDMA network to process complex tasks collaboratively. In this scenario, the RoCE HPC cluster created in Step 1 is selected.
Node labels	Labels in VKE function identically to Kubernetes labels. Labels can be used to define attributes for nodes, facilitating node operations and management. A label can be added in batches to all the nodes in the node pool. In this scenario, manually add the label vke.node.rdma.mode: {value} to configure the RDMA mode for nodes or the node pool. The following modes are available: exclusive, shared, shared-multi, and shared-host. warning Except for ecs.hpcg1ve.21xlarge and North-China ecs.hpcpni2.28xlarge, we recommend that you select the shared mode for instance types. You can select or deselect Enable auto renew for node labels and taints to determine whether to synchronously add new labels to existing nodes in the pool. Caution! Disabling this option might make the cluster unable to use existing RDMA nodes in the pool.
Kubelet configs	For businesses that are highly sensitive to latency, Kubelet custom parameters can be used to create a NUMA affinity policy. The resources allocated for a container or pod can be mapped to one or more NUMA nodes to improve business performance. If the RDMA mode is exclusive, shared, or shared-multi (1:N shared), you need to add the custom parameter topologyManagerPolicy: best-effort. This ensures that GPU and RDMA can be allocated to the same NUMA node. When the RDMA mode is shared-host, no additional configuration is required.

In the Addon configuration step, select the rdma-device-plugin add-on and configure other add-ons based on your business requirements. Click Next: Confirm Configuration to go to the next page.

After confirming that the cluster information and billing details are correct, read the Volcengine Kubernetes Engine Specific Terms of Service, select the checkbox to accept them, and then click Confirm to start creating the cluster.

Step 3: Use RDMA Resources

After the cluster is created, RDMA resources can be used when workloads are being created. In this scenario, a Deployment is used as an example.

Configuring through the console

Configuring through YAML

Log in to the VKE console. Click the name of the target cluster to go to the cluster management page.

In the left navigation pane of the cluster management page, click Workload > Deployment to go to the Deployment management page.

Click Create Deployment. In the ResourceQuota section in the Container configuration step, specify the number of RDMA cards based on your business requirements. For the descriptions of other parameters, see Creating Deployments.

warning

When the RDMA mode is exclusive, shared, or shared-multi (1:N shared), we recommend that you set the number of RDMA cards to the total number of cards in the host or half the number to achieve optimal performance for GPU training or inference tasks. For example, when the host has four RDMA NICs, you can set the number of RDMA cards to 2 or 4. When the system has eight RDMA NICs, you can set the number of RDMA cards to 4 or 8.

You can use the shared-host mode if you do not want to be concerned about the number of RDMA NICs. You cannot configure the shared-host mode in the console. To use this mode, you can configure it by using the YAML configuration based on the following instructions.

When the RDMA mode is exclusive, shared, or shared-multi (1:N shared), execute the following YAML code to configure RDMA resources for the pod.

In the spec.containers.resources field, use vke.volcengine.com/rdma to apply for RDMA NIC for the pod.

In the pod annotation, use k8s.volcengine.com/pod-networks to specify the number of NICs. We recommend setting the number of RDMA cards to the total number of cards in the host, or half the number to achieve optimal performance for GPU training or inference tasks. For example, when the host has four RDMA NICs, you can set the number of RDMA cards to 2 or 4. When the system has eight RDMA NICs, you can set the number of RDMA cards to 4 or 8.

In most cases, you need to use the spec.containers.securityContext field to request the IPC_LOCK privilege.

apiVersion: v1
kind: Pod
metadata:
name: rdma-test
annotations:
k8s.volcengine.com/pod-networks: |
[
{
"cniConf":{
"name":"rdma"
}
}
]
spec:
containers:
- name: rdma-test
image: busybox
command:
- /bin/sh
- -c
- sleep 10000
resources:
limits:
vke.volcengine.com/rdma: 1
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- mountPath: /dev/shm
name: cache-volume
- name: nvidia-topology # Non-bare metal instance types need to mount this directory. NCCL will obtain the real topology from virtualTopology.xml.
mountPath: /var/run/nvidia-topologyd
readOnly: true
volumes:
- emptyDir:
medium: Memory
name: cache-volume
- name: nvidia-topology
hostPath:
path: /var/run/nvidia-topologyd
type: DirectoryOrCreate

The preceding sample code provides an example of how to automatically create an RDMA NIC. To create multiple RDMA NICs, copy the code block in k8s.volcengine.com/pod-networks multiple times.

{
"cniConf":{
"name":"rdma"
}
}

When the RDMA mode is shared-host, execute the following YAML code to configure RDMA resources for the pod.

In the spec.containers.resources field, use vke.volcengine.com/rdma-host to apply for the RDMA resources of the entire host for the pod. The value is 1 and cannot be changed.

In the pod annotation, use k8s.volcengine.com/pod-networks to specify the number of NICs. Differing from other modes, the shared-host mode allows you to configure only one code block. If you configure more than one code block, an error will occur.

In most cases, you need to use the spec.containers.securityContext field to request the IPC_LOCK privilege.

apiVersion: v1
kind: Pod
metadata:
name: rdma-test
annotations:
k8s.volcengine.com/pod-networks: |
[
{
"cniConf":{
"name":"rdma"
}
}
]
spec:
containers:
- name: rdma-test
image: busybox
command:
- /bin/sh
- -c
- sleep 10000
resources:
requests:
vke.volcengine.com/rdma-host: 1
limits:
vke.volcengine.com/rdma-host: 1
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- mountPath: /dev/shm
name: cache-volume
- name: nvidia-topology # Non-bare metal instance types need to mount this directory. NCCL will obtain the real topology from virtualTopology.xml.
mountPath: /var/run/nvidia-topologyd
readOnly: true
volumes:
- emptyDir:
medium: Memory
name: cache-volume
- name: nvidia-topology
hostPath:
path: /var/run/nvidia-topologyd
type: DirectoryOrCreate

When you run NCCL in the container in shared mode, you need to explicitly specify the NCCL_IB_GID_INDEX environment variable. You can execute the source nccl_env.sh command to obtain the RDMA environment variables. The content of the sample script nccl_env.sh for dynamically obtaining NCCL environment variables is as follows.

Procedure: Existing resources

If you have already created a VKE cluster and node pool, you can directly use RDMA resources in the existing VKE cluster and node pool by performing the following steps.

Step 1: Install the RDMA device add-on

Log in to the VKE console. Click the name of the target cluster to go to the cluster management page.

In the left navigation pane, click Addons. Find the rdma-device-plugin add-on on the Network tab.

Move the pointer over the add-on card and click Deploy to install the RDMA device add-on in the cluster.

Step 2: Prepare node resources

Prepare nodes that support RDMA devices, such as High Performance Computing Optimized nodes. You can create nodes or use existing nodes that meet your business requirements. Manually add the label vke.node.rdma.mode: {value} to configure the RDMA usage mode for nodes or the node pool. The following modes are available: exclusive, shared, shared-multi, and shared-host.

Step 1: Add a label to an existing node

Log in to the VKE console. Click the name of the target cluster to go to the cluster management page.

In the left navigation pane, click Nodes > Node. Find the target node and in the Actions column, click ... > Label management.

Click Add new to add the preceding label to the existing node. Then, click Confirm to save the configurations.

The following sample code shows the YAML content.

type: Node
label:
vke.node.rdma.mode: shared
status:
capacity:
vke.volcengine.com/rdma: 4

Method 2: Add node labels to the node pool

Log in to the VKE console. Click the name of the target cluster to go to the cluster management page.

In the left navigation pane, click Nodes > Node pool to create a node pool, or click Edit in the Actions column that corresponds to the target cluster. Add node labels to the node pool and then click Confirm to save the configurations.

Step 3: Modify the kubelet topology management policy of the node

The node kubelet topology management policy varies based on the RDMA mode.

If the RDMA mode is exclusive, shared, or shared-multi (1:N shared), you need to configure a node kubelet topology management policy and add the custom parameter topologyManagerPolicy: best-effort. This ensures that GPU and RDMA can be allocated to the same NUMA node.

When the RDMA mode is shared-host, you do not need to configure a node kubelet topology management policy.

Log in to the VKE console. Click the name of the target cluster to go to the cluster management page.

In the left navigation pane, click Nodes > Node pool to go to the node pool management page.

Click the name of the target node pool. On the Kubelet configs tab, click Edit in the upper-right corner to manually add parameters. Then, click Confirm to save the configurations.

The following sample code shows the YAML content.

topologyManagerPolicy: best-effort

Step 4: Use RDMA resources

After the configurations are complete, RDMA resources can be used when workloads are being created. In this scenario, a Deployment is used as an example.

Configuring through the console

Configuring through YAML

Log in to the VKE console. Click the name of the target cluster to go to the cluster management page.

In the left navigation pane of the cluster management page, click Workload > Deployment to go to the Deployment management page.

Click Create Deployment. In the ResourceQuota section in the Container configuration step, specify the number of RDMA cards based on your business requirements. For other workload configurations, see Creating Deployments.

warning

When the RDMA mode is exclusive, shared, or shared-multi (1:N shared), we recommend that you set the number of RDMA cards to the total number of cards in the host or half the number to achieve optimal performance for GPU training or inference tasks. For example, when the host has four RDMA NICs, you can set the number of RDMA cards to 2 or 4. When the system has eight RDMA NICs, you can set the number of RDMA cards to 4 or 8.

You can use the shared-host mode if you do not want to be concerned about the number of RDMA NICs. You cannot configure the shared-host mode in the console. To use this mode, you can configure it by using the YAML configuration based on the following instructions.

When the RDMA mode is exclusive, shared, or shared-multi (1:N shared), execute the following YAML code to configure RDMA resources for the pod.

In the spec.containers.resources field, use vke.volcengine.com/rdma to apply for RDMA NIC for the pod.

In the pod annotation, use k8s.volcengine.com/pod-networks to specify the number of NICs. We recommend setting the number of RDMA cards to the total number of cards in the host, or half the number to achieve optimal performance for GPU training or inference tasks. For example, when the host has four RDMA NICs, you can set the number of RDMA cards to 2 or 4. When the system has eight RDMA NICs, you can set the number of RDMA cards to 4 or 8.

In most cases, you need to use the spec.containers.securityContext field to request the IPC_LOCK privilege.

apiVersion: v1
kind: Pod
metadata:
name: rdma-test
annotations:
k8s.volcengine.com/pod-networks: |
[
{
"cniConf":{
"name":"rdma"
}
}
]
spec:
containers:
- name: rdma-test
image: busybox
command:
- /bin/sh
- -c
- sleep 10000
resources:
limits:
vke.volcengine.com/rdma: 1
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- mountPath: /dev/shm
name: cache-volume
- name: nvidia-topology # Non-bare metal instance types need to mount this directory. NCCL will obtain the real topology from virtualTopology.xml.
mountPath: /var/run/nvidia-topologyd
readOnly: true
volumes:
- emptyDir:
medium: Memory
name: cache-volume
- name: nvidia-topology
hostPath:
path: /var/run/nvidia-topologyd
type: DirectoryOrCreate

The preceding sample code provides an example of how to automatically create an RDMA NIC. To create multiple RDMA NICs, copy the code block in k8s.volcengine.com/pod-networks multiple times.

{
"cniConf":{
"name":"rdma"
}
}

When the RDMA mode is shared-host, execute the following YAML code to configure RDMA resources for the pod.

In the spec.containers.resources field, use vke.volcengine.com/rdma-host to apply for the RDMA resources of the entire host for the pod. The value is 1 and cannot be changed.

In the pod annotation, use k8s.volcengine.com/pod-networks to specify the number of NICs. Differing from other modes, the shared-host mode allows you to configure only one code block. If you configure more than one code block, an error will occur.

In most cases, you need to use the spec.containers.securityContext field to request the IPC_LOCK privilege.

apiVersion: v1
kind: Pod
metadata:
name: rdma-test
annotations:
k8s.volcengine.com/pod-networks: |
[
{
"cniConf":{
"name":"rdma"
}
}
]
spec:
containers:
- name: rdma-test
image: busybox
command:
- /bin/sh
- -c
- sleep 10000
resources:
requests:
vke.volcengine.com/rdma-host: 1
limits:
vke.volcengine.com/rdma-host: 1
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- mountPath: /dev/shm
name: cache-volume
- name: nvidia-topology # Non-bare metal instance types need to mount this directory. NCCL will obtain the real topology from virtualTopology.xml.
mountPath: /var/run/nvidia-topologyd
readOnly: true
volumes:
- emptyDir:
medium: Memory
name: cache-volume
- name: nvidia-topology
hostPath:
path: /var/run/nvidia-topologyd
type: DirectoryOrCreate

When you run NCCL in the container in shared mode, you need to explicitly specify the NCCL_IB_GID_INDEX environment variable. You can execute the source nccl_env.sh command to obtain the RDMA environment variables. The content of the sample script nccl_env.sh for dynamically obtaining NCCL environment variables is as follows.

Validate results

Viewing information about RDMA NIC

After you create a pod and configure RDMA resources, you can view the RDMA NIC mounted to the pod in the console or in YAML. This topic provides an example of how to view NIC information in the console.

Click the name of the target pod to go to the pod details page.

Click the Overview tab. In the Networking section, you can view the information about the RDMA NIC mounted to the pod.

In the upper right corner of the pod details page, click ... > Edit Yaml to view the YAML configuration of the pod. You can view information about the RDMA NIC mounted to the pod, such as the IP address and MAC address.

Viewing RDMA monitoring data

You can check whether the allocated RDMA resources have been used based on the metrics provided in RDMA monitoring data. For the method of viewing RDMA monitoring data, see AI resource observability or Viewing instance GPU/RDMA monitoring data.