Docs

Volcengine Kubernetes Engine

RDMA high-performance computing

Using RDMA resources (InfiniBand) in VKE clusters

Volcengine Kubernetes Engine (VKE) can use Remote Direct Memory Access (RDMA) resources through add-ons to eliminate bottlenecks of computing tasks brought by traditional network communication. This article introduces how a VKE cluster uses RDMA resources provided by InfiniBand high-performance compute clusters.

tip

[Public beta]: VKE clusters featured with access to RDMA resources are currently in public beta.

Background information

RDMA overview

RDMA is a high-performance network protocol that reduces CPU usage and memory bandwidth bottlenecks, as well as improves bandwidth utilization. It comes with the following advantages:

Zero Copy: Applications can transfer data without involving the network software stack, allowing data to be directly sent to the buffer or directly received from the buffer without being copied to the network layer.

Kernel Bypass: RDMA provides a proprietary Verbs interface, allowing applications to transfer data directly from user space without switching context between kernel and user modes.

CPU Offload: Applications can access memory on remote hosts without consuming any CPUs on those hosts. Remote memory will be read without any help from remote processes (or CPUs), and the content read from them will not be written into the remote CPU caches.

The High Performance Compute (HPC) cluster is a cluster computing environment used to handle large-scale computing tasks, usually composed of a large number of compute nodes interconnected by an RDMA high-speed network. It supports both RoCE and InfiniBand.

An HPC cluster can provide powerful computing capability, large memory, high-speed storage, and networks to meet the demands of compute-intensive, data-intensive, or combined workloads. In an HPC cluster, computing tasks are assigned to each compute node for parallel processing, fully utilizing the computing power of multiple nodes to accelerate computation and improve efficiency. For more details, please refer to the HPC cluster overview.

RDMA add-on mode

In the InfiniBand network mode, the RDMA add-on supports the following modes. The following table describes different modes and their usage restrictions.

RDMA add-on mode	Mode description	Limit
ib-shared (shared)	Create a shared virtual network interface for pods in the cluster by virtualizing the RDMA physical network interface.	Each RDMA physical network interface can only provide one shared virtual network interface.
ib-shared-multi (1:N shared)	Create multiple shared virtual network interfaces for pods as needed in the cluster by virtualizing the RDMA physical network interface.	We recommend that you do not use this mode with GPUs.

tip

If your pods need to use HostNetwork, we recommend that you use RDMA resources provided by open source add-ons. For details, see the open source documentation.

Cautions

The following content introduces the proven practices of VKE. Reading the Limits and strictly following the steps described below are highly recommended. If you want to use another solution, consult your account manager.

Limits

A pod cannot use mGPU and RDMA add-ons simultaneously.

RDMA devices can only work with the VPC-CNI network model and it is required to use version v1.7.1 or higher VPC-CNI cluster networking components.

Kubernetes version v1.24.15-vke.12 or higher, or version starts from v1.20.15-vke.18 in v1.20 series is required. You can visit the console of the cluster and find the current version of Kubernetes in the cluster basic information page. See How to find the Kubernetes version of a cluster? for details.

RDMA metrics feedback and RDMA topology-aware scheduling are not available when using RDMA of an InfiniBand HPC cluster.

If a NCCL network error occurs in an environment running RDMA, please refer to FAQ for solutions.

Steps

Step 1: Create HPC cluster

Click create HPC cluster and finish configurations based on the parameter descriptions in the table below.

Parameters	Description
Region and AZ	Intranets within different regions are completely isolated. It is recommended to select a region as close as possible to where your business is located to reduce network latency and improve access speed. AZ (Availability Zones) refers to areas isolated from each other in terms of electricity and network. In this case, resources such as node subnets, instance types and HPC clusters are required to stay in a single availability zone. For example: Availability zone C.
Name	Enter a name for HPC cluster.
Cluster properties	Select a property for HPC cluster. Each property has its own limits. Select InfiniBand in this case. RoCE (RDMA over Converged Ethernet) is a network communication protocol designed based on Ethernet technology to realize Remote Direct Memory Access. It features high performance and high cost-efficiency, suitable for data exchange among instances in an HPC cluster. InfiniBand is a network communication protocol designed based on dedicated network hardware to implement Remote Direct Memory Access, featuring high throughput and low latency, and suitable for data exchange among instances in an HPC cluster. Cluster property needs to be selected according to the instances. tip HPC clusters of infiniBand can only work with ecs.hpcpni3.45xlarge, which is one of the High Performance Computing Optimized instances. Apart from that one, all the remaining instances can be candidates for RoCE-powered HPC clusters. InfiniBand clusters can only add or expand infiniBand-powered instances. Similarly, RoCE clusters apply the same logic.
Description	Provide descriptions or comments for the HPC cluster.

Step 2: Create a cluster and prepare resources

tip

This step only covers the key parts of configuring RDMA resources, such as creating a node pool, adding node labels, setting Kubelet custom parameters, and installing RDMA add-ons. For creating and configuring a new cluster,see Create a cluster.

Click Create cluster and finish the configurations for the step 1 cluster configuration, then click Next: node pool configuration.

The table below contains instructions as well as descriptions of key parameters in this step. Please follow the instructions in the table and configure the rest as needed.

Parameters	Description
Network model	Select a Container Network Interface (CNI) for the cluster. In this case, VPC-CNI is required. Flannel: an independent underlay network model based on which a high-performance network for clusters can be built along with the global routing capabilities of VPCs. VPC-CNI: an underlay network model built based on Elastic Network Interfaces (ENI) on VPC, providing high network performance.

On the Node pool configuration (optional) page, you need to finish key configurations, including settings of a new node pool, node labels, and Kubelet custom parameters. Click Next: Addon configuration when you finish all necessary settings on this page.

The table below contains instructions as well as descriptions of key parameters in this step. Please follow the instructions in the table and configure the rest as needed.

Parameter	Description
Worker node	Select Create now.
Node source	Select Create node in this case.
Availability zone	Availability zones (AZ) are physically isolated infrastructure units within a single region, featuring independent power and networking systems. AZs communicate over internal networks. Different regions have their own availability zones. See Regions and Availability Zones for details. In this case, resources such as node subnets, instance types and HPC clusters are required to stay in a single availability zone. For example: Availability zone C.
Node subnet	Select subnets available in the list for the nodes.
Instance type	Instance type defines resources such as CPU, memory, GPU, RDMA of a node. In this case, instance types enabled with RDMA device are required, e.g., High Performance Computing Optimized instances.
High Performance Computing Optimized	A cluster computing environment used to handle large-scale computing tasks, typically consisting of a large number of worker nodes interconnected through a high-speed RDMA network to process complex tasks collaboratively. In this case, the InfiniBand-powered HPC cluster created in step 1 should be selected.
Node labels	This is the same as the Label in Kubernetes, which is used to define attributes for nodes, facilitating later operations on nodes such as filtering. A label can be added to all the nodes in the pool in batch. In this case, the label vke.node.rdma.mode: {value} is added manually to indicate what RDMA mode the node pool is working in. The available modes include ib-shared (shared) and ib-shared-multi (1:N shared). warning Except for ecs.hpcg1ve.21xlarge and North-China ecs.hpcpni2.28xlarge , it is highly recommended to choose the ib-shared mode for instance types. The option Enable auto renew for node labels and taints allows existing nodes in the pool to automatically update labels you added on this page. Caution! Disabling this option might make the cluster unable to use existing RDMA nodes in the pool.
Kubelet configs	For businesses that are highly sensitive to latency, Kubelet custom parameters can be used to create NUMA affinity policy. The resources allocated for a Container (or Pod) can be mapped to one (or more) NUMA nodes to improve business performance. In this case, the custom parameter topologyManagerPolicy: best-effort is required, so that GPU and RDMA can be allocated to the same NUMA node.

On the Addon configuration page, ensure that rdma-device-plugin is checked. Other add-ons can be selected as needed. Click Next: Configuration confirmation to go to the next page.

Ensure that all the information of the new cluster and billing details are correct, read the "VKE Dedicated Service Terms" and click the checkbox to agree, then click Confirm on the page to start creating the cluster.

Step 3: Use RDMA Resources

Connect to the cluster using kubectl. See Connecting to cluster for details.

Use RDMA resources through YAML. The following example shows how to enable RDMA for a pod.

Last updated: 2026.07.24 14:51:38