Docs

Volcengine Kubernetes Engine

mGPU scheduling

Configuring mGPU scheduling

This topic mainly describes how to use the mGPU feature in Volcengine Kubernetes Engine (VKE) to enable multiple containers to share a GPU.

Limits

A pod cannot use mGPU and RDMA add-ons simultaneously.

mGPU cannot be used together with privileged containers. Otherwise, the containers will fail to be created or deleted.

When you use the mGPU feature, take note of the following environment requirements and usage limits on clusters and GPUs.

Project	Requirement
Kubernetes version	The version must be v1.20.15-vke.4 or later. For the method of viewing the Kubernetes version of a cluster, refer to FAQs.
Operating system	When you create a cluster, if you specify a public image provided by VKE as the operating system, the system will automatically configure an operating system of the corresponding version. Currently supported operating systems include veLinux 1.0, veLinux 1.0 CentOS Compatible, velinux 2.0, Ubuntu 20.04, and Ubuntu 22.04.
Kernel Version	When you create a cluster, if you specify a public image provided by VKE as the operating system, the system will automatically configure an operating system kernel of the corresponding version. Currently, kernel version 5.4 is supported.
NVIDIA driver version	Use v470, v535, v550 or v580. The officially supported minor versions shall prevail. For more details, refer to Overview of operating system images. When you create a GPU-accelerated node pool, the system will automatically configure an NVIDIA driver of the corresponding version.
GPU	VKE supports mainstream NVIDIA GPU types such as Turing, Volta, Ampere, Hopper, and Ada Lovelace. If you have requirements for other types of GPUs, consult official pre-sales or technical support. For node instance types supported by different types of GPUs, refer to Specifications of GPU cloud computing instances.
Number of mGPUs (corresponding to the number of containers)	Up to 16 mGPUs can be created for one GPU. Computing performance deteriorates as the number of mGPUs increases. It is recommended that you plan the number of mGPUs that are deployed on a single GPU based on the resource size requested by containers.
Number of containers	When you use mGPU, it is recommended that the number of containers in a single pod does not exceed 5.

Scheduler description

mGPU must be used together with the scheduler-plugin add-on to achieve shared GPU scheduling among containers. The scheduler-plugin add-on extends the scheduling capability and policy of the native kube-scheduler, which allows you to configure Binpack or Spread policies for GPU nodes and GPU cards. For the detailed introduction to the add-on and scheduling policy configuration descriptions, refer to the scheduler-plugin section of the Installing an add-on topic.

warning

In this topic, you need to install the scheduler-plugin add-on when you create a cluster, and cannot modify the configurations of the add-on after it is configured for the first time. Therefore, when you configure the add-on during the creation of the cluster, configure the Binpack or Spread scheduling policy based on your business requirements.

If you do not modify the scheduling policy of the scheduler-plugin add-on when you create the cluster, mGPU will adopt the following default scheduling policy:

GPU node scheduling policy	GPU card scheduling policy	Scheduling mode	Application scenarios
Spread	Binpack	Pods are spread to nodes with low GPU allocation rate. Containers are preferentially scheduled to GPUs with high packing rates on nodes.	High fault isolation performance is achieved among nodes and a small number of fragmented resources exist.

Prerequisites

The mgpu addon has been installed. For details, see Installing add-ons.

tip

If you need to use an mGPU computing power allocation policy, make sure that the version of the mgpu add-on is v0.0.3 or later. For detailed description, refer to Configuring an mGPU computing power allocation policy.

Procedure

Step 1: Create a GPU node pool

In the left navigation bar, click Cluster. In the cluster list, click the name of the target cluster name to open the cluster management page.

In the left navigation bar, click Node pool.

On the Node pool page, click Create node pool to create a node pool that meets your requirements. For parameter details, see Creating node pools.

For Instance type, select GPU Accelerated to add nodes with GPUs to the cluster.

In the Advanced settings section, click Add new to add a GPU scheduling label to the node pool.

Set the key to vke.node.gpu.schedule and the value to mgpu. This setting enables mGPU scheduling and GPU video memory isolation for nodes in this node pool.

warning

For details and considerations for using the GPU scheduling label, see GPU scheduling overview.

You can select or deselect Enable auto renew for node labels and taints to determine whether to synchronously add new labels to existing nodes in the pool. If you disable this feature, the GPU scheduling capability on existing nodes may not meet your expectations. Proceed with caution.

Set the label key to vke.volcengine.com/mgpu-compute-policy and the label value to fixed-share, guaranteed-burst-share, or native-burst-share. This sets a computing power allocation strategy for nodes in the node pool.

tip

For details on computing power allocation, see Allocating mGPU computing power.

Step 2: Configure a workload

This topic uses the creation of a stateless workload (Deployment) as an example. For detailed steps, see Creating Deployments.

Creating through the console

In the left navigation bar of the cluster management page, click Workload > Deployment.

On the Deployment page, create a Deployment or update an existing Deployment. In the ResourceQuota section of the Container configuration step, configure GPU resources based on your business requirements.

Parameter	Description
GPU resource type	In this scenario, select mGPU.
Multi-card sharing configuration	The mGPU multi-card sharing feature allows a single container to use computing power and GPU video memory resources jointly provided by multiple GPU cards on the same node. In this example, keep it off. For information about how to configure multi-card sharing, see Using mGPUs for multi-card sharing.
GPU computing power	Configure the GPU computing power of the container. You can set it to a value between 0.01 and 1 with two decimal places or to a positive integer greater than 1. For example, set it to 0.01 or 2. tip The recommended minimum GPU computing power for mGPUs is 0.05 with a step size of 0.05 (when fewer than 1 card), and the minimum GPU video memory is 1 GiB. The recommended value can improve your user experience in GPU sharing among containers.
GPU memory	Specify the GPU video memory of the container, which is provided by multiple GPUs in equal shares. If you set the GPU computing power parameter to a value between 0.01 and 1 with two decimal places, set the GPU memory parameter to a value that is greater than or equal to 256 MiB with a step of 1. If you set the GPU computing power to an integer that is greater than or equal to 1, the entire computing power of the GPU is used. Also, all the GPU video memory is used.
Graphics card type	Configure the GPU model of nodes that the Pod can be scheduled to. By default, no GPU model restriction applies. If you specify a model, the Pod will be scheduled to nodes using that GPU model.

After configuring other parameters, click Confirm.

Creating through kubectl

warning

In mGPU use scenarios, the value of the requests field must be consistent with that of the limits field.

Use YAML to create a workload that applies mGPU scheduling:

Validating results

Viewing GPU card information

You can use the mGPU Client (mgpu-cli) to view mGPU-related information in the terminal.

Log in to the node where the pod is located. You can log in to the node through ECS Terminal. For detailed operations, refer to the ECS Terminal section of the Managing nodes topic.

Run the following command to view graphics card information.

mgpu-cli device info -a

The system returns a response similar to the following one. You can view information about the graphics card, such as the percentage of its unassigned computing power and its scheduling policy.

+-----------------------------------+-------------+
| device_id | 0 |
+-----------------------------------+-------------+
| compute_free_weight | 100 |
+-----------------------------------+-------------+
| compute_free_weight_online | 70 |
+-----------------------------------+-------------+
| compute_load_balance_mode | 0 |
+-----------------------------------+-------------+
| compute_policy | 0 |
+-----------------------------------+-------------+
| compute_qos_enabled | 1 |
+-----------------------------------+-------------+
| compute_qos_enabled_native_burst | 1 |
+-----------------------------------+-------------+
| compute_timeslice | 200 |
+-----------------------------------+-------------+
| compute_timeslice_native_burst | 4096 |
+-----------------------------------+-------------+
| compute_total_weight_native_burst | 30 |
+-----------------------------------+-------------+
| device_major | 510 |
+-----------------------------------+-------------+
| instance_deploy_mode | 1 |
+-----------------------------------+-------------+
| instance_major | 509 |
+-----------------------------------+-------------+
| online_offline_switch | 2 |
+-----------------------------------+-------------+
| user_instance_major | 511 |
+-----------------------------------+-------------+
| vmem_free_bytes | 23359324160 |
+-----------------------------------+-------------+
| vmem_limit | 4096 MiB |
+-----------------------------------+-------------+

Metric name	Metric meaning
compute_free_weight_online	The percentage of unassigned computing power of the device (70% in this example).
compute_load_balance_mode	The load balancing mode among containers when the containers use the device.
compute_policy	Specify a computing power scheduling policy. Valid values: 0: fixed-share 1: guaranteed-burst-share 2: native-burst-share
compute_timeslice	The percentage of the computing power time slice.
device_major	/dev/mgpuX is the major device number.
instance_major	/dev/mgpuXinstY and /dev/mgpuXctl-$(container_id) are the major device numbers.
vmem_free_bytes	The remaining GPU memory capacity of the device. Unit: byte.

Viewing container-level mGPU information

Log in to the node where the pod is located. You can log in to the node through ECS Terminal. For detailed operations, refer to the ECS Terminal section of the Managing nodes topic.

Run the following command to view mGPU information at the container level.

mgpu-cli container info

The system returns a response similar to the following one. You can view container-level mGPU video memory allocation details.

+--------------+-----------+-------------+---------+------------------+---------------------+----------------+-----------------------------+-----------------------+-------+--------------------------+--------+----------+-------+----------------+------------+---------------+------------------+
| container_id | device_id | instance_id | pod_uid | compute_priority | compute_qos_enabled | compute_weight | compute_weight_native_burst | compute_weight_online | minor | nvidia/compute_timeslice | online | position | tasks | vmem_info | vmem_limit | vmem_priority | vmem_qos_enabled |
+--------------+-----------+-------------+---------+------------------+---------------------+----------------+-----------------------------+-----------------------+-------+--------------------------+--------+----------+-------+----------------+------------+---------------+------------------+
| 7b0622290a6e | 0 | 0 | N/A | 0 | 1 | 0 | 30 | 30 | 0 | 0 | 1 | normal | N/A | Free: 4096 MiB | 4096 MiB | 0 | 1 |
+--------------+-----------+-------------+---------+------------------+---------------------+----------------+-----------------------------+-----------------------+-------+--------------------------+--------+----------+-------+----------------+------------+---------------+------------------+

Metric name	Metric meaning
container_id	The container ID.
device_id	The device ID that corresponds to the instance.
instance_id	The instance ID.
compute_weight	The percentage of offline computing power allocated to the instance.
compute_weight_native_burst	The percentage of computing power allocated to this instance under the native burst policy.
compute_weight_online	The percentage of online computing power allocated to the instance.
online	Whether it is an online container.
vmem_info	Information about the video memory requested by a container, including free video memory, PID, and used video memory.
vmem_limit	The upper limit on the video memory requested by a container.

Viewing mGPU monitoring data

You can monitor various mGPU metrics by using cloud-native observability. For details, see AI resource observability.