Docs

Volcengine Kubernetes Engine

Topology-aware scheduling

NUMA-aware scheduling

VKE supports enhancing the resource management capability, improving the resource utilization, reducing the cost, and improving the load performance of the Kubernetes based on Katalyst.

tip

[Beta · Apply for Trial]: This feature is currently in the beta phase. Submit an application to use it.

Application scenarios

In business scenarios including searching, advertising, and recommendation, gaming, and AI distributed training, users are highly sensitive to latency, and have certain requirements for container placement at the micro-topology level. There are some limits to the micro-topology management capability of the native Kubernetes. Therefore, VKE supports the topology-aware scheduling feature of NUMA based on Katalyst.

We recommend that you use the topology-aware scheduling feature in the following scenarios:

Run EBM servers with NUMA enabled.

Compute-intensive load.

Online service load.

Database business scenario.

Notes

Take note of the following limitations and items when you are using this feature:

The requirements for Kubernetes version of the clusters

Kubernetes v1.24: v1.24.10-vke.13 and later versions.

Kubernetes v1.26: v1.26.10-vke.21 and later versions.

Kubernetes v1.28: v1.28.3-vke.15 and later versions.

Kubernetes v1.30: v1.30.4-vke.8 and later versions.

This depends on the scheduler-plugin add-on and katalyst add-on of the VKE.

This depends on the kubelet custom configuration feature of the node pool.

The QoS level of the pod must be Guaranteed. Meanwhile, when the CPU resource amount applied by the container is an integer, the container will proceed with core binding.

It is fully compatible with the topology management policy of the Kubernetes community.

tip

Currently, when you are scheduling the VKE, only the single-numa-node policy is supported, which refers to container-level affinity.

Under the single-numa-node policy, the scheduling and allocation of the CPU and devices (such as RDMA or GPU) requested by pods, need to meet the constraints of a single NUMA node. Otherwise, the scheduling will fail.

Prerequisites

The clusters that meet the limits have been created. For detailed operations, refer to Creating clusters.

In the clusters, the scheduler-plugin add-on has been installed. For detailed operations, refer to scheduler-plugin add-on.

Your cluster has custom node pools with at least one node. For detailed steps, see Creating node pools.

Procedure

Step 1: Install and configure necessary add-ons

This feature depends on the scheduler-plugin add-on and katalyst add-on of the VKE. Therefore, you need to install and configure these two add-ons.

On the left navigation bar, select Cluster, and click the name of the target cluster.

On the left navigation bar on the cluster management page, select Addons.

On the Scheduling tab on the Addons page, install and configure add-ons.

Find the Katalyst add-on and click Deploy. Select Topology Awareness Scheduling and select the CPU resource management type.

Find the scheduler-plugin add-on, click Configuration, and enable NUMA-aware scheduling.

Step 2: Enable the topology awareness capability of the node pool

On the left navigation bar on the cluster management page, select Node pool.

Under the Operation column of the target node pool, click Edit.

On the Edit node pool page, enable the label of topology awareness for the node pool configuration.

Enable the label of topology awareness for the node pool configuration. The key is vke.node.katalyst.deploy, and the value is true.

Configure the kubelet custom parameters of the node pools, and enable their topology awareness capability.

Warning

You need to specify custom kubelet parameters only for clusters of Kubernetes v1.24 and v1.26. You do not need to specify the clusters of Kubernetes v1.28 and later.

You can control whether to synchronize labels to the existing nodes in the node pools by enabling or disabling the node labels and taint automatic update features of the node pools. When you disable these features, you may not be able to achieve topology-aware scheduling on the existing nodes. You need to proceed with configurations cautiously.

Configure the kubelet custom parameters as shown in the following figure. For detailed descriptions, refer to Configuring the kubelet custom parameters for nodes.

Click Confirm to finish editing the node pool.

Step 3: Create a workload

Creating through the console

The following content uses creating a Deployment as an example. For more relevant operations of workloads, refer to Workload.

tip

For container images used in the workloads, you can refer to the Pushing and pulling images topic of CR, to upload the images to CR of Volcano Engine in advance, or directly use your own third-party image.

On the left navigation bar on the cluster management page, select Workload > Deployment, and click Create deployment.

On the Create deployment page, configure the load parameters according to actual demands.

In this example, some parameters need to be configured according to the following description. If you want to learn about the detailed configuration descriptions of the other parameters, refer to Creating Deployment.

Enable Node affinity scheduling in the Scheduling policy section of the Advanced configurations step and enter vke.node.katalyst.deploy In true in the Selector field based on the following figure.

After you create the Deployment, click the workload name to go to its details page. Then click the Instance list tab to view the status of Pod instances.

Click Node pool in the left navigation bar of the cluster management page. Find the target node pool for which the topology awareness capability is enabled, and click the name of the target node pool.

On the Node list tab of the Node pool details page, click the node name to enter the Node details page.

On the Pod list tab on the Node details page, view whether there are pods related to workloads already created.

As shown in the figure below, the pods related to the workloads created have been scheduled to the nodes that have enabled the topology awareness capability.

Creating through YAML

The following text will take using the public network image to create pods as an example to demonstrate the topology awareness scheduling. For more information about pods related to business applications, the introduction, and the creation methods of workloads, see Workloads.

Create the YAML file of the pods created. The test-topo.yaml code in the sample file is as follows:

apiVersion: v1
kind: Pod
metadata:
name: test-topo # Pod Name.
namespace: default # Namespace to which the pods belong.
spec:
affinity: # Pod affinity scheduling.
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions: # Pod affinity scheduling selector.
- key: vke.node.katalyst.deploy
operator: In
values:
- "true"
containers: # Container configurations of pods.
- image: redis # Container images of pods, which can be substituted with your own container images. If the images are from the Internet, you need to ensure the cluster has access to the Internet.
name: redis
resources:
limits:
cpu: "63"
memory: 10Gi
requests:
cpu: "63"
memory: 10Gi

Run the following command to create pods.

kubectl apply -f topo-pod.yaml

After waiting for 3-5 minutes, run the following command. Check the status of pods and ensure they are Running.

kubectl get pod -o wide

Expected output result:

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-topo 1/1 Running 0 3m11s 192.168.16.24 192.168.16.18 <none> <none>

Step 4: View scheduling and allocation results

Run the following command to view the CustomNodeResource resources of the nodes to which the pods are scheduled.

tip

Replace 192.168.16.18 with the IP address of your own nodes.

The expected output is as follows: pods have been scheduled to this node, meeting the single-numa-node policy.

What to do next

After using the NUMA-aware scheduling capability, if you need to upgrade an add-on that you rely on, carefully understand the impact of the upgrade. For example:

After upgrading the katalyst from v0.3.5-vke.5 to v0.5.x and later versions, you need to make changes to the kubelet custom parameters of the node pool associated with the add-on. For details, see FAQ.

Last updated: 2026.09.09 17:25:45