This topic describes best practices for VKE load-aware scheduling.

Applicable scenarios

Load-aware scheduling is a VKE policy that schedules workloads based on actual node loads. During scheduling, the scheduler preferentially schedules Pods to nodes with lower load based on the historical node load metrics. This balances loads among nodes and reduces application or node failures caused by a high load on a single node. The recommended usage methods for different business scenarios are as follows:

Scenario	Recommended method	Key configuration	Applicable workload
Avoid hotspot nodes for online services	Set the effective scope of load-aware scheduling to specific Pods and add an annotation to the target workload.	vke.volcengine.com/load-aware-enabled: "true", along with CPU and memory thresholds.	Web services, gateways, resident Workers, and services sensitive to P95/P99 latency.
Different thresholds for different node pools	Use NodeSchedulingConfig to override load thresholds per node pool.	Match nodes using node pool labels and set target thresholds for CPU and memory separately.	Clusters running a mix of general-purpose, memory-optimized, high-specification, and co-located node pools.
Canary rollout for existing clusters	Start with a canary rollout on a small number of workloads, expand to similar services, and then evaluate whether to enable it globally.	Select specific Pods as the effective scope first, and monitor Pending Pods, scheduling events, and changes in node hotspots.	Clusters that already run production workloads and for which scheduling policy changes pose a high risk.

Load-aware scheduling is not recommended to be enabled for short-lived Jobs by default. Jobs exit quickly after scheduling completes, and the node load observed at scheduling time may change quickly. Therefore, the benefit is usually lower than that for online services.

Pre-checks

The cluster is running and contains at least two nodes whose loads need to be balanced.

The cluster Kubernetes version meets the requirements for load-aware scheduling, and the scheduler-plugin add-on has been installed.

Load-aware scheduling has been enabled for scheduler-plugin. If a canary rollout is required, set the effective scope to specific Pods.

Node pool labels are stable. You can use cluster.vke.volcengine.com/machinepool-name or custom workload node labels to distinguish node pools.

You can check node CPU and memory utilization by using kubectl top nodes, monitoring dashboards, or node load metrics.

For more usage restrictions, see Load-aware scheduling.

Detailed parameters

Load-aware scheduling takes effect through the scheduler-plugin add-on configuration, Pod annotations, node load metrics, and node-pool-level threshold overrides. The key parameters are as follows.

Parameter	Meaning	Configuration recommendations
Effective scope	Controls which Pods load-aware scheduling applies to. You can choose specific Pods or all Pods.	For production clusters, first select specific Pods. For new clusters or fully verified homogeneous workload clusters, evaluate whether to enable the feature globally.
vke.volcengine.com/load-aware-enabled	A Pod annotation that indicates whether load-aware scheduling is enabled for the Pod.	Set it to "true" during a canary rollout. If a rollback is required, change it to "false" or remove the annotation.
Node resource utilization threshold	CPU and memory utilization thresholds used by the scheduler to filter nodes.	These thresholds are not capacity-planning limits. Set them below the maximum acceptable utilization level for the workload and leave a buffer of 5–15 percentage points.
Node scoring resource weight	The weight assigned to each resource dimension when the scheduler scores nodes by resource utilization.	For CPU-intensive workloads, increase the CPU weight. For memory-sensitive workloads, increase the memory weight. Ensure that the weights align with the resource bottleneck of the workload.
NodeSchedulingConfig.metadata.name	Name of the node-pool-level threshold override configuration.	Use a name that reflects the purpose of the node pool, such as load-threshold-general-pool and load-threshold-memory-pool.
NodeSchedulingConfig.spec.nodeSelector	Select the set of nodes to which this threshold configuration applies. The type is Kubernetes LabelSelector.	Use stable node pool labels first, such as cluster.vke.volcengine.com/machinepool-name. Prevent more than one configuration from matching the same node.
NodeSchedulingConfig.spec.targetLoadThreshold	Set the target load threshold for matching nodes, defined in the source code as a mapping from resource name to percentage.	Set CPU and memory separately based on node pool resource bottlenecks. In most cases, memory thresholds for memory-optimized pools need to be more conservative than those for general-purpose pools.
NodeMonitor.status.nodeUsage	Node load data, including node resource usage.	During troubleshooting, check whether node load data is updated as expected. If the data is missing or delayed, scheduling performance may degrade.
NodeMonitor.status.updateTime	The latest update time of the node load data.	Indicates the freshness of load data. If the update is stale, troubleshoot the monitoring collection pipeline first.

Scenario 1: Keep online services away from hotspot nodes

This scenario is suitable for long-lived Pods such as online services, gateways, and resident Workers. To reduce the impact of single-node hotspots on service latency and stability, preferentially schedule new Pods on low-load nodes and avoid nodes with CPU or memory hotspots.

Recommended configuration

Enable the load-aware scheduling feature for the scheduler-plugin add-on.

Set the effective scope to specific Pods and use an annotation to roll out the feature to target workloads in a canary manner.

Set global default CPU and memory thresholds as a fallback safeguard when no node pool override is configured.

Set node scoring resource weights based on the workload bottleneck. For CPU-intensive services, increase the CPU weight. For memory-sensitive services, increase the memory weight.

Add vke.volcengine.com/load-aware-enabled: "true" to the target Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
name: web-load-aware-verify
namespace: default
spec:
replicas: 6
selector:
matchLabels:
app: web-load-aware
template:
metadata:
labels:
app: web-load-aware
annotations:
vke.volcengine.com/load-aware-enabled: "true"
spec:
schedulerName: default-scheduler
containers:
- name: nginx
image: cr-helm-test-cn-beijing.cr.volces.com/vke/nginx:latest
resources:
requests:
cpu: 500m
memory: 512Mi

Verification methods

Connect to the target cluster. For details, see Connecting to a cluster via kubectl.

Confirm that load-aware scheduling has been enabled for scheduler-plugin and that the effective scope meets expectations.

Prepare hotspot nodes. You can use stress-test workloads or existing workloads to drive CPU/memory utilization to high levels. This ensures that at least one high-load node and one low-load node exist in the cluster.

$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
192.**.**.** 44m 0% 1811Mi 6%
192.**.**.** 25m 0% 1076Mi 3%
192.**.**.** 19m 0% 1033Mi 3%
192.**.**.** 16m 0% 935Mi 7%
192.**.**.** 14m 0% 911Mi 7%
192.**.**.** 13m 0% 937Mi 7%
192.**.**.** 4027m 51% 3462Mi 12%
192.**.**.** 1017m 13% 1169Mi 4%
vci-node1-cn-beijing-a <unknown> <unknown> <unknown> <unknown>
vci-node1-cn-beijing-b <unknown> <unknown> <unknown> <unknown>

Deploy the sample application and confirm that the Pod template contains vke.volcengine.com/load-aware-enabled: "true". The sample application uses node affinity to limit the schedulable scope to the node pool corresponding to the node pool label cluster.vke.volcengine.com/machinepool-name=pd9ji9ullqoqnlv6sok7g. The scheduler selects a more suitable node within that node pool based on load.

apiVersion: apps/v1
kind: Deployment
metadata:
name: web-load-aware-verify
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: web-load-aware
template:
metadata:
labels:
app: web-load-aware
annotations:
vke.volcengine.com/load-aware-enabled: "true"
spec:
schedulerName: default-scheduler
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cluster.vke.volcengine.com/machinepool-name
operator: In
values:
- "pd9ji9ullqoqnlv6sok7g"
containers:
- name: nginx
image: cr-helm-test-cn-beijing.cr.volces.com/vke/nginx:latest
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1Gi"

Observe Pod distribution. Newly created Pods are expected to be preferentially scheduled to low-load nodes.

$ kubectl get pods -n default -l app=web-load-aware -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-load-aware-verify-5fcf****-**** 1/1 Running 0 6s 192.**.**.** 192.**.**.** <none> <none>
web-load-aware-verify-5fcf****-**** 1/1 Running 0 6s 192.**.**.** 192.**.**.** <none> <none>
web-load-aware-verify-5fcf****-**** 1/1 Running 0 6s 192.**.**.** 192.**.**.** <none> <none>

Risks and optimization

The key risk in this scenario is a trade-off: too low a threshold reduces the number of schedulable nodes, while too high a threshold renders hotspot avoidance ineffective. First, perform a canary rollout on a single workload. Then, monitor Pending Pods, scheduling failure events, the number of hotspot nodes, and service P95/P99 latency. If newly added Pods still land on high-load nodes, check constraints such as the annotation, effective scope, node load data freshness, node affinity, taint toleration, and resource requests at the same time.

Scenario 2: Set different scheduling thresholds for different node pools

This scenario applies to clusters with multiple types of node pools, such as general-purpose compute pools, CPU-optimized pools, memory-optimized node pools, and co-located or low-priority node pools. Different node pools have different specifications, workload types, and resource bottlenecks. If only one set of global thresholds is used, issues such as general-purpose pool overload, delayed protection for memory-optimized nodes, and insufficient utilization of high-specification nodes may occur.

Recommended configuration

Identify the purpose and stable labels of each node pool. We recommend using cluster.vke.volcengine.com/machinepool-name to match node pools.

Set different CPU and memory thresholds for each type of node pool.

Create a NodeSchedulingConfig for each node pool. Each node needs to match only one expected configuration so that the source of the final threshold is clear.

Retain global default thresholds as a fallback for nodes that do not match any NodeSchedulingConfig.

During the canary rollout, select one or a small number of node pools for verification. Expand the rollout to other node pools after confirming its effectiveness.

Verification methods

Connect to the target cluster. For details, see Connecting to a cluster via kubectl.

Confirm that load-aware scheduling has been enabled for scheduler-plugin and that the effective scope meets expectations.

Create or update NodeSchedulingConfig.

apiVersion: scheduling.vke.volcengine.com/v1beta1
kind: NodeSchedulingConfig
metadata:
name: load-threshold-pool-1
spec:
nodeSelector:
matchLabels:
cluster.vke.volcengine.com/machinepool-name: pd9jkutildci5lli87oo0
targetLoadThreshold:
cpu: 75
memory: 80
---
apiVersion: scheduling.vke.volcengine.com/v1beta1
kind: NodeSchedulingConfig
metadata:
name: load-threshold-pool-2
spec:
nodeSelector:
matchLabels:
cluster.vke.volcengine.com/machinepool-name: pd9jkuh5lqoqnlv6solug
targetLoadThreshold:
cpu: 80
memory: 85
---
apiVersion: scheduling.vke.volcengine.com/v1beta1
kind: NodeSchedulingConfig
metadata:
name: load-threshold-pool-3
spec:
nodeSelector:
matchLabels:
cluster.vke.volcengine.com/machinepool-name: pd9jkv5qldci5lli87oog
targetLoadThreshold:
cpu: 85
memory: 90

kubectl apply -f nodeschedulingconfig.yaml
kubectl get nodeschedulingconfig

Check node pool labels to confirm that each node matches only one expected configuration.

kubectl get nodes --show-labels | grep cluster.vke.volcengine.com/machinepool-name

Query the details of the NodeSchedulingConfig.

kubectl get nodeschedulingconfig load-threshold-pool-1 -o yaml
kubectl get nodeschedulingconfig load-threshold-pool-2 -o yaml
kubectl get nodeschedulingconfig load-threshold-pool-3 -o yaml

Generate real load for the node pools.

$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
192.**.**.** 35m 0% 2357Mi 8%
192.**.**.** 17m 0% 1161Mi 4%
192.**.**.** 21m 0% 1085Mi 3%
192.**.**.** 5016m 64% 4567Mi 16%
192.**.**.** 3025m 77% 9057Mi 70%
192.**.**.** 9034m 57% 21875Mi 38%
vci-node1-cn-beijing-a <unknown> <unknown> <unknown> <unknown>
vci-node1-cn-beijing-b <unknown> <unknown> <unknown> <unknown>

Deploy the workload to the node pool identified by load-threshold-pool-1. The scheduler is expected to reject the workload, confirming that the threshold configured for this node pool takes effect.

apiVersion: apps/v1
kind: Deployment
metadata:
name: load-aware-threshold-reject-pool-1
namespace: default
labels:
app: load-aware-threshold-reject-pool-1
spec:
replicas: 1
selector:
matchLabels:
app: load-aware-threshold-reject-pool-1
template:
metadata:
labels:
app: load-aware-threshold-reject-pool-1
annotations:
vke.volcengine.com/load-aware-enabled: "true"
spec:
schedulerName: default-scheduler
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cluster.vke.volcengine.com/machinepool-name
operator: In
values:
- "pd9jkutildci5lli87oo0"
containers:
- name: nginx
image: cr-helm-test-cn-beijing.cr.volces.com/vke/nginx:latest
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "512Mi"

$ kubectl describe pod load-aware-threshold-reject-pool-1-5d96****-****
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 36s default-scheduler 0/8 nodes are available: 1 node(s) cpu usage exceed threshold, usage:77, threshold: 75 , 2 node(s) had untolerated taint(s), 5 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/8 nodes are available: 1 No preemption victims found for incoming pod, 7 Preemption is not helpful for scheduling.
Warning FailedScheduling 36s default-scheduler 0/8 nodes are available: 1 node(s) cpu usage exceed threshold, usage:77, threshold: 75 , 2 node(s) had untolerated taint(s), 5 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/8 nodes are available: 1 No preemption victims found for incoming pod, 7 Preemption is not helpful for scheduling.

Scale out the preceding workload, add the preceding three node pools to its node pool affinity, and observe whether Pods avoid nodes exceeding their respective node pool thresholds.

$ kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
load-aware-threshold-reject-pool-1-67f9b****-**** 1/1 Running 0 79s 192.**.**.** 192..**.**.** <none> <none>
load-aware-threshold-reject-pool-1-67f9b****-**** 1/1 Running 0 53s 192.**.**.** 192.**.**.** <none> <none>
load-aware-threshold-reject-pool-1-67f9b****-**** 1/1 Running 0 71s 192.**.**.** 192.**.**.** <none> <none>

Risks and optimization

The greatest risk when rolling out per-pool thresholds is an unclear selector design. If the same node is matched by multiple NodeSchedulingConfigs at the same time, it is difficult to determine the final threshold source during troubleshooting. Use a unique, stable label for each node pool, and run kubectl get nodes --show-labels before rollout to verify which nodes match each label.

Configure the threshold based on historical monitoring data. For general-purpose compute pools, you can set CPU to 65% to 75% and memory to 70% to 80% from the start. For high-CPU pools, you can appropriately increase the CPU threshold. For memory-optimized node pools, we recommend setting the memory threshold more conservatively, such as 60% to 70%. After rollout, continue adjusting the values based on workload latency, Pending Pods, the number of node hotspots, and resource utilization.

Rollout procedure

Select the target workload and confirm that Pod labels, resource requests, and scheduling constraints are stable.

Enable the load-aware scheduling capability of the scheduler-plugin, and select specific Pods as the effective scope first.

Add vke.volcengine.com/load-aware-enabled: "true" to the canary workload.

Set global default CPU and memory thresholds and resource weights.

To set thresholds by node pool, create NodeSchedulingConfig and verify the node matching relationship.

Deploy or scale out the canary workload and observe Pod distribution, node load, and scheduling events.

After the canary rollout is stable, expand the rollout to similar workloads. Consider a global rollout only after confirming that monitoring is stable, thresholds are clear, and the rollback path is well-defined.

Risks and limitations

Load-aware scheduling depends on node load data. If monitoring data is missing, delayed, or significantly deviates from the actual load, the scheduling performance will degrade. During troubleshooting, check the NodeMonitor update time, node monitoring metrics, Pod events, and scheduler-plugin configuration at the same time.

Load-aware scheduling does not bypass native Kubernetes scheduling constraints. Node affinity, taint tolerations, resource requests, and topology constraints still affect the final scheduling result. If these constraints have already narrowed the eligible nodes to a small set, load-aware scheduling may still be unable to avoid hotspot nodes even if it is enabled.

Rollback solution

Change the canary workload annotation to vke.volcengine.com/load-aware-enabled: "false", or delete the annotation.

If a node-pool-level NodeSchedulingConfig has been created, first delete the newly created configuration or change the Selector to an empty match to restore the global default thresholds.

If it has been enabled globally, change the effective scope back to specific Pods in the scheduler-plugin configuration or disable load-aware scheduling.

Roll out the workload again, observe whether new Pods return to normal scheduling, and confirm that Pending Pods and hotspot node metrics return to the expected range.

Volcengine Kubernetes Engine