This topic mainly describes how to use the load-aware scheduling capability provided by Volcengine Kubernetes Engine (VKE) to achieve node load balancing.

Feature overview

Load-aware scheduling is a policy provided by VKE that performs appropriate scheduling based on the load of nodes. During scheduling, pods are preferentially scheduled to nodes with lower load according to node load statistics to balance loads among nodes. This prevents application or node failures caused by a high load on a single node.

Prerequisites

A cluster is in the Running state and has two or more nodes. Load balancing among these nodes is required. For detailed operations, see Adding nodes.

The Kubernetes version of the cluster must meet the following requirements to achieve load-aware scheduling. For detailed operations, see Upgrading clusters.

Kubernetes v1.24: v1.24.15-vke.18 and later

Kubernetes v1.26: v1.26.6-vke.7 and later

Other later Kubernetes versions released by VKE

The scheduler-plugin add-on has been installed in the cluster. For detailed operations, see scheduler-plugin.

Procedure

Step 1: Enable the load-aware scheduling feature of the scheduler

Enable the load-aware scheduling feature of the scheduler in the cluster. You can enable this feature by using the VKE console or by calling the CreateAddon or UpdateAddonConfig API.

Enabling the feature in the console

In the cluster that meets the requirements, configure the scheduler-plugin add-on to enable the load-aware scheduling feature. For detailed operations, see scheduler-plugin.

Parameters	Description
Effective scope	Specify the effective scope of load-aware scheduling. If you select Specific pod, configure the load-aware scheduling annotation for the specified pods based on the instructions in this topic. If you select Global, all pods in the cluster use load-aware scheduling by default, and no additional configuration is required. tip The Effective scope parameter is displayed only for clusters of the following Kubernetes versions: Kubernetes v1.26: v1.26.15-vke.33 and later Kubernetes v1.28: v1.28.15-vke.26 and later Kubernetes v1.30: v1.30.4-vke.11 and later Other later Kubernetes versions released by VKE If load-aware scheduling is enabled for the cluster for the first time, the feature globally takes effect by default.
Advanced configuration	Specify the parameters of node resources for load-aware scheduling based on your business requirements. Node resource utilization threshold: Specify the node CPU and memory utilization. Load-aware scheduling filters nodes based on these values. Node scoring resource weight: Specify the weight for different resources when the load-aware scheduling add-on scores nodes based on resource utilization. Valid values: 1 to 10. The higher the weight, the higher the score.

Enabling the feature by calling the API

Call the CreateAddon API to install the scheduler-plugin add-on in the cluster that meets the requirements, or specify the following parameters to enable load-aware scheduling when you update the add-on configurations by calling the UpdateAddonConfig API:

Set the LoadAwareSchedulingEnabled parameter to true.

Set the Scope sub-parameter of the LoadAwareSchedulingConfig parameter to pod.

Step 2: Add an annotation that enables load-aware scheduling for a pod

Add annotations to a pod to indicate whether to enable load-aware scheduling for the pod based on the actual load of nodes.

Annotation key	Annotation sample value	Description
vke.volcengine.com/load-aware-enabled	true	Whether to enable load-aware scheduling. Valid values: true false

The following YAML file serves as an example.

Examples

This topic uses a cluster with three nodes (each with 4 cores and 16 GiB of memory) as an example. It compares node load differences before and after load-aware scheduling is used. It also introduces methods of scheduling pods to lower-load nodes for load balancing.

When load-aware scheduling is not used

Connect to the target cluster. For details, see Connecting to clusters.

Create the YAML file of the workload. The following code of the sample file stress-test.yaml shows how to create a Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
name: stress-test # The name of the Deployment.
namespace: default # The namespace of the Deployment.
labels:
app: stress-test
spec:
replicas: 2 # The number of pods.
selector:
matchLabels:
app: stress-test
template:
metadata:
name: stress-test
labels:
app: stress-test
spec:
containers:
- args:
- '--vm'
- '3'
- '--vm-bytes'
- '1800M'
- '-c'
- '3'
- '--vm-hang'
- '10'
command:
- stress
image: polinux/stress
imagePullPolicy: Always
name: stress
resources:
limits:
cpu: '3'
memory: 6Gi
requests:
cpu: '3'
memory: 6Gi
restartPolicy: Always

Execute the following command to create the workload.

kubectl apply -f stress-test.yaml

View the nodes to which the pods of the workload created in the previous step are scheduled.

kubectl get pods -o wide

The expected response is as follows, which indicates that the pods have been scheduled to the 192.168.1.5 and 192.168.1.3 nodes.

View the resource usage of nodes.

kubectl top nodes

The expected response is as follows, which indicates that the nodes 192.168.1.5 and 192.168.1.3 are high-load nodes.

When load-aware scheduling is used

The preceding result indicates that load balancing among nodes is not achieved. To schedule as many pods as possible to the 192.168.1.4 node in subsequent operations, add annotations to the pods to enable load-aware scheduling based on the following descriptions.

Create a workload with six pods with load-aware scheduling enabled and request 500 mCore of CPU for each pod.

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx # The name of the Deployment.
namespace: default # The namespace of the Deployment.
labels:
app: nginx
spec:
replicas: 6
selector:
matchLabels:
app: nginx
template:
metadata:
annotations:
vke.volcengine.com/load-aware-enabled: "true" # Enable load-aware scheduling.
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx-cantainer
image: nginx:latest # The container image address and tag of the pod.
resources:
limits:
cpu: 500m # The upper limit of the container CPU.
requests:
cpu: 500m # The container CPU request.

View the nodes on which the pods of the workload created in the previous step are scheduled.

kubectl get pods -owide

The expected response is as follows, which indicates that all six pods have been scheduled to the 192.168.1.4 node.