Volcengine Kubernetes Engine
Volcengine Kubernetes Engine
Document
- Documentation
Volcengine Kubernetes EngineVolcengine Kubernetes EngineBest practicesSchedulingBest practices for ResourcePolicy scheduling
Copy page
Download PDFScheduling
Best practices for ResourcePolicy scheduling
Copy pageDownload PDFBest practices for ResourcePolicy scheduling
This topic describes best practices for the ResourcePolicy scheduling capability.
Applicable scenarios
ResourcePolicy scheduling allows you to specify a custom ResourcePolicy to set the order in which pods of a workload are scheduled to different types of nodes, such as subscription ECS instances, pay-as-you-go ECS instances, autoscaling ECS instances, and VCI virtual nodes. This capability is suitable for businesses that need to balance cost, elasticity, and stability across multiple resource pools. The recommended approaches for different business scenarios are as follows:
Scenario | Recommended resource pool order | Key configuration | Applicable business |
VCI and autoscaling ECS | VCI first, autoscaling ECS second |
| Batch jobs with large, bursty resource demands. |
ECS first, VCI for burst traffic | ECS node pool first, VCI second. |
| Online services with a certain amount of long-term stable traffic and occasional sudden spikes. |
ECS and VCI scheduling by ratio | Usually ECS first, VCI second. | Set maxReplicasRatio for both resource pools, and set the sum of the ratios to 1. | Gradual migration from ECS to VCI or cost structure optimization. |
Pre-checks
- The scheduler-plugin has been installed in the target cluster, and ResourcePolicy scheduling has been enabled.
- If VCI is involved, ensure that a VCI-related cluster or virtual node has been created. We also recommend enabling VCI inventory-aware scheduling.
- The target workload uses stable Labels, and the ResourcePolicy and Pod are in the same namespace.
- Pods already scheduled in the target namespace will not be mistakenly selected by the new ResourcePolicy.selector.
- Kubernetes pod-deletion-cost is not used at the same time to control the scale-in order.
Detailed parameters
A ResourcePolicy only applies to workload Pods that match the selector in its own namespace. subsets indicate the resource pool configurations. Pods will be scheduled first to the resource pool that appears earliest in the sequence. During a scale-in, the pods in the resource pool that appears last in the sequence will be scaled in first.
Parameter | Meaning | Configuration recommendations |
metadata.name | ResourcePolicy object name. | Name it to reflect the business and resource pool model, such as web-ecs-vci-ratio or batch-vci-first. This enables the purpose of the policy to be quickly identified during troubleshooting. |
metadata.namespace | The namespace to which the ResourcePolicy belongs. The policy applies only to Pods selected by the selector in the same namespace. | It must match the namespace of the target workload Pods. Do not split the policy and workload of the same business across different namespaces. |
spec.selector | Select the Pods managed by the ResourcePolicy in the form of a set of label key-value pairs. | Use business-specific and stable labels, such as app: web. Before creating, confirm that existing Pods in the namespace will not be mistakenly selected. Otherwise, existing Pods may be preferentially scaled in. |
spec.subsets | The resource pool list. Pods will be scheduled first to the resource pool that appears earliest in the sequence. When pods are being scaled in, the pods in the resource pool that appears last in the sequence will be scaled in first. | Sort by the priority of "stability, cost, elasticity". In fixed-ECS-first scenarios, ECS usually comes first and VCI second. In VCI-first batch job scenarios, VCI comes first and autoscaling ECS second. |
spec.subsets[].name | The resource pool name, which also appears in status.policyStatuses[].name. | Name the resource pool to reveal the resource type and purpose, such as pool-ecs-stable, pool-vci-burst, and pool-ecs-autoscale. |
spec.subsets[].maxReplicas | The maximum number of Pods scheduled to this resource pool. If you do not specify this parameter, the maximum number of Pods is not limited. | Estimate the value based on the available capacity, VCI quota, per-Pod requests, and budget cap of the resource pool. We recommend configuring it for VCI resource pools to avoid exhausting the VCI quota in burst traffic scenarios. |
spec.subsets[].maxReplicasRatio | The threshold for the ratio of Pods scheduled to this resource pool to all managed Pods. The value range provided by the official documentation is [0, 1]. | Suitable for proportional allocation or migration between ECS and VCI. Ratios across all resource pools are recommended to sum to 1. When the number of replicas is small, rounding errors exist. We recommend not using this parameter with whenNotReachMax: DoNotSchedule. |
spec.subsets[].whenNotReachMax | The handling policy when the resource pool has not reached maxReplicas or maxReplicasRatio but no schedulable resources are available temporarily. DoNotSchedule and ScheduleAnyWay are supported. |
|
spec.subsets[].nodeSelectorTerm | Select the set of nodes that form the resource pool via node labels. The structure is the same as Kubernetes NodeSelectorRequirement. |
|
spec.subsets[].tolerations | The tolerations that a Pod must carry to be scheduled onto this resource pool. The format follows Kubernetes pod.spec.tolerations. |
|
Scenario 1: VCI and autoscaling ECS
In this scenario, workloads preferentially use VCI. When the VCI quota or capacity is insufficient, Pods are scheduled to the ECS node pool, and additional resources are provisioned through ECS node pool autoscaling. This method is mainly suitable for workloads such as batch jobs with sudden, high resource demands, eliminating the need to maintain fixed ECS resources over the long term. When the VCI capacity is insufficient, ECS resources can be autoscaled to ensure sufficient resource supply.
Recommended configuration
- Specify the VCI virtual node as the first subset to preferentially receive Pods.
- Set maxReplicas for the VCI subset to prevent the VCI quota from being exhausted.
- Specify the ECS node pool with autoscaling enabled as the second subset.
- Set maxReplicas for the ECS subset to specify the upper limit of ECS resource usage.
- Set the whenNotReachMax of the ECS subset to DoNotSchedule. In this way, Pods briefly remain Pending when ECS resources are insufficient to trigger node pool autoscaling.
-
Verification methods
- Connect to the target cluster. For details, see Connecting to a cluster via kubectl.
- Confirm that the VCI virtual node exists, that the workload has the Toleration required for scheduling to VCI, and that the ECS node pool has autoscaling enabled.
- To observe the behavior during creation more clearly, set the ECS node pool to an empty node pool with zero nodes.
- Deploy the sample application.
-
- When VCI reaches maxReplicas or the quota is insufficient, new Pods enter the scheduling path of the ECS subset.
-
- When ECS resources are insufficient, Pods briefly remain Pending, and the node pool scales out.
- After the scale-out, Pods can enter the Running state instead of immediately entering the Failed state due to insufficient VCI inventory.
-
Risks and optimization
The main risk in this scenario is the wait time. DoNotSchedule makes Pods remain Pending before the ECS node pool scales out. Therefore, it is not suitable for online traffic that is extremely sensitive to startup latency.
Scenario 2: Schedule Pods proportionally to ECS and VCI
In this scenario, maxReplicasRatio is used to control the replica ratio between ECS and VCI. It is suitable for businesses transitioning from ECS to VCI, optimizing their cost structure, or maintaining a long-term balance between fixed resources and elastic resources.
Recommended configuration
- Specify the ECS node pool as the first subset, with a ratio such as 0.6.
- Specify the VCI virtual node as the second subset, with a ratio such as 0.4.
- Set maxReplicasRatio for both subsets. We recommend setting the sum of the ratio thresholds to 1.
- We recommend setting whenNotReachMax to ScheduleAnyWay. Ratio calculations may be inaccurate when only a few Pods exist. We recommend that you do not configure maxReplicasRatio together with whenNotReachMax: DoNotSchedule. Otherwise, some Pods may unexpectedly remain Pending.
-
Verification methods
- Connect to the target cluster. For details, see Connecting to a cluster via kubectl.
- Verify that the ResourcePolicy object meets expectations.
-
- Deploy the test workload with 10 replicas.
-
- Observe the replica distribution.
-
- During the scale-in (10 to 8), preferentially release the Pods in the resource pool that appears last in the sequence.
-
Risks and optimization
Proportional scheduling is not suitable for workloads with very few replicas. For example, when the total number of replicas is only 2 or 3, the target ratio of 0.6/0.4 will deviate due to rounding. We recommend using this mode when a large number of replicas exist and the business can accept short-term ratio deviations. During migration, you can set the ratio to 0.9/0.1 from the start, observe VCI stability and cost, and then gradually adjust the ratio to 0.7/0.3 or 0.6/0.4.
Scenario 3: ECS first, VCI for burst traffic
In this scenario, fixed ECS resources and VCI resources coexist in the cluster. Workloads are preferentially scheduled to fixed ECS resources. When ECS resources are insufficient or the maxReplicas limit is reached, VCI provides additional elastic resources. During scale-in, ResourcePolicy preferentially deletes Pods from the later VCI resource pool, releasing on-demand elastic resources first, and then releasing fixed ECS resources.
This mode applies to online services with a certain amount of long-term stable traffic and occasional sudden spikes. Compared with "VCI first + autoscaling ECS", it emphasizes fixed resource utilization and controllable costs. Compared with "ratio-based scheduling", it emphasizes resource pool order and scale-in release order rather than maintaining a fixed ratio over the long term.
Recommended configuration
- Specify the fixed ECS node pool as the first subset to preferentially handle stable baseline traffic.
- Set maxReplicas for the ECS subset. This value is recommended to be close to the number of replicas required for stable traffic.
- Specify the VCI virtual node as the second subset to handle insufficient ECS resources or sudden business traffic spikes.
- Set maxReplicas for the VCI subset to control the amount of elastic resources used based on peak traffic, the VCI quota, and the budget cap.
- Set whenNotReachMax to ScheduleAnyWay for the ECS subset so that Pods can quickly overflow to VCI when fixed ECS resources are insufficient. You can only consider DoNotSchedule if the ECS node pool has autoscaling enabled and you want to wait for the high-priority pool to scale out.
-
Verification methods
- Connect to the target cluster. For details, see Connecting to a cluster via kubectl.
- Verify that the ResourcePolicy object meets expectations.
-
- Deploy the test workload.
-
- Observe the workload replica distribution.
-
- When scaling in the workload, preferentially delete Pods on the VCI resource pool.
-
Risks and optimization
In this mode, ecs-stable.maxReplicas needs to be close to the number of replicas required for stable traffic. For example, if 20 replicas under normal conditions can handle baseline traffic, the ECS resource pool upper limit can be set to 20. vci-burst.maxReplicas needs to be set based on the peak traffic, VCI quota, and budget cap. If the fixed ECS resource pool is configured with insufficient capacity, workloads will start using VCI prematurely, potentially increasing elastic resource costs. If the VCI limit is set too high, burst traffic may quickly consume the VCI quota or budget.
Note that if you set the whenNotReachMax parameter of the ECS node pool to DoNotSchedule but autoscaling is not enabled for the ECS node pool, business Pods will remain Pending and will not be created using VCI. Therefore, if you want the fixed ECS resource pool to scale out before using VCI, enable autoscaling for the ECS node pool and carefully set whenNotReachMax to DoNotSchedule.
During rollout verification, we recommend first selecting a canary workload and performing both scale-in and scale-out drills. If existing Pods are found to be mistakenly selected by the policy, first narrow the selector scope and then re-run the canary rollout.
Rollout procedure
- Select the target workload and confirm that the Pod Labels of the Deployment are stable and will not mistakenly select other Pods.
- Select the resource pool model: fixed ECS first, VCI first with ECS as fallback, or ECS/VCI by ratio.
- Create a ResourcePolicy and limit the selector to the canary workload.
- Scale out the workload and observe whether Pods are scheduled based on resource pool order.
- Scale in the workload and verify whether Pods are preferentially released from the low-priority resource pool.
- If VCI is involved, observe inventory awareness, VCI Pod creation events, and cost metrics.
- After the canary rollout is stable, extend the rollout to similar workloads.
Risks and limitations
This feature cannot be used together with Kubernetes pod-deletion-cost. Before creating a ResourcePolicy, you also need to confirm that Pods already scheduled in the target namespace will not be mistakenly selected by the policy. Otherwise, existing Pods may be preferentially scaled in.
When an existing ResourcePolicy is modified, already scheduled Pods will be scaled in based on the original policy order, and newly created Pods will be scaled in based on the updated policy order. Therefore, before the resource pool order is adjusted, scale-out and scale-in need to be paused or controlled to prevent the same batch of Pods from being affected by both the old and new policies at the same time.
Rollback solution
- Pause scale-out and scale-in operations on the target workload.
- Change the label in the Deployment template so that it no longer matches ResourcePolicy.selector, or delete the ResourcePolicy.
- Roll out the workload again and observe whether new Pods return to normal scheduling.
- If the scale-in order deviates from expectations, scale out first to restore the number of replicas and then redesign the resource pool order.
- If the VCI inventory or quota is abnormal, first reduce the VCI maxReplicas value or ratio, and then restore the replica count of the online service.
FAQ
Why do Pods not enter VCI?
First, check whether the VCI virtual node exists, whether the Pod has the corresponding Toleration, whether ResourcePolicy.selector matches, whether the VCI subset has reached maxReplicas or maxReplicasRatio, and whether VCI inventory-aware scheduling is keeping the Pod Pending due to insufficient inventory.
Why does the actual distribution not fully match the ratio specified in the ResourcePolicy?
When the number of replicas is small, ratio calculation has rounding errors. We recommend using ratio-based scheduling when a large number of replicas exist, and avoid configuring maxReplicasRatio together with whenNotReachMax: DoNotSchedule.
Do all workloads support reverse scale-in?
Currently, only Deployments support reverse scale-in through ResourcePolicy. Other workloads do not support reverse scale-in.
Last updated: 2026.09.24 14:56:10
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
