Docs

Volcengine Kubernetes Engine

Large-scale cluster management

Large-scale cluster limits and suggestions

In large-scale Kubernetes clusters, performance bottlenecks are primarily reflected in two aspects: the total number of resource objects the cluster can accommodate (static capacity), and the processing rate the system can sustain when performing change operations (dynamic throughput). This topic describes the static scale and request limits of VKE in detail and provides optimization suggestions. This helps you better handle the challenges posed by scale expansion.

Resource size and quantity limits

To ensure the stability and performance of large-scale clusters, carefully read the following limits and the corresponding descriptions. If these limits are exceeded, cluster performance degradation or stability issues may occur. The following table describes the general usage limits of VKE in detail. We also recommend understanding the relevant limits of open-source Kubernetes for large-scale clusters.

Category	Limitation item	Limit	Description and impact
Data volume	Total ETCD capacity	The default value is 8 GiB, and the maximum value is 16 GiB.	If this value is too high, this will affect performance, including data read/write latencies, system resource usage, and election latencies. This also leads to longer service and data recovery. The ETCD database can be up to 16 GiB in size. If this value is exceeded, the cluster will enter the read-only state.
		Total size of ETCD objects per type	800 MiB	If a type of resource object occupies too much ETCD space, clients (such as Controllers) may exert tremendous pressure on kube-apiserver when listing the resource. This may lead to timeouts or OOM.
		Single resource size	1 MiB	An oversized resource object severely impacts ETCD write performance and API Server processing efficiency. For frequently modified resources, we recommend keeping the size of a resource under 10 KiB. In ETCD, each update to a key-value pair generates a new historical version. In scenarios with frequent updates to large data objects, ETCD consumes more resources when saving historical versions of the data.
Maximum Number of Instances	Node quantity	For the Flannel cluster, the default value is 200, and the maximum value is 1000. VPC-CNI cluster: 5000	When a cluster has more than 5,000 nodes, control plane add-ons face significant pressure. The Flannel model is limited by VPC route entries, with an upper limit of 1,000 nodes. For large-scale clusters, we recommend selecting the VPC-CNI model.
		Number of pods	With NetworkPolicy enabled: 50,000 With NetworkPolicy disabled: 150,000	Excess Pods will lead to the following issues: greater ETCD storage pressure, higher API Server List/Watch operation latencies, longer resource synchronization cycle of the Controller Manager, higher scheduling decision latencies of the Scheduler, and higher load on the control plane add-ons. We recommend keeping the total number of Pods in a single cluster within 150,000.
		Pending Pod quantity	10000	When excess pending Pods exist, newly submitted Pods may remain in a waiting state for a long time, and the scheduler will periodically generate events. This may lead to event flooding.
		CiliumIdentities quantity	65535	Cilium uses identities to implement security policies. Each Pod with a unique label combination consumes one identity. When the number of identities approaches the upper limit, new Pods may fail to obtain a network identity. This causes network connectivity issues.
		HPA quantity	2000	Each HPA runs every 15 seconds. When the number of HPA objects exceeds 2,000, performance degrades linearly. This increases the load on kube-controller-manager and the API Server.
Density	Number of Pods per node	For the Flannel cluster, the default value is 64, and the maximum value is 256. For the VPC-CNI cluster, Pod density is related to ECS specifications. For details, see Calculating schedulable Pod density for nodes.	Flannel cluster: Configure custom settings based on the console guidance when you create the cluster. The settings cannot be modified after the cluster is created. VPC-CNI cluster: To run more Pods on a single node, select ECS specifications that support more ENIs.

Configuration optimization for application workloads

Category	Description	Recommendation
Control the Pod object size	For large workloads requiring frequent operations (such as scaling), keep the Pod size under 10 KiB. In Kubernetes, resource manifests are stored in the ETCD database. Each time a resource is retrieved (via List or Watch), the full manifest is transmitted. Excess data significantly increases API Server load and network bandwidth usage. This affects the overall performance of the control plane. When a cluster has a large number of Pods, the impact of the Pod size on performance is multiplied.	Avoid directly embedding large amounts of configuration data into env or args of the Pod YAML file. Use ConfigMap or Secret for mounting instead.
Control the total number of Watch connections in the cluster	For each Secret and ConfigMap referenced by a Pod, the kubelet on each node creates a corresponding Watch connection to listen for changes. The total number of Watches generated by all nodes can impose a high load on the control plane. When the number of Watches in a cluster exceeds 200,000, it may affect cluster initialization time and even cause frequent control plane restarts.	For ConfigMaps and Secrets that do not need to change, use Immutable ConfigMap to avoid creating Watches. Immutable ConfigMap example: apiVersion: v1 kind: ConfigMap metadata: ... data: ... immutable: true Use nodes with higher specifications to increase Pod density and reduce the number of nodes with the same computing power. This reduces the total number of Watches.
Prohibit percentage-based updates for DaemonSets	In large-scale clusters with thousands of nodes, batch creation or updating of DaemonSets imposes a high load on the control plane. If Pods in the DaemonSets initiate a large number of requests to the API Server during startup, more control plane resources will be consumed.	Strictly control the concurrency of rolling updates by configuring updateStrategy: Use absolute values rather than percentages for rolling update concurrency. We recommend keeping maxUnavailable no more than 10. We recommend setting minReadySeconds to 5 seconds or more to ensure new Pods are fully ready before proceeding with the next batch of updates. Configure readinessProbe and livenessProbe to ensure new Pods are marked Ready only after passing health checks. Set maxSurge to 0 to avoid resource pressure from additional scaling. Perform DaemonSet updates for large clusters during off-peak hours. The sample YAML file for updateStrategy is as follows: apiVersion: apps/v1 kind: DaemonSet metadata: ... spec: minReadySeconds: 5 updateStrategy: rollingUpdate: maxSurge: 0 maxUnavailable: 10 type: RollingUpdate
Set Job ttlSecondsAfterFinished	For a large number of Job objects created via CronJob or other mechanisms, configure the ttlSecondsAfterFinished field to automatically clean up the objects after they are complete. Without automatic cleanup, completed and failed Jobs and their Pods will continuously use ETCD storage space and control plane resources. This may lead to increased latencies of the API Server List operation, ETCD storage bloat, and exceeded namespace object count limits.	Configure ttlSecondsAfterFinished for all CronJobs and one-off Jobs. We recommend setting it to 86400 (24 hours). Regularly check and manually clean up historical Jobs that are in the Completed or Failed state and are not configured with TTL. For more configuration information, see Kubernetes documentation. The following sample code shows the YAML content. apiVersion: batch/v1 kind: Job metadata: ... spec: ttlSecondsAfterFinished: 100 template: spec: containers: - name: pi
Disable automatic ServiceAccount mounting for Pods	By default, Kubernetes automatically mounts ServiceAccount credential Secrets into Pods, and the kubelet creates a Watch for each Secret on each node to listen for changes, which imposes a high load on the API Server.	For Pods that do not need to access the API Server, set automountServiceAccountToken: false in the Pod spec. The following sample code shows the YAML content. spec: automountServiceAccountToken: false # Key configuration: disable automatic token mounting containers: - name: app image: nginx:latest
Disable automatic injection of Service environment variables into Pods	By default, Kubernetes injects environment variables (such as MY_SERVICE_SERVICE_HOST and MY_SERVICE_SERVICE_PORT) of all active Services in the namespace into each Pod. When excess services exist, Pods may crash at startup due to excess environment variables.	Set enableServiceLinks: false in the Pod spec to disable service environment variable injection. The following sample code shows the YAML content. spec: enableServiceLinks: false # Key configuration: disable Service environment variable injection containers: - name: app image: nginx:latest
Set revisionHistoryLimit to reduce the number of ReplicaSets	By default, Kubernetes retains 10 historical ReplicaSet versions. When a large number of Deployments exist in a large-scale cluster, the historical ReplicaSet versions of each Deployment are stored in ETCD. This continuously uses ETCD storage space.	Set revisionHistoryLimit of the Deployment to 3 to 5. This retains necessary rollback capability while reducing the accumulation of historical objects. For Deployments that do not need historical versions, you can set it to 0. For more configuration instructions, see Kubernetes documentation. The following sample code shows the YAML content. apiVersion: apps/v1 kind: Deployment metadata: ... spec: replicas: 5 revisionHistoryLimit: 3
Prohibit DaemonSets from performing List and Watch operations on ClusterScope resources	Avoid deploying independent controllers on each node to listen for the full cluster data. In this mode, a large number of List requests are simultaneously sent to the API Server during node startup to synchronize cluster status, and each node maintains its own Watch connection. This leads to a surge in Watch connections and request volume, and great pressure on the control plane.	Deploy controllers that listen for full data as Deployments rather than as DaemonSets on every node. For an add-on that must run on a node (such as the log collection add-on), configure FieldSelector or LabelSelector to only listen for Pods on the node.

APIServer access optimization

Category	Description	Recommendation
Access APIServer	In large-scale clusters, the way clients access kube-apiserver directly affects control plane stability. Inappropriate API call modes can cause a surge in CPU, memory, and network load on kube-apiserver and ETCD. This may even lead to control plane overload or request timeouts. VKE has built-in rate-limiting mechanisms to protect the API Server, but optimizing API Server access mode from the client side remains a core measure for ensuring cluster performance.	For details, see Kubernetes best practices for developers.

Last updated: 2026.05.26 17:42:55