Docs

Volcengine Kubernetes Engine

Clusters

Kubernetes best practices for developers

This topic provides recommendations and best practices for accessing control-plane components and optimizing client access modes in VKE.

Using the list-watch mode instead of full listing

Background information

As a Kubernetes developer, you may need your components to:

Periodically query lists of certain Kubernetes objects.

Run on multiple nodes (especially those components deployed as DaemonSets).

Typically, object status is queried by periodically calling the List API. Each call to this method requires loading all objects into kube-apiserver memory for serialization and transmission. These requests consume large amounts of CPU and memory on the control plane, leading to control plane overload or request timeouts. Even if the status of periodically queried Kubernetes objects hasn't changed, this still places high load on the kube-apiserver.

Recommended solution

We recommend replacing the full listing approach with the list-watch mode, in which the list API is first called to load all objects and then the watch API is called to obtain incremental object status changes. Compared with full listing, the list-watch mode can effectively reduce the time that the kube-apiserver spends in processing requests and decrease request traffic. If object status does not change, no additional load is generated.

To implement the list-watch mode in Golang, see SharedInformer and SharedInformerFactory for relevant Golang packages.

Supplementary description

Whether you use the full listing approach or the list-watch mode, a recommended practice is to set the ResourceVersion parameter (ResourceVersion=0) in listing requests to read data from the kube-apiserver cache. This reduces the number of interactions between kube-apiserver and the etcd database.

tip

For details about the ResourceVersion parameter, see the official Kubernetes documentation.

The kube-apiserver in VKE has built-in rate limiting rules to prevent out-of-memory (OOM) errors caused by a large number of listing requests. However, client-go automatically makes retries after it receives a rate limiting response. So in general, your listing requests will not fail.

Reducing unnecessary traffic generated by watch and list calls

Kubernetes internally uses the watch API to send notifications about object updates. Although watch calls require far fewer resources than periodic list calls, watch calls in large clusters can still consume a significant amount of cluster resources and impact cluster performance. The greatest impact comes from creating watch calls to watch frequently changing objects from multiple locations, such as watching pod data corresponding to add-ons running on multiple nodes. Third-party add-ons or extensions installed in the cluster may create such watch calls in the background.

We recommend the following best practices:

Reduce unnecessary processing and traffic generated by watch and list API calls.

Avoid creating watch calls from multiple nodes to watch frequently changing objects (for example, DaemonSet).

We recommend creating a central controller on a single node to implement monitoring and processing of object data.

Perform watch operations only on necessary objects, and avoid watching all objects in the cluster. For example, kubelet on each node should only watch pods scheduled to the same node.

If you use the controller-runtime library, you can refer to the official Kubernetes documentation to set watch filters. The following code provides an example:

ctrl.Options.NewCache = cache.BuilderWithOptions(cache.Options{
SelectorsByObject: cache.SelectorsByObject{
&corev1.Pod{}: {
Field: fields.SelectorFromSet(fields.Set{"spec.nodeName": "node01"}),
},
},
},
)

If you use the client-go library directly, you can set a watch filter when constructing the Informer. The following code provides an example:

lw := cache.NewListWatchFromClient(c.CoreV1().RESTClient(), "pods", metav1.NamespaceAll, fields.OneTermEqualSelector("spec.nodeName", string(nodeName)))
informer := cache.NewSharedInformer(lw, &apiv1.Pod{}, resyncPeriod)

Avoid deploying third-party add-ons or extensions that require a large number of watch or list calls, which may impact cluster performance.

Disabling automatic mounting of API access credentials

In Kubernetes versions earlier than 1.22, if you do not specify a ServiceAccount when creating a pod, Kubernetes will automatically perform the following actions:

Allocate the default ServiceAccount to the pod.

Mount the API access credentials of the ServiceAccount as a secret for the pod.

For each mounted Secret, kubelet creates a watch to monitor changes to that Secret on each node.

In large-scale clusters, the above actions mean thousands of unnecessary watch calls, which may generate a significant load on the kube-apiserver.

Therefore, if the code running in the pod does not need to access the Kubernetes API, it is recommended to disable automatic mounting of the API access credentials of the ServiceAccount to avoid creating related Secrets and watch calls. For a detailed explanation, see Opt out of API credential automounting.

Using Protobuf for API data serialization

We recommend using Protobuf (Protocol Buffers) as the mechanism for Kubernetes API serialization. Compared with JSON, Protobuf is more memory- and traffic-efficient. For details, see Alternate representations of resources.

Whether you use the controller-runtime library or the client-go library, you can set the encoding format in KubeConfig. The following code provides an example:

Last updated: 2026.10.23 10:41:06