Docs

Volcengine Kubernetes Engine

Clusters

Best practices for cluster upgrades

This topic mainly describes recommended operations and considerations for each phase of a cluster upgrade.

Scenario introduction

Kubernetes is a powerful container orchestration platform that is widely used for deploying and managing cloud-native applications. With the rapid development of Kubernetes, new versions are continuously released, which offer more features and better performance. VKE guarantees the stable operation of the latest three minor versions of Kubernetes. Clusters running expired versions may encounter security and stability risks. Therefore, to ensure the continuity of your business, we recommend that you upgrade your cluster version in a timely manner. This topic will introduce recommended operations and considerations related to the upgrades of the Kubernetes version. This helps you upgrade the Kubernetes version of your cluster or node pool to the latest version in a secure and efficient manner. For more information about upgrade policies and cluster upgrade principles, see Kubernetes version support policy and Upgrading clusters.

Prerequisites

The cluster is in the Running state, and an upgradable Kubernetes version is available.

Considerations

During the upgrade of the Kubernetes version for the cluster control plane, applications in the cluster are not interrupted. However, if the applications strongly depend on the API server, they may be affected for a short period of time.

Upgrading the Kubernetes version of a node can affect business pods running on the node. During the upgrade, pod probers and lifecycle hooks may fail, and in-place restarts of pods may occur. Therefore, before the upgrade, you need to check whether the cluster is in a healthy state and whether its business operations are performed during off-peak hours. This helps you determine whether to start upgrading nodes.

You cannot roll back the upgrade of the cluster. Proceed with caution.

If you have multiple environments (test, pre-release, production), first verify the upgrade feature in the test environment. After validating the feature in all environments, upgrade your production cluster.

When you consecutively upgrade the minor version of Kubernetes for the same cluster, we recommend that you perform an upgrade at an interval of more than 24 hours to ensure stable cluster operation. For example, after you upgrade the cluster control plane from Kubernetes v1.20 to v1.24 and the cluster has stably run for 24 hours, upgrade the cluster to v1.26.

For more considerations during cluster upgrade, see Upgrading clusters.

Preparation

Determine the upgrade time window

Select off-peak hours based on the cluster size and business characteristics, and determine the upgrade time window based on the following descriptions:

Upgrade the control plane first and then upgrade nodes based on your business requirements.

The control plane upgrade takes 5 to 10 minutes.

When you select multiple nodes for simultaneous upgrade, the system automatically upgrades the nodes in batches based on a specific concurrency level. The upgrade of a single node or a single batch of nodes takes about 3 minutes.

After the upgrade passes verification in the test or pre-release environment, a two-hour time window will be reserved for production cluster upgrades. The window includes one hour for the upgrade and one hour for observation.

Making sure that business workloads are distributed

When you upgrade a node, the containerd and kubelet add-ons will restart. In this case, the containers of the workloads that are configured with readiness probes and liveness probes in the cluster may be in the NotReady state during this period. We recommend that you configure multiple replicas for a workload and configure topology spread constraints or anti-affinity for pods. In this way, multiple replicas are distributed across different nodes, and these nodes are upgraded in batches.

This topic takes the ingress-nginx add-on as an example. After you install this add-on, it will be deployed as a Deployment named ingress-nginx-controller in the kube-system namespace in the cluster. For the method of installing add-ons, see Installing add-ons.

Configure topology spread constraints in the spec.template.spec field of the corresponding YAML file for ingress-nginx-controller. In this way, when the number of nodes meets the requirements, pods are distributed across nodes in different availability zones as much as possible. The sample code snippet is as follows:

Configuring monitoring and alerting

Deploy a monitoring and alerting add-on in the cluster. For example, install the Prometheus-agent monitoring add-on and configure the workspace. In this way, you can identify whether exceptions occur in important metrics in a timely manner during the add-on upgrade process. For more information about cluster monitoring, see Cluster monitoring.

Enabling cluster auditing

Enable the cluster auditing feature in the VKE console to record the daily operations of users in the cluster by using the API server audit logs. This helps cluster administrators perform security audits of user operations in the cluster and ensures that the cluster runs securely and stably. For more information about cluster auditing, see Cluster auditing.

Upgrading the control plane

This topic uses the scenario of upgrading the Kubernetes version of a cluster from v1.20 to v1.24 as an example to introduce the procedure and considerations for upgrading the cluster control plane and nodes.

Step 1: Confirm the cluster version

In the left navigation bar, click Cluster.

On the Cluster page, click the name of the target cluster.

On the Overview page of the target cluster, confirm the current Kubernetes version (control plane version) of the cluster: v1.20.15-vke.30 .

Step 2: Perform control plane upgrade check

In the left navigation bar of the cluster management page, click Cluster upgrade.

On the Cluster upgrade page, click the Kubernetes upgrade check tab to run the pre-upgrade check.

Click Control plane upgrade check and confirm that the target version for this upgrade is the minor version v1.24.15-vke.26 .

Click Confirm to start the pre-upgrade check for the control plane Kubernetes version.

Based on the control plane upgrade check result, fix failed check items. Fix or skip warning items as appropriate. For more descriptions and fix suggestions for upgrade check items, see Cluster upgrade check items.

The abnormal check items in the control plane check report in this example are as follows:

Addon component compatibility: The version of the ingress-nginx add-on in the cluster is outdated. You need to upgrade the add-on before upgrading the cluster. The check result of this check item is Not passed. You need to resolve the issue to ensure the check item passes the check.

Deprecated k8s API Resource Check: Deprecated Kubernetes API resources exist in the cluster. You need to handle these objects. This check item returns a Warning. Determine whether to skip this check item based on your business requirements during the formal upgrade process.

Resolve issues in the upgrade check results. After resolving the issues, perform the control plane upgrade check again until all check items pass.

Upgrade the ingress-nginx add-on to the latest version. For detailed operations, see Upgrading add-ons.

Identify which services in the cluster use deprecated Kubernetes API resources, and assess whether the cluster or your business is affected. If the cluster or your business is not affected, you can ignore this warning item. Otherwise, upgrade the Kubernetes resources in the affected services to a later version. For detailed descriptions of resource changes and deprecations, see Kubernetes version release notes.

Click Control plane upgrade check again, confirm the upgrade version information, and then click Confirm to perform the control plane upgrade check.

Step 3: Perform control plane upgrade

On the Cluster upgrade page, click the Kubernetes upgrade tab. Then, click Control plane upgrade.

In the Control plane Kubernetes upgrade dialog box, check the upgrade information.

Click Confirm to upgrade the control plane.

During the upgrade process, do not perform any operations on the cluster. You can click View details in the Actions column to view the control plane Kubernetes upgrade progress. Monitor business metrics and pods in the cluster to identify unexpected exceptions.

When the task is in the Pass state on the cluster upgrade page, go to the Overview page of the cluster to confirm that the cluster is in the Running state and the Kubernetes version is the target v1.24.15-vke.26 version. The Kubernetes version of nodes has not been upgraded and remains the v1.20.15-vke.30 version. In this case, the cluster control plane is upgraded.

Upgrading nodes

Step 1: Select nodes to be upgraded

When you select nodes to be upgraded, for pods that are sensitive to traffic fluctuations, divide the nodes that host these pods into different batches for upgrade.

In this example, the pods of ingress-nginx-controller have been distributed across different nodes. Therefore, when you select nodes to be upgraded, the two nodes hosting these pods are divided into two batches for upgrade. node-xptzps that corresponds to the IP address 192.168.4.7 is the first batch to be upgraded.

The remaining node is designated as the second batch for upgrade.

Step 2: Perform node upgrade check

On the Cluster upgrade page, click the Kubernetes upgrade check tab. Then, click Node upgrade check.

On the Node upgrade check page, select the target version for upgrade. Select nodes or node pools to be upgraded. Click Confirm to start the node upgrade pre-check.

In this example, select node-xptzps that corresponds to the node IP address 192.168.4.7 for the node upgrade check.

After you confirm that node-xptzps has passed the node upgrade check, select the remaining nodes in batches for the node upgrade check.

After the remaining nodes pass the node upgrade check, you can upgrade the nodes. If any check items have a Fail or Warning result, click View details in the Actions column. Resolve failed items following the on-screen instructions. After fixing issues, perform the node upgrade check again. Ensure Pass shows in the Check status column before upgrading. For more details and fixes, see Cluster upgrade check items.

Step 3: Perform node upgrade

On the Cluster upgrade page, click the Kubernetes upgrade tab.

Click Node upgrade. On the Node Kubernetes upgrade page, select the upgradable nodes and then click Confirm to upgrade the nodes.

In this example, first select node-xptzps that corresponds to the node IP address 192.168.4.7 for the node upgrade check.

Click Confirm to upgrade the node-xptzps node.

During the upgrade process, do not perform any operations on the cluster and nodes. View details in the Actions column to view the node Kubernetes upgrade progress. Monitor business metrics and pods in the cluster to identify exceptions.

When the upgrade state Pass is displayed on the Node Kubernetes upgrade progress page, go to the corresponding node details page to confirm that the node is in the Running state, the node Kubelet version has been upgraded to the target version v1.24.15-vke.26, and no exceptions occur in existing pods. This indicates that the current node is upgraded.

Select the remaining nodes in batches for upgrade based on the planned node upgrade schedule.

On the Kubernetes upgrade tab, click View details in the Actions column that corresponds to the upgrade task. On the details page of the upgrade task, you can observe that the selected nodes are being upgraded in batches. When the upgrade status of all nodes is Pass, it indicates that all nodes have been upgraded.

Performing post-upgrade checks

In the left navigation bar of the cluster management page, click Overview.

In the Basic information section, verify that the cluster is in the Running state, and both the Kubernetes version and node version are the target v1.24.15-vke.26 version.

Monitor workload metrics in the cluster to check whether obvious exceptions occur. For detailed operations, see Monitoring management.

Create and delete test pods. If pods can be scheduled and terminated as expected, this means the cluster and nodes are upgraded, and all cluster services run as expected.

(Optional) If exceptions occur in your services and can't be automatically resolved through Observability tips, events, logs, or other descriptions, you can submit a ticket to get technical support.

Last updated: 2026.07.23 11:53:57