Docs

Volcengine Kubernetes Engine

Node check and self-healing

Configuring node check and self-healing

The operation status of nodes, as the infrastructure of clusters, is critical to users. The VKE team provides the self-developed node check and self-healing feature for common failure detection and self-healing for nodes. This topic describes how to configure the node check and self-healing feature.

tip

This feature is available only to enterprises. If you cannot view the corresponding page in the console, make sure that your account has completed enterprise verification.

Prerequisites

The node-problem-detector add-on has been installed. For details, see node-problem-detector.

(Optional) A Volcengine Managed Service for Prometheus (VMP) workspace has been created. For details, see Creating common workspaces.

(Optional) To use the custom failure detection feature, make sure that you have configured custom failure check items. For details, see Custom node check items.

Procedure

Step 1: Configure a check and self-healing rule

In the left navigation bar, click Cluster.

On the Cluster page, click the name of the target cluster to go to the cluster management page.

In the left navigation pane of the cluster management page, click Check and self-healing. Then, click Create check and self-healing rules.

tip

When you configure check and self-healing rules, if multiple check items are selected and their self-healing rules differ, the system will automatically split them into multiple rules.

Parameter	Description
Rule name	Specify a name for the check and self-healing rule. The rule name must be unique in the same cluster.
Rules
Failure type	Select the type of failures to be checked. Supported failure types include GPU failures, XID exception, Node failure, System and add-on failures, and custom fault.
Check item	Select one or more items to be checked for the specified failure type. Different types of failures support different check items. For details, see Check and self-healing overview.
Event listening	When the Failure type is Node failure , and the check item is ECS failure, specific events can be selected for listening. All events: Listens to all events. Specified events: Listens to specified events. Multiple selection allowed. For event details, see Check and self-healing overview.
Failure handling approach	Select an automatic failure handling method. Valid values: Anomaly prompt: Only includes the Anomaly prompt process. Cordon nodes: Includes Anomaly prompt, Process intervention (optional), and Cordon nodes. Drain nodes: Includes Anomaly prompt, Process intervention (optional), Cordon nodes, and Node draining. Auto recovery: Includes Anomaly prompt, Process intervention (optional), Cordon nodes, Node draining, Auto recovery, and Uncordon (optional). tip When Failure type is set to custom fault, only the Auto recovery failure handling approach is supported.

Configure the troubleshooting process.

Anomaly prompts: When a node fails, the system converts failure information into metrics. You can use the cloud-native observability feature to obtain metrics and view them in the VMP console.

Process intervention: After you enable this feature, the system will add the specified labels to nodes after the preceding steps are complete, to identify failed nodes. You can filter specified nodes by label to perform manual intervention and troubleshooting.

Cordon nodes: When a node fails, the NoSchedule taint will be added to the node to prevent new pods from being scheduled to the node.

Node draining: When a node fails, add the NoExecute taint to the node and evict all pods on it.

Parameter

Description

Prohibit scheduling

Specify the time period before a node starts evicting pods after the node is cordoned. Multiple time periods are available, such as immediately, 15 minutes, 30 minutes, 1 hour, 2 hours, 3 hours, 6 hours, 12 hours, 24 hours, 36 hours, 2 days, 3 days, 4 days, 5 days, 6 days, and 7 days.

warning

Before node draining, the system checks whether one of the following workloads is running on the nodes. You need to make sure that these workloads do not exist on the nodes. Otherwise, node draining will not be performed.

ingress-controller Pods

Workloads that are strongly bound to the nodes

Single-replica workloads

Static pods

Node draining does not guarantee the termination or eviction of workloads on a node. If a workload cannot be evicted and no manual intervention is performed, the next self-healing operation will not be performed until the node is forcibly restarted at the specified time.

Auto recovery: When you enable this feature, the system will attempt to fix node failures to achieve self-healing after node draining.

Parameter	Description
Reboot ECS nodes if necessary	Automatic fixing operations vary by failure type. If you select this option, the node will be automatically restarted regardless of whether it is drained or not. The node will not be automatically restarted unless it is automatically fixed. Specify the waiting period before a node restarts after it has been drained, such as immediately, 30 minutes, 1 hour, 2 hours, 3 hours, and 6 hours.

Uncordon: Specify whether to automatically uncordon a node after the node restarts and its failure is fixed. When the feature is enabled, the system will perform health checks again after the node restarts. If all check items pass the checks, the node will be uncordoned.

warning

The system can only make sure that no exceptions occur in the existing check items. If a failure occurs beyond the scope of node check items, the operation of training or inference tasks may become abnormal. If issues persist after the self-healing process is complete, submit a ticket to obtain technical support.

(Optional) Click Add rules to add multiple rules.

After the configuration is complete, carefully read Considerations, select I acknowledge and consent to, and then click Confirm to save the configurations.

Step 2: Associate a node pool with the check and self-healing rule

You can associate the check and self-healing rule with a node pool when creating it, or with an existing node pool. This topic provides an example of how to associate the check and self-healing rule with a node pool when you create the node pool.

tip

Only managed node pools can be associated with check and self-healing rules. If your node pool is a normal node pool, you need to enable hosting for the node pool first and then associate a check and self-healing rule with the node pool.

In the left navigation bar of the cluster management page, click Nodes > Node pool. On the page that appears, click Create node pool. In the pop-up dialog box, select Managed mode pool.

When configuring node pool details, enable the Check and self-healing feature under Operation and maintenance configuration and bind self-healing rules.

Parameter	Description
Check and self-healing	Specify whether to enable the node check and self-healing feature. Enable: After you configure node self-healing rules, the system will automatically check node status and promptly handle node failures to recover nodes. Disable: Does not enable this feature. This is the default setting.
Check and self-healing rules	Configure a check and self-healing rule. Valid values: Recommendation rule: Use the system default recommended rule. For details, see Recommended rules. Custom rule: To use your custom rules, select a configured check and self-healing rule from the drop-down list.

tip

For more configurations of node pools, see Creating node pools.

Click Confirm to complete the binding.

Step 3: (Optional) Enable cloud-native observability

If you need to use node failure metrics to obtain node status, enable the cloud-native observability feature. The steps are as follows:

Bind the cluster to a VMP workspace, install the prometheus-agent add-on, and enable cloud-native observability. For details, see Enabling observability.

Enable check and self-healing observability. For details, see Check and self-healing observability.

Validate results

Viewing the check and self-healing rule

In the left navigation pane of the cluster management page, click Health check & auto recovery.

In the self-healing rule list, click the name of the target self-healing rule to view its details.

Viewing self-healing task details

After self-healing is complete for a failed node, you can view the self-healing task list and details on the Node details page.

In the left navigation bar of the cluster management page, click Nodes > Node.

Click the name of the target node to go to the Node details page.

Click the Self-healing from failure tab to view the list of failure self-healing tasks for the current node.

Click the name of the target self-healing task to view the self-healing details for the current node.

Viewing check and self-healing observability data

You can view check and self-healing observability data on the cloud-native observability dashboard and configure the corresponding alerts. For details, see Check and self-healing observability.

Viewing alert rule details

After you configure check and self-healing alerts, you can view the details of the corresponding alert rule in the VMP console.

Click Alert Center > Alert rule in the left navigation bar. Click the Alert rule group tab. On this tab, you can view the configured check and self-healing alert rule groups.

Click the name of the target alert rule group to go to its details page. Then, click the Alert rule list tab to view the alert rule details.

Other operations

Terminating the self-healing process

During the execution of a check and self-healing task, you can manually terminate the self-healing process at any time.

warning

After the self-healing process is terminated, the step in progress will continue until completion, but subsequent steps will not be performed. You need to manually resolve the remaining issues. No troubleshooting will be performed on the node again until it is uncordoned.

In the left navigation bar of the cluster management page, click Nodes > Node.

Click the name of the target node to go to the Node details page.

Click the Self-healing from failure tab and then find the target self-healing task in the failure self-healing task list.

Click the name of the self-healing task to go to its details page. Click Stop failure processing to terminate the self-healing task.

Last updated: 2026.07.16 16:23:16