Docs

Volcengine Kubernetes Engine

Other observation

Check and self-healing observability

Check and self-healing observability allows you to monitor node pool check and self-healing in the cluster. This observability enables you to monitor nodes for kubelet, runtime, and GPU failures based on metrics and receive alerts. This topic describes how to configure check and self-healing observability.

tip

For details on self-healing, see Configuring node check and self-healing.

Prerequisites

Cloud-native observability has been enabled. For details, see Enabling observability.

The VKE observability feature has been enabled. For details, see VKE observability.

(Optional) An alert notification policy has been created in the Observability Platform or VMP. For details, see Creating notification policies and Creating alert notification policies.

(Optional) An alert aggregation policy has been created in the Observability Platform or VMP. For details, see Creating aggregation policies and Creating alert aggregation policies.

Enabling observability

Step 1: Configure observability items

In the left navigation bar, click Cluster. Find the target cluster and click its name.

In the left navigation pane of the cluster management page, click Observability settings, and select the Basic observability tab.

Find the AIOps remediation card and click Enable observation to go to the observability item configuration page.

The system automatically checks the necessary conditions for enabling observability, such as the workspace configuration and add-on status.

Click Next to go to the alert and notification configuration page.

Step 2: Configure alerts and notifications

In the Alert and notification step, select the platform to receive alerts. Currently, Unified Alert Center and The original VMP alarm strategy are supported.

For the Notification strategy name parameter, select the created alert notification policy and specify the notification object and Notification method parameters based on different alert levels.

tip

If you do not need to configure alerts for a certain level, you can unselect the corresponding alert level.

If the existing notification policies do not meet your needs, you can select vke_default_notify_policy (automatically created) for the Notification strategy name parameter, and specify the notification object and Notification method parameters for the required alert levels. After you save the configuration, the system will automatically create a custom notification policy named vke_default_notify_policy in the unified Alert Center or VMP.

In the Alerting aggregation policy section, select the created alert aggregation policy.

Configure preset alert rules.

Enable or disable the switch next to a preset alert rule. After it is enabled, the system detects metrics based on the configured trigger conditions and generates alerts. When it is disabled, the rule will no longer trigger new alerts.

Click View PromQL to view the description and corresponding PromQL statement of the current preset alert rule.

Click the name of the preset rule to be configured to expand the preset alert rule configuration page, and configure the trigger conditions for different levels (P0, P1, or P2) of alerts in the Triggering condition section.

Parameter	Description
Alert level	Select the check box in the Enable column that corresponds to each alert level (P0, P1, or P2) to activate its trigger condition. If you do not select an option, alerts at this level are not configured.
Comparison condition	Specify the comparison condition between the PromQL statement and the threshold. You can configure multiple comparison conditions, including Greater than, Greater than or equal to, Less than, Less than or equal to, Equal to, and Not equal to.
Threshold	Configure the threshold of the metric.
The duration of the condition	Configure the duration for which the metric continuously meets the trigger condition. The system triggers the alert at the corresponding level only if the metric continuously meets the comparison condition within this duration.
Label	Configure the label of the alert rule. It is mainly used for alert identification, grouping, deduplication, and routing matching. A label consists of a key and a value. Where: Key: A string. It can start with a letter or an underscore (_), but cannot start with a double underscore (__). It can contain letters, digits, and underscores (_) and can be 1 to 256 characters in length. Value: A string. It can contain Chinese characters, English letters, digits, and special characters, and can be 1 to 512 characters in length. Common configuration example: labels: severity: critical # The alert level. business: pay # The business line. alert_type: resource # The alert type.
Annotation	Configure the annotation of the alert rule. It is mainly used to display key information. An annotation consists of a key and a value. Where: Key: A string. It can start with a letter or an underscore (_), but cannot start with a double underscore (__). It can contain letters, digits, and underscores (_) and can be 1 to 256 characters in length. Value: A string. It can contain Chinese characters, English letters, digits, and special characters, and can be 1 to 512 characters in length. Common configuration example: annotations: summary: "High CPU usage on instance {{ $labels.instance }}" description: "Current CPU usage is {{ $value \| humanizePercentage }}, threshold is 80%" runbook_url: "https://xxx/docs/cpu-alert" dashboard_url: "https://grafana/xxx"

Click Complete the configuration.

Observability dashboards

tip

The built-in dashboards provide features such as period-over-period comparisons of monitoring data and the ability to copy dashboard queries. For details, see Usage of dashboards.

You can view monitoring information related to node check and self-healing. This includes the number of abnormal nodes, number of nodes with GPU abnormalities, number of self-healing actions, number of successful self-healing actions, and more. You can specify a custom time range for querying information and set the information refresh mode to manual refresh or automatic refresh.

In the left navigation pane of the cluster management page, click Monitoring center > Dashboard.

In the left dashboard list, click AIOps > Check and self-healing to view the monitoring dashboard.

View metrics

To query integrated or non-integrated metrics in the cluster, or to understand metric meanings, see Querying metrics.

You can use the Explore feature of VMP to quickly query and display metric data. For details, see Querying metrics.

Last updated: 2026.10.22 17:24:38