Docs

Volcengine Kubernetes Engine

Metric alerts

Preset alert templates

Preset alert templates are used to monitor the running status of the VKE cluster control plane, workloads, nodes, and related add-ons. This topic describes the trigger conditions, PromQL configurations, rule descriptions, and alert handling procedures for each preset alert rule.

VKE control plane add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
APIServer exception	P0: Trigger immediately when the value is == 0	sum(up{cluster=~"$Cluster", job="kubernetes-apiserver"}) by (cluster)	An alert is triggered when no APIServer instances are available. This is a high-priority failure of the cluster control plane.	Check whether the control plane add-on status and cluster access are affected. Check whether all API requests fail, and contact the platform team for processing.
Scheduler exception	P0: Value == 0 for 5 minutes	sum(up{cluster=~"$Cluster", job="kube-scheduler"}) by (cluster)	An alert is triggered when no Scheduler instances are available. This may affect Pod scheduling.	Check whether a large number of Pods are in the Pending state. Check the scheduling add-on health and control plane events.
ControllerManager exception	P0: Value == 0 for 5 minutes	sum(up{cluster=~"$Cluster", job="kube-controller-manager"}) by (cluster)	An alert is triggered when no Controller Manager instances are available. This may affect replica control and controller convergence.	Check controller-related events and Deployment/StatefulSet convergence status. Contact the platform team for troubleshooting.
Etcd leader election failure	P0: Value == 0 for 5 minutes	etcd_server_has_leader{cluster=~"$Cluster"}	When etcd has no Leader, control plane reads and writes may be abnormal.	Check whether exceptions occur in API requests, node heartbeats, and resource updates. Investigate the etcd cluster status and escalate the issue as needed.
High duration of APIServer requests	P1: Value > 2 for 10 minutes P2: Value > 2 for 5 minutes	histogram_quantile(0.90, sum(rate(apiserver_request_duration_seconds_bucket{cluster=~"$Cluster", verb!~"WATCH\|CONNECT",resource!=""}[5m])) by (cluster, instance, le, verb))	The APIServer P90 request latency is consistently high.	Analyze whether sudden traffic spikes, slow resource objects, or backend storage jitter exist. Identify high-frequency callers and check the health of the control plane.
High APIServer request error rate	P1: Value > 0.1 for 10 minutes P2: Value > 0.1 for 5 minutes	sum(rate(apiserver_request_total{cluster=~"$Cluster", code=~"50.*"}[5m])) by (cluster) / sum(rate(apiserver_request_total{cluster=~"$Cluster"}[5m])) by (cluster)	The APIServer 5xx error rate exceeds 10%.	Check request error distribution, resource types, and call sources. Perform root cause analysis in conjunction with etcd, webhook, and controller status.
APIServer client certificate expires within 1 day	P0: Trigger immediately when the value is > 0	sum(increase(apiserver_client_certificate_expiration_seconds_bucket{cluster=~"$Cluster", job="kubernetes-apiserver", le="86400"}[5m])) by (cluster)	The API Server has received a client certificate with a remaining validity period of no more than 1 day within the last 5 minutes.	Identify the cluster that triggered the alert, and revoke the Kubeconfig and create a new Kubeconfig via the console or OpenAPI.
APIServer client certificate expires within 7 days	P1: Trigger immediately when the value is > 0	clamp_min(sum(increase(apiserver_client_certificate_expiration_seconds_bucket{cluster=~"$Cluster", job="kubernetes-apiserver", le="604800"}[5m])) by (cluster) - sum(increase(apiserver_client_certificate_expiration_seconds_bucket{cluster=~"$Cluster", job="kubernetes-apiserver", le="86400"}[5m])) by (cluster), 0)	The API Server has received a client certificate with a remaining validity period of more than 1 day and no more than 7 days within the last 5 minutes.	Identify the cluster that triggered the alert, create a certificate renewal plan, and revoke the Kubeconfig and create a new Kubeconfig through the console or OpenAPI before the certificate expires.
APIServer client certificate expires within 30 days	P2: Trigger immediately when the value is > 0	clamp_min(sum(increase(apiserver_client_certificate_expiration_seconds_bucket{cluster=~"$Cluster", job="kubernetes-apiserver", le="2.592e+06"}[5m])) by (cluster) - sum(increase(apiserver_client_certificate_expiration_seconds_bucket{cluster=~"$Cluster", job="kubernetes-apiserver", le="604800"}[5m])) by (cluster), 0)	The APIServer has received client certificates with a remaining validity greater than 7 days and no more than 30 days in the last 5 minutes.	Identify the cluster that triggered the alert, create a certificate renewal plan, and revoke the Kubeconfig and create a new Kubeconfig through the console or OpenAPI before the certificate expires.
High P99 latency in Admission Webhook	P2: Value > 1.5 for 10 minutes	histogram_quantile( 0.99, sum by(cluster, name, type, le) ( rate( apiserver_admission_webhook_admission_duration_seconds_bucket{ cluster=~"$Cluster" }[5m] ) ) ) and on(cluster, name, type) ( sum by(cluster, name, type) ( increase( apiserver_admission_webhook_admission_duration_seconds_count{ cluster=~"$Cluster" }[5m] ) ) >= 20 )	Monitors the P99 processing latency of Admission Webhook requests to identify Webhooks with persistently slow responses.	Identify the Webhook by name and type. Check the Pod status, resource utilization, and network latency of the corresponding service, and optimize the processing logic or perform scale-out.
APIServer APF request rejected	P1: Value >= 5 for 5 minutes	sum by(cluster, priority_level, reason) ( increase( apiserver_flowcontrol_rejected_requests_total{ cluster=~"$Cluster" }[5m] ) )	Monitors the number of requests rejected by API Priority and Fairness (APF) to identify request failures caused by throttling.	Identify rejected requests by priority_level and reason. Check the APF configuration and control plane load, and adjust throttling configurations or reduce request pressure.
APIServer write request throttled	P2: Value > 0 for 5 minutes	sum by(cluster) ( rate( apiserver_dropped_requests_total{ cluster=~"$Cluster", request_kind="mutating" }[1m] ) )	Monitors write requests dropped due to concurrency throttling on the APIServer.	Check write request traffic and control plane resource utilization. Identify high-frequency callers and reduce request frequency, or contact the platform team to adjust the upper limit of concurrent requests.
APIServer read-only request throttled	P2: Value > 0 for 5 minutes	sum by(cluster) ( rate( apiserver_dropped_requests_total{ cluster=~"$Cluster", request_kind="readOnly" }[1m] ) )	Monitors read-only requests dropped due to concurrency throttling on the APIServer.	Check read-only request traffic and control plane resource utilization. Identify high-frequency callers and reduce request frequency, or contact the platform team to adjust the concurrent request limit.
Using Kubernetes APIs to be removed	P2: == 1 for 10m	apiserver_requested_deprecated_apis{ cluster=~"$Cluster" }	Monitors cluster calls to APIs that will be removed in subsequent Kubernetes versions.	Identify deprecated APIs by group, version, resource, and removed_release. Upgrade related clients or modify resource manifests to ensure supported APIs are used after the upgrade.
High etcd DB capacity utilization	P1: Value > 0.9 for 10 minutes P2: Value > 0.8 for 10 minutes	max by(cluster, instance) ( etcd_mvcc_db_total_size_in_bytes{cluster=~"$Cluster"} ) / on(cluster) group_left() max by(cluster) ( vke_cluster_level_threshold{ cluster=~"$Cluster", type="cluster_db_size" } )	Monitors the ratio of the etcd database physical size to the capacity threshold.	Check the data volume and growth trends of various resource objects. Clean up unused objects and perform compaction or defragmentation during off-peak hours.
Persistent etcd proposal backlog	P1: Value > 5 for 10 minutes	max by(cluster, instance) ( etcd_server_proposals_pending{ cluster=~"$Cluster" } )	Monitors the number of pending proposals in etcd to identify persistent request processing backlogs.	Check the etcd Leader and member status. Investigate inter-member network, disk latency, and instance load, and contact the platform team if necessary.
Excessive data for a single resource type in etcd	P1: Value > 2147483648 for 30 minutes P2: Value > 838860800 for 30 minutes	vke_cluster_etcd_size_total{ cluster=~"$Cluster", resource!="global" }	Monitors the data space occupied by a single Kubernetes resource type in etcd.	Identify resource types occupying a large amount of space by resource. Investigate the reasons for object growth and clean up useless or abnormally inflated data.
Excessive objects of a single resource type in etcd	P1: Value > 100000 for 30 minutes P2: Value > 50000 for 30 minutes	vke_cluster_etcd_count_total{ cluster=~"$Cluster", resource!="global" }	Monitors the number of objects of a single Kubernetes resource type in etcd.	Identify resource types with a large number of objects by resource. Investigate abnormal creation behavior and clean up unused or residual objects.
High etcd fragmentation rate	P2: Value > 0.5 for 30 minutes	( 1 - max by(cluster, instance) ( etcd_mvcc_db_total_size_in_use_in_bytes{cluster=~"$Cluster"} ) / clamp_min( max by(cluster, instance) ( etcd_mvcc_db_total_size_in_bytes{cluster=~"$Cluster"} ), 1 ) ) and on(cluster, instance) ( max by(cluster, instance) ( etcd_mvcc_db_total_size_in_bytes{cluster=~"$Cluster"} ) > 1073741824 )	Monitors the fragmentation rate of the Etcd database to identify decreased physical space utilization.	Confirm the database size and fragmentation rate. Check the business load and contact the platform team to perform defragmentation during off-peak hours.
High P99 latency in ControllerManager work queues	P2: Value > 30 for 10 minutes	histogram_quantile( 0.99, sum by(cluster, name, le) ( rate( workqueue_work_duration_seconds_bucket{ cluster=~"$Cluster", job="kube-controller-manager" }[5m] ) ) )	Monitors the P99 latency of kube-controller-manager work queue items.	Identify the slow controller by name. Check related resource events, dependent services, and controller logs to confirm whether resources converge as expected.
High ControllerManager work queue depth	P2: Value > 10000 for 10 minutes	max by(cluster, name) ( workqueue_depth{ cluster=~"$Cluster", job="kube-controller-manager" } )	Monitors the kube-controller-manager work queue depth to identify continuous task backlog.	Identify the backlogged controller by name. Check controller logs, dependent services, and resource convergence, and contact the platform team.
High Scheduler scheduling P99 latency	P2: Value > 5 for 10 minutes	histogram_quantile( 0.99, sum by(cluster, profile, result, le) ( rate( scheduler_scheduling_attempt_duration_seconds_bucket{ cluster=~"$Cluster" }[5m] ) ) ) and on(cluster, profile, result) ( sum by(cluster, profile, result) ( increase( scheduler_scheduling_attempt_duration_seconds_count{ cluster=~"$Cluster" }[5m] ) ) >= 20 )	Monitors the P99 latency of Scheduler scheduling attempts to identify persistently increasing scheduling latencies.	Locate slow scheduling requests by profile and result. Check the scheduler load, plugin latency, and cluster resource status, and investigate a large number of Pending Pods.
High CPU utilization of control plane add-ons	P1: Value > 90 for 10 minutes P2: Value > 80 for 10 minutes	vke_cluster_cpu_utilization{ cluster=~"$Cluster", component=~"kube-apiserver\|kube-controller-manager\|kube-scheduler" }	Monitors the CPU utilization of individual APIServer, ControllerManager, and Scheduler instances.	Locate high-load instances by component and pod. Check request traffic, add-on logs, and resource specifications, and contact the platform team for scale-out if necessary.
High memory utilization of control plane add-ons	P1: Value > 90 for 10 minutes P2: Value > 80 for 10 minutes	vke_cluster_memory_utilization{ cluster=~"$Cluster", component=~"kube-apiserver\|kube-controller-manager\|kube-scheduler\|etcd" }	Monitors the memory utilization of control plane add-ons and individual etcd instances.	Identify high-memory instances by component, pod, or endpoint. Check memory growth trends and add-on logs, and contact the platform team for scale-out if necessary.
Cluster Autoscaler anomalies	P0: Value == 0 for 5 minutes	sum(up{cluster=~"$Cluster", job=~"cluster-autoscaler\|autocluster"}) by (cluster)	Monitors the availability status of Cluster Autoscaler instances.	Check the status and logs of the Cluster Autoscaler instance. Confirm whether monitoring collection is functioning as expected, and observe whether Pending Pods and node scaling are affected.
Karpenter anomalies	P0: Value == 0 for 5 minutes	sum(up{cluster=~"$Cluster", job="karpenter"}) by (cluster)	Monitors the availability status of Karpenter instances.	Check Karpenter instance status and logs. Verify whether monitoring collection is functioning as expected and observe whether node provisioning, reclamation, and Pending Pods are affected.
Cluster Autoscaler cannot scale safely	P1: Value == 0 for 60 minutes	max(cluster_autoscaler_cluster_safe_to_autoscale{cluster=~"$Cluster"}) by (cluster)	Monitors whether Cluster Autoscaler satisfies the secure scaling conditions of the cluster.	Check for nodes that are in the NotReady state, unregistered, or stuck in an intermediate scaling state for an extended period. After recovering the abnormal nodes, verify whether node scaling operations have resumed as expected.
Karpenter cluster status not synchronized	P1: Value == 0 for 60 minutes	max(karpenter_cluster_state_synced{cluster=~"$Cluster"}) by (cluster)	Monitors whether the cluster status of Karpenter is synchronized with the actual node and Pod information.	Check Karpenter controller logs and resource access permissions. Troubleshoot node and Pod information synchronization exceptions, and confirm whether node provisioning and reclamation have recovered.

VKE workload

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
Multiple container restarts	P1: Trigger immediately when the value is >= 5 P2: Trigger immediately when the value is >= 2	increase(kube_pod_container_status_restarts_total{cluster=~"$Cluster",namespace=~"$Namespace"}[10m])	Monitors the number of container restarts within 10 minutes.	Check container logs, events, probes, resource limits, and dependent service status.
Unexpected number of Deployment Pods	P2: Value > 0 for 5 minutes	kube_deployment_spec_replicas{cluster=~"$Cluster",namespace=~"$Namespace"} - kube_deployment_status_replicas_available{cluster=~"$Cluster",namespace=~"$Namespace"}	The expected number of available Deployment replicas is not reached.	Check issues related to rolling update status, Pod scheduling, probes, and images.
Unexpected number of StatefulSet Pods	P2: Value > 0 for 5 minutes	kube_statefulset_replicas{statefulset!="", cluster=~"$Cluster",namespace=~"$Namespace"} - kube_statefulset_status_replicas_ready{statefulset!="",cluster=~"$Cluster",namespace=~"$Namespace"}	The expected number of ready StatefulSet replicas is not reached.	Check PVCs, storage mounting, startup order, and Pod events.
High Pod CPU limit saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"$Namespace",container!=""}[5m]))by(cluster,namespace,pod,container)/sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"$Namespace",resource="cpu",container!=""})by(cluster,namespace,pod,container)	The container CPU usage approaches its limit, which may cause throttling.	Check whether the resource limit is low and whether hot traffic and CPU throttling exist.
High Pod memory limit saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"$Namespace",container!=""})by(cluster,namespace,pod,container)/sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"$Namespace",container!="",resource="memory",unit="byte"})by(cluster,namespace,pod,container)	The container memory usage approaches its limit.	Check for memory leaks, cache bloat, or low memory limits. Scale up the memory capacity if necessary.
Abnormal Pod status	P2: Value == 1 for 5 minutes	kube_pod_status_phase{cluster=~"$Cluster",namespace=~"$Namespace",job=~"kube-state-metrics", phase=~"Pending\|Unknown\|Failed"}	Check whether Pods enter the Pending, Unknown, or Failed state.	Check Pod events, focusing on scheduling failures, image pull failures, and node exceptions.
Pods in the Backoff state	P2: Value == 1 for 5 minutes	kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff\|CrashLoopBackOff", cluster=~"$Cluster",namespace=~"$Namespace"}	Containers enter the ImagePullBackOff or CrashLoopBackOff states.	Check image accessibility, startup commands, dependency settings, and probe settings.
Persistently blocked due to container CPU pressure	P2: Value > 0.25 for 5 minutes	sum by (cluster, namespace, pod, container) (rate(container_pressure_cpu_stalled_seconds_total{cluster=~"$Cluster",namespace=~"$Namespace",container!=""}[5m]))	Monitors the proportion of time containers are completely blocked due to CPU pressure.	Identify the abnormal container by namespace, pod, and container. Check the CPU utilization, resource limits, and node load, and then adjust the resource configuration or migrate the workload.
Persistent wait due to container CPU pressure	P2: Value > 0.4 for 5 minutes	sum by (cluster, namespace, pod, container) (rate(container_pressure_cpu_waiting_seconds_total{cluster=~"$Cluster",namespace=~"$Namespace",container!=""}[5m]))	Monitors the proportion of time containers wait for execution due to CPU pressure.	Identify the abnormal container by namespace, pod, and container. Check CPU quotas, resource contention, and node load, and adjust resource configuration.
Persistently blocked due to container IO pressure	P2: Value > 0.25 for 5 minutes	sum by (cluster, namespace, pod, container) (rate(container_pressure_io_stalled_seconds_total{cluster=~"$Cluster",namespace=~"$Namespace",container!=""}[5m]))	Monitors the proportion of time containers are completely blocked due to IO pressure.	Identify the abnormal container by namespace, pod, and container. Check the disk latency, IO load, and storage status, and reduce IO pressure or change the storage specification.
Persistent wait due to container IO pressure	P2: Value > 0.35 for 5 minutes	sum by (cluster, namespace, pod, container) (rate(container_pressure_io_waiting_seconds_total{cluster=~"$Cluster",namespace=~"$Namespace",container!=""}[5m]))	Monitors the proportion of time containers wait due to IO pressure.	Identify the abnormal container by namespace, pod, and container. Check disk throughput, IO contention, and storage status, and optimize read/write behavior.
Persistently blocked due to container memory pressure	P2: Value > 0.15 for 5 minutes	sum by (cluster, namespace, pod, container) (rate(container_pressure_memory_stalled_seconds_total{cluster=~"$Cluster",namespace=~"$Namespace",container!=""}[5m]))	Monitors the proportion of time containers are completely blocked due to memory pressure.	Identify the abnormal container by namespace, pod, and container. Check memory utilization, resource limits, and node memory pressure, and adjust memory configurations.
Persistent wait due to container memory pressure	P2: Value > 0.25 for 5 minutes	sum by (cluster, namespace, pod, container) (rate(container_pressure_memory_waiting_seconds_total{cluster=~"$Cluster",namespace=~"$Namespace",container!=""}[5m]))	Monitors the proportion of time containers wait due to memory pressure.	Locate the abnormal container by namespace, pod, and container. Check memory contention, resource limits, and node memory pressure, and adjust memory configurations.
DaemonSet replicas unavailable	P1: > 0 for 15 minutes	kube_daemonset_status_number_unavailable{cluster=~"$Cluster", namespace=~"$Namespace"}	Monitors the number of unavailable Pods in the DaemonSet.	Identify the abnormal workload by namespace and daemonset. Check Pod events, node status, and scheduling constraints, and restore unavailable replicas.
DaemonSet Pod scheduling error	P1: > 0 for 15 minutes	kube_daemonset_status_number_misscheduled{cluster=~"$Cluster", namespace=~"$Namespace"}	Monitors DaemonSet Pods scheduled to unexpected nodes.	Identify the abnormal workload by namespace and daemonset. Check node selectors, affinity, taints, and tolerations, and adjust the scheduling policy.
Running Pod not ready for an extended period	P1: Value == 1 for 10 minutes	(kube_pod_status_phase{cluster=~"$Cluster", namespace=~"$Namespace", phase="Running"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster", namespace=~"$Namespace", condition="false"} == 1)	Monitors Pods that are in the Running phase but persistently not in the Ready state.	Identify the abnormal Pod by namespace and pod. Check the container status, readiness probes, events, and dependent services, and restore the Pod to the Ready state.
Pod unschedulable for an extended period	P1: Value == 1 for 15 minutes	kube_pod_status_unschedulable{cluster=~"$Cluster", namespace=~"$Namespace"}	Monitors Pods that are persistently in the Unschedulable state.	View Pod scheduling events. Check remaining node resources, affinity, taints, tolerations, and storage constraints, and adjust resources or scheduling configurations.
Pod terminated for an extended period	P1: Value > 600 for 5 minutes	(time() - kube_pod_deletion_timestamp{cluster=~"$Cluster", namespace=~"$Namespace"}) and on(cluster, namespace, pod) (kube_pod_status_phase{cluster=~"$Cluster", namespace=~"$Namespace", phase!="Succeeded"} == 1)	Monitors Pods that have not been terminated for a long period after a deletion request is received.	Identify the abnormal Pod by namespace and pod. Check finalizers, container exit status, and storage unmounting, remove the blocker, and delete the Pod again.
Persistently blocked Pod creation	P1: Value == 1 for 15 minutes	kube_pod_container_status_waiting_reason{ cluster=~"$Cluster", namespace=~"$Namespace", reason=~"ErrImagePull\|CreateContainerConfigError\|ContainerCreating" }	Monitors Pods persistently in a waiting state due to image, configuration, or container creation issues.	View Pod events and container waiting reasons. Check image pulls, configuration references, container runtime, and storage mounts, and recreate the Pod after fixing.
Container OOMKilled	P1: Trigger immediately when the value is > 0	( increase( kube_pod_container_status_restarts_total{ cluster=~"$Cluster", namespace=~"$Namespace", container!="" }[10m] ) > 0 ) and on(cluster, namespace, pod, container) ( kube_pod_container_status_last_terminated_reason{ cluster=~"$Cluster", namespace=~"$Namespace", container!="", reason="OOMKilled" } == 1 )	Monitors containers terminated by the system due to insufficient memory.	Identify the abnormal container by namespace, pod, and container. Check memory usage trends, memory limits, and application logs, and optimize memory usage or increase limits.
Job failed	P1: Trigger immediately when the value is > 0	max_over_time( kube_job_status_failed{cluster=~"$Cluster", namespace=~"$Namespace"}[15m] )	Monitors Jobs that have recently had failed Pods.	Identify failed tasks by namespace and job_name. Check Pod events, container logs, and exit codes, and then fix task configurations or adjust retry policies.
Job not completed for an extended period	P1: Trigger immediately when the value is > 0	(time() - kube_job_status_start_time{cluster=~"$Cluster", namespace=~"$Namespace"} > 21600) and on(cluster, namespace, job_name) (kube_job_status_active{cluster=~"$Cluster", namespace=~"$Namespace"} > 0) and on(cluster, namespace, job_name) ( kube_job_spec_completions{cluster=~"$Cluster", namespace=~"$Namespace"} - kube_job_status_succeeded{cluster=~"$Cluster", namespace=~"$Namespace"} > 0 )	Monitors Jobs that have not reached the desired completion count long after startup.	Locate the abnormal job by namespace and job_name. Check active Pods, job logs, and external dependencies, and handle blocked steps or terminate the abnormal job.
HPA replicas inconsistent for an extended period	P1: > 0 for 15 minutes	abs( kube_horizontalpodautoscaler_status_desired_replicas{ cluster=~"$Cluster", namespace=~"$Namespace" } - kube_horizontalpodautoscaler_status_current_replicas{ cluster=~"$Cluster", namespace=~"$Namespace" } )	Monitors whether the current replicas of HPA persistently differ from the desired number of replicas.	Identify the abnormal HPA by namespace and horizontalpodautoscaler. Check scaling events, resource metrics, resource quotas, and target workload status.
HPA reaches maximum replicas	P2: Value >= 1 for 15 minutes	kube_horizontalpodautoscaler_status_desired_replicas{ cluster=~"$Cluster", namespace=~"$Namespace" } / kube_horizontalpodautoscaler_spec_max_replicas{ cluster=~"$Cluster", namespace=~"$Namespace" }	Monitors cases where the desired number of replicas for an HPA persistently reaches the configured upper limit.	Check business load, target metrics, and remaining cluster resources. After confirming capacity requirements, increase the maximum number of replicas or optimize application resource usage.
Deployment version not converged	P1: > 0 for 15 minutes	abs( kube_deployment_metadata_generation{cluster=~"$Cluster", namespace=~"$Namespace"} - kube_deployment_status_observed_generation{cluster=~"$Cluster", namespace=~"$Namespace"} )	Monitors continuous inconsistency between the version observed by the Deployment controller and the desired version.	Identify the abnormal workload by namespace and deployment. Check controller events, rolling update status, and Pod status, and address the causes blocking updates.
StatefulSet version not converged	P1: > 0 for 15 minutes	abs( kube_statefulset_metadata_generation{cluster=~"$Cluster", namespace=~"$Namespace"} - kube_statefulset_status_observed_generation{cluster=~"$Cluster", namespace=~"$Namespace"} )	Monitors continuous inconsistency between the version observed by the StatefulSet controller and the desired version.	Locate the abnormal workload by namespace and statefulset. Check controller events, rolling update status, and Pod order, and address the causes blocking updates.
High Pod startup P99 latency	P2: Value > 60 for 10 minutes	histogram_quantile( 0.99, sum by(cluster, le) ( rate(kubelet_pod_worker_start_duration_seconds_bucket{cluster=~"$Cluster"}[5m]) ) )	Monitors the P99 latency of cluster Pod worker startup.	Check the kubelet, container runtime, and resource load on the node where the slow-starting Pod resides. Investigate the time consumed by image pulling, storage mounting, and container initialization.
ResourceQuota nearly exhausted	P2: Value > 0.9 for 15 minutes	( sum by(cluster, namespace, resourcequota, resource) ( kube_resourcequota{cluster=~"$Cluster", namespace=~"$Namespace", type="used"} ) / sum by(cluster, namespace, resourcequota, resource) ( kube_resourcequota{cluster=~"$Cluster", namespace=~"$Namespace", type="hard"} > 0 ) ) < 1	Monitors the resource usage ratio of namespace ResourceQuotas.	Identify quotas that are nearly exhausted by namespace, resourcequota, and resource. Clean up unused resources or adjust quotas after assessing business requirements.
Overcommitted CPU quota	P2: Value > 1.5 for 5 minutes	sum by(cluster) ( kube_resourcequota{ cluster=~"$Cluster", namespace=~"$Namespace", type="hard", resource="requests.cpu" } ) / sum by(cluster) ( kube_node_status_allocatable{cluster=~"$Cluster", resource="cpu"} )	Monitors the ratio of the total CPU request quota in a namespace to the allocatable CPU of the cluster.	Check the CPU request quota and actual usage of each namespace. Adjust inappropriate quotas or scale out cluster resources based on capacity requirements.
Overcommitted memory quota	P2: Value > 1.5 for 5 minutes	sum by(cluster) ( kube_resourcequota{ cluster=~"$Cluster", namespace=~"$Namespace", type="hard", resource="requests.memory" } ) / sum by(cluster) ( kube_node_status_allocatable{cluster=~"$Cluster", resource="memory"} )	Monitors the ratio of the total memory request quota in a namespace to the allocatable memory of the cluster.	Check the memory request quota and actual usage of each namespace. Adjust inappropriate quotas or scale out cluster resources based on capacity requirements.

VKE storage delivery

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
PVC Pending for an extended period	P1: Value == 1 for 15 minutes	kube_persistentvolumeclaim_status_phase{ cluster=~"$Cluster", namespace=~"$Namespace", phase="Pending" }	Monitors PVCs that are persistently in the Pending state.	Locate the abnormal PVC by namespace and persistentvolumeclaim. Check the StorageClass, storage capacity, access mode, and CSI Provisioner status.
PVC terminated for an extended period	P1: Value > 600 for 5 minutes	(time() - kube_persistentvolumeclaim_deletion_timestamp{ cluster=~"$Cluster", namespace=~"$Namespace" }) and on(cluster, namespace, persistentvolumeclaim) ( kube_persistentvolumeclaim_status_phase{ cluster=~"$Cluster", namespace=~"$Namespace", phase="Bound" } == 1 )	Monitors PVCs that have not been terminated for a long period after a deletion request is received.	Locate the abnormal PVC by namespace and persistentvolumeclaim. Check finalizers, Pod references, and storage controller status, remove dependencies, and delete the PVC again.
PVC scale-out not completed for a long period	P2: Value == 1 for 30 minutes	kube_persistentvolumeclaim_status_condition{ cluster=~"$Cluster", namespace=~"$Namespace", condition=~"FileSystemResizePending\|Resizing", status="true" }	Monitors PVCs that have been scaling out or waiting for file system scale-out for a long period.	Identify the abnormal PVC by namespace and persistentvolumeclaim. Check the CSI Resizer, storage backend, and Pod restart requirements, and expand volumes and file systems.
Low PVC available space	P1: Value < 0.05 for 5 minutes P2: Value < 0.15 for 15 minutes	max by(cluster, namespace, persistentvolumeclaim) ( kubelet_volume_stats_available_bytes{ cluster=~"$Cluster", namespace=~"$Namespace" } ) / max by(cluster, namespace, persistentvolumeclaim) ( kubelet_volume_stats_capacity_bytes{ cluster=~"$Cluster", namespace=~"$Namespace" } > 0 )	Monitors the available storage space ratio of PVCs.	Identify the PVC with insufficient space by namespace and persistentvolumeclaim. Clean up useless data, adjust data retention policies, or expand the storage volume.
Low PVC inode availability	P1: Value < 0.03 for 5 minutes P2: Value < 0.05 for 15 minutes	max by(cluster, namespace, persistentvolumeclaim) ( kubelet_volume_stats_inodes_free{ cluster=~"$Cluster", namespace=~"$Namespace" } ) / ( max by(cluster, namespace, persistentvolumeclaim) ( kubelet_volume_stats_inodes_free{ cluster=~"$Cluster", namespace=~"$Namespace" } ) + max by(cluster, namespace, persistentvolumeclaim) ( kubelet_volume_stats_inodes_used{ cluster=~"$Cluster", namespace=~"$Namespace" } ) )	Monitors the available inode ratio of PVCs.	Locate the PVC with insufficient inodes by namespace and persistentvolumeclaim. Check the growth of small files and clean up unnecessary files. Migrate or scale up the volume if necessary.
Abnormal PV status	P1: Value == 1 for 5 minutes	kube_persistentvolume_status_phase{ cluster=~"$Cluster", phase=~"Failed\|Pending" }	Monitors PVs that are persistently in the Failed or Pending state.	Identify the abnormal PV by persistentvolume. Check the PVC binding relationship, reclaim policy, StorageClass, and storage backend status.
PV terminated for an extended period	P1: Value > 600 for 5 minutes	(time() - kube_persistentvolume_deletion_timestamp{cluster=~"$Cluster"}) and on(cluster, persistentvolume) ( kube_persistentvolume_status_phase{cluster=~"$Cluster", phase="Bound"} == 1 )	Monitors PVs that have not been terminated for a long period after a deletion request is received.	Identify the abnormal PV by persistentvolume. Check finalizers, PVC references, and storage backend resources, and delete the PV again after removing dependencies.
VolumeAttachment not completed for an extended period	P1: Value > 600 for 5 minutes	(time() - kube_volumeattachment_created{cluster=~"$Cluster"}) and on(cluster, volumeattachment) ( kube_volumeattachment_status_attached{cluster=~"$Cluster"} == 0 )	Monitors VolumeAttachments that have not been mounted for an extended period after creation.	Locate the abnormal record by volumeattachment. Check the target node, volume status, CSI Attacher logs, and storage backend, and restore the volume attachment.
Persistently failed storage operations	P2: Value > 0 for 5 minutes	sum by(cluster, operation_name, volume_plugin, status) ( increase(storage_operation_duration_seconds_count{ cluster=~"$Cluster", status!="success" }[10m]) )	Monitors continuously failing kubelet storage operations.	Identify failed operations by operation_name, volume_plugin, and status. Check kubelet, CSI add-on, and storage backend logs, and fix the issue at the corresponding stage.

VKE service delivery

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
No available Endpoints in the Service	P1: Value > 0 for 5 minutes	( kube_service_spec_type{ cluster=~"$Cluster", namespace=~"$Namespace", type=~"ClusterIP\|NodePort\|LoadBalancer" } == 1 ) and on(cluster, namespace, service) kube_service_info{ cluster=~"$Cluster", namespace=~"$Namespace", cluster_ip!="None", external_name="" } unless on(cluster, namespace, service) label_replace( kube_endpoint_address_available{ cluster=~"$Cluster", namespace=~"$Namespace" } > 0, "service", "$1", "endpoint", "(.*)" )	Monitors Services that are persistently without available Endpoints.	Identify the abnormal service by namespace and service. Check the selector, backend Pod Ready status, and EndpointSlice, and restore available backends.
LoadBalancer ingress not ready	P1: Value > 900 for 5 minutes	( time() - kube_service_created{ cluster=~"$Cluster", namespace=~"$Namespace" } ) and on(cluster, namespace, service) ( kube_service_spec_type{ cluster=~"$Cluster", namespace=~"$Namespace", type="LoadBalancer" } == 1 ) unless on(cluster, namespace, service) kube_service_status_load_balancer_ingress{ cluster=~"$Cluster", namespace=~"$Namespace" }	Monitors LoadBalancer Services that have not obtained an ingress address for an extended period after creation.	Identify the abnormal service by namespace and service. Check Service events, load balancer controllers, and cloud resource status, and fix reconciliation failures.
No local endpoints available on the kube-proxy Local Service	P2: Value > 0 for 10 minutes	max by(cluster, node, ip_family, traffic_policy) ( kubeproxy_sync_proxy_rules_no_local_endpoints_total{ cluster=~"$Cluster", job="kube-proxy" } )	Monitors Services that use the Local traffic policy but have no local Endpoints on the node.	Identify the abnormal node by node, ip_family, and traffic_policy. Check the distribution and health status of backend Pods, and adjust scheduling or traffic policies.

VKE node

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
Node in non-ready status	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	kube_node_status_condition{cluster=~"$Cluster", condition="Ready", status!="true", node!~"vci.*"}	A node is not in the Ready state.	Prioritize checking whether the node is unexpectedly taken offline. If so, check the heartbeat, network, disk, kubelet, and runtime of the node.
High node CPU utilization	P1: Value > 0.8 for 10 minutes P2: Value > 0.7 for 5 minutes	sum(rate(node_cpu_seconds_total{cluster=~"$Cluster", mode="user"}[5m])) by(cluster, instance) / count(count(node_cpu_seconds_total{cluster=~"$Cluster"}) by (cluster, cpu, instance)) by(cluster, instance)	The utilization of the user-mode CPU on a node is high.	Identify hot nodes and high-consumption Pods. Migrate workloads, scale out nodes, or optimize resource requests/limits if necessary.
High node memory utilization	P1: Value > 0.8 for 10 minutes P2: Value > 0.7 for 5 minutes	(node_memory_MemTotal_bytes{cluster=~"$Cluster"} - node_memory_MemAvailable_bytes{cluster=~"$Cluster"}) / node_memory_MemTotal_bytes{cluster=~"$Cluster"}	The node memory utilization is high.	Check for hotspot Pods, cache usage, and memory leaks to prevent triggering OOM or eviction.
Low file system capacity	P1: Value < 0.03 for 10 minutes P2: Value < 0.1 for 5 minutes	(node_filesystem_avail_bytes{cluster=~"$Cluster", fstype!=""} / node_filesystem_size_bytes{cluster=~"$Cluster", fstype!=""}) and node_filesystem_readonly{cluster=~"$Cluster", fstype!=""} == 0	The remaining capacity on writable filesystems is insufficient.	Promptly clean up logs, container images, and temporary files, or expand disk capacity to prevent node disks from being filled up.
Node restart	P2: Trigger immediately when the value is > 0	changes(node_boot_time_seconds{cluster=~"$Cluster"}[15m])	A node has restarted within the last 15 minutes.	Check node events, system logs, and host status to confirm whether the node restart is planned or unexpected.
High virtual node Pod utilization	P1: Value > 0.9 for 10 minutes P2: Value > 0.8 for 5 minutes	count by (cluster, node) ( max by (cluster, node, namespace, pod) ( kube_pod_info{cluster=~"$Cluster", node=~"vci."} )) /max by (cluster, node) ( kube_node_status_allocatable{cluster=~"$Cluster", node=~"vci.", resource="pods"})	Detect whether the remaining capacity for Pods schedulable to virtual nodes is insufficient.	Clean up terminated Pods promptly. If you need to increase the Pod capacity on virtual nodes, contact the development team.
Virtual node in non-ready status	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	kube_node_status_condition{cluster=~"$Cluster", condition="Ready", status!="true", node=~"vci.*"}	Detects virtual nodes that are not in the Ready state.	First, confirm whether the node goes offline as expected. For example, clearing the subnet configuration of a virtual node in a specific availability zone causes the corresponding virtual node to become abnormal, and the virtual node must be manually deleted. If not, check the node heartbeat and contact the development team if necessary.
Frequent flapping of node Ready status	P2: Value > 2 for 15 minutes	changes( kube_node_status_condition{ cluster=~"$Cluster", condition="Ready", status="true", node!~"vci.*" }[15m] )	Monitors frequent changes in node Ready status within a short period.	Locate the abnormal node by node. Check the kubelet, container runtime, node network, and system logs, and restore node stability.
Node resource pressure	P2: Value == 1 for 5 minutes	kube_node_status_condition{ cluster=~"$Cluster", condition=~"MemoryPressure\|DiskPressure\|PIDPressure", status="true", node!~"vci.*" }	Monitors nodes continuously under memory, disk, or PID pressure.	Identify the pressure type by node and condition. Check the corresponding resource utilization and abnormal processes, release resources, or migrate workloads.
Low file system inode availability	P1: Value < 0.03 for 10 minutes P2: Value < 0.1 for 5 minutes	node_filesystem_files_free{ cluster=~"$Cluster", fstype!~"tmpfs\|overlay", mountpoint!~"/var/lib/kubelet/pods." } / node_filesystem_files{ cluster=~"$Cluster", fstype!~"tmpfs\|overlay", mountpoint!~"/var/lib/kubelet/pods." }	Monitors the available inode ratio of node file systems.	Identify the file system with insufficient inodes by instance, device, and mountpoint. Check for small-file increases and clean up unused files. Perform expansion or migration if necessary.
High node conntrack utilization	P1: Value > 0.95 for 5 minutes P2: Value > 0.8 for 10 minutes	node_nf_conntrack_entries{cluster=~"$Cluster"} / node_nf_conntrack_entries_limit{cluster=~"$Cluster"}	Monitors the usage ratio of the node conntrack table.	Identify the abnormal node by instance. Check the number of short-lived connections, connection leaks, and network traffic, and optimize connection usage or adjust the conntrack capacity.
High node network error rate	P2: Value > 0.001 for 10 minutes	( ( sum by(cluster, instance) ( rate(node_network_receive_errs_total{cluster=~"$Cluster", device!="lo"}[5m]) ) + sum by(cluster, instance) ( rate(node_network_transmit_errs_total{cluster=~"$Cluster", device!="lo"}[5m]) ) ) / clamp_min( ( sum by(cluster, instance) ( rate(node_network_receive_packets_total{cluster=~"$Cluster", device!="lo"}[5m]) ) + sum by(cluster, instance) ( rate(node_network_transmit_packets_total{cluster=~"$Cluster", device!="lo"}[5m]) ) ), 1 ) ) and on(cluster, instance) ( ( sum by(cluster, instance) ( rate(node_network_receive_packets_total{cluster=~"$Cluster", device!="lo"}[5m]) ) + sum by(cluster, instance) ( rate(node_network_transmit_packets_total{cluster=~"$Cluster", device!="lo"}[5m]) ) ) > 100 )	Monitors the receive/transmit error rate of node network interfaces.	Locate the abnormal node by instance. Check NIC statistics, link status, drivers, and node network configuration, and handle hardware or network faults.
Node NTP not synchronized	P2: Value == 0 for 10 minutes	node_timex_sync_status{cluster=~"$Cluster"}	Monitors whether the node clock stays synchronized with the time source.	Identify the abnormal node by instance. Check the NTP service, time source configuration, and node network, and restore clock synchronization.
Node PLEG stalled for an extended period	P1: Value > 180 for 5 minutes	time() - kubelet_pleg_last_seen_seconds{cluster=~"$Cluster"}	Monitors whether the kubelet Pod Lifecycle Event Generator (PLEG) has been inactive for an extended period.	Identify the abnormal node by instance. Check the status and logs of kubelet and the container runtime. Restart the abnormal add-ons and migrate workloads if necessary.
High node file handle utilization	P1: Value > 0.95 for 5 minutes P2: Value > 0.8 for 10 minutes	node_filefd_allocated{cluster=~"$Cluster"} / node_filefd_maximum{cluster=~"$Cluster"}	Monitors the ratio of allocated file handles to the system limit on nodes.	Identify the abnormal node by instance. Check processes with high file handle utilization and connection leaks, release abnormal usage, or adjust system limits.
Collection exceptions on file system devices	P2: Value == 1 for 5 minutes	node_filesystem_device_error{ cluster=~"$Cluster", fstype!="" }	Monitors failures in collecting node file system device information.	Identify the abnormal file system by instance, device, and mountpoint. Check mount point accessibility, file system status, and IO blocking.
Node kernel OOM kill	P1: Trigger immediately when the value is > 0	increase( node_vmstat_oom_kill{ cluster=~"$Cluster" }[10m] )	Monitors kernel-triggered process termination due to insufficient memory on nodes.	Identify the abnormal node by instance. Check memory utilization, terminated processes, and workload resource configurations, and release memory or migrate workloads.

VKE CNI add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
Abnormal VPC-CNI Pod status	P0: Value > 0 for 10 minutes P1: Value > 0 for 3 minutes P2: Trigger immediately when the value is > 0	sum(kube_pod_container_status_waiting_reason{cluster=~"$Cluster",reason=~"CrashLoopBackOff\|ErrImagePull\|ImagePullBackOff\|Init:CrashLoopBackOff", pod=~".cello."}) by (cluster, namespace, pod, reason)	CNI-related Pods enter an abnormal waiting state.	Check the Pod events, image pulling, Crash logs, and node network status of the cello add-on. Recreate Pods if necessary.
Abnormal number of available replicas for the VPC-CNI operator	P0: Value > 0 for 10 minutes P1: Value > 0 for 3 minutes P2: Trigger immediately when the value is > 0	sum(kube_deployment_status_replicas_unavailable{cluster=~"$Cluster", deployment=~".cello."}) by (cluster, namespace, deployment) > 0 or sum(kube_deployment_status_replicas{cluster=~"$Cluster", deployment=~".cello."}) by (cluster, namespace, deployment) == 1	CNI Operator replicas are unavailable or insufficient	Check Deployments, node scheduling, images, resource limits, and pod logs to ensure the availability of the CNI control plane.
High VPC-CNI container memory usage	P1: Trigger immediately when the value is > 0.8	max(container_memory_usage_bytes{cluster=~"$Cluster", namespace=~"kube-system", pod=~".cello."} / (10241024)) by (cluster, namespace, pod, container) / max(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~".cello.", resource="memory"} / (10241024)) by (cluster, namespace, pod, container)	The CNI container memory usage is approaching the limit.	Check for abnormal traffic, memory leaks, or low limits. Adjust resources and restart add-ons if necessary.
High VPC-CNI container CPU usage	P1: Trigger immediately when the value is > 0.8	sum(rate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace="kube-system",pod=~".cello."}[1m])) by (cluster, namespace, pod, container) / max(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~".cello.", resource="cpu"}) by (cluster, namespace, pod, container)	The CNI container CPU usage almost reaches the limit.	Check for network storms, excessive retries, or insufficient resource allocations. Scale up appropriately and monitor recovery.
Insufficient available IPs in the cluster resource pool	P1: Value == 0 for 10 minutes	sum(resource_pool_available{cluster=~"$Cluster",job="vpc-cni"}) by (cluster, name)	Available IPs in the resource pool have been exhausted.	Immediately check subnet capacity, node allocation policies, and business expansion situation. Scale out the subnet or clean up unused resources if necessary.
VPC CNI call failure	P0: Value > 0 for 10 minutes P1: Value > 0 for 5 minutes	sum(rate(rpc_latency_ms_count{cluster=~"$Cluster",job="vpc-cni",error="true"}[1m])) by (cluster, rpc_api, node)	Internal CNI RPC calls persistently fail.	Check failed APIs, node distribution, and logs. Determine whether the alert is triggered by a dependency exception, a network issue, or an add-on bug.
Internal subsystem exception of VPC CNI	P0: Trigger immediately when the value is > 0	sum(rate(sub_sys_error_count{cluster=~"$Cluster",job="vpc-cni"}[1m])) by (cluster, subsys, node)	An exception occurs in internal CNI subsystems.	Identify the specific subsystem by using the subsys dimension. Check node logs and recent changes for troubleshooting.
Throttling in cluster APIs	P2: Value > 0 for 10 minutes	sum(rate(openapi_statistic{cluster=~"$Cluster",job="vpc-cni",errCode=~"FlowLimitExceeded\|AccountFlowLimitExceeded"}[2m])) by (cluster, api)	VPC CNI triggers throttling when calling cloud APIs.	Check whether large-scale scaling or retry storms exist. Reduce calls and use cloud resource quota policies for optimization.
API call failure	P1: Value > 0 for 10 minutes P2: Value > 0 for 5 minutes	sum(rate(openapi_statistic{cluster=~"$Cluster",job="vpc-cni",errCode!~"FlowLimitExceeded\|AccountFlowLimitExceeded\|InsufficientIpInSubnet", errCode!~"QuotaExceeded.*", errCode!~""}[1m])) by (cluster, api, errCode)	API errors other than throttling or quota errors occur.	Identify the cause by api and errCode. Check cloud resource status, permissions, and network connectivity.
VPC-CNI dependent resource quota exceeded	P0: Trigger immediately when the value is > 0	sum(rate(openapi_statistic{cluster=~"$Cluster",job="vpc-cni",errCode=~"InsufficientIpInSubnet\|QuotaExceeded.*"}[1m])) by (cluster, api, errCode)	Dependent resources are insufficient, or quotas have been used up.	Prioritize scaling out subnets, increasing quotas, or reclaiming idle resources to ensure that networks can be allocated to new Pods.
Metadata call failure	P1: Value > 0 for 5 minutes P2: Value > 0 for 3 minutes	sum(rate(metadata_statistic{cluster=~"$Cluster",job="vpc-cni", errCode!~""}[1m])) by (cluster, url, node)	Access to the Metadata service fails.	Check network connectivity, routing, and security policies from nodes to the Metadata service. Focus on affected nodes.
ACL rule calculation failure	P1: Trigger immediately when the value is > 0	sum(rate(cello_acl_recalculate_time_status_second_count{cluster=~"$Cluster",job="vpc-cni", status=~"fail"}[2m])) by (cluster, kind, node)	The ACL rule recalculation fails.	Check policy changes, the number of ACLs, and add-on logs. Check whether network policy enforcement is affected.
High P99 latency of ACL rule calculation	P2: Trigger immediately when the value is > 1 P1: Trigger immediately when the value is > 2	histogram_quantile(0.99, sum(increase(cello_acl_recalculate_time_status_second_bucket{cluster=~"$Cluster",job="vpc-cni", status=~"success"}[2m])) by (cluster, node, kind, le))	ACL recalculation takes a long period of time.	Analyze policy scale and node distribution. Evaluate whether rule explosions, hot nodes, or add-on performance bottlenecks exist.
Failure in processing ACL watch events	P2: Trigger immediately when the value is > 0 P1: Trigger immediately when the value is > 0.2	sum(rate(cello_acl_watcher_event_process_latency_count{cluster=~"$Cluster",job="vpc-cni", status=~"fail"}[2m])) by (cluster, event, node)	ACL watch events fail to be consumed.	Check the failed event types and nodes. Check listener pipeline issues, queue backlog, or add-on exceptions.
High P99 latency for ACL watch event processing	P2: Trigger immediately when the value is > 1000	histogram_quantile(0.99, sum(increase(cello_acl_watcher_event_process_latency_bucket{cluster=~"$Cluster",job="vpc-cni", status=~"success"}[1m])) by (cluster, node, event, le))	The ACL watch event processing latency is high.	Focus on the control plane change volume and node processing capacity. Reduce change frequency if necessary and investigate performance bottlenecks.
Data plane BPF MAP operation failure	P0: Value > 0 for 10 minutes P1: Trigger immediately when the value is > 0	sum(rate(cello_bpf_map_ops_total{cluster=~"$Cluster",job="vpc-cni", result!="success"}[2m])) by (cluster, map_name, node)	Data plane BPF Map operations fail, which may directly affect network forwarding.	Immediately check Cilium/BPF logs and kernel status on affected nodes. Remove the nodes if necessary.
Cilium controller execution failure	P1: Value > 0 for 1 minute	sum(rate(cilium_controllers_runs_total{cluster=~"$Cluster",job="vpc-cni", status=~"failure"}[1m])) by (cluster, node)	The Cilium controller fails to be executed.	Check Cilium logs on the node and the name of the failed controller to troubleshoot configuration synchronization or dependency exceptions.
Cilium subsystem exception	P1: Value > 0 for 1 minute	sum(rate(cilium_errors_warnings_total{cluster=~"$Cluster",job="vpc-cni", level=~"error"}[1m])) by (cluster, node, subsystem)	Errors on Cilium subsystems are continuously reported.	Identify the issue by subsystem and handle it based on the node network symptoms.
Cilium policy generation failure	P1: Trigger immediately when the value is > 0	sum(rate(cilium_policy_regeneration_time_stats_seconds_count{cluster=~"$Cluster",job="vpc-cni", status !="success"}[1m])) by (cluster, node)	Cilium policy regeneration fails.	Check policy changes and node Cilium status. Check whether network policies take effect.
High P99 latency for Cilium policy enforcement	P2: Trigger immediately when the value is > 1	avg(histogram_quantile(0.99, rate(cilium_policy_implementation_delay_bucket{cluster=~"$Cluster",job="vpc-cni"}[1m]))) by (cluster, node, source)	The network policy distribution and enforcement are slow.	Check the policy scale, node pressure, and control plane change frequency. Avoid changing a large number of policies simultaneously.
High Cilium bpf map usage	P0: Trigger immediately when the value is > 0.9 P1: Trigger immediately when the value is > 0.8	max(cilium_bpf_map_pressure{cluster=~"$Cluster",job="vpc-cni"}) by (cluster, map_name, node)	The BPF map capacity is approaching its upper limit.	Immediately check hot nodes and Map types. Evaluate connection scale, policy scale, and expansion requirements.
Large number of Cilium identities	P0: Trigger immediately when the value is > 50000 P1: Trigger immediately when the value is > 40000	max(apiserver_storage_objects{resource=~"ciliumidentities.cilium.io",cluster=~"$Cluster"}) by (cluster)	The number of Cilium Identity objects is approaching the threshold.	Evaluate whether the namespace, label, and policy design lead to an identity explosion. Optimize labels and policies if necessary.
Cilium BPF map operation failed	P0: Value > 0 for 10 minutes P1: Trigger immediately when the value is > 0	sum(rate(cilium_bpf_map_ops_total{cluster=~"$Cluster",job="vpc-cni", outcome!="success", map_name!~"nodeport_neigh4\|throttle"}[2m])) by (cluster, map_name, operation, node)	Unsuccessful read or write operations have occurred on BPF maps, which may affect the correctness of data-plane forwarding for network policies, connection tracking, and related functions.	Identify the nodes with operation failures. Check the Cilium status and logs on the affected nodes to investigate whether the map is full or whether there are permission anomalies.

VKE CSI add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High CSI IaaS API latency	P1: Value > 5 for 2 minutes	histogram_quantile(0.90, sum by(cluster,method,le)(rate(volc_api_request_duration_seconds_bucket{cluster=~"$Cluster",job="csi.*"}[2m])))	High P90 latency of CSI calls to cloud IaaS APIs	Check the specific method, cloud disk API status, and CSI logs. Check whether cloud‑side jitter or request backlogs exist.
High CSI IaaS API error rate	P1: Value > 5 for 2 minutes	rate(volc_api_request_errors{cluster=~"$Cluster",job="csi.*"}[2m])	The error rate of CSI calls to cloud APIs is continuously increasing.	Check authentication, quotas, network connectivity, and cloud disk API response codes.
Increase in Kubelet CSI RPC failures - NAS	P1: Value > 1 for 5 minutes	sum(increase(csi_operations_seconds_count{cluster=~"$Cluster",job="kubelet", grpc_status_code!="OK",driver_name=~"nas.*"}[1m])) by (cluster, node, method_name, driver_name)	CSI RPC related to the NAS driver fails.	Check NAS mounting, network connectivity, node permissions, and target node logs.
Increase in Kubelet CSI RPC failures - TOS	P2: Value > 1 for 10 minutes	sum(increase(csi_operations_seconds_count{cluster=~"$Cluster",job="kubelet", grpc_status_code!="OK",driver_name=~"tos.*"}[1m])) by (cluster, node, method_name, driver_name)	CSI RPC related to the TOS driver fails.	Check the failed method, node, and TOS dependency chain. Check whether the driver is in a normal status.
Increase in Kubelet CSI RPC failures - EBS	P1: Value > 1 for 5 minutes	sum(increase(csi_operations_seconds_count{cluster=~"$Cluster",job="kubelet", grpc_status_code!="OK",driver_name=~"ebs.*"}[1m])) by (cluster, node, method_name, driver_name)	Monitors RPC failures related to operations on the EBS driver, such as mounting, unmounting, and expansion.	Check volume status, node attach/mount logs, and CSI add-on health.
Increase in Kubelet CSI RPC failures - VEPFS	P1: Value > 1 for 5 minutes	sum(increase(csi_operations_seconds_count{cluster=~"$Cluster",job="kubelet", grpc_status_code!="OK",driver_name=~"vepfs.*"}[1m])) by (cluster, node, method_name, driver_name)	RPC related to the VEPFS driver fails.	Check file system status, networks, and CSI node add-on logs.
Increase in Kubelet CSI RPC failures - FSX	P1: Value > 1 for 5 minutes	sum(increase(csi_operations_seconds_count{cluster=~"$Cluster",job="kubelet", grpc_status_code!="OK",driver_name=~"fsx.*"}[1m])) by (cluster, node, method_name, driver_name)	RPC related to the FSX driver fails.	Check FSX storage instance status, network paths, and node add-on logs.
Increase in FSX client uninstallation failures	P2: Value > 0 for 5 minutes	sum(increase(fsx_agent_pod_lifecycle_error_total{cluster=~"$Cluster",method="Uninstall"}[5m])) by (cluster, instance, method)	The FSX Agent failed to uninstall.	Check instance-level error logs to identify residual resources from reclamation, permission issues, or abnormal node states.

VKE CSI-EBS service

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
Unavailable csi-ebs controller Pods	P0: Value == 1 for 10 minutes P1: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"csi-ebs-controller.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"csi-ebs-controller.",condition="false"} == 1)	The csi-ebs controller Pod is in the NotReady state. As a result, control plane operations such as volume creation, deletion, mounting, expansion, and snapshotting cannot be completed.	Check the status and events of the csi-ebs-controller Pod in the kube-system namespace. Check image pulling, startup parameters, and resource limits. Check whether dependencies (APIServer and cloud credentials) function as expected. View the logs of the controller and sidecar containers to identify the root cause.
Unavailable csi-ebs node Pods	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"csi-ebs-node.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"csi-ebs-node.",condition="false"} == 1)	The csi-ebs node Pod is in the NotReady state, which may cause volume mount/unmount (NodeStage/NodePublish) failures on the node. This will affect Pod startup.	Identify the csi-ebs-node Pods and logs of the corresponding node. Check whether the kubelet add-on (driver-registrar) is registered. Check node disks, kernel modules, and mount environments. Restart the node Pod or drain the node if necessary.
csi-ebs control plane operation backlog	P1: Value > 20 for 10 minutes P2: Value > 10 for 5 minutes	max(workqueue_depth{cluster=~"$Cluster",job=~"csi-ebs-.*"}) by (cluster, job, name)	The pending task queues of control plane add-ons (responsible for volume provisioning, mounting, expansion, and snapshotting) persistently remain high. This indicates that the event processing rate cannot keep up with the event generation rate, resulting in backlogs and delays in control plane operations.	Identify the specific add-on (job) and queue (name) with backlogs. Check the logs of the corresponding add-on and driver. Check whether the API Server and cloud APIs are experiencing slowness. Evaluate whether a scale-out or a QPS/concurrency limit increase is needed.
csi-ebs control plane RPC call failure	P1: Value > 5 for 5 minutes P2: Value > 0 for 5 minutes	sum(increase(csi_sidecar_operations_seconds_count{cluster=~"$Cluster",job=~"csi-ebs-.*",grpc_status_code!="OK"}[2m])) by (cluster, job, method_name)	Control plane add-ons fail to call the driver's gRPC operations. This will directly cause failures in control plane operations such as volume creation, deletion, mounting, expansion, and snapshotting.	Identify the failed operation by using the failed method_name and job. View error codes and stack traces in the driver logs. Check the cloud credentials, quotas, and cloud storage APIs for issues. Check whether the status of related CRDs, PVs, or PVCs is stuck.
csi-ebs node RPC call failures	P1: Value > 5 for 5 minutes P2: Value > 0 for 5 minutes	sum(increase(csi_operations_seconds_count{cluster=~"$Cluster",job="kubelet",driver_name=~"ebs.*",grpc_status_code!="OK"}[2m])) by (cluster, method_name)	kubelet fails to call the csi-ebs node driver's gRPC operations. This causes volume mount, unmount, and expansion failures on the node, thereby blocking Pod startup.	Identify the failure stage based on the failed method_name. View the csi-ebs-node logs on the corresponding node. Check the node disk status, device path, and file system. Check whether the volume is attached on the cloud side.
csi-ebs cloud storage API call failure	P1: Value > 5 for 5 minutes P2: Value > 0 for 5 minutes	sum(increase(volc_api_request_errors{cluster=~"$Cluster",job="csi-ebs"}[2m])) by (cluster, action)	The csi-ebs controller fails to call the APIs of Volcano Engine cloud storage (IaaS). It usually indicates a dependency exception or a credential/quota issue on the cloud and will cause failures in volume creation, deletion, mounting, expansion, and other operations.	Identify the specific cloud API based on the failed action. Check access credentials (AK/SK and STS) and permissions. Check the account quota, throttling, and cloud service status. Contact the cloud platform for assistance in troubleshooting if necessary.
High latency in csi-ebs cloud storage APIs	P1: Value > 8s for 5 minutes P2: Value > 5s for 5 minutes	histogram_quantile(0.99, sum by(cluster, action, le)(rate(volc_api_request_duration_seconds_bucket{cluster=~"$Cluster",job="csi-ebs"}[2m])))	P99 latency continuously remains high when the csi-ebs controller calls Volcano Engine cloud storage APIs (IaaS). This slows down control plane operations such as volume creation, deletion, mounting, and expansion, resulting in PVCs remaining Pending for extended periods or slow mounting.	Identify slow APIs by using action. Check the health status of cloud-side APIs and regional network links. Observe whether throttling or error rate spikes occur. Contact the cloud platform team to confirm backend latency if necessary.
High csi-ebs node RPC latency	P1: Value > 30s for 5 minutes P2: Value > 15s for 5 minutes	histogram_quantile(0.99, sum by(cluster, method_name, le)(rate(csi_operations_seconds_bucket{cluster=~"$Cluster",job="kubelet",driver_name=~"ebs.*"}[2m])))	When kubelet calls the csi-ebs node driver's gRPC operations, P99 latency remains high. This will slow down volume mount/unmount/expansion on the node and delay Pod startup.	Identify the slow stage by using method_name (such as NodeStageVolume). View the csi-ebs-node logs on the corresponding node. Check the node disk I/O, file system, and device scanning latency. Check node load and kernel status.

VKE DNS service

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
CoreDNS panic occurred	P0: Value > 0 for 10 minutes P1: Trigger immediately when the value is > 0	sum(increase(coredns_panics_total{cluster=~"$Cluster",job="core-dns"}[5m]))	CoreDNS panics occurred within the last 5 minutes.	Immediately check CoreDNS Pod logs, recent configuration changes, and upstream resolution dependencies.
High CoreDNS error rate	P1: Value > 0.3 for 10 minutes P2: Value > 0.3 for 5 minutes	sum(increase(coredns_dns_responses_total{cluster=~"$Cluster",job="core-dns",rcode=~"SERVFAIL\|REFUSED"}[5m]))/sum(increase(coredns_dns_requests_total{cluster=~"$Cluster",job="core-dns"}[5m]))	The failed response ratio of CoreDNS is high.	Check the failed domain names, upstream DNS, Corefile settings, and network connectivity.
Long CoreDNS DNS resolution duration	P1: Value > 0.5 for 5 minutes P2: Value > 0.1 for 5 minutes	histogram_quantile(0.99, rate(coredns_dns_request_duration_seconds_bucket{cluster=~"$Cluster",job="core-dns"}[5m]))	The CoreDNS P99 resolution latency is high.	Determine whether the alert is triggered by slow upstream resolution, a low cache hit ratio, or insufficient CoreDNS resources. Scale out replicas if necessary.
Cache hit ratio of CoreDNS DNS resolution	P1: Value < 0.3 for 5 minutes P2: Value < 0.5 for 5 minutes	sum(increase(coredns_cache_hits_total{cluster=~"$Cluster",job="core-dns"}[5m]))/sum(increase(coredns_dns_requests_total{cluster=~"$Cluster",job="core-dns"}[5m]))	The cache hit ratio is decreased, which indicates increased pressure on backend resolution.	Check the business request mode, cache settings, and hotspot domains. Optimize TTL or deploy NodeLocalDNS if necessary.
Unavailable CoreDNS Pods	P0: Value == 1 for 10 minutes P1: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster", namespace=~"kube-system",pod=~"coredns.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster", namespace=~"kube-system",pod=~"coredns.",condition="false"} == 1)	CoreDNS Pods are unavailable.	Prioritize restoring Pod availability. Check images, scheduling, probes, and resource limits.
CoreDNS configuration reload failed	P2: Trigger immediately when the value is > 0	sum(increase(coredns_reload_failed_total{cluster=~"$Cluster",job="core-dns"}[5m]))	The configuration change did not take effect successfully, which may cause unexpected resolution behavior.	Check the CoreDNS Pod logs for reload-related errors and confirm whether the cause is a Corefile syntax error, plugin configuration exception, or permission issue. After correcting the Corefile configuration, manually fix the configuration and trigger a reload if necessary.
CoreDNS upstream health check anomaly	P2: Trigger immediately when the value is > 2	sum(increase(coredns_proxy_healthcheck_failures_total{proxy_name="forward",cluster=~"$Cluster",job="core-dns"}[10m])) or sum(increase(coredns_forward_healthcheck_broken_total{cluster=~"$Cluster",job="core-dns"}[10m]))	The health status of upstream DNS is abnormal.	Check the network connectivity and service status of the upstream DNS. Temporarily switch to an available upstream DNS address if necessary.
Upstream CoreDNS concurrent request rejection	P2: Triggered immediately when the value is > 4	sum(increase(coredns_forward_max_concurrent_rejects_total{cluster=~"$Cluster",job="core-dns"}[10m]))	Forwarding concurrency has reached the upper limit, indicating upstream pressure or insufficient CoreDNS configuration capacity.	View the current DNS query count and confirm whether a sudden increase in business requests has caused concurrency to exceed the CoreDNS max_concurrent limit. Increase the max_concurrent value if necessary.
Health check anomalies continuously reported for CoreDNS Pods	P2: Trigger immediately when the value is > 2	sum by (pod) (increase(prober_probe_total{cluster=~"$Cluster", namespace=~"kube-system",pod=~"coredns.*",container="coredns",result="failed"}[10m]))	The CoreDNS Pod probe fails 3 times within a 10-minute window. This is suitable for early detection of health check fluctuations.	Identify the specific Pod that triggered the alert. Check its logs and events to confirm the probe failure type and error message.
High CoreDNS CPU Limits saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"coredns.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"coredns.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the CoreDNS Pod continuously approaches the container CPU Limit, which may slow down or throttle DNS resolution requests.	Check the resource configuration and load of the corresponding Pod.
High CoreDNS memory Limits saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"coredns.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"coredns.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the CoreDNS Pod continuously approaches the container memory limit, posing a risk of OOM events or DNS service instability.	Check the memory usage and resource configuration of the corresponding Pod.
NodeLocalDNS panic	P0: Value > 0 for 10 minutes P1: Trigger immediately when the value is > 0	sum(increase(coredns_panics_total{cluster=~"$Cluster",job="node-local-dns"}[5m]))	NodeLocalDNS panics occur.	Check the logs of the corresponding DaemonSet Pods and the node scope to determine whether local node resolution is affected.
High NodeLocalDNS error rate	P1: Value > 0.3 for 10 minutes P2: Value > 0.3 for 5 minutes	sum(increase(coredns_dns_responses_total{cluster=~"$Cluster",job="node-local-dns",rcode=~"SERVFAIL\|REFUSED"}[5m]))/sum(increase(coredns_dns_requests_total{cluster=~"$Cluster",job="node-local-dns"}[5m]))	The error response ratio of NodeLocalDNS is high.	Check the distribution of faulty nodes, the status of upstream CoreDNS, and the local cache configuration.
Long DNS resolution duration on NodeLocalDNS	P2: Value > 0.5 for 5 minutes	histogram_quantile(0.99, rate(coredns_dns_request_duration_seconds_bucket{cluster=~"$Cluster",job="node-local-dns"}[5m]))	The local DNS resolution latency of nodes is high.	Check node resources, upstream resolution latencies, and hotspot domain request patterns.
Cache hit ratio of NodeLocalDNS DNS resolution	P2: Value < 0.3 for 5 minutes	sum(increase(coredns_cache_hits_total{cluster=~"$Cluster",job="node-local-dns"}[5m]))/sum(increase(coredns_dns_requests_total{cluster=~"$Cluster",job="node-local-dns"}[5m]))	The NodeLocalDNS cache hit ratio is low.	Check whether a large number of requests are for cold data, and optimize caches and traffic paths if necessary.
Unavailable NodeLocalDNS Pods	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster", namespace=~"kube-system",pod=~"node-local-dns.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster", namespace=~"kube-system",pod=~"node-local-dns.",condition="false"} == 1)	A NodeLocalDNS Pod is unavailable.	Prioritize locating the abnormal nodes and restoring DaemonSet Pods to prevent DNS resolution failures on some nodes.
High NodeLocalDNS CPU Limits saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-local-dns-[^-]+",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-local-dns-[^-]+",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the NodeLocalDNS Pod continuously approaches the container's CPU Limit, which may slow down or throttle node-side DNS resolution requests.	Check the resource configuration and load of the corresponding Pod.
High NodeLocalDNS memory Limits saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-local-dns-[^-]+",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-local-dns-[^-]+",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the NodeLocalDNS Pod continuously approaches the container's Memory Limit, posing a risk of OOM or node-side DNS service instability.	Check the memory usage and resource configuration of the corresponding Pod.
NodeLocalDNS NodeCache network configuration failed	P2: Trigger immediately when the value is > 0	sum(increase(coredns_nodecache_setup_errors_total{cluster=~"$Cluster",job="node-local-dns"}[10m])) by (cluster, pod, errortype)	Periodic NodeLocalDNS network configuration failed.	View the Pod logs to determine whether the issue is caused by iptables rules, network interface configuration, or ConfigMap read failure. After fixing the corresponding error, recreate the abnormal Pod.
NodeLocalDNS upstream health check anomaly	P2: Trigger immediately when the value is > 2	sum(increase(coredns_forward_healthcheck_failures_total{cluster=~"$Cluster",job="node-local-dns"}[10m])) by (cluster, pod) or sum(increase(coredns_proxy_healthcheck_failures_total{proxy_name="forward",cluster=~"$Cluster",job="node-local-dns"}[10m])) by (cluster, pod) or sum(increase(coredns_forward_healthcheck_broken_total{cluster=~"$Cluster",job="node-local-dns"}[10m])) by (cluster, pod)	Failures in health checking from NodeLocalDNS to upstream CoreDNS are increasing, which may cause node-side DNS back-to-origin anomalies or higher resolution latency.	Check the status of the upstream CoreDNS Pod and the network connectivity from the node to CoreDNS. When the upstream service recovers, observe whether the health check failure metric sample count decreases.
Upstream NodeLocalDNS concurrent request rejection	P2: Triggered immediately when the value is > 4	sum(increase(coredns_forward_max_concurrent_rejects_total{cluster=~"$Cluster",job="node-local-dns"}[10m])) by (cluster, pod)	NodeLocalDNS has reached its maximum forward concurrency limit and is rejecting requests. This usually indicates slow upstream responses, insufficient concurrency configuration, or a backlog of node-side DNS requests.	Check the upstream CoreDNS response speed and node-side DNS request volume. Increase the max_concurrent configuration or scale out CoreDNS replicas if necessary.
NodeLocalDNS configuration reload failed	P2: Trigger immediately when the value is > 0	sum(increase(coredns_reload_failed_total{cluster=~"$Cluster",job="node-local-dns"}[5m])) by (cluster, pod)	The NodeLocalDNS configuration reload did not take effect successfully, which may cause unexpected node-side DNS behavior.	Check the Pod logs to identify the cause of the reload failure, check recent ConfigMap changes, and correct the configuration. Recreate the Pod after correction.
High NodeLocalDNS Admission Webhook error rate	P2: Value > 0.1 for 5 minutes	sum(increase(controller_runtime_webhook_requests_total{cluster=~"$Cluster",job="vke-node-local-dns-admission",code!~"2.."}[10m])) / clamp_min(sum(increase(controller_runtime_webhook_requests_total{cluster=~"$Cluster",job="vke-node-local-dns-admission"}[10m])), 1)	The proportion of non-2xx responses from the NodeLocalDNS Admission Webhook has increased, which may impact NodeLocalDNS admission capabilities.	View Webhook Pod logs to confirm the error type. Check TLS certificate validity and Service connectivity, and recreate the Pod if necessary.
NodeLocalDNS Admission certificate read failed	P2: Trigger immediately when the value is > 0	sum(increase(certwatcher_read_certificate_errors_total{cluster=~"$Cluster",job="vke-node-local-dns-admission"}[10m]))	NodeLocalDNS Admission certificate read failed, which may impact the availability of the Webhook TLS service.	Check whether the certificate Secret exists or has expired. After updating the certificate, recreate the Webhook Pod.
High NodeLocalDNS Admission Webhook fail-open rate	P2: Value > 0.3 for 5 minutes	(sum(increase(apiserver_admission_webhook_fail_open_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="vke-node-local-dns-admission.kube-system.mutating",type="admit"}[5m])) by (cluster, name, type) / clamp_min(sum(increase(apiserver_admission_webhook_admission_duration_seconds_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="vke-node-local-dns-admission.kube-system.mutating",type="admit"}[5m])) by (cluster, name, type), 1)) and on(cluster, name, type) (sum(increase(apiserver_admission_webhook_admission_duration_seconds_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="vke-node-local-dns-admission.kube-system.mutating",type="admit"}[5m])) by (cluster, name, type) >= 20) and on(cluster, name, type) (sum(increase(apiserver_admission_webhook_fail_open_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="vke-node-local-dns-admission.kube-system.mutating",type="admit"}[5m])) by (cluster, name, type) >= 3)	kube-apiserver calls to the NodeLocalDNS Admission mutating webhook fail and are allowed through by failurePolicy=Ignore, indicating that the NodeLocalDNS Admission capability is degraded. The anomaly may be caused by an abnormal Service/Endpoint, TLS, network, or webhook process. At least 20 calls and 3 fail-open events within 5 minutes are required to trigger this rule. This prevents false positives caused by a single failure in low-traffic scenarios.	Check the status of the Webhook Pod and the connectivity from kube-apiserver to the Webhook. After recovery, continue to observe whether the metric decreases.

VKE Ingress-Nginx add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High Ingress-Nginx request latency	P1: Value > 3 for 5 minutes P2: Value > 1 for 5 minutes	histogram_quantile(0.90, sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{cluster=~"$Cluster"}[5m])) by (cluster,host,path,method,le))	The P90 request latency of inbound traffic is high.	Check affected domain names, paths, and methods. Check whether the alert is triggered by slow ingress forwarding or slow backend service response.
High Ingress-Nginx request error rate	P1: Value > 0.5 for 5 minutes P2: Value > 0.3 for 5 minutes	sum by (cluster) (rate(nginx_ingress_controller_requests{cluster=~"$Cluster",status!~"2.."}[5m])) / sum by (cluster) (rate(nginx_ingress_controller_requests{cluster=~"$Cluster"}[5m]))	The ratio of non-2xx requests is high.	Check status code distribution, upstream service health, and Ingress rule settings. Roll back recent changes if necessary.
Unavailable Ingress-Nginx Pod	P0: Value == 0 for 10 minutes P1: Value == 0 for 5 minutes	kube_pod_container_status_ready{cluster=~"$Cluster",pod=~"ingress-nginx.*"}	The Ingress controller Pods are unavailable.	Prioritize restoring the controller Pod. Check whether exceptions occur in scheduling, probes, images, and resources.
Ingress-Nginx certificate expiring soon	P0: Value < 7 for 1 minute P1: Value < 14 for 1 minute	(nginx_ingress_controller_ssl_expire_time_seconds{cluster="$Cluster"} - time())/24/3600	The certificate is about to expire.	Renew, re‑issue, or update the Secret in advance to prevent ingress HTTPS interruption.
High Ingress-Nginx CPU utilization	P1: Value > 90 for 5 minutes	sum(rate(container_cpu_usage_seconds_total{cluster=~"$Cluster", pod=~"ingress-nginx-."}[1m])) by (cluster, namespace, pod, container) / max(kube_pod_container_resource_limits{resource="cpu",cluster=~"$Cluster",pod=~"ingress-nginx-."}) by (cluster, namespace, pod, container) * 100	The Ingress Pod CPU usage is approaching the upper limit.	Check for sudden traffic spikes, insufficient throttling, or insufficient resources. Scale out replicas and resources if necessary.
High Ingress-Nginx memory utilization	P1: Value > 90 for 5 minutes	(container_memory_working_set_bytes{cluster=~"$Cluster",pod=~"ingress-nginx-."}>0) / (container_spec_memory_limit_bytes{cluster=~"$Cluster",pod=~"ingress-nginx-."}>0) * 100	The Ingress Pod memory usage is approaching the upper limit.	Check the number of connections, caches, configurations, and memory limits. Scale up or optimize the configurations if necessary.

VKE AI training task

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
GPU Xid error	P0: Value > 30 for 10 minutes P1: Value > 0 for 10 minutes P2: Value > 0 for 5 minutes	increase(DCGM_CUSTOM_XID_ERRORS_TOTAL_COUNTER{cluster=~"$Cluster", job=~"dcgm\|dcgm-vci"}[5m])	Xid errors occur on GPUs within 5 minutes, and alerts are triggered by severity.	Check GPU node events, DCGM metrics, and driver logs. Check whether the alert is triggered by a hardware failure, driver exception, or abnormal task pressure. Isolate the node and migrate training tasks if necessary.
ENI link down	P0: Value > 0 for 5 minutes	increase(node_infiniband_link_downed_total{cluster=~"$Cluster"}[5m])	An RDMA/IB ENI link becomes unavailable.	Check switched fabrics, optical modules, cables, and node ENI status. Check whether the training task communication is affected. Remove the faulty node if necessary.
High rate of ENI error recovery	P1: Value > 5 for 10 minutes P2: Value > 5 for 5 minutes	increase(node_infiniband_link_error_recovery_total{cluster=~"$Cluster"}[5m])	The ENI link repeatedly recovers from error status, which poses instability risks.	Check for network jitter, switch port errors, driver versions, and node hardware health. Check whether the metric values continuously increase.
Excessive ENI errors	P1: Value > 5 for 10 minutes P2: Value > 5 for 5 minutes	increase(node_infiniband_local_link_integrity_errors_total{cluster=~"$Cluster"}[5m])	The total number of IB link integrity errors is large.	Check physical link quality, port error correction information, and switched fabric alerts. Replace links or nodes if necessary.
GPU Xid V2 error	P0: Value > 30 for 10 minutes P1: Value > 0 for 10 minutes P2: Value > 0 for 2 minutes	increase(DCGM_EXP_XID_ERRORS_COUNT{cluster=~"$Cluster", job=~"dcgm\|dcgm-vci"}[5m])	A GPU Xid error is also detected in another set of DCGM metrics as supplementary.	Check this error together with the GPU Xid error. Focus on the driver, firmware, and card health, as well as task exceptions.
Suspected GPU removal from nodes due to Xid error	P2: Value > 0 for 5 minutes	sum(kube_node_status_capacity{cluster=~"$Cluster", node!~"vci.", resource=~"nvidia_com_gpu\|vke_volcengine_com_mgpu_core"} - kube_node_status_allocatable{cluster=~"$Cluster", node!~"vci.", resource=~"nvidia_com_gpu\|vke_volcengine_com_mgpu_core"}) by (cluster, node, resource)	Determine whether GPU resources are abnormally lost by comparing capacity with allocatable resources.	Compare reported node GPU resources, device add-on status, and GPU driver logs. If necessary, restart related add-ons or replace the node.

VKE AIOPS service

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
GPU-related exception on VKE node	P2: Value > 0 for 1 minute	aiops_resource_condition_abnormal{cluster_id=~"$Cluster",resource_kind="VkeNode",condition=~"GpuCardFallen\|GpuMemoryErrorRecoverable\|GpuMemoryErrorUnrecoverable\|GpuLinkError\|GpuLinkUnhealthy\|GpuDriverError"}	AIOPS detects exceptions on the GPU, memory, link, or driver of a node.	Check node diagnostic results and GPU add-on logs. Decide whether to remove the node based on the impact on training tasks.
Kubelet exceptions	P2: Value > 0 for 1 minute	aiops_resource_condition_abnormal{cluster_id=~"$Cluster",resource_kind="VkeNode",condition=~"KubeletUnhealthy"}	AIOPS detects that the kubelet of a node is in an abnormal state.	Log in to the node and check kubelet logs, certificates, disks, runtime connections, and node heartbeats.
ContainerRuntime exception on VKE node	P2: Value > 0 for 1 minute	aiops_resource_condition_abnormal{cluster_id=~"$Cluster",resource_kind="VkeNode",condition=~"RuntimeUnhealthy"}	AIOPS identifies container runtime exceptions, such as containerd exceptions.	Check the runtime process, image pulling, disk, and socket connection. Restart the runtime if necessary and monitor node recovery.
NTP exceptions on VKE nodes	P2: Value > 0 for 1 minute	aiops_resource_condition_abnormal{cluster_id=~"$Cluster",resource_kind="VkeNode",condition=~"NTPProblem"}	AIOPS identifies node clock synchronization exceptions.	Check the NTP service, time source connectivity, and clock drift. Restore time synchronization first to prevent certificate and scheduling exceptions.
Self-healing starts	P2: Value > 0 for 1 minute	aiops_resource_remediation_in_progress{cluster_id=~"$Cluster"}	The AIOPS self-healing operation has started.	Focus on the self-healing target, scope of impact, and whether manual intervention is required. Avoid making repeated changes during self-healing.
Self-healing execution failure	P2: Value > 0 for 1 minute	aiops_resource_remediation_completed{cluster_id=~"$Cluster",phase="Failed"}	The self-healing operation failed to execute.	Check self-healing task logs and failure reasons. Check whether permissions, node status, or dependent resources are abnormal. Perform manual intervention if necessary.
Issues remaining after self-healing	P2: Value > 0 for 1 minute	aiops_resource_remediation_completed{cluster_id=~"$Cluster",phase="Completed",completion_status="completed_without_success"}	The self-healing process has completed, but the exception has not been resolved.	Continue manual troubleshooting based on root cause analysis, and evaluate whether to escalate the issue or remove the node.

VKE vArmor add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High average processing latency for workload admission requests	P0: Trigger immediately when the value is > 8 P1: Trigger immediately when the value is > 4 P2: Trigger immediately when the value is > 2	sum (rate(varmor_webhook_latency_sum{cluster=~"$Cluster"}[5m])) / sum (rate(varmor_webhook_latency_count{cluster=~"$Cluster"}[5m]))	The average processing latency of the vArmor webhook is high. This may affect workload admission.	Check the admission chain, webhook service replicas, resources, and downstream dependencies. Temporarily downgrade policies if necessary.
High ArmorProfile processing error rate	P0: Trigger immediately when the value is > 2 P1: Trigger immediately when the value is > 1 P2: Trigger immediately when the value is > 0.3	sum(rate(varmor_profile_processing_failure_total{cluster=~"$Cluster"}[5m]))	The ArmorProfile processing failures continue to increase.	Check profile settings, add-on logs, and recent policy changes. Check whether security policy distribution is affected.

VKE Katalyst add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
Katalyst Controller exception	P1: Value = 0 for 5 minutes P0: Value = 0 for 10 minutes	sum(up{cluster=~"$Cluster", job="katalyst"}) by (cluster, namespace, pod, instance)	Monitors whether the katalyst-controller pod is unavailable.	Prioritize restoring Pod availability. Check images, scheduling, probes, Pod logs, and resource limits.
Katalyst Metric exception	P1: Value = 0 for 5 minutes P0: Value = 0 for 10 minutes	sum(up{cluster=~"$Cluster", job="katalyst-metric"}) by (cluster, namespace, pod, instance)	Monitors whether the katalyst-metric Pod is unavailable.	Prioritize restoring Pod availability. Check images, scheduling, probes, Pod logs, and resource limits.
Katalyst Agent exception	P1: Value = 0 for 5 minutes P0: Value = 0 for 10 minutes	sum(up{cluster=~"$Cluster", job="katalyst-agent"}) by (cluster, namespace, pod, instance)	Monitors whether the katalyst-agent Pod is unavailable.	Prioritize restoring Pod availability. Check images, scheduling, probes, Pod logs, and resource limits.
High CPU Burst sync latency	P1: Value > 200000 for 5 minutes	sum(rate(sync_cpu_burst_duration_us_total{cluster=~"$Cluster", job="katalyst-agent"}[5m])) by (cluster, namespace, pod, instance)	Monitors the total time taken by the CPU Burst feature to update kernel configurations for all containers on the node.	Identify nodes where a large number of containers frequently update the cpu.cfs_quota_us kernel configuration, optimize CPU Burst feature-related parameters, and migrate workloads or scale out nodes if necessary.

VKE FSX service

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High error rate of csi-fsx server-side calls	P1: Value > 20 for 10 minutes P2: Value > 5 for 10 minutes	100 * (sum(rate(fsx_server_rpc_error_total{cluster=~"$Cluster",job="csi-fsx-node"}[5m])) by (cluster, instance, method) / clamp_min(sum(rate(fsx_server_rpc_total{cluster=~"$Cluster",job="csi-fsx-node"}[5m])) by (cluster, instance, method), 1e-9))	Monitors the RPC error rate on the csi-fsx server side. This metric is instrumented by the server-side ObserveFSXServerRPC, and is suitable for identifying whether server-side methods such as Mount, Umount, Recover, and Status are persistently failing.	Confirm whether the anomalies are concentrated on a few instances or a specific method. Then check the csi-fsx logs on the corresponding nodes, gRPC call trace, and dependency status. If the issues are mainly concentrated on mount or unmount methods, further cross-check node-side RPC errors and latency.
High P99 latency for fsx-agent node-side calls	P1: Value > 60000 for 10 minutes P2: Value > 30000 for 10 minutes	histogram_quantile(0.99, sum(rate(fsx_agent_pod_rpc_time_bucket{cluster=~"$Cluster",job="csi-fsx-node"}[5m])) by (cluster, instance, method, le)) * 1000	Monitors the P99 latency of node-side FSX client RPC calls. Query results are in milliseconds, so 60000 represents 60 seconds, and 30000 represents 30 seconds. This metric covers node-side operations such as Mount, Umount, Start, Stop, and Uninstall.	Identify slow nodes and slow operations by instance and method. Check FSX client logs, pod resource usage, node load, FSX instance status, and network connectivity. If necessary, cross-check whether fusedaemon control plane failures are concurrently increasing.
RPC errors occur on the fsx-agent node	P2: Trigger immediately when the value is > 0	sum(increase(fsx_agent_pod_rpc_error_total{cluster=~"$Cluster",job="csi-fsx-node"}[5m])) by (cluster, instance)	Monitors whether RPC errors occur on the node-side FSX client within a 5-minute window. This template adopts a zero-tolerance policy for such errors, triggering an alert as soon as any failure occurs, in order to detect anomalies in critical operations such as mount, unmount, and status queries as early as possible.	Confirm whether the issue is concentrated on specific nodes, then drill down by method and code dimensions. Check for FSX client response errors, gRPC connectivity issues, permission problems, mount target leftovers, or the client not being ready. If errors are mainly concentrated on Mount and the error code is 17, evaluate whether it is a duplicate mount scenario based on the code semantics.
Continuous increase in Fusedaemon control plane failures on fsx-agent	P1: Value > 0 for 5 minutes	sum(rate(fsx_fusedaemon_ctl_op_failed_ops{cluster=~"$Cluster",job="csi-fsx-agent"}[5m])) by (cluster, instance, instance_name, method, status)	Monitors whether control plane operation failures on the FSX client are continuously increasing. This metric is more indicative of client-side control link anomalies, such as control operation failures, process state maintenance anomalies, authentication issues, or configuration problems. It should not be directly equated to data plane I/O read/write failures.	Confirm whether anomalies are concentrated on a specific instance_name, instance, or method. Check the FSX client/fusedaemon logs on the corresponding nodes, client resource usage, authentication Secret, version status, and recent changes. If accompanied by a rise in node-side RPC latency, prioritize treating it as a node-level anomaly.

VKE Snapshot-controller add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High CSI call latency for snapshot creation	P1: Value > 15 for 10 minutes P2: Value > 5 for 5 minutes	histogram_quantile(0.99, sum by(cluster, driver_name, le)(rate(snapshot_controller_operation_total_seconds_bucket{cluster=~"$Cluster",operation_name="CreateSnapshot"}[5m])))	The P99 latency of the snapshot-controller calling the CSI driver to create a snapshot (CreateSnapshot) remains high. This stage only initiates the snapshot request and does not include the wait time for the snapshot to become ready. Under normal circumstances, it completes within a few seconds. Sustained high latency usually indicates slow processing by the CSI driver or abnormal responses from the underlying cloud storage APIs.	Check the snapshot-controller logs to confirm whether CreateSnapshot requests time out or return errors. Check the status and logs of the CSI driver (csi-ebs) Pod. Confirm whether the underlying cloud storage API responds normally.
Snapshot operation stalled	P1: > 0 for 15 minutes P2: Value > 0 for 10 minutes	snapshot_controller_operations_in_flight{cluster=~"$Cluster"} > 0 unless on(cluster) sum by(cluster)(rate(snapshot_controller_operation_total_seconds_count{cluster=~"$Cluster",operation_status="success"}[15m])) > 0	snapshot-controller has ongoing snapshot operations, but none have completed successfully within the specified duration. Normally, regardless of cluster size, snapshot operations should complete continuously. A complete stall usually indicates that the CSI driver is unavailable, the underlying cloud storage API is faulty, or the snapshot-controller itself is abnormal.	Check the status and logs of the csi-ebs driver Pod. Check VolumeSnapshot/VolumeSnapshotContent events. Check the snapshot-controller logs to locate the root cause.
High snapshot CPU saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"snapshot-controller.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"snapshot-controller.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the snapshot-controller add-on's Pod continuously approaches the container CPU Limit, which may cause slow responses or throttling.	Check the CPU usage trend and load of the corresponding Pod. Evaluate whether to adjust the resource Limits configuration. Check for abnormally high-frequency snapshot operations.
High snapshot memory saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"snapshot-controller.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"snapshot-controller.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the snapshot-controller add-on's Pod continuously approaches the container Memory Limit, posing a risk of OOM or service instability.	Check the memory usage trend of the corresponding Pod. Evaluate whether to adjust the memory Limits configuration. Check for memory leaks or abnormal growth.
snapshot-controller Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"snapshot-controller.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"snapshot-controller.",condition="false"} == 1)	The snapshot-controller add-on's Pod is not in the Ready state, which impacts the cluster's snapshot capability provided by this add-on.	Check the status and events of the snapshot-controller Pod in the kube-system namespace. Check image pulling, startup parameters, and resource limits. Confirm whether dependencies (APIServer and cloud credentials) are normal. View the container logs to locate the root cause.

VKE cr-credential-controller add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High CR credential controller CPU saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cr-credential-controller.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cr-credential-controller.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the cr-credential-controller add-on's Pod continuously approaches the container's CPU Limit, which may cause slow responses or throttling.	Check the CPU usage trend and load of the corresponding Pod. Check for high-frequency credential synchronization, Secret/ConfigMap changes, or abnormal retries. Evaluate whether to adjust the CPU Limits configuration.
High CR credential controller memory saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cr-credential-controller.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cr-credential-controller.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the cr-credential-controller add-on's Pod continuously approaches the container's Memory Limit, posing a risk of OOM or service instability.	Check the memory usage trend of the corresponding Pod. ‌Check for container restarts, OOMKilled events, and signs of GC/cache growth. Check for abnormal object backlog or memory leaks. Evaluate whether to adjust the memory Limits configuration.
CR credential controller Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cr-credential-controller.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cr-credential-controller.",condition="false"} == 1)	The cr-credential-controller add-on's Pod is not in the Ready state, which impacts the cluster capabilities provided by this add-on.	Check the status and events of the cr-credential-controller Pod in the kube-system namespace. Examine image pulls, startup parameters, resource limits, and probe configurations. Check whether the dependent services and the cluster APIServer are normal. View the container logs to locate the root cause.

VKE AIBrix add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
AIBrix controller instance anomalies	P0: Value == 0 for 10 minutes P1: Value == 0 for 5 minutes	sum(up{cluster=~"$Cluster", job="aibrix"}) by (cluster, namespace, pod, instance)	AIBrix controller-manager instance liveness check. When the up metric is 0, Prometheus cannot scrape the aibrix metrics endpoint. The Pod may be offline, in CrashLoop, or the network may be disconnected.	Check the aibrix Pod status, container logs, and ServiceMonitor configuration.
Critical panic detected	P0: Value > 0 for 10 minutes P1: Trigger immediately when the value is > 0	(sum(increase(controller_runtime_reconcile_panics_total{cluster=~"$Cluster",job="aibrix"}[5m])) or vector(0)) + (sum(increase(controller_runtime_webhook_panics_total{cluster=~"$Cluster",job="aibrix"}[5m])) or vector(0))	The AIBrix controller-runtime encountered a panic during reconcile or webhook processing.	Check the panic stack trace in controller-manager logs and contact the development team.
High reconcile error rate	P1: Value > 0.3 for 10 minutes P2: > 0.15 for 10 minutes	sum(rate(controller_runtime_reconcile_total{cluster=~"$Cluster",job="aibrix",result="error"}[5m])) / sum(rate(controller_runtime_reconcile_total{cluster=~"$Cluster",job="aibrix"}[5m]))	The error ratio of reconcile operations for each AIBrix controller remains high.	A high error rate usually indicates abnormal dependent services, resource conflicts, or controller logic bugs. Identify the specific error source by the controller and result labels.
High aibrix CPU Limits saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"aibrix-controller-manager.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"aibrix-controller-manager.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the aibrix add-on's Pod continuously approaches the container's CPU Limit, which may cause slow responses or throttling.	Check the resource configuration and load of the corresponding Pod.
High aibrix memory Limits saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"aibrix-controller-manager.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"aibrix-controller-manager.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the aibrix add-on's Pod continuously approaches the container's Memory Limit, posing a risk of OOM or service instability.	Check the memory usage and resource configuration of the corresponding Pod.
aibrix Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"aibrix-controller-manager.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"aibrix-controller-manager.",condition="false"} == 1)	The aibrix add-on's Pod is not in the Ready state, which impacts the cluster capabilities provided by this add-on.	Check the events, image pulls, dependent services, and container logs of the corresponding Pod.

VKE keda add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High keda CPU Limits saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(keda-admission-webhooks\|keda-operator\|keda-operator-metrics-apiserver).",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(keda-admission-webhooks\|keda-operator\|keda-operator-metrics-apiserver).",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the keda add-on's Pod continuously approaches the container CPU Limit, which may cause slow responses or throttling and affect the timeliness of autoscaling decisions.	Check the CPU usage trend and load of the corresponding Pod. Evaluate whether to adjust the resource Limits configuration. Check for abnormally high-frequency scaling events or a sudden increase in ScaledObjects.
High keda memory Limits saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(keda-admission-webhooks\|keda-operator\|keda-operator-metrics-apiserver).",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(keda-admission-webhooks\|keda-operator\|keda-operator-metrics-apiserver).",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the Pod for the keda add-on continuously approaches the container Memory Limit, posing a risk of OOM or service instability, which may interrupt the autoscaling controller.	Check the memory usage trend of the corresponding Pod. Evaluate whether to adjust the memory Limits configuration. Troubleshoot memory leaks or abnormal growth of ScaledObject or TriggerAuthentication resources.
keda Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(keda-admission-webhooks\|keda-operator\|keda-operator-metrics-apiserver).",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(keda-admission-webhooks\|keda-operator\|keda-operator-metrics-apiserver).",condition="false"} == 1)	The keda add-on's Pod is not in the Ready state, which impacts the cluster's event-driven autoscaling capabilities, including ScaledObject reconciliation and external metrics collection.	Check the status and events of the keda-related Pods (keda-operator, keda-admission-webhooks, and keda-operator-metrics-apiserver) in the kube-system namespace. Check image pulling, startup parameters, and resource limits. Confirm whether the dependent services (APIServer and metrics-server) are normal. View the container logs to locate the root cause.

VKE metrics-server

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High metrics-server CPU Limits saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"metrics-server.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"metrics-server.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the metrics-server add-on's Pod continuously approaches the container CPU Limit, which may cause slow responses or throttling. Check the resource configuration and workload of the corresponding Pod.	Check the CPU usage trend, current requests/limits, and recent load changes of the metrics-server Pod. Check the metrics-server logs for scraping timeouts, request backlogs, or throttling. Evaluate whether to adjust the CPU Limits or scale out the replicas.
High metrics-server memory Limits saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"metrics-server.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"metrics-server.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the metrics-server add-on's Pod continuously approaches the container's Memory Limit, posing a risk of OOM or service instability. Check the memory usage and resource configuration of the corresponding Pod.	Check the Pod memory trend, restart records, and OOMKilled events. Check the metrics-server logs for continuous memory growth, abnormal scraping, or object accumulation. Evaluate whether to adjust the memory Limits, and identify the root cause together with the cluster's recent scale changes.
metrics-server Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"metrics-server.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"metrics-server.",condition="false"} == 1)	The metrics-server add-on's Pod is not in the Ready state, which impacts the cluster capabilities provided by this add-on. Check the events, image pull, dependent services, and container logs of the corresponding Pod.	Check the status, events, and recent restart records of the metrics-server Pod in the kube-system namespace. Check image pulls, startup parameters, certificate configurations, and connectivity to kubelet and APIServer. View the container logs to locate the root cause, such as startup failure, authentication failure, or probe failure.
metrics-server container metrics unavailable	P1: Value == 0 for 10 minutes P2: == 0 for 5 minutes	max(metrics_server_storage_points{cluster=~"$Cluster",type="container"}) by (cluster)	metrics-server currently stores no container metric samples, indicating that the resource metrics directly relied on by HPA are entirely unavailable. Prioritize checking the status of metrics-server itself, the overall scraping path of kubelet, and the component logs.	Check the metrics-server Pod status and logs to confirm whether the add-on is serving metrics normally. Verify whether the scraping path from metrics-server to kubelet, the certificate configuration, and the network connectivity are normal. Use the resource metrics API or verify metric recovery on the HPA side. If necessary, further troubleshoot the issue using the kubelet metric endpoint.

VKE cronhpa add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High cronhpa CPU Limits saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cronhpa.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cronhpa.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the cronhpa add-on's Pod continuously approaches the container's CPU Limit, which may cause slow responses or throttling, affecting the on-time execution of CronHPAs.	Check the CPU usage trend and load of the corresponding Pod. Evaluate whether to adjust the resource Limits configuration. Check for abnormally high-frequency CronHPA rules or a sudden increase in CronHPA resources.
High cronhpa memory Limits saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cronhpa.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cronhpa.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the cronhpa add-on's Pod continuously approaches the container's Memory Limit, posing a risk of OOM or service instability. This may cause scheduled scaling tasks to be lost or delayed.	Check the memory usage trend of the corresponding Pod. Evaluate whether to adjust the memory Limits configuration. Check for memory leaks or abnormal growth of CronHPA resources.
cronhpa Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cronhpa.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"cronhpa.",condition="false"} == 1)	The cronhpa add-on's Pod is not in the Ready state, which impacts the cluster's autoscaling capability. Scaling based on Cron schedules cannot be executed on time.	Check the status and events of the cronhpa Pod in the kube-system namespace. Check image pulling, startup parameters, and resource limits. Confirm whether the dependent service (APIServer) is normal. View the container logs to locate the root cause.

VKE prometheus-adapter add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High prometheus-adapter CPU Limits saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"prometheus-adapter.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"prometheus-adapter.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the prometheus-adapter add-on's Pod continuously approaches the container's CPU Limit, which may cause slow responses or throttling. Check the resource configuration and workload of the corresponding Pod.	Check the CPU usage trend, resource configuration, and request volume changes of the prometheus-adapter Pod. Check the adapter logs for Prometheus query timeouts, slow rule evaluation, or request backlog. Evaluate whether to adjust the CPU Limits, and check whether the upstream call volume is abnormal.
High prometheus-adapter memory Limits saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"prometheus-adapter.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"prometheus-adapter.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the prometheus-adapter add-on's Pod continuously approaches the container Memory Limit, posing a risk of OOM or service instability. Check the memory usage and resource configuration of the corresponding Pod.	Check the Pod memory trend, restart records, and OOMKilled events. Check the adapter logs for large query results, cache accumulation, or abnormal retries. Evaluate whether to adjust the memory Limits, and check for abnormal query rules or call traffic.
prometheus-adapter Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"prometheus-adapter.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"prometheus-adapter.",condition="false"} == 1)	The prometheus-adapter add-on's Pod is not in the Ready state, which impacts the cluster capabilities provided by this add-on. Check the events, image pull, dependent services, and container logs of the corresponding Pod.	Check the status, events, and recent restart records of the prometheus-adapter Pod in the kube-system namespace. Check the image, startup parameters, rule configuration, and connectivity to Prometheus and the APIServer aggregation layer. View the container logs to locate the root cause, such as startup failure, registration anomalies, or unreachable dependencies.

VKE p2p-accelerator add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High p2p-accelerator CPU Limits saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(cr-)?p2p-accelerator.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(cr-)?p2p-accelerator.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the p2p-accelerator add-on's Pod continuously approaches the container's CPU Limit, which may cause slow responses or throttling. The Pod matching rule covers both cr-p2p-accelerator and p2p-accelerator rendered resources.	Check the CPU usage trend and load of the corresponding Pod. Check for large-scale image distribution, warm-up tasks, or abnormal download requests. Confirm whether both p2p-accelerator and the optional preheat add-on are within the expected scope. Evaluate whether to adjust the CPU Limits configuration.
Too high p2p-accelerator memory limits saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(cr-)?p2p-accelerator.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(cr-)?p2p-accelerator.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the p2p-accelerator add-on's Pod continuously approaches the container's Memory Limit, posing a risk of OOM or service instability. The Pod matching rule covers both cr-p2p-accelerator and p2p-accelerator rendered resources.	Check the memory usage trend of the corresponding Pod. View signs of container restarts, OOMKilled events, and cache growth. Check for abnormal download queues, warm-up tasks, or growth in the connection count. Evaluate whether to adjust the memory Limits configuration.
p2p-accelerator Pod unavailable	P1: Value == 1 for 10 minutes P2: == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(cr-)?p2p-accelerator.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(cr-)?p2p-accelerator.",condition="false"} == 1)	The p2p-accelerator add-on's Pod is not in the Ready state, which impacts the cluster P2P acceleration capabilities provided by this add-on.	Check the status and events of the p2p-accelerator-related Pods in the kube-system namespace. Examine image pulls, startup parameters, resource limits, and probe configurations. Confirm whether the network, storage, and cluster APIServer are normal. View the container logs to locate the root cause.

VKE image-accelerator add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High image-accelerator CPU Limits saturation	P2: > 80% for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace="kube-system",pod=~"image-accelerator.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace="kube-system",pod=~"image-accelerator.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	Monitors the CPU Usage / Limit ratio of the image-accelerator add-on's Pod. This rule is triggered when it continuously approaches the container CPU Limit. It detects CPU resource insufficiency or throttling risk of the add-on.	Identify highly-saturated containers by cluster, namespace, pod, and container. Check the Pod CPU usage curve, CPU Limit configuration, add-on request pressure, and throttling status. Increase resource limits or scale out the add-on if necessary.
High image-accelerator memory Limits saturation	P2: > 80% for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace="kube-system",pod=~"image-accelerator.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace="kube-system",pod=~"image-accelerator.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Monitors the Memory Working Set / Memory Limit ratio of the image-accelerator add-on's Pod. This rule is triggered when the value continuously approaches the container's Memory Limit. It helps detect the risk of OOM or service instability.	Identify highly saturated containers‌ by cluster, namespace, pod, and container. Check memory usage trends, OOMKilled/restart events, memory limit configurations, and recent traffic fluctuations. ‌Adjust memory quotas‌ or ‌investigate abnormal memory growth‌ as needed.
No available replicas for image-accelerator Pods	P1: Value > 0 for 10 minutes P2: Value > 0 for 5 minutes	(max(kube_deployment_spec_replicas{cluster=~"$Cluster",namespace="kube-system",deployment="image-accelerator"}) by (cluster, namespace, deployment) > 0) and on(cluster, namespace, deployment) (max(kube_deployment_status_replicas_available{cluster=~"$Cluster",namespace="kube-system",deployment="image-accelerator"}) by (cluster, namespace, deployment) == 0)	Monitors whether the image-accelerator Deployment continuously has no available replicas when the desired replica count is greater than 0. It is not triggered when the add-on is not installed or the replica count is manually scaled to 0.	Confirm whether Deployment spec.replicas is greater than 0. Check Deployment rollout, Pod events, image pulls, resource scheduling, probe failures, and container logs. Check whether the add-on is not installed or has been manually scaled in.
High image-accelerator webhook admission P90 latency	P1: > 1s for 10 minutes P2: > 0.5s for 10 minutes	histogram_quantile(0.90, sum(rate(apiserver_admission_webhook_admission_duration_seconds_bucket{cluster=~"$Cluster",job="kubernetes-apiserver",name="image-accelerator-admission.kube-system.mutating",type="admit",rejected="false"}[5m])) by (cluster, name, type, operation, le)) and on(cluster, name, type, operation) (sum(increase(apiserver_admission_webhook_admission_duration_seconds_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="image-accelerator-admission.kube-system.mutating",type="admit",rejected="false"}[5m])) by (cluster, name, type, operation) >= 20) and on(cluster) (max(kube_deployment_spec_replicas{cluster=~"$Cluster",namespace="kube-system",deployment="image-accelerator"}) by (cluster) > 0)	Monitors the P90 latency of image-accelerator mutating webhook admission calls based on kube-apiserver admission webhook histogram metrics. At least 20 calls within 5 minutes are required to trigger the rule, preventing false positives from low-traffic samples.	Confirm whether the call volume meets the threshold, and locate slow calls by cluster, name, type, and operation. Check the image-accelerator Pod resources, server-side logs, Service/Endpoint/TLS status, and the apiserver admission path. Check whether the Pod creation or update path is slowed down by the webhook.
High image-accelerator webhook error rate	P2: > 30% for 5 minutes	(sum(increase(apiserver_admission_webhook_fail_open_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="image-accelerator-admission.kube-system.mutating",type="admit"}[5m])) by (cluster, name, type) / clamp_min(sum(increase(apiserver_admission_webhook_admission_duration_seconds_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="image-accelerator-admission.kube-system.mutating",type="admit"}[5m])) by (cluster, name, type), 1)) and on(cluster, name, type) (sum(increase(apiserver_admission_webhook_admission_duration_seconds_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="image-accelerator-admission.kube-system.mutating",type="admit"}[5m])) by (cluster, name, type) >= 20) and on(cluster, name, type) (sum(increase(apiserver_admission_webhook_fail_open_count{cluster=~"$Cluster",job="kubernetes-apiserver",name="image-accelerator-admission.kube-system.mutating",type="admit"}[5m])) by (cluster, name, type) >= 3) and on(cluster) (max(kube_deployment_spec_replicas{cluster=~"$Cluster",namespace="kube-system",deployment="image-accelerator"}) by (cluster) > 0)	Monitors the proportion of kube-apiserver calls to the image-accelerator mutating webhook that fail and are allowed through by failurePolicy=Ignore. At least 20 calls and fail-open events within 5 minutes are required to trigger this rule, preventing false positives caused by a single failure in low-traffic scenarios.	Confirm whether the call volume and fail-open count meet the threshold. Check MutatingWebhookConfiguration, Service/Endpoint, TLS certificates, network connectivity, and webhook process logs. Evaluate the impact of the mutation capability downgrade on Pod creation or updates.

VKE mgpu add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
mgpu Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(ek8s-gpu-agent\|ek8s-gpu-exporter).",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(ek8s-gpu-agent\|ek8s-gpu-exporter).",condition="false"} == 1)	The Pod of the mgpu add-on is not in the Ready state, which impacts the GPU virtualization capabilities of the add-on.	Check the status and events of the ek8s-gpu-agent and ek8s-gpu-exporter Pods in the kube-system namespace. Examine image pulls, startup parameters, probes, and resource configurations. Check whether the node status and add-on dependencies are normal. View the container logs to locate the root cause.
Unhealthy mgpu add-on	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	mgpu_healthy{cluster=~"$Cluster",job="mgpu-agent"}	Monitors the health status of the mgpu add-on.	Identify the abnormal mgpu add-on by cluster and instance. Check add-on Pod status, logs, and node GPU status, and restore add-on operation.
High failure rate of the mgpu device plugin	P1: Value > 0.5 for 10 minutes P2: Value > 0.2 for 5 minutes	( sum by (cluster, nodename, resource, operation) ( increase(mgpu_device_plugin_operations_total{cluster=~"$Cluster",job="mgpu-agent",result="error"}[5m]) ) / sum by (cluster, nodename, resource, operation) ( increase(mgpu_device_plugin_operations_total{cluster=~"$Cluster",job="mgpu-agent"}[5m]) ) ) and on (cluster, nodename, resource, operation) ( sum by (cluster, nodename, resource, operation) ( increase(mgpu_device_plugin_operations_total{cluster=~"$Cluster",job="mgpu-agent"}[5m]) ) > 0 )	Monitors the failure rate of various operations of the mgpu device plugin.	Locate failed operations by nodename, resource, and operation. Check device plugin logs, mGPU resource status, and Pod events, and fix device allocation exceptions.

VKE rdma-device-plugin add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High RDMA plugin CPU saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"network-device-plugin.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"network-device-plugin.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the rdma-device-plugin add-on's Pod continuously approaches the container's CPU Limit, which may cause slow responses or throttling.	View the actual CPU usage / Limit ratio of the network-device-plugin Pod and check for abnormally high-frequency calls. Increase the CPU Limit or recreate the Pod if necessary.
High RDMA plugin memory saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"network-device-plugin.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"network-device-plugin.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the rdma-device-plugin add-on's Pod continuously approaches the container Memory Limit, posing a risk of OOM or service instability.	Check the Memory Working Set / Limit ratio of the network-device-plugin Pod and check for memory leaks or abnormal growth. Increase the Memory Limit or recreate the Pod to prevent OOM if necessary.
rdma-device-plugin Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"network-device-plugin.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"network-device-plugin.",condition="false"} == 1)	The rdma-device-plugin add-on's Pod is not in the Ready state, which impacts the cluster capabilities provided by this add-on.	View the status and events of the target Pod to identify why it is not ready (such as image pull failure, insufficient resources, or health check failure). Confirm the Pod is ready after remediation.

VKE node-problem-detector add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High node-problem-detector CPU saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-problem-detector.",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-problem-detector.",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	CPU usage of the node-problem-detector add-on's Pod continuously approaches the container CPU Limit, which may cause slow responses or throttling.	Check the CPU usage trend and load of the corresponding Pod. Evaluate whether to adjust the resource Limits configuration. Check for abnormal detection loops or sudden growth in log volume.
High node-problem-detector memory saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-problem-detector.",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-problem-detector.",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	Memory usage of the node-problem-detector add-on's Pod continuously approaches the container's Memory Limit, posing a risk of OOM or service instability.	Check the memory usage trend and restart records of the corresponding Pod. Evaluate whether to adjust the memory Limits configuration. Check for memory leaks or abnormal growth.
node-problem-detector Pod unavailable	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-problem-detector.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"node-problem-detector.",condition="false"} == 1)	The node-problem-detector add-on's Pod is not in the Ready state, which impacts monitoring and reporting of abnormal node status.	Check the status and events of the node-problem-detector Pod in the kube-system namespace. Examine image pulls, startup parameters, probes, and resource configurations. Confirm whether the node and dependent services are normal. View the container logs to locate the root cause.

VKE kube-proxy service

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
kube-proxy Pod not ready for an extended period	P1: Value == 1 for 15 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace="kube-system",pod=~"kube-proxy-.+",phase!="Succeeded"} == 1) and on(cluster, namespace, pod)(kube_pod_status_ready{cluster=~"$Cluster",namespace="kube-system",pod=~"kube-proxy-.+",condition="false"} == 1)	The kube-proxy Pod remains in a not-ready state, and the corresponding node may not be able to maintain Service forwarding rules promptly.	Check the status, events, and node of the alerting Pod in the kube-system namespace. Troubleshoot image pulls, startup parameters, readiness probes, container logs, node status, and connectivity to the API Server. After remediation, confirm that the Pod recovers to Ready and the corresponding DaemonSet replicas are available.
kube-proxy DaemonSet replicas unavailable	P1: > 0 for 15 minutes P2: Value > 0 for 5 minutes	max by (cluster, namespace, daemonset) (kube_daemonset_status_number_unavailable{cluster=~"$Cluster",namespace="kube-system",daemonset="kube-proxy"})	The kube-proxy DaemonSet has desired replicas that have not reached an available state, indicating that at least one node is missing an available kube-proxy Pod.	Check the desired, current, ready, and available replica counts of the kube-proxy DaemonSet in the kube-system namespace. Identify the nodes that lack available Pods, and check Pod scheduling, node taints, node status, and related events. View the corresponding Pod logs, fix the anomalies, and confirm that the number of unavailable replicas returns to 0.
Frequent kube-proxy Pod restarts	P1: Value > 2 for 5 minutes	max by (cluster, namespace, pod, container)(increase(kube_pod_container_status_restarts_total{cluster=~"$Cluster",namespace="kube-system",pod=~"kube-proxy-.+",container="kube-proxy"}[15m]))	The kube-proxy container has restarted more than twice in the last 15 minutes, which may cause the corresponding node to temporarily lose its rule maintenance capability.	Check the events, container exit reason, last termination state, and restart timestamp of the alerting Pod. Review the current and previous container logs to check for OOM kills, startup failures, insufficient node resources, and API Server connectivity issues. After remediation, confirm that the container is running stably and that the restart count is no longer increasing.
kube-proxy Endpoint changes persistently backlogged	P1: Value > 0 for 30 minutes P2: Value > 0 for 10 minutes	max by (cluster, node) (kubeproxy_sync_proxy_rules_endpoint_changes_pending{cluster=~"$Cluster",job="kube-proxy"}) and on (cluster, node)max by (cluster, node) (up{cluster=~"$Cluster",job="kube-proxy"} == 1)	kube-proxy instances with normal metric collection have a sustained backlog of unprocessed Endpoint changes. Related backend changes may not be reflected in node forwarding rules promptly. A brief non-zero value of this metric is acceptable. This alert rule only triggers on persistent backlogs.	Identify the backlogged instances by cluster and node, and check kube-proxy logs and rule synchronization latency. Check the API Server watch pipeline, EndpointSlice change rate, node load, and iptables/IPVS rule programming errors. After recovery, confirm that the number of pending Endpoint changes returns to 0, and verify that the backends of the affected Services are accessible.
kube-proxy Service changes persistently backlogged	P1: Value > 0 for 30 minutes P2: Value > 0 for 10 minutes	max by (cluster, node) (kubeproxy_sync_proxy_rules_service_changes_pending{cluster=~"$Cluster",job="kube-proxy"}) and on (cluster, node) max by (cluster, node) (up{cluster=~"$Cluster",job="kube-proxy"} == 1)	kube-proxy instances with normal metric collection have a sustained backlog of unprocessed Service changes. Related Service configurations may not be reflected in node forwarding rules promptly. A brief non-zero value of this metric is acceptable. This alert rule only triggers on persistent backlogs.	Identify the backlogged instances by cluster and node, and check kube-proxy logs and rule synchronization latency. Check the API Server watch pipeline, recent Service changes, node load, and iptables/IPVS rule programming errors. After remediation, confirm that the number of pending Service changes returns to 0 and verify the access paths of the affected Services.
kube-proxy rule synchronization P99 is too high	P1: > 5s for 10m P2: > 1s for 15m	histogram_quantile(0.99,sum by (cluster, node, ip_family, le)(rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket{cluster=~"$Cluster",job="kube-proxy"}[5m]))) and on (cluster, node, ip_family)(sum by (cluster, node, ip_family) (increase(kubeproxy_sync_proxy_rules_duration_seconds_count{cluster=~"$Cluster",job="kube-proxy"}[5m])) >= 5)	The P99 latency for proxy rule synchronization on a single node and single IP family has remained elevated. The speed at which Service or EndpointSlice changes are written to the node forwarding rules may be slowing down. This rule is only evaluated when at least 5 synchronization observations have been recorded in the last 5 minutes.	Identify slow instances by cluster, node, and ip_family, and confirm that the synchronization sample size in the last 5 minutes meets the evaluation conditions. Check kube-proxy logs, rule scale, recent Service/EndpointSlice changes, node CPU, and system load. Investigate iptables/IPVS rule programming failures or lock contention. After recovery, confirm that the synchronization P99 latency has returned to the cluster's historical baseline.
kube-proxy API 429 or 5xx ratio is too high	P1: > 30% for 10m P2: > 10% for 10m	(sum by (cluster) (increase(rest_client_requests_total{cluster=~"$Cluster",job="kube-proxy",code=~"429\|5.."}[10m])) / clamp_min(sum by (cluster) (increase(rest_client_requests_total{cluster=~"$Cluster",job="kube-proxy"}[10m])), 1)) and on (cluster) (sum by (cluster) (increase(rest_client_requests_total{cluster=~"$Cluster",job="kube-proxy"}[10m])) >= 20) and on (cluster) (sum by (cluster) (increase(rest_client_requests_total{cluster=~"$Cluster",job="kube-proxy",code=~"429\|5.."}[10m])) >= 3)	The combined proportion of HTTP 429 and 5xx responses returned by the API Server in kube-proxy's client-go requests has remained persistently high, which may delay or block the retrieval of Service and EndpointSlice changes. Status code 429 indicates that the API Server is rate-limiting requests. Status code 5xx indicates that the API Server failed to process the request. This rule is only evaluated when at least 20 requests and a combined total of at least 3 429/5xx responses have been recorded in the last 10 minutes. This does not include network transport errors and is not equivalent to client-side local rate-limiting waits.	Distinguish between 429 and 5xx by HTTP status code, and confirm that the total request volume, abnormal request count, and error ratio all meet the trigger conditions. For 429 errors, check the API Server load and rate-limiting status. For 5xx errors, check the API Server health, logs, and its dependent components. Check kube-proxy watch/reconnect logs and verify whether Endpoint and Service change events are backlogged. After recovery, confirm that the error rate and backlog metrics have returned to normal levels.

VKE flannel add-on

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
Frequent flannel Pod restarts	P0: > 2 for 5m	max by (cluster, namespace, pod, container) ( increase(kube_pod_container_status_restarts_total{cluster=~"$Cluster",namespace="kube-flannel",pod=~"kube-flannel-ds-.*",container="kube-flannel"}[15m]))	Monitors the number of container restarts of the flannel Pods within 15 minutes. Frequent restarts may cause repeated interruptions to the Pod network configuration of the corresponding node.	Check the status, events, and restart records of the alerting Pod in the kube-flannel namespace. View the current and previous container logs. Check for PodCIDR conflicts, APIServer connectivity, iptables configurations, and process anomalies. After remediation, confirm that the restart count stops growing.
Flannel Pod not ready for an extended period	P2: == 1 for 10m	(kube_pod_status_phase{cluster=~"$Cluster",namespace="kube-flannel",pod=~"kube-flannel-ds-.",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace="kube-flannel",pod=~"kube-flannel-ds-.",condition="false"} == 1)	Monitors whether the flannel Pods remain in a not-ready state. An anomaly prevents the corresponding node from completing Pod network configuration normally.	Check the status, events, readiness probes, and recent restarts of the alerting Pod. Review image pulls, startup configuration, and resource limits. Check the connectivity from the Pod to the API Server and view the container logs. After remediation, confirm that the Pod recovers to Ready.
flannel DaemonSet has unavailable replicas on nodes	P0: Value > 0 for 10 minutes	max by (cluster, namespace, daemonset) ( kube_daemonset_status_number_unavailable{cluster=~"$Cluster",namespace="kube-flannel",daemonset="kube-flannel-ds"})	Monitors whether kube-flannel-ds has unavailable replicas. An anomaly indicates that at least one node expected to run flannel does not have an available Pod.	Check the desired, current, ready, and available replica counts of the DaemonSet and identify the affected nodes. Check node status, taints, resource shortages, and Pod scheduling events. Check the status and logs of the corresponding flannel Pods. After remediation, confirm that the number of unavailable replicas returns to 0.
Node network not ready for an extended period	P2: == 1 for 10m	kube_node_status_condition{cluster=~"$Cluster",condition="NetworkUnavailable",status="true",node!~"vci.*"}	Monitors whether the NetworkUnavailable condition of non-VCI nodes remains true, indicating that the node network is not yet ready.	Identify the node based on alert labels and check the node conditions and events. Confirm whether the node's PodCIDR is correctly assigned. Check the flannel Pod, subnet configuration, and container logs on the node. Examine the node route and iptables status. After remediation, confirm that NetworkUnavailable returns to false.
High flannel CPU limits saturation	P2: > 0.4 for 10m	sum by (cluster, namespace, pod, container) ( rate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace="kube-flannel",pod=~"kube-flannel-ds-.",container="kube-flannel",image!=""}[5m])) / sum by (cluster, namespace, pod, container) ( kube_pod_container_resource_limits{cluster=~"$Cluster",namespace="kube-flannel",pod=~"kube-flannel-ds-.",resource="cpu",container!=""})	Monitors the ratio of the flannel container's average CPU usage over the last 5 minutes to its CPU limit. A ratio persistently exceeding 40% may indicate frequent events, abnormal add-on retries, or busy looping.	Identify high-saturation containers by cluster, namespace, pod, and container. Check CPU usage trends, limit configurations, and container logs. Troubleshoot frequent node or subnet events, APIServer watch/reconcile anomalies, and busy loops in processes. Evaluate whether to adjust the resource limit, restart, or upgrade the add-on based on historical baselines.
High flannel memory limits saturation	P2: > 0.25 for 10m	sum by (cluster, namespace, pod, container) ( container_memory_working_set_bytes{cluster=~"$Cluster",namespace="kube-flannel",pod=~"kube-flannel-ds-.",container="kube-flannel",image!=""}) / sum by (cluster, namespace, pod, container) ( kube_pod_container_resource_limits{cluster=~"$Cluster",namespace="kube-flannel",pod=~"kube-flannel-ds-.",resource="memory",unit="byte",container!=""})	Monitors the ratio of the flannel container's memory working set to its memory limit. A ratio persistently exceeding 25% may indicate object cache growth, event backlog, or a potential memory leak.	Identify high-saturation containers by cluster, namespace, pod, and container. Check memory usage trends, restart records, and OOMKilled events. Use logs to investigate object cache growth, event backlog, or memory leaks. Review the memory limit and evaluate whether the limit needs to be adjusted or whether the add-on needs to be restarted or upgraded, based on historical baseline data.

VKE VEPFS service

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High csi-vepfs CPU limit saturation	P1: Value > 0.9 for 5 minutes P2: Value > 0.8 for 5 minutes	sum(rate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-vepfs-controller\|csi-vepfs-node\|vepfs-client).",container!=""}[5m])) by (cluster, namespace, pod, container) / clamp_min(sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-vepfs-controller\|csi-vepfs-node\|vepfs-client).",resource="cpu",container!=""}) by (cluster, namespace, pod, container), 1e-9)	The CPU usage of the csi-vepfs add-on Pod is continuously approaching the container CPU limit. This may cause the controller or node add-on to respond slowly, experience request queuing, or be throttled by CFS.	Identify the highly saturated container by cluster, namespace, pod, and container. View the Pod CPU usage trend and throttling situation. Check for recent batch volume operations, client Pod maintenance, or abnormal retries. Evaluate whether the CPU limit needs to be adjusted or the pressure needs to be split.
High csi-vepfs memory limit saturation	P1: Value > 0.9 for 5 minutes P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-vepfs-controller\|csi-vepfs-node\|vepfs-client).",container!=""}) by (cluster, namespace, pod, container) / clamp_min(sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-vepfs-controller\|csi-vepfs-node\|vepfs-client).",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container), 1e-9)	The memory working set of the csi-vepfs add-on Pod is continuously approaching the container memory limit. This poses a risk of out of memory (OOM) events, restarts, or service instability.	Check the memory working set and limit configuration by pod and container. Check whether OOMKilled, restarts, or GC exceptions have occurred. Check the controller/node logs to determine whether an abnormal accumulation of objects exists. Adjust resource configurations or troubleshoot memory leaks if necessary.
Unavailable csi-vepfs Pods	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-vepfs-controller\|csi-vepfs-node\|vepfs-client).",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-vepfs-controller\|csi-vepfs-node\|vepfs-client).",condition="false"} == 1)	The csi-vepfs controller, node, or vepfs-client Pod is in a non-ready state, which will affect volume lifecycle operations, node mount service maintenance, or client Pod capabilities.	Check the status and events of the corresponding Pod in the kube-system namespace. Troubleshoot image pulling, scheduling, probes, insufficient resources, and container startup failures. View controller/node/client Pod logs. Confirm whether the dependent Kubernetes API, OpenAPI, and backend storage services are normal.
csi-vepfs control plane RPC call failures	P1: Value > 5 for 5 minutes P2: Value > 0 for 5 minutes	sum(increase(csi_sidecar_operations_seconds_count{cluster=~"$Cluster",job=~"csi-vepfs-.*",grpc_status_code!="OK"}[2m])) by (cluster, job, method_name)	The csi-vepfs control plane sidecar fails to call the driver's gRPC operations. This may affect control plane operations such as volume creation, deletion, publishing, unpublishing, or expansion.	Identify the failed operation by cluster, job, and method_name. Check the logs of csi-vepfs-controller and related sidecars. Confirm user-side impact by checking PVC/PV/VolumeAttachment events. Check APIs, backend storage, permissions, and network dependencies.
csi-vepfs node RPC call failures	P1: Value > 5 for 5 minutes P2: Value > 0 for 5 minutes	sum(increase(csi_operations_seconds_count{cluster=~"$Cluster",job="kubelet",driver_name="vepfs.csi.volcengine.com",grpc_status_code!="OK"}[2m])) by (cluster, node, method_name)	gRPC operations fail when kubelet calls the csi-vepfs node driver. This may cause volume mount, unmount, or expansion failures on the node, thereby blocking Pod startup.	Identify the failed node and CSI method by node and method_name. Check the csi-vepfs-node logs on the corresponding node. View the kubelet logs and Pod mount events. Check the GPFS mount status, kernel modules, and mount service connectivity on the node.
High error rate of the csi-vepfs API	P1: Value > 20 for 10 minutes P2: Value > 5 for 10 minutes	100 * ( sum(rate(csi_vepfs_openapi_request_total{cluster=~"$Cluster",result="error"}[5m])) by (cluster, service, operation) / clamp_min(sum(rate(csi_vepfs_openapi_request_total{cluster=~"$Cluster"}[5m])) by (cluster, service, operation), 1e-9) )	The error rate of the csi-vepfs controller calling vePFS, HPC, ECS, or VPC APIs continues to rise. This may affect dynamic volume creation, fileset changes, HPC binding, or node security group configurations.	Identify the failed API by service and operation. Check the request error codes and RequestId in the csi-vepfs-controller logs. Verify cloud service quotas, permissions, regions, and network connectivity. If failures are concentrated on a single external service, contact the corresponding service team to troubleshoot the issue.
High P99 latency of the csi-vepfs API	P1: Value > 25 for 10 minutes P2: Value > 10 for 10 minutes	histogram_quantile(0.99, sum(rate(csi_vepfs_openapi_request_duration_seconds_bucket{cluster=~"$Cluster"}[5m])) by (cluster, service, operation, le))	The P99 latency of external API calls from the csi-vepfs controller remains persistently high. This may cause CSI control plane operations to slow down or approach the 30-second single-request timeout budget.	Identify the source of slow calls by service and operation. Check for an increase in the error rate during the same period. View slow requests and retry records in the controller log. Confirm the root cause by combining external cloud service status, throttling, and network latency.
csi-vepfs fileset unavailability observed	P2: Value > 0 for 0 seconds	sum(increase(csi_vepfs_openapi_fileset_unavailable_observed_total{cluster=~"$Cluster"}[30m])) by (cluster)	csi-vepfs observed that the fileset is not in the Running state in the DescribeFilesets result. This metric indicates that an unavailable state has been observed. The 30-minute observation window is specified in the alert rule.	Check the filesystemID, filesetID, filesetName, and status in the controller logs. Confirm the current status of the fileset in the vePFS control plane. Determine whether it is a brief state transition or a persistent unavailability. If the exception persists, continue troubleshooting by combining backend file system events.
High csi-vepfs native command error rate	P1: Value > 20 for 10 minutes P2: Value > 5 for 10 minutes	100 * ( sum(rate(csi_vepfs_native_command_execution_total{cluster=~"$Cluster",job=~"csi-vepfs-(controller\|node)",result="error"}[5m])) by (cluster, job, owner, instance, location, command, error_class) / on(cluster, job, owner, instance, location, command) group_left(error_class) clamp_min(sum(rate(csi_vepfs_native_command_execution_total{cluster=~"$Cluster",job=~"csi-vepfs-(controller\|node)"}[5m])) by (cluster, job, owner, instance, location, command), 1e-9) )	The error rate of csi-vepfs executing native commands (GPFS, system, or within containers) continues to rise. This may lead to failures in node mount service changes, mount recovery, or kernel module processing.	Identify the failure source by job, owner, instance, location, command, and error_class. Distinguish between controller-side and node-side command failures. Check the corresponding add-on logs and command outputs. For GPFS command failures, focus on checking the admin node, mount status, SSH/exec paths, and backend cluster status.
csi-vepfs native command timeout or transmission exception	P1: Value > 5 for 5 minutes P2: Value > 0 for 0 seconds	sum(increase(csi_vepfs_native_command_execution_total{cluster=~"$Cluster",job=~"csi-vepfs-(controller\|node)",result="error",error_class=~"timeout\|transport\|remote_shell"}[5m])) by (cluster, job, owner, instance, location, command, error_class, error_code)	csi-vepfs encountered timeout, transport, or remote_shell errors while executing native commands. This usually indicates an exception in the execution trace of remote admin, kube exec, CRI exec, or local commands.	Distinguish between timeout, transport errors, and remote shell errors by error_class and error_code. Check the network connectivity of the node or controller corresponding to the instance. Troubleshoot admin access, kube exec, CRI exec, and the local command environment. Determine the stage where the failure occurred by combining command and location.
Abnormal nodes exist in the csi-vepfs file system cluster	P1: Value > 0 for 30 minutes P2: Value > 0 for 10 minutes	max(csi_vepfs_fs_cluster_abnormal_nodes_count{cluster=~"$Cluster"}) by (cluster, mount_service_id)	csi-vepfs has observed that the GPFS cluster corresponding to the mount service has abnormal nodes that are neither in the active nor the down state. This may affect node mount service changes, recovery, or subsequent mounts.	Identify the corresponding file system cluster by mount_service_id. Check the node status collection results in the controller logs. Confirm the abnormal node status in the backend GPFS cluster. If the exception persists, prioritize restoring the GPFS node status and then observe whether the csi-vepfs metrics recover.

VKE NAS service

Rule name	Trigger condition	PromQL configuration	Rule description	Alert handling process
High csi-nas CPU limit saturation	P2: Value > 0.8 for 5 minutes	sum(irate(container_cpu_usage_seconds_total{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-nas-controller\|csi-nas-node).",container!=""}[5m])) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-nas-controller\|csi-nas-node).",resource="cpu",container!=""}) by (cluster, namespace, pod, container)	The CPU usage of the Pod for the csi-nas add-on continuously approaches the CPU limit of the container. This may cause slow responses or throttling.	Identify the Pod and container whose CPU is approaching the limit. Check whether the recent load and request volume of the add-on have suddenly increased. Evaluate whether to increase the CPU limit or optimize concurrency. Check container logs to confirm whether abnormal busy-waiting or retrying exists.
High csi-nas memory limit saturation	P2: Value > 0.8 for 5 minutes	sum(container_memory_working_set_bytes{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-nas-controller\|csi-nas-node).",container!=""}) by (cluster, namespace, pod, container) / sum(kube_pod_container_resource_limits{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-nas-controller\|csi-nas-node).",resource="memory",unit="byte",container!=""}) by (cluster, namespace, pod, container)	The memory usage of the csi-nas add-on Pod is continuously approaching the container memory limit. This poses a risk of out of memory (OOM) or service instability.	Identify the Pod and container whose memory is approaching the limit. Observe the memory growth trend to determine if a leak exists. Evaluate whether the memory limit needs to be increased. Identify the source of memory usage by combining logs and pprof.
Unavailable csi-nas Pods	P1: Value == 1 for 10 minutes P2: Value == 1 for 5 minutes	(kube_pod_status_phase{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-nas-controller\|csi-nas-node).",phase!="Succeeded"} == 1) and on(cluster, namespace, pod) (kube_pod_status_ready{cluster=~"$Cluster",namespace=~"kube-system",pod=~"(csi-nas-controller\|csi-nas-node).",condition="false"} == 1)	The csi-nas add-on Pod is in a non-ready state, which will affect capabilities such as volume creation, deletion, mounting, and unmounting.	Check the status and events of the csi-nas-controller/node Pod in the kube-system namespace. Check image pulling, startup parameters, and resource limits. Check whether dependencies (APIServer and NFS server) function as expected. View the logs of the driver and sidecar containers to identify the root cause.
csi-nas control plane operation backlog	P1: Value > 20 for 10 minutes P2: Value > 10 for 5 minutes	max(workqueue_depth{cluster=~"$Cluster",job=~"csi-nas-.*"}) by (cluster, job, name)	The pending task queues of the control plane add-on csi-nas (responsible for volume provisioning) remain persistently high. This indicates that the event processing rate cannot keep up with the event generation rate, resulting in backlogs and delays in volume creation and deletion.	Identify the specific queue (name) with backlogs. Check the logs of the csi-provisioner and driver. Check whether the APIServer is experiencing slowness. Evaluate whether a scale-out or a concurrency increase is needed.
csi-nas control plane RPC call failures	P1: Value > 5 for 5 minutes P2: Value > 0 for 5 minutes	sum(increase(csi_sidecar_operations_seconds_count{cluster=~"$Cluster",job=~"csi-nas-.*",grpc_status_code!="OK"}[2m])) by (cluster, job, method_name)	The csi-nas control plane add-on fails to call the driver's gRPC operations, which will directly lead to the failure of control plane operations such as volume creation and deletion.	Identify the failed operation by using the failed method_name and job. View error codes and stack traces in the driver logs. Check the NFS server, quota, and network. Check whether the status of related PVs or PVCs is stuck.
csi-nas node RPC call failures	P1: Value > 5 for 5 minutes P2: Value > 0 for 5 minutes	sum(increase(csi_operations_seconds_count{cluster=~"$Cluster",job="kubelet",driver_name=~"nas.*",grpc_status_code!="OK"}[2m])) by (cluster, method_name)	The kubelet fails to call the csi-nas node driver's gRPC operations. This causes volume mount and unmount failures on the node, thereby blocking Pod startup.	Identify the failure stage based on the failed method_name. View the csi-nas-node logs on the corresponding node. Troubleshoot NFS mounting, network connectivity, and credentials. Confirm the volume parameter and mount point status.
High csi-nas node RPC latency	P1: Value > 30s for 5 minutes P2: Value > 15s for 5 minutes	histogram_quantile(0.99, sum by(cluster, method_name, le)(rate(csi_operations_seconds_bucket{cluster=~"$Cluster",job="kubelet",driver_name=~"nas.*"}[2m])))	When kubelet calls the csi-nas node driver's gRPC operations, P99 latency remains continuously high. This will slow down volume mount/unmount on the node and delay Pod startup.	Identify the slow stage by using method_name (such as NodePublishVolume). View the csi-nas-node logs on the corresponding node. Check the NFS server latency and network connection. Check node load and mount duration.
csi-nas operation failure	P1: Value > 5 for 5 minutes P2: Value > 0 for 5 minutes	sum(increase(csi_nas_operations_errors_total{cluster=~"$Cluster",job=~"csi-nas.*"}[2m])) by (cluster, job, method, operation)	Fine-grained operations within the csi-nas driver (NFS mount/unmount, subdirectory creation/deletion/archiving, and volume usage statistics) failed.	Identify the failed operation by job (to distinguish between the controller and the node), method, and operation. View the corresponding add-on logs (mount/subdirectory/archiving). Check the NFS server and network. Confirm subdirectory permissions and capacity.
High csi-nas operation latency	P1: Value > 15s for 5 minutes P2: Value > 8s for 5 minutes	histogram_quantile(0.99, sum by(cluster, job, method, operation, le)(rate(csi_nas_operations_duration_seconds_bucket{cluster=~"$Cluster",job=~"csi-nas.*"}[2m])))	The P99 latency of fine-grained operations inside the csi-nas driver remains persistently high (especially the NFS mount path). This will slow down volume creation/mounting.	Identify slow operations by job, method, and operation. Focus on troubleshooting the latency of the NFS mount path. Check the network and NFS server load. Observe whether it is accompanied by an increase in the error rate.

Last updated: 2026.10.22 17:24:38