This topic introduces common alerts and rule recommendations.

Common alerts

Common alerts for Kubernetes

Alert type	Monitoring object	Metric source	Monitoring purpose	Typical PromQL
Pod or container	Pod/Container	kube-state-metrics	This alert is triggered when a container remains in the CrashLoopBackOff state for the last 5 minutes. This identifies repeated startup failures or ongoing crashes.	max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}[5m]) >= 1
Pod/Container	Pod	kube-state-metrics	This alert is triggered when a Pod remains in the Pending or Unknown state and the Pod is not part of a Job. This indicates that the Pod is not ready.	sum by (namespace, pod, cluster) ( max by (namespace, pod, job, cluster) ( kube_pod_status_phase{phase=~"Pending\|Unknown"} ) * on (namespace, pod, cluster) group_left() topk by (namespace, pod, cluster) ( 1, max by (namespace, pod, owner_kind, cluster) (kube_pod_owner{owner_kind!="Job"}) ) ) > 0
Pod/Container	Container	kube-state-metrics	This alert is triggered when a container remains in the Waiting state for a long period due to a reason other than CrashLoopBackOff. This indicates issues with image pulling, creation, or dependency initialization.	kube_pod_container_status_waiting_reason{reason!="CrashLoopBackOff"} > 0
Workload rollout	Deployment	kube-state-metrics	This alert is triggered when the desired number of replicas for a Deployment exceeds that of available replicas, and the number of updated replicas remains unchanged for 10 minutes. This indicates that the rollout is stuck or replicas are unavailable for a long period.	( kube_deployment_spec_replicas > kube_deployment_status_replicas_available ) and ( changes(kube_deployment_status_replicas_updated[10m]) == 0 )
Workload rollout	Deployment	kube-state-metrics	This alert is triggered when the Progressing condition of a Deployment is false. This indicates that the rolling update is not progressing as expected.	kube_deployment_status_condition{condition="Progressing", status="false"} != 0
Workload rollout	StatefulSet	kube-state-metrics	This alert is triggered when the current and updated revisions of a StatefulSet differ, updated replicas are not fully ready, and no progress is made for 5 minutes. This indicates that the StatefulSet update is stuck.	( max by (namespace, statefulset, job, cluster) ( kube_statefulset_status_current_revision unless kube_statefulset_status_update_revision ) * on (namespace, statefulset, job, cluster) (kube_statefulset_replicas != kube_statefulset_status_replicas_updated) ) and on (namespace, statefulset, job, cluster) ( changes(kube_statefulset_status_replicas_updated[5m]) == 0 )
Workload rollout	DaemonSet	kube-state-metrics	This alert is triggered when the number of scheduled, updated, available, or mis-scheduled replicas for a DaemonSet is abnormal, and no progress is made within 5 minutes. This identifies that the DaemonSet is not covering nodes as expected.	( (kube_daemonset_status_current_number_scheduled != kube_daemonset_status_desired_number_scheduled) or (kube_daemonset_status_number_misscheduled != 0) or (kube_daemonset_status_updated_number_scheduled != kube_daemonset_status_desired_number_scheduled) or (kube_daemonset_status_number_available != kube_daemonset_status_desired_number_scheduled) ) and ( changes(kube_daemonset_status_updated_number_scheduled[5m]) == 0 )
Autoscaling	HPA	kube-state-metrics	This alert is triggered when the desired number of replicas for an HPA differs from the current number of replicas, and the current number remains unchanged within the range between the minimum number and the maximum number for 15 minutes. This indicates that scaling does not take effect.	(kube_horizontalpodautoscaler_status_desired_replicas != kube_horizontalpodautoscaler_status_current_replicas) and (kube_horizontalpodautoscaler_status_current_replicas > kube_horizontalpodautoscaler_spec_min_replicas) and (kube_horizontalpodautoscaler_status_current_replicas < kube_horizontalpodautoscaler_spec_max_replicas) and changes(kube_horizontalpodautoscaler_status_current_replicas[15m]) == 0
Autoscaling	HPA	kube-state-metrics	This alert is triggered when the current number of replicas for an HPA equals the maximum number of replicas, and the maximum number is not equal to the minimum number. This indicates that the autoscaling policy has reached its upper limit.	( kube_horizontalpodautoscaler_status_current_replicas == kube_horizontalpodautoscaler_spec_max_replicas ) and on (namespace, horizontalpodautoscaler, cluster) ( kube_horizontalpodautoscaler_spec_max_replicas != kube_horizontalpodautoscaler_spec_min_replicas )
Scheduling/Maintainability	PDB	kube-state-metrics	This alert is triggered when the desired number of healthy replicas for a PDB exceeds the current number of healthy replicas. This indicates insufficient replicas for secure evictions.	( kube_poddisruptionbudget_status_desired_healthy - kube_poddisruptionbudget_status_current_healthy ) > 0
Node resource	Node	node-exporter	This alert is triggered when the average node CPU utilization (excluding idle and iowait) exceeds 90 percent. This indicates a sustained high CPU load on the node.	sum without(mode) (avg without (cpu) (rate(node_cpu_seconds_total{mode!~"idle\|iowait"}[2m]))) * 100 > 90
Node resource	Node	node-exporter	This alert is triggered when the node memory utilization exceeds 90 percent. This indicates imminent exhaustion of available memory on the node.	100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) > 90
Node resource	Node filesystem	node-exporter	This alert is triggered when available filesystem space drops below 15 percent, and it is predicted that the space will be exhausted within 24 hours based on the trend in the last 6 hours, while the filesystem is not read‑only. This provides early warning of a disk space shortage.	( node_filesystem_avail_bytes{mountpoint!=""} / node_filesystem_size_bytes{mountpoint!=""} * 100 < 15 and predict_linear(node_filesystem_avail_bytes{mountpoint!=""}[6h], 246060) < 0 and node_filesystem_readonly{mountpoint!=""} == 0 )
Node resource	Node filesystem	node-exporter	This alert is triggered when available filesystem space falls below 5 percent, and the filesystem is not read‑only. This indicates imminent exhaustion of disk space.	( node_filesystem_avail_bytes{mountpoint!=""} / node_filesystem_size_bytes{mountpoint!=""} * 100 < 5 and node_filesystem_readonly{mountpoint!=""} == 0 )
Node resource	Node filesystem	node-exporter	This alert is triggered when the remaining filesystem inode percentage drops below 40 percent, and it is predicted that inodes will be exhausted within 24 hours based on the trend in the last 6 hours, while the filesystem is not read‑only. This provides early warning of an inode shortage.	( node_filesystem_files_free{mountpoint!=""} / node_filesystem_files{mountpoint!=""} * 100 < 40 and predict_linear(node_filesystem_files_free{mountpoint!=""}[6h], 246060) < 0 and node_filesystem_readonly{mountpoint!=""} == 0 )
Node resource	Node disk	node-exporter	This alert is triggered when the weighted I/O time rate of block devices persistently exceeds a threshold. This indicates disk I/O queue congestion or storage layer saturation.	rate(node_disk_io_time_weighted_seconds_total[5m]) > 10
Node network	Node network	node-exporter	This alert is triggered when the ratio of received error packets to total received packets exceeds 1 percent. This indicates errors on the network receiving path of nodes.	rate(node_network_receive_errs_total[2m]) / rate(node_network_receive_packets_total[2m]) > 0.01
Node network	Node network	node-exporter	This alert is triggered when the ratio of transmitted error packets to total transmitted packets exceeds 1 percent. This indicates errors on the network transmission path of nodes.	rate(node_network_transmit_errs_total[2m]) / rate(node_network_transmit_packets_total[2m]) > 0.01
Control plane	Kubernetes component	kube-apiserver/kubernetes_build_info	This alert is triggered when the number of major and minor versions of Kubernetes add-ons in a cluster exceeds 1. This indicates version inconsistency among control plane add-ons.	count by (cluster) ( count by (git_version, cluster) ( label_replace(kubernetes_build_info{job!~"kube-dns\|coredns"},"git_version","$1","git_version","(v[0-9].[0-9]).*") ) ) > 1
Control plane	API server client	kube-apiserver/rest_client_requests_total	This alert is triggered when the ratio of 5xx error requests to total requests from clients accessing the API Server exceeds 1 percent. This indicates control plane access exceptions or dependency failures.	(sum(rate(rest_client_requests_total{code=~"5.."}[5m])) by (cluster, instance, job, namespace) / sum(rate(rest_client_requests_total[5m])) by (cluster, instance, job, namespace)) > 0.01

Common alerts for etcd

Category	Common alerts	Metric source	Description	Typical PromQL
Availability	EtcdNoLeader	etcd native metrics	This indicates that etcd has lost its leader. This is a fundamental control‑plane availability indicator.	etcd_server_has_leader == 0
Performance	EtcdHighFsyncDurations	etcd native metrics	This indicates increased WAL fsync latencies. This typically indicates disk write jitters or storage performance degradation.	histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (le)) > 0.5
Capacity	EtcdBackendQuotaLowSpace	etcd native metrics	This indicates that the backend space approaches the upper limit. This prevents impacts on control plane stability.	(etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes) > 0.95
Write failure	EtcdProposalFailures/EtcdProposalsPendingHigh	etcd native metrics	This indicates proposal failures or backlogs. This helps discover write failures, slow consensus processing, or internal blocking.	rate(etcd_server_proposals_failed_total[5m]) > 0 or etcd_server_proposals_pending > 0

Rule recommendations

Item	Recommended practice
Alert level	We recommend configuring alerts with severity levels P0, P1, or P2 to distinguish between trends and critical failures.
Condition duration	We recommend setting a "for" condition duration for most alert rules to reduce false positives caused by transient jitters.
Namespace filtering	We recommend excluding test environments, temporary tasks, and short‑lived objects to reduce noise.
Tag specification	We recommend consistently adding tags such as cluster, namespace, and pod to improve routing and troubleshooting efficiency.
Metric availability confirmation	We recommend confirming whether the corresponding metric already exists on the VMP Explore page before you configure an alert rule. If the metric is not found, evaluate and supplement the collection configuration on the source side to prevent the alert rule from being invalid.

Volcengine Managed Service for Prometheus