Docs

Volcengine Kubernetes Engine

Other observation

Monitoring Service observability

Monitoring service observability allows you to monitor key metrics of the prometheus-adapter add-on, including request load, error rate, cache refresh, process resources, and more. This helps you quickly locate add-on anomalies and assess the operational health status. This topic describes how to enable monitoring service observability, configure collection rules, and view the preset observability dashboard for prometheus-adapter.

Prerequisites

Cloud-native observability has been enabled. For details, see Enabling observability.

The VKE observability feature has been enabled. For details, see VKE observability.

The prometheus-adapter add-on has been installed. For details, see prometheus-adapter.

(Optional) An alert notification policy has been created in the Observability Platform or VMP. For details, see Creating notification policies and Creating alert notification policies.

(Optional) An alert aggregation policy has been created in the Observability Platform or VMP. For details, see Creating aggregation policies and Creating alert aggregation policies.

Enabling observability

Step 1: Configure observability items

In the left navigation bar, click Cluster. Find the target cluster and click its name.

In the left navigation pane of the cluster management page, click Observability settings, and select the Basic observability tab.

Find the Monitoring services card and click Enable observation to go to the observability item configuration page.

The system automatically checks the necessary conditions for enabling observability, such as the workspace configuration and add-on status.

For the Observation item parameter, select observability items for which you want to collect metrics. For enabled parameters, the system will not collect the corresponding metrics; meanwhile, the system will prompt you to install add-ons. You can decide whether to install the relevant add-ons and enable the corresponding observability items according to your needs.

After the configuration is complete, click Next to go to the metric configuration page.

Step 2: Configure metric collection

On the metric configuration page, view the collection interval and metric quantity corresponding to the enabled observability items.

Click Edit metrics. This enables you to select or discard specific metrics of the add-on, modify the collection interval, and configure metric relabeling.

For Collection interval, select a collection interval for the metrics of the add-on. The supported collection intervals vary for different add-ons.

In the metric list, select a metric to collect. To stop collecting a metric, uncheck it. Click the All, Basic metrics, or Other metrics tab to filter metrics by type.

tip

The impact of adjusting the metric collection interval is as follows. Adjust the interval as needed:

Reducing the collection interval: This increases the number of metrics reported per unit time and improves monitoring accuracy. However, the cost of standard VMP workspaces increases.

Increasing the collection interval: This decreases the number of metrics reported per unit time and lowers the cost of standard VMP workspaces. However, this decreases monitoring accuracy.

Cloud service metrics are categorized into Basic metrics and Other metrics. The billing method varies based on different types of metrics. For details, see VMP billing.

In the metricRelabelings section, click Add rule to configure metric relabeling rules. This allows you to rewrite the labels of each metric after collection and before writing to storage. Common use cases include modifying label values, adding new labels, and renaming labels to standardize label naming for queries. For details, see Metric relabeling.

No.	Feature	Replacement value	Description
①	Action type	action	The action to apply to the metrics. Currently, only replace is supported. The logic is as follows: If the values of targetLabel and sourceLabels are the same, sourceLabels is overwritten. If the key specified by targetLabel does not exist, a new label is created by default.
②	Source labels	sourceLabels	Specify one or more source labels, separated by \| if multiple labels are provided. The system reads the values of these labels for matching or extraction.
③	Regular expression	regex	The regular expression pattern used to match source label values. The default is (.*), which matches any value. You can use capture groups such as (.+) to extract partial content.
④	Replacement value	replacement	The value written to the target label after a successful match. The default is $1, which refers to the content of the first capture group in the regular expression. A fixed string is also supported.
⑤	Target label	targetLabel	The name of the target label to which the processing result is written. If the target label already exists, its existing value is overwritten. If the target label does not exist, a new label is created.

After the configuration is complete, click Next to go to the alert and notification configuration page.

Step 3: Configure alerts and notifications

In the Alert and notification step, select the platform to receive alerts. Currently, Unified Alert Center and The original VMP alarm strategy are supported.

For the Notification strategy name parameter, select the created alert notification policy and specify the notification object and Notification method parameters based on different alert levels.

tip

If you do not need to configure alerts for a certain level, you can unselect the corresponding alert level.

If the existing notification policies do not meet your needs, you can select vke_default_notify_policy (automatically created) for the Notification strategy name parameter, and specify the notification object and Notification method parameters for the required alert levels. After you save the configuration, the system will automatically create a custom notification policy named vke_default_notify_policy in the unified Alert Center or VMP.

In the Alerting aggregation policy section, select the created alert aggregation policy.

Configure preset alert rules.

Enable or disable the switch next to a preset alert rule. After it is enabled, the system detects metrics based on the configured trigger conditions and generates alerts. When it is disabled, the rule will no longer trigger new alerts.

Click View PromQL to view the description and corresponding PromQL statement of the current preset alert rule.

Click the name of the preset rule to be configured to expand the preset alert rule configuration page, and configure the trigger conditions for different levels (P0, P1, or P2) of alerts in the Triggering condition section.

Parameter	Description
Alert level	Select the check box in the Enable column that corresponds to each alert level (P0, P1, or P2) to activate its trigger condition. If you do not select an option, alerts at this level are not configured.
Comparison condition	Specify the comparison condition between the PromQL statement and the threshold. You can configure multiple comparison conditions, including Greater than, Greater than or equal to, Less than, Less than or equal to, Equal to, and Not equal to.
Threshold	Configure the threshold of the metric.
The duration of the condition	Configure the duration for which the metric continuously meets the trigger condition. The system triggers the alert at the corresponding level only if the metric continuously meets the comparison condition within this duration.
Label	Configure the label of the alert rule. It is mainly used for alert identification, grouping, deduplication, and routing matching. A label consists of a key and a value. Where: Key: A string. It can start with a letter or an underscore (_), but cannot start with a double underscore (__). It can contain letters, digits, and underscores (_) and can be 1 to 256 characters in length. Value: A string. It can contain Chinese characters, English letters, digits, and special characters, and can be 1 to 512 characters in length. Common configuration example: labels: severity: critical # The alert level. business: pay # The business line. alert_type: resource # The alert type.
Annotation	Configure the annotation of the alert rule. It is mainly used to display key information. An annotation consists of a key and a value. Where: Key: A string. It can start with a letter or an underscore (_), but cannot start with a double underscore (__). It can contain letters, digits, and underscores (_) and can be 1 to 256 characters in length. Value: A string. It can contain Chinese characters, English letters, digits, and special characters, and can be 1 to 512 characters in length. Common configuration example: annotations: summary: "High CPU usage on instance {{ $labels.instance }}" description: "Current CPU usage is {{ $value \| humanizePercentage }}, threshold is 80%" runbook_url: "https://xxx/docs/cpu-alert" dashboard_url: "https://grafana/xxx"

Click Complete the configuration.

Description

In addition to using preset alert templates to configure alert rules directly, you can customize the template configurations or directly use PromQL statements to configure alert rules. For details, see Alert configuration overview.

Details of the preset alert rule templates, including rule descriptions, PromQL configurations, and common alert handling procedures. See Preset alert templates.

Observability dashboards

tip

The built-in dashboards provide features such as period-over-period comparisons of monitoring data and the ability to copy dashboard queries. For details, see Usage of dashboards.

You can view the built-in monitoring dashboard in the VKE console. This dashboard provides information such as API service quality, metric discovery cache, and process resources.

In the left navigation pane of the cluster management page, click Monitoring center > Dashboard.

In the left dashboard list, select Monitoring services > Prometheus Adapter monitoring to view the monitoring dashboard.

The following table describes the charts on the prometheus-adapter dashboard in detail.

Description

For the method of obtaining the PromQL statements that correspond to the charts, see Using dashboards.

Chart name	Unit	Chart description
Overview/One Screen Overview
API QPS	req/s	The total request rate across all API traffic (calculated based on data from the past 5 minutes), including requests to /healthz, /apis, and the metrics API. This information reflects the request load of the add-on. Under normal circumstances, this value should fluctuate periodically and steadily in line with the collection interval of the HPA controller. If there is a sustained increase or spike, it usually indicates that a large number of HPA objects are frequently pulling custom metrics. We recommend checking whether any HPA has a configuration with an excessively short collection interval or whether there are abnormal metric query requests.
5xx Error Ratio	%	The proportion of 5xx error requests to total requests (calculated based on data from the past 5 minutes), reflecting the add-on's request error rate. Under normal circumstances, the error rate should be close to 0%, and occasional brief fluctuations are within an acceptable range. If the error rate remains above 0% or rises significantly, it typically indicates that the add-on is encountering errors when processing custom metric queries. Check the add-on logs and verify whether the backend Prometheus is reachable.
API P99 Latency	Milliseconds	The P99 latency of API requests (calculated based on data from the past 5 minutes), reflecting the tail latency of request processing by the adapter. Under normal circumstances, the P99 latency should remain low, fluctuating with request volume but remaining overall stable. If the P99 latency increases significantly or remains persistently high, it usually indicates that the add-on's response is slowing down, which may impact the timeliness of HPA scaling that depends on custom metrics. Check the add-on's resource usage and the performance of backend Prometheus queries.
Panics (1h)	Count	The cumulative number of panics (calculated based on data from the past 1 hour), reflecting the stability of the add-on process. Under normal circumstances, this value should always be 0. If a value greater than 0 is observed, it indicates that the add-on process has crashed and restarted. Investigate the add-on logs immediately to identify the cause of the panic and resolve the issue...
Cache Refreshes (10m)	Count	The cumulative number of cache refresh triggers (calculated based on data from the past 10 minutes), reflecting the update frequency of the add-on's metric cache. Under normal circumstances, this value should increase periodically at a fixed cycle, consistent with the cache refresh interval specified in the add-on configuration. If this value remains at 0 for an extended period, it indicates that the cache is not refreshed and custom metric data may have stopped updating. In this case, check whether the add-on is running normally. If the refresh count is high, there may be frequent restarts or a configuration anomaly.
Process Uptime	Automatic conversion (seconds / minutes / hours)	The continuous execution time of the prometheus-adapter process (current time minus process start time), reflecting the stability of the Adapter instance. Under normal circumstances, this value should keep increasing monotonically, indicating that the add-on process is running stably without a restart. If this value drops sharply (resets to zero and starts growing again), it indicates that the add-on process has restarted. The Reason for restart should be investigated by correlating the Panics metric and Addon log.
API Service/API Quality of Service
API Request Rate by Subresource / Code	req/s	The average API request rate by subresource and status code (calculated based on data from the past 5 minutes), reflecting the request volume and response distribution for different subresources. Under normal circumstances, the request rate varies with the query frequency of clients such as HPA, and success status codes should account for the majority. If the 4xx or 5xx request rate for a certain subresource continues to rise, we recommend troubleshooting request failures by correlating the corresponding status code, add-on logs, and Prometheus backend status.
5xx Error Ratio	%	The proportion of 5xx error requests to total requests (calculated based on data from the past 5 minutes), reflecting server-side errors encountered by the add-on when processing API requests. Under normal circumstances, this proportion should be close to 0%. If this proportion continues to rise, it usually indicates that an abnormal condition occurred when the add-on processed custom metric queries. Check the add-on logs, resource usage, and whether the backend Prometheus is reachable.
API Request Latency	Milliseconds (ms)	The P50, P95, and P99 latency of API requests (calculated based on data from the past 5 minutes), reflecting the overall latency and tail latency of API requests. Under normal conditions, all percentile values should remain stable, with P95 and P99 higher than P50. If P95 or P99 continues to rise, it indicates that some requests are becoming slower, which may affect the timeliness of HPA retrieving custom metrics. Check the component resource usage and the query performance of the backend Prometheus.
Inflight Requests	Count	The number of API requests being processed by prometheus-adapter at the current time, shown by request type. This information reflects the add-on's real-time concurrent request pressure. Under normal circumstances, this value fluctuates as requests arrive and are processed. If this value remains high or continues to increase, it indicates that requests may be accumulating. Troubleshoot process bottlenecks using API request Latency, CPU usage, and add-on logs.
API Request Rate by Verb / Subresource	req/s	The average API request rate calculated by request method and subresource (based on data from the past 5 minutes), reflecting the access frequency of different types of custom metric queries. Under normal circumstances, the request rate changes with the query frequency of clients such as HPA. When the request rate of a certain type increases abnormally, we recommend checking whether there are a large number of HPAs performing high-frequency queries on the same subresource or whether the client is configured with an excessively short query interval.
TLS Handshake Errors	Count	The cumulative number of TLS handshake errors for prometheus-adapter (calculated based on data from the past 5 minutes), reflecting anomalies encountered when clients establish encrypted connections to the add-on. Under normal circumstances, this value should be 0. If this value is greater than 0 or continues to increase, check whether the aggregation API certificates are valid, whether the TLS configurations of the client and the add-on are consistent, and review the handshake error messages in the add-on logs.
Metric Discovery/Metric Discovery Cache
Cache Refresh Rate	ops/s	The average cache refresh rate by refresh type (calculated based on data from the past 5 minutes), reflecting the update frequency of the metric discovery cache. Under normal circumstances, this value should change regularly according to the cache refresh cycle specified in the add-on configuration. If this value remains at 0 for an extended period, it may indicate that the cache is not being refreshed properly. If the value is significantly higher than expected, check whether the add-on is restarting frequently or whether the cache refresh configuration is abnormal.
Cache Refresh Latency	Milliseconds (ms)	Cache refresh latency (calculated based on data from the past 5 minutes), showing the average, P95, and P99 values broken down by refresh type. This information reflects the refresh efficiency and tail latency of the metric discovery cache. Under normal conditions, all latency values should remain stable, with P95 and P99 higher than the average. If P95 or P99 continues to rise, it indicates that some cache refreshes are slowing down, which may cause new metrics or metric changes not to be discovered in time. Check backend Prometheus query performance, rule quantity, and add-on resource usage.
Cache Refresh Attempts (10m)	Count	The number of cache refresh attempts counted by refresh type (calculated based on data from the past 10 minutes), reflecting the refresh frequency of the metric discovery cache. Under normal circumstances, this value should match the cache refresh cycle specified in the add-on configuration. If this value remains at 0 for a long time, it may indicate that cache refresh is not being executed. If it is significantly higher than expected, we recommend checking whether the add-on is frequently restarted or if the cache refresh configuration is abnormal.
Refreshes over 10s (15m)	Count	The number of cache refreshes that took longer than 10 seconds, broken down by refresh type (calculated based on data from the past 15 minutes). This information reflects the occurrence of slow cache refresh requests. Under normal circumstances, this value should be 0 or remain low. If this value continues to increase, it indicates that cache refreshes are frequently taking longer than 10 seconds, which may cause delays in updating metric discovery results. Check the query performance of the backend Prometheus, the number of rules, and the resource usage of the component.
Runtime/Process Resources
CPU Usage	cores	Average CPU usage of each prometheus-adapter instance (calculated based on data from the past 5 minutes). The value represents the number of CPU cores used, reflecting the add-on's computational resource consumption. Under normal conditions, this value should vary with API request volume and cache refresh task count and remain within the CPU limit of the instance. If this Value continues to rise OR approaches the CPU limit, requests and cache refreshes may slow down. Troubleshoot the issue by combining API request latency, cache refresh latency, and instance resource configuration.
Resident Memory	byte	The resident memory size used by each prometheus-adapter instance at the current moment, reflecting the amount of data actually resident in physical memory for the add-on. Under normal circumstances, this value fluctuates with cache size and request load, but should remain generally stable. If this value continues to grow or approaches the memory limit, it may indicate an excessively large cache size or abnormal memory growth. Combine the instance memory limit, restart records, and add-on logs to troubleshoot the issue.
File Descriptor Utilization	%	The proportion of open file descriptors to the maximum allowed by the system across all prometheus-adapter instances at the current moment, reflecting the usage of file descriptor resources. Under normal circumstances, this proportion should remain low and stable. If this proportion continues to rise or approaches 100%, the add-on may not be able to establish new network connections or open files. Check whether connections are properly released and examine the maximum process file descriptor limit.
Goroutines	Count	The current Goroutine quantity for individual prometheus-adapter instances. This information reflects the scale of concurrent tasks within the add-on. Under normal circumstances, this value fluctuates slightly with request volume and cache refresh tasks and remains relatively stable. If this value continues to increase and does not decrease, request blocking or abnormal Goroutine exits may have occurred. Troubleshoot the issue using the add-on logs, request latency, and resource usage.

View metrics

To query integrated or non-integrated metrics in the cluster, or to understand metric meanings, see Querying metrics.

You can use the Explore feature of VMP to quickly query and display metric data. For details, see Querying metrics.

Last updated: 2026.10.22 17:24:39