Docs

Volcengine Managed Service for Prometheus

Prometheus data collection and integration

Best practices for improving Prometheus-agent stability

Prometheus-agent is a collector add-on that allows Volcengine Managed Service for Prometheus (VMP) to collect information from Volcengine Kubernetes Engine (VKE) clusters. The stability of Prometheus-agent affects cluster observability results and quality. This topic describes how to improve the stability of Prometheus-agent in practical scenarios.

Background

For enhanced performance and stability, a newer version of the Prometheus-agent add-on may be available in your VKE cluster. We recommend reviewing the release notes and upgrading to the latest version. For details, see Add-on release notes.

An increase in cluster size can cause performance issues, which can lead to unexpected faults, such as metric loss. You are also advised to perform the following common operations to mitigate the risk and improve the stability of the Prometheus-agent add-on.

Common operations

Upgrading the version

We recommend that you upgrade the Prometheus-agent add-on in your cluster to the latest version. For details, see Upgrading add-ons.

warning

Prometheus-agent V2.8.0 has a new capability: add-on self-monitoring metrics collection, which enables the use of cloud-native VKE observability to monitor the metrics and status of the add-on itself.

If the add-on version is earlier than v2.8.0, upgrade it to the latest version.

If upgrading is not feasible, configure add-on self-monitoring for data collection based on the instructions in Step 2.

Ensure that Prometheus-agent in your VKE cluster is deployed using the standard method. Otherwise, you may not be able to upgrade it in the VKE console. To deploy it in the standard way, perform the following steps:

In the left navigation bar, click Cluster. In the cluster list, click the name of the target cluster.

In the left navigation bar of the cluster management page, click Addons. On the Addons page, click the Monitoring tab.

Find the Prometheus-agent card, and in the upper right corner, click ... > Install. For a detailed description of configuration items for the installation, see Installing add-ons

If you want to collect basic metrics of containers in the clusters, enable cloud native observability and basic observability. For details, see Enabling observability and Basic observability.

Creating self-monitoring collection configurations

If the Prometheus-agent add-on in your cluster is earlier than V2.8.0 and cannot be upgraded, you can follow the steps below to create a self-monitoring collection configuration by using a ServiceMonitor.

In the left navigation pane, click Cluster. In the cluster list on the right side, find the target cluster.

Click the name of the cluster to go to the cluster configuration page.

In the left navigation bar, click Workload > Resource Explorer. Click Create with Yaml to configure service discovery through ServiceMonitor.

Select Custom from the Type drop-down menu.

Enter the YAML configuration in the Yaml editor.

Enabling autoscaling

As the cluster scales and the number of metrics being collected grows, Prometheus-agent may face insufficient resource capacity, leading to issues like OOM (Out of Memory), restricted metric collection, or even data loss in severe cases. To prevent these problems, we recommend enabling the autoscaling feature , which allows the system to automatically adjust the replica count of the collector add-on based on its real-time resource usage.

You can enable add-on autoscaling when or after you deploy the add-on. This topic describes how to enable the feature after the add-on is deployed.

In the left navigation bar, click Cluster. In the cluster list, click the name of the target cluster.

In the left navigation bar of the cluster management page, click Addons. On the Addons page, click the Monitoring tab.

Click Configuration on the Prometheus-agent add-on card. In the dialog box that appears, enable add-on autoscaling and specify the maximum shard number. Then, click Confirm.

Parameter

Description

Addon scaling

Specify whether to enable the automatic scaling feature for the vm-agent collector and the kube-state-metrics agent.

Disable: The number of shards for the agent is the specified initial number of shards. The agent will not be automatically scaled out as resource usage increases.

Enable: You need to specify the maximum number of shards for the agent. The agent will be automatically scaled based on resource usage by the following rules:

vm-agent scale-out: vm-agent is automatically scaled out when CPU or memory usage is greater than 70%.

vm-agent scale-in: vm-agent is automatically scaled in when CPU and memory usage are less than 30%.

kube-state-metrics scale-out: kube-state-metrics is automatically scaled out when CPU or memory usage is greater than 70%.

kube-state-metrics scale-in: kube-state-metrics is automatically scaled in when CPU and memory usage are less than 70%.

warning

The cooldown period of vm-agent scaling is 30 minutes. kube-state-metrics is scaled through HorizontalPodAutoscaler.

After you enable the automatic scaling feature for the agent, ensure that cluster resources are sufficient. Otherwise, the agent scale-out may fail caused by insufficient resources.

When you scale out the vm-agent collector and the kube-state-metrics agent, it is recommended to specify the number of shards for the kube-state-metrics agent as an integer multiple of the number of shards for the vm-agent collector to ensure load balancing. For example, maintain a 1:1 or 2:1 ratio of the number of shards for the kube-state-metrics agent to the number of shards for the vm-agent collector.

Configure alerts

VMP provides alert rule templates for the Prometheus-agent add on. We recommend that you configure add-on alert rules to detect add-on exceptions in real time.

tip

The default thresholds in these templates are strict. If the default thresholds cause too many unnecessary alerts, adjust these thresholds to meet your actual requirements.

In the top navigation bar, select the target region.

In the left navigation bar, click Alert center > Alert rule template. On the page that appears, select the Preset template tab.

On the Preset template page, find the vmagent alert rule template. Click Apply template in the Actions column.

Adjust thresholds for alert rules that you want to modify. For details, see Creating alert rule groups.

Viewing dashboards

Built-in dashboards

If a cluster has cloud native observability enabled, you can check the Prometheus-agent monitoring dashboard.

tip

When you check the dashboard, select the VMP workspace that is bound to the cluster.

In the top navigation bar, select the target region.

In the left navigation bar, select Monitor dashboard. In the dashboard list, search for vmagent.

We recommend that you pay special attention to the following charts in the dashboard:

CPU and RSS memory % usage: show the resource usage of the add-on.

Persistent queue dropped: reflects whether persistent queues have dropped data. If vmagent has performance issues or other exceptions, the corresponding metrics cannot be sent to the VMP workspace. If that happens, the add-on stores metrics in a persistent queue and recovers metric sending when exceptions are fixed. If any metric is displayed in the chart, metric sending has encountered data losses.

RemoteWrite connection saturation: shows the connection saturation. If the saturation is greater than 80%, relatively high latency is observed in sending of each packet. In this case, consider increasing the CPU limit for vmagent to increase the concurrency level.

Custom dashboards

You can also create a Grafana dashboard and import it as the Prometheus-agent dashboard. For details, see the following documents.

Deployment: Deploying Grafana in a VKE cluster and accessing workspaces.

Dashboard: Import the following dashboard configuration file into Grafana:

vmagent-dashboard.json

Last updated: 2026.10.10 11:04:51