Docs

Volcengine Kubernetes Engine

Setup & Configuration

Custom metric labels

Volcengine Kubernetes Engine (VKE) allows you to define custom metric labels within clusters to improve the flexibility and information richness of the monitoring system. This topic describes how to configure custom metric labels.

Background information

Metrics and labels

In the Prometheus monitoring system, metrics can have multiple labels attached. These labels are used to group and distinguish data in greater detail. Each label consists of a key and a value, which enriches the dimensionality of metrics and makes it easier to monitor and analyze specific scenarios.

For example, when monitoring the CPU usage of multiple servers, you can use labels to denote the ID or location of each server, as in cpu_usage{server_id="ecs1", location="datacenter1"}. In this case, you can not only view the overall CPU usage but also filter by labels to get the CPU usage of a specific server or data center.

Labels enable highly flexible data querying, filtering, or aggregation. You can use them to perform comparative analysis, calculate sums or averages, and generate other statistical metrics. This is extremely helpful for understanding and optimizing system performance.

tip

For details about metrics and labels, see Format of Prometheus time series data.

Custom labels

In addition to the default labels included in metrics, you often need custom labels to enhance the flexibility and information richness of the monitoring system. For example:

Environment differentiation. When you deploy the same application in development, test, and production environments, labels (such as env="production" or env="development") can distinguish metric data from different environments. This allows you to aggregate and easily view metrics from multiple environments in a single Prometheus instance, and quickly identify which environment is experiencing issues.

Service instance differentiation. When multiple instances of the same service are running, labels (such as instance="server1" or instance="server2") can be used to identify each service instance. This helps monitor the performance of individual instances, identify unbalanced load allocation, or detect failures in specific instances.

Version tracking. During software deployment or updates, you can use labels (for example, version="1.0" or version="1.1") to track the performance of different application versions. This helps development teams understand how new versions impact performance or compare performance differences between versions.

Geographic location tagging. For distributed services, geographic labels (such as region="cn-beijing-1" and region="cn-beijing-2") enable monitoring of performance across regions and analysis of key metrics like response time and availability between geographic locations.

Business-related information. You can use business-level labels (such as customer="customerA" and project="projectX") to segment monitoring data. This allows teams to view resource usage for specific customers or projects, supporting cost analysis and resource optimization for business units.

Warning

Adding custom labels to metrics doubles the number of active time series in the workspace within a short period (1 hour). Confirm in advance that the VMP workspace bound to the cluster has sufficient specifications to handle the increased number of active time series. For details, see Workspace specifications.

Prerequisites

A Volcengine Kubernetes Engine (VKE) cluster has been created and cloud native observability has been enabled for the cluster. For details, see Enabling observability.

The prometheus-agent add-on has been installed and upgraded to v2.6.0-vke.1.24 or v2.6.0-vke.1.20 or later. For details, see Add-on release records.

Procedure

In the left navigation bar, click Cluster.

In the cluster list, click the target cluster.

In the left navigation bar of the cluster management page, click Configurations > ConfigMap. On the page that appears, click Create with Yaml to create a ConfigMap.

tip

This configuration is a global configuration. After a custom metric label is configured for the cluster, all metrics collected by the prometheus-agent add-on in the cluster will include this label.

This configuration modifies the global parameter in native Prometheus settings. Only some parameters are supported for this configuration. For details, see Global prometheus-agent parameters.

Upgrading the prometheus-agent add-on does not overwrite this configuration.

If you specify built-in labels such as _cloud_product and cluster in this configuration, they will not take effect.

Validating results

In the VMP console , use the Explore feature to quickly query cluster metric data. You can view the custom labels in the metrics.

References

Global prometheus-agent parameters

Only some native Prometheus parameters are supported, as shown in the table below. Unmentioned parameters are not supported.

Parameters	Supported	Default configuration
external_labels	Yes	None
remote_write	Yes	None

Last updated: 2026.07.24 10:53:29