After you enable observability in a cluster and install the prometheus-agent add-on in the cluster, you can use a PodMonitor to configure custom collection rules. This topic describes how to configure service discovery by using a PodMonitor.

Verify the demo

You can use the following demo code to integrate a Go application and use a PodMonitor for service discovery. In the following example, Go is used to create an application, and a metric named rpc_durations_seconds is exposed in the application to collect statistics on the RPC latency of the service.

package main​
​
import (​
        "math"​
        "math/rand"​
        "net/http"​
        "time"​
​
        "github.com/prometheus/client_golang/prometheus"​
        "github.com/prometheus/client_golang/prometheus/collectors"​
        "github.com/prometheus/client_golang/prometheus/promhttp"​
)​
​
var rpcDurations = prometheus.NewSummaryVec(​
        prometheus.SummaryOpts{​
                Name:       "rpc_durations_seconds",​
                Help:       "RPC latency distributions.",​
                Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},​
        },​
        []string{"service"},​
)​
​
func main() {​
        // Create a new registry.​
        reg := prometheus.NewRegistry()​
​
        // Add Go module build info.​
        reg.MustRegister(collectors.NewBuildInfoCollector())​
        // Add go runtime metrics and process collectors.​
        reg.MustRegister(​
                collectors.NewGoCollector(),​
                collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),​
        )​
​
        reg.MustRegister(rpcDurations)​
​
        start := time.Now()​
​
        oscillationFactor := func() float64 {​
                return 2 + math.Sin(math.Sin(2*math.Pi*float64(time.Since(start))/float64(10*time.Minute)))​
        }​
​
        // Periodically record some sample latencies for the three services.​
        go func() {​
                for {​
                        v := rand.Float64() * 0.0002​
                        rpcDurations.WithLabelValues("uniform").Observe(v)​
                        time.Sleep(time.Duration(100*oscillationFactor()) * time.Millisecond)​
                }​
        }()​
​
        go func() {​
                for {​
                        v := (rand.NormFloat64() * 0.0002) + 0.00001​
                        rpcDurations.WithLabelValues("normal").Observe(v)​
                        time.Sleep(time.Duration(75*oscillationFactor()) * time.Millisecond)​
                }​
        }()​
​
        go func() {​
                for {​
                        v := rand.ExpFloat64() / 1e6​
                        rpcDurations.WithLabelValues("exponential").Observe(v)​
                        time.Sleep(time.Duration(50*oscillationFactor()) * time.Millisecond)​
                }​
        }()​
​
        http.Handle("/metrics", promhttp.HandlerFor(​
                reg, promhttp.HandlerOpts{},​
        ))​
        http.ListenAndServe(":2023", nil)​
}​

Prerequisites

Cloud native observability has been enabled. For details, see Enabling observability.

The demo code has been packaged as an image and uploaded to CR. For details, see Go application integration.

The target cluster has been connected through kubectl. For details, see Connecting to clusters.

Procedure

Step 1: Deploy the application

You can use the image from CR to deploy the application in the target VKE cluster. The steps are as follows:

Log in to the VKE console.

In the left navigation bar, click Cluster. On the page that appears, click the name of the target cluster to go to the configuration page of the cluster.

In the left navigation bar, click Workload > Deployment. Click Create with Yaml, and deploy the application.

apiVersion: apps/v1
kind: Deployment
metadata:
name: golang-demo # Specify a name for the application.
namespace: volcano-metrics # Specify the namespace where the application resides.
labels:
app: golang-demo # Specify application labels.
spec:
replicas: 2 # Specify the number of replicas for the application.
selector:
matchLabels:
app: golang-demo
template:
metadata:
labels:
app: golang-demo
spec:
containers:
- name: golang-demo # Specify a name for the container.
image: doc-cn-beijing.cr.volces.com/vmp/golang-demo:1.0 # Specify the address and version of the application image.
ports:
- name: metric-port # Specify a name for the container port.
containerPort: 2023 # Specify the container port.

Step 2: Configure a PodMonitor

In the left navigation bar, click Workload > Resource Explorer. Click Create with Yaml and configure a PodMonitor.

Select Custom from the Type drop-down menu.

Edit the configurations in the Yaml box.

Click Confirm.

Validating results

Log in to the VMP console.

In the top navigation bar, choose the target region.

In the left navigation bar, click Explore to go to the Explore page.

In the upper-right corner, specify the Workspace from which metrics will be queried.In the Query text box, enter up{job="volcano-metrics/golang-demo-discover"} to verify if the corresponding pod has been discovered as expected.

Advanced configuration

Specify a custom name for the job

By default, the name of the job discovered through the PodMonitor is <namespace>/<name>. In this example, the default job name is volcano-metrics/golang-demo-discover. You can specify a custom name for the job based on your business requirements.

Method 1

You can use the jobLabel field in the PodMonitor to change the job name to the value of a specified pod label. In this example, you can change the job name to the value of the label app: golang-demo. As a result, the job name is golang-demo.

The expected result is as follows: You can query metrics by using the job name golang-demo.

Method 2

You can use the spec.podMetricsEndpoints.N.relabelings field in the PodMonitor to configure relabeling and specify a custom job name.

The expected result is as follows: You can query metrics by using the job name jobname-demo.

Adding pod labels to metrics

By default, pod labels are not added to metrics. You can use the spec.podTargetLabels field in the PodMonitor to add pod labels to metrics.

The expected result is as follows:

Metric relabeling

You can use metric relabeling to add, modify, or delete metric labels before metrics are collected or pushed to managed Prometheus workspaces. You can configure metric relabeling by using the following two methods:

Use relabelings:

Modify or filter target labels before target collection.

Modify or filter target labels during service discovery.

Use metricRelabelings:

Modify or filter metric labels after data collection.

Delete specific metrics after data collection.

Typical scenarios include deleting unnecessary metrics, renaming metric labels, and adding new metric labels.

Adding global labels

If you want to add a specified label to all collected metrics, you can specify the spec.podMetricsEndpoints.N.relabelings parameter. For example, use the following configuration to add the label key-value pair label: value to all collected metrics.

Retaining specified metrics

If you want to write only specified metrics, you can configure action: keep. For example, use the following configuration to retain only metrics whose names match the regular expression vmagent_remotewrite.*.

Deleting metrics

If you do not want to write some metrics, you can discard these metrics by configuring action: drop. For example, use the following configuration to discard the vmagent_daily_series_limit_rows_dropped_total metric.

Modifying the value of a label

You can configure action: replace to modify the value of a label. For example, use the following configuration to remove the /api/v1 prefix for the path label value. As a result, path: /api/v1/targets will change to path: /targets.

Volcengine Kubernetes Engine