Docs

Volcengine Managed Service for Prometheus

Middleware integration

Kafka integration

Volcengine Managed Service for Prometheus (VMP) allows you to monitor Kafka instances that are deployed in Volcengine Kubernetes Engine (VKE) clusters and Elastic Compute Service (ECS) instances, or Message Queue for Kafka instances. This topic describes how to monitor Kafka instances that are deployed in different environments.

Background

As a high-throughput, distributed streaming platform, Apache Kafka is used for building real-time data pipelines and streaming applications. With the publish/subscribe pattern at its core, Apache Kafka supports massive data persistence and horizontal scaling, which is used in scenarios such as log collection and event sourcing.

You can use VMP to monitor Kafka instances that are deployed in VKE clusters and ECS instances, or Message Queue for Kafka instances to achieve key metric alerting and optimization.

tip

If you want to monitor the instances of Message Queue for Kafka, we recommend using the Cloud Monitor integration feature in the integration center. The feature supports the collection of cloud service metrics with a few clicks.

Prerequisites

A VMP workspace has been created. For details, see Creating workspaces.

Elastic Compute Service (ECS)

A Volcano Engine root user has been created, and ECS has been activated.

The Kafka application is deployed in an ECS instance.

Volcengine Kubernetes Engine (VKE)

VKE has been activated.

A cluster has been created. For details, see Creating clusters.

The Kafka application is deployed in the VKE cluster.

The cloud native observability feature is enabled, and prometheus-agent is installed in the cluster. For more details, refer to Enabling observability.

Message Queue for Kafka

A Volcano Engine root user has been created, and Message Queue for Kafka has been activated.

A Message Queue for Kafka instance has been created. For more details, see Creating instances.

The endpoint address of the Message Queue for Kafka instance is obtained. For more details, refer to Viewing endpoints.

tip

The VMP workspace and the monitoring targets (ECS instance, VKE cluster, and Message Queue for Kafka instance) must be deployed in the same region.

Integration methods

Method 1: integration center (recommended)

In the top navigation bar, select the target region.

In the left navigation pane, click Access Center > Components. On the page that appears, click Kafka and configure the integration task.

Specify the deployment environment of the collector.

When the monitoring target is deployed in the ECS instance, click Hosting environment.

When the monitoring target is a Message Queue for Kafka instance or a self-built Kafka instance that is deployed in the VKE cluster, click VKE environment.

Click Create Integration Task to specify the details of the integration task.

Basic information

Parameter
Description
Task name
Configure the name of the integration task.
workspace
Select a created VMP workspace.

Scrape configuration

Parameter	Description
JobLabel	Configure the label of the collection job. If this parameter is not specified, the ID of the integration task will be used as the metric by default.
Metric scraping interval	Configure the collection interval of the collector.
ECS VPC	Select the virtual private cloud (VPC) where the ECS instance is deployed. This parameter is required only when the monitoring target is deployed in the hosting environment.
Subnet	Select the subnet of the VPC. This parameter is required only when the monitoring target is deployed in the hosting environment.
Service addresses	Specify the server (broker) address for the Kafka instance. The address varies based on different environments. When the monitoring target is deployed in the ECS instance, the service address is in the format of ECS instance IP address + port number. Example: 192.168.1.10:9092. When the monitoring target is deployed in the VKE cluster, you can access the Kafka instance by using ClusterIP or NodePort. For example, when you use a ClusterIP service, the address is kafka-demo.kafka.svc.cluster.local:9092. For more details, refer to ClusterIP and NodePort. When the monitoring target is a Message Queue for Kafka instance, the service address is the domain name or IP address of the endpoint for the instance such as 192.168.1.10:9092. For more details, refer to Viewing endpoints.
Versions	Specify the version of the Kafka instance such as 3.9.1.
Topic Regular Filtering	Use a regular expression to collect topic metrics that meet the filter rules in the Kafka instance.
Group Regular Filtering	Use a regular expression to collect group metrics that meet the filter rules in the Kafka instance.
Enable SASL	Specify whether to enable Simple Authentication and Security Layer (SASL). After SASL is enabled, you need to configure the following authentication information: SASL username: Specify the username for SASL authentication. SASL password: Specify the password for SASL authentication. SASL method: Configure the SASL authentication method. Valid values: PLAIN, SCREAM-SHA256, and SCREAM-SHA512.

Click Confirm to complete the configuration.

Method 2: custom deployment

You can also deploy kafka-exporter in the VKE cluster to monitor the Kafka instance.

Step 1: Deploy kafka-exporter

In the left navigation pane, click Cluster. In the cluster list on the right side, find the target cluster.

Click the name of the cluster to go to the cluster details page.

Click Workload > Deployment in the left navigation pane. Click Create with Yaml to deploy kafka-exporter.

tip

For detailed parameters of kafka-exporter, refer to kafka-exporter.

Step 2: Configure service discovery

Click Workload > Resource Explorer in the left navigation pane.

Click Create with Yaml to configure service discovery by using PodMonitor.

Select Custom from the Type drop-down menu.

Enter the YAML configuration in the Yaml field.

Validate results

View the integration task

After the configuration is complete, you can view the status and details of the task. Check whether the integration task is running as expected.

In the left navigation pane, click Access Center > access task.

Click Hosting environment or VKE environment. You can view the status of integration tasks in the task list.

After the integration task is created, you can perform the following O&M operations:

Click the name of the task to go to the task details page.

Click Edit in the Actions column to edit the task.

Click Suspend task in the Actions column to suspend the task.

Click Delete in the Actions column to delete the corresponding integration task.

Query task metrics

Click Explore in the left navigation pane. On the page that appears, use the up statement to view the status of collection jobs for the integration task. In this example, the statement is up{job="kafka_job"}. When the metric value is 1, it indicates that the collector is created and working as expected.

tip

当 Kafka 部署在 VKE 集群中时，采集任务的名称为kafka-exporter-***. You can use up{job="kafka-exporter-***"} to view the status of collection jobs.

Monitoring information

Query metrics

You can quickly query and display metric data by using the Explore feature of VMP. For more details, refer to Querying metrics.

tip

For common Kafka metrics, refer to the official documentation.

Build dashboards

You can install Grafana in a VKE cluster or an ECS instance and then use Grafana to view metrics and create dashboards.

Deploying Grafana in a VKE cluster and accessing workspaces

Deploying Grafana on an ECS instance and integrating Grafana with a workspace

Configure alerts

You can configure alerts for the Kafka instance in the alerting center of VMP. For details, see Creating alert rules.

Last updated: 2026.10.12 10:28:46

Parameter	Description
Task name	Configure the name of the integration task.
workspace	Select a created VMP workspace.