Docs

Volcengine Managed Service for Prometheus

AI add-on integration

vLLM integration

Volcengine Managed Service for Prometheus (VMP) allows you to monitor vLLM instances that are deployed in Volcengine Kubernetes Engine (VKE) clusters.

Background

vLLM is an open-source framework ‌dedicated to‌ Large Language Model (LLM) inference and deployment. It supports popular models (such as Llama and GPT), is compatible with quantization methods (such as GPTQ and AWQ), enables multi-node/multi-GPU scaling, and provides an OpenAI-compatible API. vLLM ‌optimizes and accelerates LLM deployment‌. ‌Leveraging the virtual memory concept from operating systems‌, its ‌innovative memory management transforms LLM inference efficiency‌. ‌Splitting the KV Cache into dynamically allocated memory pages increases GPU memory utilization from 60% (traditional frameworks) to over 95%‌. vLLM can help implement high-concurrency scenarios.

You can use VMP to monitor vLLM instances that are deployed in VKE clusters to achieve key metric alerting and optimization.

Prerequisites

A VMP workspace has been created. For details, see Creating workspaces.

VKE has been activated.

A cluster has been created. For details, see Creating clusters.

A vLLM instance has been deployed in the cluster.

Cloud native observability has been enabled for the cluster, and prometheus-agent has been installed in the cluster. For details, see Enabling observability.

tip

The VMP workspace and the VKE cluster must be deployed in the same region.

Procedure

Configure an integration task

In the top navigation bar, select the target region.

In the left navigation bar, click Integration Center > Component. On the page that appears, find and click vLLM to configure the integration task.

Configure the deployment environment of the collector. Click VKE environment and select an associated cluster.

Click Create Integration Task to specify the details of the integration task.

Parameter	Description
Basic information
Task name	Specify the name of the integration task. It can contain Chinese characters, letters (uppercase and lowercase), digits, and underscores (_), and must be 1 to 40 characters in length.
Project	The project to which the instance belongs. ‌A project is a resource management mechanism that Volcano Engine provides to enable grouping of cloud resources by project. Project-based authorization with Identity and Access Management (IAM) facilitates resource isolation and data security, as well as cost calculation across cloud resources. For details, see Project management.
Tags	Labels of the instance. A tag is a key-value pair that allows you to classify and aggregate cloud resources from different dimensions. You can use tags for resource authorization and bill splitting. For details, see Managing tags. tip You can add up to 50 labels.
Scrape configuration
JobLabel	You can select labels from Kubernetes Pod objects, and the label value will be used as the job label for all metrics. For example, if you set the JobLabel parameter to foo and the Pod label is foo: bar, the system will add a job="bar" label to all collected metrics. If this parameter is left empty, the default job label for the metrics consists of the namespace and name of the PodMonitor object, which is in the format <namespace>/<name>.
Metric scraping interval	Configure the collection interval of the collector.
Namespace	Specify the namespace where the collection target is deployed in the cluster. Values: arbitrary: indicates that metrics of resources in any namespace in the cluster can be collected. custom: indicates that you can specify custom namespaces in the cluster, and only metrics of resources in the specified namespaces can be collected. For example, if you set the value to default, only metrics of resources in the default namespace can be collected.
Metric acquisition port	Configure the metric port for the collection target. You can set the value to a port name or a port number.
Metric acquisition path	Configure the metric path of the collection target.
LabelSelector	Select collection targets by using the labels of Kubernetes Pod objects.
Advanced configuration
Metric Relabel configuration	Configure metric relabeling. For more details, refer to Metric relabeling.

Click Confirm to complete the configuration.

Obtain monitoring dashboards

Click the Effect preview tab to view sample monitoring dashboards. Click Get the Grafana template to download the Grafana monitoring dashboard template in JSON format.

Query metrics

Click the Acquisition metric tab to view the list of metrics for the collection targets.

View alert templates

Click the Alert template tab to view the preset alert templates for vLLM. Click Turn on alert. You are redirected to the alert group creation page, where you can swiftly create and enable alerts based on the current alert template.

Validate results

After the configuration is complete, you can view the status and details of the task. Check whether the integration task is running as expected.

In the left navigation bar, click Access center > Access task.

Click VKE environment. You can view the status of the integration task in the task list.

After the integration task is created, you can perform the following O&M operations:

Click the name of the task to go to the task details page.

Click Edit in the Actions column to edit the task.

Click Suspend task in the Actions column to suspend the task.

Click Delete in the Actions column to delete the integration task.

Monitoring information

Query metrics

You can quickly query and display metric data by using the Explore feature of VMP. For more details, refer to Querying metrics.

Create dashboards

You can install Grafana in a VKE cluster or an ECS instance and then use Grafana to view metrics and create dashboards.

Deploying Grafana in a VKE cluster and accessing workspaces

Deploying Grafana on an ECS instance and integrating Grafana with a workspace

Configure alerts

You can configure alerts in the alerting center of VMP. For more details, see Creating alert rules.

Last updated: 2026.10.10 10:16:50