Docs

Observability Platform

Best practices of AI application monitoring

vLLM inference engine trace monitoring

Application Performance Management (APM) provides end-to-end tracing capabilities across multiple platforms. This topic describes how to instrument the vLLM inference engine with APM, and view the trace and metric data for vLLM-related tracing.

Background

vLLM is an open-source framework dedicated to‌ the inference and deployment of large language models (LLMs). It is designed to optimize and accelerate LLM usage, especially for efficient inference. vLLM processes an enormous volume of data, which makes collecting observability data for the framework highly challenging. Furthermore, effective observability of vLLM may require annotation of the collected data. APM can be seamlessly integrated with the vLLM inference engine to address these challenges: it ingests vLLM observability data, enables dialogue analysis, and delivers in-depth insights into the LLM operations.

Prerequisites

The basic environment has been prepared, and information such as the service name has been obtained.

APM has been activated. For detailed steps, refer to Activating APM.

Procedure

Step 1: Report observability data

You can report observability data to APM via apmplus-opentelemetry-collector, or directly report observability data to the APM backend based on your needs.

Data reporting via Otel Collector

Direct data reporting

Log in to the VKE console, select Cluster in the left navigation bar. On the cluster list page, click the target cluster name to go to the cluster management page.

On the Monitoring tab of the Addons page, install the apmplus-opentelemetry-collector add-on. Additionally, to collect metric data at the host and process levels, we recommend that you install the apmplus-server-agent add-on at the same time.

Configure the following environment variables in the environment to use apmplus-opentelemetry-collector to report observability data to APM.

Step 2: Instrument vLLM

Execute the following command in your environment to install vLLM.

Execute the following command in your environment to install the OpenTelemetry package.

Execute the following command in your environment to start vLLM.

Sample command for the default configuration

Sample command for optional configurations

Add the collect-detailed-traces parameter to the startup command to specify the module and collect detailed trace information, such as the time_in_model_execute and time_in_model_forward information.

warning

This parameter involves potentially costly and blocking operations and may impact inference performance. We recommend that you use this parameter based on your needs.

Step 3: View vLLM tracing data

In the left navigation bar, select Service Monitoring > Service List to go to the service list page.

Filter the service associated with the target vLLM by service name, click the service name to access the service details page, and view the vLLM golden signals on the Service overview tab. Such golden signals include QPS, number of requests, error rate, number of errors, number of errors per second, and response latency.

On the Trace analysis tab, click the target trace ID to go to the details page of a single trace. View the vLLM instrumentation data in the list view.

(Optional) If the collect-detailed-traces parameter is added in Step 2, two additional attributes are added to the inference span.

Last updated: 2026.07.09 20:16:23