Docs

Volcengine Kubernetes Engine

ServingKit

AI application observability

ServingKit supports custom configuration of AI application observability dashboards through cloud-native basic observability and deep observability capabilities. Basic observability provides the monitoring metrics for AI resources and inference services. Deep observability provides an end-to-end performance monitoring service for inference applications.

This topic uses an SGLang-based AI application as an example to demonstrate how to create your own AI application observability dashboard.

Basic observability

In the left navigation bar, click Cluster. In the cluster list, find the heterogeneous computing cluster in which the AI application was deployed, and click the cluster name.

In the left navigation pane of the cluster management page, click Cloud native observability > Overview. On the page that appears, click Enable basic observability to enable cloud-native observability and VKE observability. For more configuration details, refer to Enabling observability and VKE observability.

After turning on observability, the application metrics of SGLang will be collected into the Volcengine Managed Service for Prometheus (VMP). You can use Explore to query metrics in the VMP console. An example is as follows:

There is a built-in VMP dashboard that allows you to view the metrics collected from SGLang-based applications. To do that, you can go to the VMP console, select Monitoring dashboard in the left navigation bar and search for SGlang to open dashboard as shown below:

You can also use self-built Grafana and set up an SGLang monitoring dashboard. For more details, see Deploying Grafana in a VKE cluster and integrating Grafana with a workspace.

SGLang Dashboard-1747206543364.json

The effect after importing is shown in the figure:

Deep observability

Search for SGlang and click the service name in the results to open the monitoring dashboard page.

View metrics.

LLM metrics

On tabs such as Service Overview, LLM performance, and Token usage, you can view large model core metrics and key service metrics, such as the number of large model calls, the token usage, response latency, time to first token (TTFT), time per output token (TPOT), queries per second (QPS), and request error rate.

tip

TTFT and TPOT are effective only when LLMs are called with streaming-enabled requests.

The example of a streaming call is as follows:

curl localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "/data/models/DeepSeek-R1",
"messages": [
{
"role": "user",
"content": "hello"
}
],
"stream": true,
"stream_options": {
"include_usage": true,
"continuous_usage_stats": true
}
}'

Runtime metrics

On the Runtime monitoring tab, you can view runtime-related metrics, including CPU time, CPU usage, Memory usage, Number of GC objects, Number of threads, and Number of context switches.

View trace analysis.

On the Trace analysis tab, you can search for trace information based on different conditions. Click the ID of the target trace to go to the details page of the trace. View trace information in the list.

Last updated: 2026.07.24 15:50:48