Docs

Volcengine Kubernetes Engine

Qwen practices

Quickly deploying Qwen3-235B with Dynamo and vLLM (PD disaggregation)

This topic mainly describes how to quickly deploy the Qwen3-235B model inference service (PD disaggregation) based on Dynamo and vLLM by using Volcengine Kubernetes Engine (VKE).

Solution introduction

The solution integrates multiple acceleration technologies of Volcano Engine to quickly deploy the Qwen3-235B-A22B model in a Kubernetes cluster.

Qwen3: the latest Qwen large language model (LLM) launched by Alibaba Cloud, which features six dense models and two Mixture-of-Experts (MoE) models. With 0.6 billion to 235 billion parameters, Qwen3 can meet diverse business requirements.

Dynamo: NVIDIA Dynamo is an open-source, low-latency, modular inference framework for serving generative AI models in distributed environments.

GDKV model acceleration: a fast model loading solution achieved through the Remote Direct Memory Access (RDMA) and GPU Direct RDMA (GDR) technologies provided by Volcano Engine.

Elastic Instant Cache (EIC): a high-performance distributed KV Cache service launched by the Volcano Engine storage team for LLM inference.

PD disaggregation technology: The technology divides the inference process into two phases, which are deployed in different hardware resources to optimize performance:

Prefill: A compute-intensive phase.

Decode: A memory-intensive phase.

Deployment information

Model: Qwen3-235B-A22B

Inference engine: Dynamo and vLLM

Instance specification: ecs.hpcpni3ln.45xlarge

Prefill: ecs.hpcpni3ln.45xlarge instance* 1

Decode: ecs.hpcpni3ln.45xlarge instance* 1

Supported capabilities

Parallel strategy: PD disaggregation

Precision: FP8

Model acceleration: GDKV

KV Cache: EIC

Monitoring: Volcengine Managed Service for Prometheus (VMP) and Application Performance Management Plus (APMPlus)

Access: API Gateway (APIG)

Performance evaluation

The following table shows the performance differences between the open source scenario and the solution provided in this topic.

tip

The delivered performance is related to machine specifications, distribution configuration, and network bandwidth. The actual test result shall prevail.

Category	Open source	Optimized
Inference engine	vLLM (open source edition)	Dynamo and vLLM (Volcano Engine optimized version)
Input/Output	3500/1500	3500/1500
TTFT (ms)	<=5000	<=5000
TPOT (ms)	<=50	<=50
Parallel policy	TP	PD disaggregation and TP
Single card throughput (TPS)	897	1595

Usage notes

To ensure successful deployment, which involves a bunch of add-ons dependent on each other and a variety of configurations, you need to strictly follow the instructions in this document.

When both GDKV model acceleration and EIC are enabled, the memory of the instance node may be insufficient. If you have high requirements for performance, we recommend that you only enable EIC. For more details, refer to Method of testing EIC performance.

To avoid violation of the VKE limit, we recommend that you proceed based on this solution (or on the resources mentioned in the documents). If you need to change your solution, you can consult the Account Manager of the Volcano Engine.

Do not use mGPU on GPU resources.

A full deployment depends on some beta capabilities of the Volcano Engine. Contact the Account Manager of the Volcano Engine to apply for the beta trial of the following features:

Beta feature/resource	Product involved	Description
hpcpni3ln instance optimized for High-Performance Computing based on GPU	GPU Elastic Compute Service	The HPC instance, providing computing power. The Qwen3-235B model has a large number of parameters. Therefore, you need to apply for a GPU instance with large memory, such as ecs.hpcpni3ln.45xlarge.
Capability of mounting nodes to local disks	Volcengine Kubernetes Engine	You can configure the local disk mounting in the VKE cluster.
EIC	EIC	A high-performance distributed KV Cache service launched by the Volcano Engine storage team for LLM inference.

Prerequisites

Creating an HPC cluster

The HPC cluster can achieve the logical isolation of high-performance computing GPU instances (computing nodes). Instances in the same cluster are interconnected by Remote Direct Memory Access (RDMA) networks. For details, see Creating High-Performance Compute Clusters.

tip

The region and availability zone where the high-performance compute cluster is located need to be consistent with those of the VKE nodes.

Creating a VKE cluster

The following parameters need to be configured as described in the table when creating a cluster by using VKE. For details on other parameters, see Creating clusters.

Configuration item	Description
Cluster configuration
Kubernetes version	Select v1.28 or later.
Network model	Select VPC-CNI.
Node pool configuration
Managed node pool	Enable Managed node pool.
Availability zone	Same availability zone as the High-Performance Compute Cluster.
Instance type	Select ecs.hpcpni3ln, the GPU Accelerated specification. Ensure you already obtained permission for trial of the beta version.
High Performance Computing Cluster	Select the HPC cluster you created.
Number of nodes	2 nodes.
Data volume	Configure the following two data volumes: ESSD: set capacity to 1024 GiB and the performance level to the default PL0. Local SSD: Specify /data01 as the mount directory. warning The data01 directory has subsequent dependencies. Therefore, the directory cannot be modified or specified. Ensure the data that needs long-term storage has been backed up properly in case of computer crashes or disk damage which causes data loss.
Node labels	When you deploy the model by using VKE, you need to specify the RDMA usage mode by using node labels. The label key is set to: vke.node.rdma.mode, and the label value is set to: shared.
Kubelet configs	When you deploy the model by using VKE, you need to configure the non-uniform memory access (NUMA) affinity policy of the RDMA device by specifying the parameters of Kubelet. Select the parameter topology-manager-policy and its value best-effort.
Addon configuration
nvidia-device-plugin	(Required) The NVIDIA device driver. It allows use of management add-ons for GPUs in containers.
rdma-device-plugin	(Required) The add-on manages RDMA network interface cards (NICs) and enables pod containers to use RDMA devices.
apig-controller	(Required) API Gateway controller provides the cloud-native AI Gateway capability, which can complete the intelligent load balancing of the inference traffic according to different policies.
csi-ebs	(Required) It allows VKE clusters to use the standard Container Storage Interface (CSI) of the TOS.
prometheus-agent	(Required) Volcengine Managed Service for Prometheus (VMP) monitoring add-on collects the metrics data of the Kubernetes cluster and send the data to VMP.
apmplus-opentelemetry-collector	(Required) Collect OpenTelemetry data and Prometheus metrics, and send them to APMPlus.
apmplus-server-agent	(Required) Send the collection service and host monitoring data in parallel to APMPlus.
cr-credential-controller	(Recommended) It can pull Container Registry (CR) images without passwords, which simplifies deployment operations.
keda	(Recommended) The event-driven autoscaler based on Kubernetes. It bComplete the autoscaler tasks of the inference service based on complex metrics.
vci-virtual-kubelet	(Recommended) It can deploy CPU-consuming workloads such as system add-ons by using Volcengine Container Instance (VCI), which eliminates the need for operation and maintenance (O&M) and does not occupy the computing resources of GPU nodes.

Procedure

Step 1: Create an EIC instance

With approaches such as storage-instead-of-computation and zero-copy GDR, EIC significantly reduces GPU resource consumption and inference latency.

Click Instance management in the left navigation bar and click Create instance.

Configure parameters required on the page. The parameters mentioned below need to be configured as described. Other parameters can be configured as needed.

Configuration item	Description
VKE cluster	Select the created VKE cluster.
Storage type	Select DRAM.
Enable RDMA	Enable RDMA.
Node	Select the nodes created when the VKE cluster is created.
DRAM	Set to 500 GiB.
Network interface card	Set to eth1, eth2, eth3, eth4.

Click Create instance in the lower-right corner of the page. If Running is displayed in the Status column, it indicates that the EIC instance is created.

If an error occurs when creating an instance, the status will change to Failed. You can hover the mouse over the instance status to view the details. You can delete the instance that fails to be created by clicking Delete in the Actions column and then create a new one. If you cannot handle the issue, contact your account manager of Volcano Engine for support.

Step 2: Check the StorageClass required by NATS and ETCD

In the VKE console, click Storage > StorageClass in the left navigation bar of the management page for the target cluster. On the StorageClass page, check whether the StorageClass named ebs-ssd exists, which is used for storage related to NATS and ETCD.

tip

The StorageClass is automatically created after the csi-ebs add-on is installed in the cluster. If the StorageClass does not exist, install the csi-ebs add-on in the cluster first. For detailed operations, refer to Installing an add-on.

Step 3: Deploy qwen3-dynamo-vllm-pd

VKE provides the Qwen3-235B-A22B model application (PD disaggregation) based on the Dynamo and vLLM frameworks. The application can be quickly deployed through the Application template page of the VKE console.

In the left navigation bar of the VKE console, click Application center > Application template.

On the Application template page, find qwen3-dynamo-vllm-pd and click Deploy.

On the Deploy application page, specify the parameters of qwen3-dynamo-vllm-pd. Configure the parameters mentioned below as described. Other parameters can be configured according to the prompts on the page or by referring to the descriptions in Application template.

Configuration item	Description
application template
Application template	Select the Chart version of qwen3-dynamo-vllm-pd. We recommend that you select the latest version.
Parameter configuration	The configuration method of qwen3-dynamo-vllm-pd. Click the edit icon next to values.yaml to edit the YAML file of the current application. For the descriptions of some parameters, refer to the following table.
Name	Customize the application name according to the naming requirements prompted by the interface.
Deployment environment
Cluster	Select the VKE cluster created as described in Prerequisites.
Namespace	Select a namespace under the target cluster.
Protocol	Read and understand the agreement before selecting it.

You can modify the configurations of the values.yaml file based on your business requirements. The following table describes some parameters. We recommend that you keep the default configurations.

Configuration item	Description	Sample value
modelConfig.local.path	The local path of the ECS instance used to store the model file, which must be the created data volumes.	/data01/models
prefillWorker.replicas	Prefill the number of replicas to deploy on the node.	1
decodeWorker.replicas	Decode the number of replicas deployed on the node.	1
decodeWorker.router	Decode the Router algorithm of the node.	round-robin
etcd.persistence.storageClass	The StorageClass used by ETCD.	ebs-ssd
etcd.persistence.size	The storage capacity of ETCD.	10Gi
nats.config.cluster.enabled	Does NATS use cluster configuration?	true
nats.config.cluster.replicas	The number of replicas in a NATS cluster is larger than or equal to 3.	3
nats.config.jetstream.fileStore.pvc.storageClassName	The StorageClass used by NATS.	ebs-ssd
nats.config.jetstream.fileStore.pvc.size	The size of NATS storage.	10Gi

If you want to enable EIC, you can modify the configurations of the values.yaml file for the application based on the following code block and annotations.

kvcacheConfig:
eic:
enabled : true
# EIC instance ID. Log in to the EIC console and click the EIC instance to find the endpoint on the "Instance Information" tab.
instanceId: "eicydx97nk******"
# Endpoint of the EIC instances. Log in to the EIC console and click the EIC instance to find the endpoint on the "Instance Information" tab.
endpoint: "192.168.xx.xx:12500;192.168.xx.xx:12500;192.168.xx.xx:12500"
# EIC node network interface card. Log in to the EIC console and click the EIC instance to find the NIC on the "Node List" tab.
ClientMultiNetLocalInterfaceNames: "eth1;eth2;eth3;eth4"

Click Confirm to start deployment.

The applications that are already deployed can be found on the Helm page. Select the target project, cluster, and namespace, and click the name of the deployed qwen3-dynamo-vllm-pd application to go to the details page. On the details page, view the basic information about the application and the created workloads, services, and other resources.

tip

<application name>-qwen3-dynamo-vllm-pd indicates the name of the service externally exposed by the application. You need to associate the service in the subsequent steps, such as Ingress configuration and model validation.

View the status of pods that correspond to the qwen3-dynamo-vllm-pd application on the Pod page of the VKE cluster.

On the left navigation bar of the VKE console, select Cluster, find the target cluster, and click the cluster name.

In the left navigation bar of the target cluster management page, click Workload > Pod. Then, enter the application name qwen3-dynamo-vllm-pd in the search box to view the status of related pods.

Step 4: Configure an Ingress

Configure model routing through APIG Ingress. APIG Ingress can function as a managed cloud-native gateway based on the Volcano Engine APIG, realizing high availability and scalability in Ingress traffic management that is required in cloud-native applications.

On the left navigation bar on the cluster management page, select Services and Ingresses > Ingress, and click Create Ingress. Use API Gateway to expose the model service externally in the cluster. The parameters mentioned below need to be configured as described. Other parameters can be configured by referring to the Creating APIG Ingress through the console.

Configuration item	Description
Basic configuration
Namespace	Select the namespace where the qwen3-dynamo-vllm-pd application template is deployed to associate the service exposed by the application.
Ingress configuration
Ingress type	Select cloud-native API Gateway.
APIG specification settings
Gateway specifications	You can select Automatically create, or select Use existing APIG instances. tip When you are using APIG to expose the model service in the clusters, we recommend that you configure the following parameters or features (Some parameters can only be configured in the APIG console, and cannot be configured in the VKE console): Instance specification: We recommend that you select 4c8g, with two availability zones to realize high availability by default. Bandwidth limit: This option is used to set the maximum bandwidth of the CLB instance working with the APIG instance. If more bandwidth is required, you can specify a custom CLB instance in the APIG console. For details, see Adding access entrance. Circuit breaking: When some GPU instances malfunction, common practice is taking the faulty instances offline to eliminate failures. At the same time, it is necessary to retain the faulty instances for troubleshooting later. You can isolate the faulty instances by configuring a circuit breaking rule for the APIG instance. Request timeout: Timeout is not applied to API Gateway by default. We recommend that you keep the default settings. If timeout is needed, you can configure it on the API Gateway console. For details, refer to Advanced routing policy. Observability: Monitoring alert and log feature are not turned on by default for the API Gateway instances created in the VKE console. If needed, we recommend you to turn on the log and monitoring alarm capabilities of the instances in the APIG console. Instance monitoring: This can help you to detect the anomalies in the whole system chain from the gateway entry level. For example: Unhealthy back end and back-end crashes will cause the APIG to return 503 error code and specific errors. Instance logs: You can view the request errors, request time consumption, and the allocation of requests in various Pods in the back end from the access logs.
Backend upstream	We recommend that you set the Load balancing algorithm parameter to Poll. It is more applicable to the scenarios of upgrading back-ends if the back-end loads are approximately balanced.
Listener configuration
Protocol	Select a protocol as needed. If HTTPS is selected, you need to configure the certificate as described in the document APIG Ingress.
Forwarding configuration
Domain name	Fill in the Ingress domain name to be exposed. The A (Address) record of this domain name needs to be configured later with the public or private IP address generated when the Ingress is successfully created.
Service	Select a service that is created when you deploy the qwen3-dynamo-vllm-pd application template and set the port number to 8000. The name of the service is in the format of <application name>-qwen3-dynamo-vllm-pd.

(Optional) Configure authentication for an APIG.

Create consumer: On the left navigation bar, select Consumer management, and click Create consumers.

Generate an API-KEY: In the consumer list, click the name of the target consumer, to enter the consumer details page. In the API Key information area, click Generate API Key.

In the pop-up dialog box, select Automatically create.

Click Confirm to generate API Key credential information.

Enable Key Auth add-on: in the left navigation bar, select Plugin Center > Key Auth and click Create Plugin on the page.

Effect level: select the Routing level and configure the instance, service, and route to which the add-on belongs.

Authentication configuration: select the header and enter the text "Authorization" in the textbox on the right side to indicate that the API Key will be included in the Authorization header of requests.

Consumer: Select the consumer created in Step 1.

Other options can remain as the default.

Obtain Ingress address.

VIP: includes both public and private IP addresses.

backend service: includes the backend service name and access domain name.

Configure domain name resolution.

To test the DNS in the local environment, you can implement a domain-to-IP resolution locally.

In the production environment, configure the access domain name of Ingress among the domain name providers, and ensure the A record of the Ingress access domain name has been configured with the public IP address or private IP address of Ingress.

Step 5: enable observability

Basic observability provides the monitoring metrics for AI resources and inference services. Deep observability provides an end-to-end performance monitoring service for inference applications.

Basic observability

In the left navigation bar of the page for a specific cluster, select Overview under the gray title Cloud native observability, then click Enable on the page to start monitoring. For more configuration details, refer to Enable observability and VKE observability.

After you enable observability, Dynamo-based application metrics will be collected into VMP, and you can query the metrics on the Explore page in the VMP console. The following figure shows an example.

View Dynamo-based application metrics.

You can view Dynamo-based application metrics on the preset dashboards of VMP. In the left navigation bar of the VMP console, click Monitoring dashboard. Then, enter dynamo in the search box to view the dashboards. The following figure shows the sample Dynamo-based application dashboards.

You can install Grafana and create Dynamo monitoring dashboards. For more details, refer to Deploying Grafana in a VKE cluster and integrating Grafana with a workspace.

dynamo_default_dashboard.json

The following figure shows the result after importing.

Deep observability

Search for the keyword Dynamo and click the service name to go to the monitoring dashboard page.

View metrics.

LLM metrics

On tabs such as Service Overview, LLM Performance, and Token Usage, you can view the core metrics of the LLM service, including the number of LLM calls, token usage, response latency, TTFT, TPOT, QPS, and request error rate.

tip

TTFT and TPOT are effective only when LLMs are called with streaming-enabled requests.

The example of streaming call is as follows:

curl localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-235B-A22B",
"messages": [
{
"role": "user",
"content": "hello"
}
],
"stream": true,
"stream_options": {
"include_usage": true,
"continuous_usage_stats": true
}
}'

Runtime metrics

On the Runtime monitoring tab, you can view runtime-related metrics, including CPU time, CPU usage, Memory usage, Number of GC objects, Number of threads, and Number of context switches.

View link analysis.

On the Trace analysis tab, you can search for trace information based on different conditions. Click a Trace ID you want to open the details page of the trace. View Trace information in the list.

Model verification

Method 1: Access through Service

After you deploy the qwen3-dynamo-vllm-pd application template, the created service whose name is in the format of <application name>-qwen3-dynamo-vllm-pd will provide an HTTP API endpoint (port 8000). The endpoint can be called as you do with OpenAI APIs.

The sample code is as follows.

tip

Replace the value of localhost:8000 with the endpoint of your <application name>-qwen3-dynamo-vllm-pd service.

The parameter content is what you want to type to chat with LLMs.

In the Workload section of the Helm details page that corresponds to the qwen3-dynamo-vllm-pd application, you can select a network-reachable workload to go to the details page. In the upper-right corner of the details page, click Console to log in to the pod that corresponds to the application to call the API.

Method 2: Access via APIG

When enabling the authentication feature of the APIG, you need to add an authentication token in the request.

The following example is provided for reference.

tip

Replace the example.com in http://example.com with the domain name of the back-end services obtained in Step 4: configure Ingress.

Replace eyJhbGciOiJSUzI****** with the API Key generated in Step 4: configure Ingress.

The content parameter specifies the text you input to chat with LLMs.

Last updated: 2026.07.24 15:50:47