Docs

Volcengine Kubernetes Engine

Deepseek practices

Quickly deploying DeepSeek-V3/R1 (full version) with xLLM (PD disaggregation)

This topic mainly describes how to quickly deploy a model inference service with the DeepSeek-R1 or DeepSeek-V3 full version based on xLLM by using Volcengine Kubernetes Engine (VKE).

Solution introduction

This solution achieves the prefill-decode (PD) disaggregation deployment of the DeepSeek-R1 or DeepSeek-V3 large language model (LLM) in Kubernetes clusters.

DeepSeek-V3 model: Released by the Chinese AI startup DeepSeek, the DeepSeek-V3 model is a general LLM with 671 billion parameters and 37 billion activated parameters, featuring a mixture of experts (MoE) architecture. Pre-trained on 14.8 trillion tokens, the DeepSeek-V3 model provides services such as information retrieval, data analysis, and natural language processing, which apply to various application scenarios such as customer service, educational tutoring, and content creation.

DeepSeek-R1 model: Released by Chinese AI startup DeepSeek, the R1 model is an inference model trained based on DeepSeek-V3-Base. It excels in mathematics, code, and natural language reasoning and has gained wide attention for its open source, algorithm innovation, and cost-effectiveness.

xLLM engine: As an efficient inference framework independently developed by , the xLLM engine provides PD disaggregation, elastic scaling, and other features, outperforming other open source frameworks.

PD disaggregation technology: The technology divides the inference process into two phases, which are deployed in different hardware resources to optimize performance:

Prefill: A compute-intensive phase.

Decode: A memory-intensive phase.

Advantages of the solution

Improved resource utilization: The prefill phase is separated from the decode phase, and resources are allocated to different phases.

Increased throughput: New requests are being processed in the Prefill phase while the old ones are being handled in the Decode phase.

Reduced latency: The wait time of concurrent requests is reduced and the overall efficiency is improved.

Deployment information

Model: DeepSeek-V3-671B and DeepSeek-R1-671B

Inference engine: xLLM

Instance specification: ecs.hpcpni3ln.45xlarge

Prefill: ecs.hpcpni3ln.45xlarge instance* 1

Decode: ecs.hpcpni3ln.45xlarge instance* 1

Supported capabilities

Parallel strategy: PD disaggregation, Data Parallelism (DP), Tensor Parallelism (TP), and Expert Parallelism (EP).

Observability: Volcengine Managed Service for Prometheus (VMP)

Access: API Gateway (APIG)

Performance evaluation

The following table shows the performance differences between the open source scenario and the solution provided in this topic.

tip

The delivered performance is related to machine specifications, distribution configuration, and network bandwidth. The actual test result shall prevail.

Category	open source	Optimized
Inference engine	SGLang (open source edition)	xLLM (self-developed by Volcano Engine)
Input/Output	3500/1500	3500/1500
TTFT (ms)	<=5000	<=5000
TPOT (ms)	<=50	<=50
Parallel strategy	TP	PD disaggregation, DP, TP, and EP.
Single-GPU throughput (TPS)	167.4	669

Usage notes

In this example, many add-ons are used, which are dependent on each other and have complex configurations. To achieve the expected results, you must perform operations according to the guidance in this article.

To achieve the expected results and conform to the limits of VKE, you need to perform operations by following the solution described in this article or on the recommended resources. If you want to use a different solution, contact your Volcano Engine account manager for assistance.

Do not use mGPU on GPU resources.

A full deployment depends on some beta capabilities of the Volcano Engine. Contact the Account Manager of the Volcano Engine to apply for the beta trial of the following features:

Beta feature/resource	Product involved	Description
xLLM inference acceleration solution	ModelArk	The core capability of this best practice needs to be used with an xLLM License.
hpcpni3ln instance optimized for High-Performance Computing based on GPU	GPU Elastic Compute Service	The HPC instance, providing computing power. The Deepseek model has a large number of parameters and therefore requires a GPU instance with large memory, e.g., ecs.hpcpni3ln.45xlarge.
Capability of mounting nodes to local disks	Volcengine Kubernetes Engine	You can configure the local disk mounting in the VKE cluster.

Prerequisites

Creating an HPC cluster

The high-performance compute cluster is used to achieve logical isolation of HPC GPU instances (compute nodes). Instances in the same cluster are interconnected by Remote Direct Memory Access (RDMA) networks. For details, see Creating HPC clusters.

tip

The region and availability zone where the high-performance compute cluster is located need to be consistent with those of the VKE nodes.

Creating a VKE cluster

The following parameters need to be configured as described in the table when creating a cluster by using VKE. For details on other parameters, see Creating clusters.

Configuration item	Description
Cluster configuration
Kubernetes version	Select v1.28 or later.
Network model	Select VPC-CNI.
Node pool configuration
Managed node pool	Enable managed node pool.
Availability zone	Same availability zone as the High-Performance Compute Cluster.
Instance type	Select ecs.hpcpni3ln, the GPU Accelerated specification. Ensure you already obtained permission for trial of the beta version.
High Performance Computing Cluster	Select the HPC cluster you created.
Number of nodes	2 nodes.
Data volume	Configure the following two data volumes: ESSD: set capacity to 1024 GiB and the performance level to the default PL0. Local SSD: Specify /data01 as the mount directory. warning The data01 directory is required by subsequent steps and must not be changed or customized. If the physical host fails or the local disk is damaged, data loss may occur. Do not store data that must be retained long-term on the local disk, and make sure your data is backed up in case of physical host failures or local disk damage.
Node labels	When you deploy the DeepSeek model by using VKE, you need to specify the RDMA usage mode by using node labels. Set the label key to vke.node.rdma.mode and the label value to shared.
Kubelet configs	The non-uniform memory access (NUMA) affinity policy for RDMA devices needs to be configured by using custom parameters with the Kubelet when deploying DeepSeek model in the VKE. Select the parameter topology-manager-policy and parameter value best-effort.
Addon configuration
nvidia-device-plugin	(Required) The NVIDIA device driver. It allows use of management add-ons for GPUs in containers.
rdma-device-plugin	(Required) The add-on manages RDMA network interface cards (NICs) and enables pod containers to use RDMA devices.
apig-controller	(Required) The API Gateway controller provides cloud-native AI gateway capabilities and enables intelligent load balancing for inference traffic based on different policies.
csi-ebs	(Required) It enables VKE clusters to use the standard CSI implementation for Elastic Block Storage (EBS).
prometheus-agent	(Required) Volcengine Managed Service for Prometheus (VMP) monitoring add-on collects the monitoring metrics data of the Kubernetes cluster to VMP.
apmplus-opentelemetry-collector	(Not supported yet) Collect OpenTelemetry data and Prometheus metrics, and send them to APMPlus.
apmplus-server-agent	(Not supported yet) It collects and sends monitoring data of services and hosts to APMPlus.
cr-credential-controller	(Recommended) It can pull Container Registry (CR) images without passwords, which simplifies deployment operations.
keda	(Recommended) The event-driven autoscaler based on Kubernetes completes the autoscaler tasks of the inference service based on complex metrics.
vci-virtual-kubelet	(Recommended) It can deploy CPU-consuming workloads such as system add-ons by using Volcengine Container Instance (VCI), which eliminates the need for operation and maintenance (O&M) and does not occupy the computing resources of GPU nodes.

After the VKE cluster and nodes are created, bind an elastic IP address (EIP) to each ECS instance corresponding to the nodes. For more details, refer to Associating EIPs.

Locally run the following command to upload xLLM License to all GPU nodes used in this solution.

tip

Replace the value of [your_ip_address] with the EIP of the GPU node.

[your_ip_address] can contain only one IP address. If multiple GPU nodes exist, run the command multiple times.

scp license.lic root@[your_ip_address]:/root/license.lic

Procedure

Step 1: Deploy the application

In the left navigation bar, select Application center > Application template.

On the Application template page, find deepseek-v3-xllm-pd or deepseek-r1-xllm-pd and click Deploy.

On the Deploy application page, specify the parameters of deepseek-v3-xllm-pd or deepseek-r1-xllm-pd.

Configuration item	Description
Orchestration template
Application template	Select the Chart version of deepseek-v3-xllm-pd or deepseek-r1-xllm-pd. We recommend that you select the latest version.
Parameter configuration	The configuration method of deepseek-v3-xllm-pd or deepseek-r1-xllm-pd. Click the edit icon to edit the values.yaml file, and update the YAML file based on the YAML file parameter description in the following table.
Name	Customize the application name according to the naming requirements prompted by the interface.
Deployment environment
Cluster	Select the VKE cluster created in Prerequisites.
Namespace	Customize and select the namespace under the target cluster.
Protocol	Read and understand the agreement, and then select the checkbox to accept it.

YAML file parameters

Configuration item	Description	Example
modelConfig.local.path	The local path of the ECS instance used to store the model file. If you want to use a local disk, the path needs to be consistent with the mount target configured in the node pool.	/data01/models
prefillWorker.resources.limits.vke.volcengine.com/rdma	The number of RDMA network interface cards (RNICs) used by the prefill worker. It is recommended that you set the value based on the number of RNICs (4 or 8) corresponding to the model.	4
prefillWorker.resources.requests.vke.volcengine.com/rdma			4
decodeWorker.resources.limits.vke.volcengine.com/rdma	The number of RNICs used by the decode worker. We recommend that you set the value based on the number of RNICs (4 or 8) corresponding to the model.	4
decodeWorker.resources.requests.vke.volcengine.com/rdma			4

Click Confirm to start deployment.

The deployed application will be displayed on the Helm page. Select the target project, cluster, and namespace and click the name of the deployed deepseek-v3-xllm-pd or deepseek-r1-xllm-pd application to go to the details page. On the details page, view the basic information about the application and the created workloads, services, and other resources.

tip

<application name>-<template name>-httpsvc indicates the name of the service externally exposed by the application. You need to associate the service in the subsequent steps, such as Ingress configuration and model validation.

Step 2: Configure an Ingress

Configure model routing through APIG Ingress. APIG Ingress can function as a managed cloud-native gateway based on the Volcano Engine APIG, realizing high availability and scalability in Ingress traffic management that are required in cloud-native applications.

On the left navigation bar on the cluster management page, select Services and Ingresses > Ingress, and click Create Ingress. Use API Gateway to expose the model service externally in the cluster. The parameters mentioned below need to be configured as described. Other parameters can be configured by referring to the Creating APIG Ingress through the console.

Configuration item	Description
Basic configuration
Namespace	Select the namespace where the deepseek-v3-xllm-pd or deepseek-r1-xllm-pd application template is deployed to associate the service exposed by the application.
Ingress configuration
Ingress type	Select Cloud-native API gateway.
APIG specification settings
Gateway specifications	You can select Automatically create, or select Use existing APIG instances. tip When you are using the API Gateway to expose the model service in the clusters, we recommend you to configure the following parameters or features (Some parameters can only be configured in the APIG console, and cannot be configured in the VKE console): Instance specification: We recommend that you select 4c8g, with two availability zones to realize high availability by default. Bandwidth limit: This option is used to set the maximum bandwidth of the CLB instance working with the APIG instance. If more bandwidth is required, you can specify a custom CLB instance on the APIG console. For details, see Adding access entrance. Circuit breaking: When some GPU instances malfunction, the common method is to take the faulty instances offline to eliminate failures. At the same time, it is necessary to retain the faulty instances for troubleshooting later. You can isolate the faulty instances by configuring a circuit breaking rule for the APIG instance. Request timeout: Timeout is not applied to API Gateway by default. We recommend that you keep the default settings. If timeout is needed, you can configure it on the API Gateway console. For details, refer to Advanced routing policy. Observability: Monitoring alert and log feature are not turned on by default for the API Gateway instances created in the VKE console. If needed, we recommend that you enable the logging, monitoring, and alerting capabilities of the instances in the APIG console. Instance monitoring: This can help you to detect the unexpected anomalies in the whole system chain from the gateway entry level. For example: Unhealthy back end and back-end crashes will cause the APIG to return 503 error code and specific errors. Instance logs: You can view the request errors, request time consumption, and the allocation of requests in various Pods in the back end from the access logs.
Backend upstream	We recommend that you set the Load balancing algorithm parameter to Poll. It is more applicable to the scenarios of upgrading back-ends if the back-end loads are approximately balanced.
Listener configuration
Protocol	Select a protocol as needed. If HTTPS is selected, you need to configure the certificate as described in the document APIG Ingress.
Forwarding configuration
Domain name	Fill in the Ingress domain name to be exposed. The A (Address) record of this domain name needs to be configured later with the public or private IP address generated when the Ingress is successfully created.
Service	Select a service that is created when you deploy the deepseek-v3-xllm-pd or deepseek-r1-xllm-pd application template and set the port number to 80. The name of the service is in the format of <application name>-<template name>-httpsvc.

(Optional) Configure authentication for an APIG.

Create consumer: On the left navigation bar, select Consumer management, and click Create consumer.

Generate an API-KEY: In the consumer list, click the name of the target consumer, to enter the consumer details page. In the API Key information area, click Generate API Key.

In the pop-up dialog box, select Automatically create.

Click Confirm to generate API Key credential information.

Enable Key Auth add-on: In the left navigation bar, click Plugin Center. On the page that appears, select Key Auth and then click Create Plugin.

Effective level: Select Routing level and configure the instance, service, and route to which the add-on belongs.

Authentication configuration: Select header and enter the text "Authorization" in the textbox on the right side to indicate that API Key will be included in the Authorization header of requests.

Consumer: select the consumer created in step 1.

Other options can remain as default.

Obtain Ingress address.

VIP: includes both public and private IP addresses.

Backend service: includes the backend service name and access domain name.

Configure domain name resolution.

To test the DNS in the local environment, you can implement a domain-to-IP resolution locally.

In a production environment, configure an access domain name for the Ingress at the domain name provider, and make sure that the public or private IP address of the Ingress is configured in the A record of the Ingress access domain name.

Step 3: Enable observability configuration

In the left navigation bar of the cluster management page, click Cloud native observability > Overview. On the page that appears, click Enable basic observability to enable cloud native observability and VKE observability. For more configuration details, refer to Enabling observability and VKE observability.

After you enable cloud native observability, xLLM metrics will be collected into VMP, and you can view xLLM monitoring charts on the preset dashboards in the VMP console.

Search xllm to view xLLM dashboards, which are shown in the following figure.

(Optional) Use Grafana to view xLLM monitoring charts.

If you want to use Grafana to view xLLM monitoring charts, you can deploy Grafana or import dashboards into your self-built Grafana.

For the method of installing the Grafana service, refer to Deploying Grafana in a VKE cluster and integrating Grafana with a workspace.

xllm_default_dashboard.json

The following figure shows the result after importing.

Model validation

Method 1: Access the model by using the service

After you deploy the deepseek-v3-xllm-pd or deepseek-r1-xllm-pd application template, the created service whose name is in the format of <application name>-<template name>-httpsvc will provide an HTTP API endpoint (port 80). You can call the service endpoint by using an OpenAI-compatible API.

The sample code is as follows.

tip

Replace the value of localhost:80 with the endpoint of your <application name>-<template name>-httpsvc service.

Set the content parameter to your conversation content with the model.

In the Workload section of the Helm details page that corresponds to the deepseek-v3-xllm-pd or deepseek-r1-xllm-pd application, you can select a network-reachable workload to go to the details page. In the upper-right corner of the details page, click Console to log in to the pod that corresponds to the application to call the API.

Method 2: Access the model by using APIG

When enabling the authentication feature of the APIG, you need to add an authentication token to the request.

The following example is provided for reference.

tip

Replace the value of localhost:80 with the access domain name of the Ingress backend service obtained in Step 2.

Replace the value of eyJhbGciOiJSUzI****** with the API key generated in Step 2.

The parameter content is the conversation content you want to send to the model.

Last updated: 2026.07.24 15:50:47