Docs

Volcengine Kubernetes Engine

Deepseek practices

Deploying DeepSeek-V3.1 with SGLang

This topic describes how to deploy the DeepSeek-V3.1 large language model (LLM) as an inference service via SGLang in VKE.

Solution introduction

The solution leverages multiple acceleration technologies from Volcano Engine to enable rapid deployment of the DeepSeek-V3.1 model within a Kubernetes cluster.

DeepSeek-V3.1, developed by Chinese startup DeepSeek, is a hybrid model that supports both thinking and non-thinking modes.

Deployment information

Model: DeepSeek-V3.1

Inference engine: SGLang

Instance specification: ecs.hpcpni3ln.45xlarge * 2 units

Capabilities

Precision: FP8

Observability: Volcengine Managed Service for Prometheus (VMP)

Access: API Gateway (APIG)

Notes

To ensure successful deployment, which involves a bunch of add-ons depending on each other and a variety of configurations, you need to exactly follow the instructions in this document.

To achieve the expected results and conform to the limits of VKE, you need to perform operations by following the solution described in this article or on the recommended resources. If you need to change your solution, you can consult the Account Manager of the Volcano Engine.

Do not use mGPU on GPU resources.

A full deployment depends on some beta capabilities of Volcano Engine. Contact the account manager of Volcano Engine to apply for the beta trial of the following features:

Beta features/resources	Product involved	Description
hpcpni3ln instance optimized for High-Performance Computing based on GPU	GPU cloud computing servers	The HPC instance, providing computing power. The Deepseek model has a large number of parameters and therefore requires a GPU instance with large memory, e.g., ecs.hpcpni3ln.45xlarge.
Capability of mounting nodes to local disks	Volcengine Kubernetes Engine	You can configure the local disk mounting in the VKE cluster.

Prerequisites

Creating an HPC cluster

The high-performance compute cluster is used to achieve logical isolation of HPC GPU instances (compute nodes). Instances in the same cluster are interconnected by Remote Direct Memory Access (RDMA) networks. For details, see Creating HPC clusters.

tip

The HPC cluster must be deployed in the same region and availability zone as the VKE nodes.

Creating a VKE cluster

The following parameters need to be configured as described in the table when creating a cluster by using the VKE. For details on other parameters, see Creating clusters. You can also use the managed cluster template for heterogeneous computing to quickly create a VKE cluster for this scenario.

Parameter	Description
Cluster configuration
Kubernetes version	Select version v1.28 or later.
Network model	Select VPC-CNI.
Node pool configuration
Managed node pool	Enable managed node pool.
Availability zone	The VKE cluster is deployed in the same availability zone as the HPC cluster.
Instance type	Select the ecs.hpcpni3ln.45xlarge instance family of the High Performance Computing Optimized type that you applied for the beta trial.
High Performance Computing Cluster	Select the HPC cluster you created.
Number of nodes	2
Data volume	Configure the following two data volumes: ESSD: set capacity to 1024 GiB and the performance level to the default PL0. Local SSD: Specify /data01 as the mount directory. Warning The data01 directory will be used as a dependency later and therefore cannot be modified. Ensure the data that needs long-term storage has been backed up properly in case of computer crashes or disk damage which causes data loss.
Node labels	When you deploy the DeepSeek model by using VKE, you need to specify the RDMA usage mode by using node labels. Set the label key to vke.node.rdma.mode and the label value to shared.
Kubelet configs	The non-uniform memory access (NUMA) affinity policy for RDMA devices needs to be configured by using custom parameters with the Kubelet when deploying DeepSeek model in the VKE. Select topology-manager-policy and set the value to best-effort.
Addon configuration
nvidia-device-plugin	(Required) NVIDIA device driver, which is a management add-on that allows the use of GPU devices in containers.
rdma-device-plugin	(Required) Management add-on for RDMA NICs, which enables pod containers to use RDMA devices.
apig-controller	(Required) APIG controller, which works as a cloud-native AI gateway providing intelligent load balancing for inference traffic based on policies.
csi-ebs	(Required) It allows VKE clusters to use the standard Container Storage Interface (CSI) of TOS.
prometheus-agent	(Required) Monitoring add-on for Volcengine Managed Service for Prometheus (VMP), which collects and sends metrics of Kubernetes clusters to VMP.
cr-credential-controller	(Recommended) It can pull images from Container Registry (CR) without passwords, which facilitates deployment process.
keda	(Recommended) Event-driven autoscaler based on Kubernetes. It performs auto scaling for reference services based on complex metrics.
vci-virtual-kubelet	(Recommended) It can deploy CPU-consuming workloads such as system add-ons by using Volcengine Container Instance (VCI), which eliminates the need for operation and maintenance (O&M) and does not occupy the computing resources of GPU nodes.

Procedure

Step 1: deploy LeaderWorkerSet

This topic uses the LeaderWorkerSet (LWS) to orchestrate inference workloads across multiple nodes (for example, DeepSeek-V3.1 requires deployment across two nodes). LWS supports deploying a group of pods as a unit of replication.

In the left navigation bar, click Application center > Application template. Scroll to the bottom of the page, click More application templates.

Click the AI tab. Find the leader-worker-set application template, move the pointer over the application template, and then click Deploy.

On the Deploy application page, configure the parameters for leader-worker-set.

Configuration item	Description
Orchestration template
Application template	Select the latest Chart version of leader-worker-set.
Name	Customize the application name according to the naming requirements prompted by the interface.
Deployment environment
Cluster type	Keep the default value as Managed version.
Cluster	Select the VKE cluster created in the Prerequisites.
Namespace	Select a namespace under the target cluster.
Protocol	Read and understand the agreement before selecting it.

Click Confirm to start deployment.

The applications that are already deployed can be found on the Helm page. Find leader-worker-set application by specifying target project, cluster, and namespace, and then click it to view detailed information of the application.

Step 2: Deploy deepseek-v3.1-sglang

VKE allows you to quickly deploy the DeepSeek-V3.1 model based on the SGLang framework by using an application template.

In the left navigation bar of the VKE console, click Application center > Application template.

On the Application template page, find deepseek-v3.1-sglang, and click Deploy.

On the Deploy application page, configure parameters for deploying deepseek-v3.1-sglang.

Configuration item	Description
Orchestration template
Application template	Select the Chart version of deepseek-v3.1-sglang. We recommend that you select the latest version.
Parameter configuration	Configure parameters for the application deployment. Click the Edit icon to view the configuration description of various parameters in the values.yaml file or edit the YAML file.
Name	Customize the application name according to the naming requirements prompted by the interface.
Deployment environment
Cluster	Select the VKE cluster created in Prerequisites.
Namespace	Select a namespace under the target cluster.
Protocol	Read and understand the agreement before selecting it.

Click Confirm to start deployment.

The applications that are already deployed can be found on the Helm page. Select the target project, cluster, and namespace, find the application deployed based on the deepseek-v3.1-sglang template, and click the application name to go to the details page. On the details page, view basic information about the application, workload, service, and other resources.

tip

The services whose names match the pattern <application name>-api-svc are external access points of their applications and will be used in subsequent steps such as routing configuration and model verification.

On the StatefulSet page of the VKE cluster, view the status of the workload that corresponds to the deepseek-v3.1-sglang application.

In the left navigation bar of the VKE console, click Cluster. Find the target cluster and click the cluster name.

In the left navigation bar, click Workload > StatefulSet. In the search box, enter the application name of deepseek-v3.1-sglang and click the search icon to view the status.

Step 3: Configure the Ingress

Configure routing for the model through APIG Ingress. Built based on Volcano Engine APIG, APIG Ingress functions as a managed cloud-native gateway that features Ingress traffic management that ensures high availability and scalability of cloud-native applications.

On the left navigation bar on the cluster management page, select Services and Ingresses > Ingress, and click Create Ingress. Use API Gateway to expose the model service externally in the cluster. The parameters mentioned below need to be configured as described. Other parameters can be configured by referring to the Creating APIG Ingress through the console.

Configuration item	Description
Basic configuration
Namespace	Select the namespace where you want to deploy the deepseek-v3.1-sglang application template. The namespace is used to associate the service exposed externally by the application.
Ingress configuration
Ingress type	Select Cloud-native API gateway.
APIG specification settings
Gateway specifications	You can select Automatically create, or select Use existing APIG instances. tip When you use the API Gateway to expose the model service in the clusters, we recommend you to configure the following parameters or features (some parameters can only be configured in the APIG console, and cannot be configured in the VKE console): Instance specification: We recommend that you select 4c8g, with two availability zones to realize high availability by default. Bandwidth limit: This option is used to set the maximum bandwidth of the CLB instance working with the APIG instance. If more bandwidth is required, you can specify a custom CLB instance in the APIG console. For details, see Adding access entrance. Circuit breaking: When some GPU instances malfunction, the common method is to take the faulty instances offline to eliminate failures. At the same time, it is necessary to retain the faulty instances for troubleshooting later. You can isolate the faulty instances by configuring a circuit breaking rule for the APIG instance. Request timeout: Timeout is not applied to API Gateway by default. We recommend that you keep the default settings. You can configure a timeout setting in the API Gateway console. For details, see Advanced routing policy. Observability: By default, the alerting and logging capabilities are disabled for APIG instances that are created via the VKE console. You can enable logging and alerting in the APIG console to meet your requirements. Instance monitoring: This feature helps you detect the issues of the entire system link from the perspective of the Ingress in a timely manner. For example, when the back end is unhealthy or crashes, the API Gateway reports error code 503 and error details. Instance logs: You can view the request errors, request time consumption, and the allocation of requests in various Pods in the back end from the access logs.
Backend upstream	We recommend that you set the Load balancing algorithm parameter to Poll. It is more applicable to the scenarios of upgrading back-ends if the back-end loads are approximately balanced.
Listener configuration
Protocol	Select a protocol as needed. If HTTPS is selected, you need to configure the certificate as described in the document APIG Ingress.
Forwarding configuration
Domain name	Fill in the Ingress domain name to be exposed. The A (Address) record of this domain name needs to be configured later with the public or private IP address generated when the Ingress is successfully created.
Service	Select the service that was named in the pattern <application name>-api-svc and associated with the application created based on the deepseek-v3.1-sglang application template, and then set the port number to 8080.

(Optional) Configure APIG authentication.

Create consumer: On the left navigation bar, select Consumer management, and click Create consumer.

Generate an API-KEY: In the consumer list, click the name of the target consumer, to enter the consumer details page. In the API Key information area, click Generate API Key.

In the pop-up dialog box, select Automatically create.

Click Confirm to generate API Key credential information.

Enable Key Auth add-on: In the left navigation bar, click Plugin Center. On the page that appears, select Key Auth and then click Create Plugin.

Effective level: Select Routing level and configure the instance, Service, and route to which the add-on belongs.

Authentication configuration: Select header and enter the text "Authorization" in the textbox on the right side to indicate that API Key will be included in the Authorization header of requests.

Consumer: select the consumer created in step 1.

Other options can remain as default.

Obtain Ingress address.

VIP: includes both public and private IP addresses.

Backend service: includes the backend service name and access domain name.

Configure domain name resolution.

To test the DNS in the local environment, you can implement a domain-to-IP resolution locally.

To test in the production environment, you need to register the Ingress domain name with your DNS provider and add public or private IPs correspondingly.

Step 4: enable observability

In the left navigation bar of the page for a specific cluster, select Overview under the gray title Cloud native observability, then click Enable on the page to start monitoring. For more configuration details, refer to Enable observability and VKE observability.

After turning on observability, the application metrics of SGLang will be collected into the Volcengine Managed Service for Prometheus (VMP). You can use Explore to query metrics in the VMP console. An example is as follows:

There is a built-in VMP dashboard that allows you to view the metrics collected from SGLang-based applications. To do that, you can go to the VMP console, select Monitor dashboard in the left navigation bar and search for SGLang to open the dashboard as shown below:

You can also use self-built Grafana and set up an SGLang monitoring dashboard. For more details, refer to Deploying Grafana in a VKE cluster and integrating Grafana with a workspace.

SGLang Dashboard-1747206543364.json

The effect after import is shown in the figure:

Model verification

Method 1: Access through Service

After you deploy the deepseek-v3.1-sglang application template, the created service whose name is in the format of <application name>-api-svc serves an HTTP API endpoint (port 8080). The endpoint can be called as you do with OpenAI APIs.

The sample code is as follows.

tip

Replace ****** in http://****** with the endpoint of your <application name>-api-svc service.

The parameter content specifies the text you input to chat with LLMs.

In the Workload section of the Helm details page that corresponds to the deepseek-v3.1-sglang application, click the name of the created StatefulSet to go to the StatefulSet page. In the upper-right corner of the details page, click Console to log in to the pod that corresponds to the application to perform API calls.

Method 2: Access via APIG

When enabling the authentication feature of the APIG, you need to add an authentication token in the request.

The following example is provided for reference.

tip

Replace the example.com in http://example.com with the domain name of the back-end services obtained in Step 3: Configure the Ingress.

Replace eyJhbGciOiJSUzI****** with the API Key generated in Step 4: configure Ingress.

The parameter content is what you want to type to chat with LLMs.

Last updated: 2026.07.24 15:50:48