Docs

Volcengine Kubernetes Engine

Kimi model deployment

Deploying Kimi-K3 model quickly via LLM general template

This topic describes how to deploy the Kimi-K3 model in a VKE cluster using the ServingKit LLM general template.

Background

Kimi K3 is an open-source multimodal LLM released by Moonshot AI, and it is also the first open-source model in the world to exceed the 3T-parameter landmark. The model is built on the KDA architecture and attention residual (AttnRes) structure, with 2.8T parameters, native visual understanding, and a million-token ultra-long context window. Kimi K3 is designed for long-horizon programming, complex reasoning, and deep knowledge work, and performs excellently across evaluations in multiple domains. This topic describes how to quickly deploy the Kimi-K3 model on a Kubernetes cluster using the LLM general template.

Notes

To ensure successful deployment, which involves multiple interdependent add-ons and various configurations, you need to precisely follow the instructions in this document.

To achieve the expected results and conform to the limits of VKE, you need to perform operations by following the solution described in this article or on the recommended resources. If you need to use a different solution, contact your Volcano Engine account manager for assistance.

Do not use mGPU to partition GPU resources.

Preparation

Some resources of Volcano Engine at the beta stage are required to deploy AI services. To use the following resources, contact your account manager.

Beta resource	Related service	Description
hpcpni3ln HPC-optimized instance	GPU Cloud Computing	The HPC instance, providing computing power. When deploying Kimi-K3, we recommend requesting 4 GPU instances of the ecs.hpcpni3ln.45xlarge type.

In this example, APIG is used to expose the service. Therefore, you need to prepare an APIG instance. You can create an APIG instance in advance, use an existing APIG instance, or leverage automatic instance creation during application deployment on ServingKit.

tip

If you create an APIG instance in advance or use an existing one, ensure that the region of the instance matches the region of the VKE cluster. We recommend an instance with both Public network and Private network selected for Network type. For details about creating APIG instances, see Creating instances.

You need to create a high-performance compute (HPC) cluster to achieve logical isolation of HPC GPU instances (compute nodes). Instances in the same cluster are interconnected by Remote Direct Memory Access (RDMA) networks. For details, see Creating HPC clusters.

tip

The HPC cluster must be deployed in the same region and availability zone as the VKE cluster.

Procedure

Step 1: Create a VKE heterogeneous computing cluster

Quickly create a heterogeneous computing cluster by using a VKE cluster template. Take note of the parameter configurations listed in the following table. For more details, see Using a cluster template to create managed clusters for heterogeneous computing.

Parameter	Description
Cluster and node pool configuration
Region	The Region and Availability zone of the cluster must be consistent with those of the prepared HPC cluster. You can switch regions in the top navigation bar.
Availability zone
Instance type	Select the ecs.hpcpni3ln.45xlarge specification of the High Performance Computing Optimized type.
Number of nodes	Set it to 4.
Advanced settings
Data volume	Configure the following two data volumes: ESSD: Select 1024 GiB and retain the default performance level PL0. Local SSD: Specify /data01 as the mount directory. warning The /data01 directory is required by subsequent steps and must not be changed or customized. If the physical host fails or the local disk is damaged, data loss may occur. Do not store data that must be retained long-term on the local disk, and make sure backups are in place.

Step 2: Deploy Kimi-K3

Download and deploy the model using the LLM General Template provided by ServingKit. During deployment, take note of the parameter configurations listed in the following table. For descriptions of other parameters and detailed procedures, see Deploying general-purpose LLM template applications.

Parameter	Description
Select framework and orchestration
Image	Select Predefined, then in the Advanced filter drop-down list, select POC and SGLang, find the v0.5.16.iaas.202607280000-kimi-k3-cu130 image, and select the image.
Model configuration	StorageClass type: Select Data volume. Select models: Click Add predefined model, select Kimi-K3 and Kimi-K3-DSpark models, and configure the Mount path as /data01 and the Container mounting path as /data/models. The model data will be downloaded to the /data01 directory.
Inference mode	Select PD co-location.
Instance specification	Number of instances: Set the value to 1. Set Single instance GPUs to 32.
Resource configuration	Select the VKE heterogeneous computing cluster created in Step 1 and the node pool containing nodes with the ecs.hpcpni3ln.45xlarge specification in the cluster.
Start command	Add the following startup command. For a detailed description of each field in the startup command, see the "Startup command descriptions" table below. source /workspace/nccl_env.sh && GLOO_SOCKET_IFNAME=eth0 NCCL_MIN_NCHANNELS=24 NCCL_IB_QPS_PER_CONNECTION=8 NCCL_CUMEM_ENABLE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_K3_ATTN_RES_MODE=jit SGLANG_MOE_FUSED_GATE_RADIX=1 sglang serve --trust-remote-code \ --model-path /data/models/Kimi-K3 \ --served-model-name kimi-k3 \ --tp-size 32 \ --nnodes 4 \ --node-rank $POD_GROUP_INDEX \ --dist-init-addr "${ROLESET_NAME}-${ROLE_NAME}-${ROLE_TEMPLATE_HASH}-${ROLE_REPLICA_INDEX}-0.${STORM_SERVICE_NAME}:20000" \ --ep-size 32 \ --moe-runner-backend marlin \ --decode-attention-backend flashmla \ --mem-fraction-static 0.85 \ --mamba-radix-cache-strategy extra_buffer_lazy \ --dist-timeout 3600 \ --reasoning-parser kimi_k3 \ --tool-call-parser kimi_k3 \ --mamba-full-memory-ratio 0.8 \ --host 0.0.0.0 \ --port 8080 \ --speculative-algorithm DSPARK \ --speculative-draft-model-path /data/models/Kimi-K3-DSpark \ --speculative-dspark-block-size 7 \ --enable-linear-replayssm-spec \ --cuda-graph-max-bs 128 \ --max-running-requests 128 \ --enable-metrics \ --enable-cache-report \ --show-time-cost Set the port to 8080.
API Gateway	Select Enable and associate the prepared APIG instance. tip You can also select Automatically create to have an automatically created APIG instance with a fixed instance specification and configuration specified by ServingKit. Creating an APIG instance will incur fees. For details, see APIG billing.

Startup command descriptions

Parameter	Type	Description	Sample value
sglang serve	Startup command	Start the model service backend via the SGLang CLI. This field is a command entry instead of a parameter.	None
--model	Path/Model ID	The loaded LLM model. You can set this field to the following values: Hugging Face model ID Local model path	/data/models/Kimi-K3
--tp-size	Integer	Configure tensor parallelism.	32
--nnodes	Integer	Specify the total number of nodes in the distributed cluster.	4
--node-rank	Integer	Specify the current node's index in the cluster, starting from 0.	0
--dist-init-addr	IP: Port	The initialization address (manager node address) of the distributed cluster, which is used for multi-node communication synchronization.	Variable injection
--ep-size	Integer	Configure expert parallelism.	32
--host	IP address	The host address bound to the service.	0.0.0.0
--port	Integer	The port number that the service listens on, which is used to connect to the client.	8000
--mem-fraction-static	Float	Restricts the static GPU memory ratio, leaving more room for the runtime workspace.	0.85
--cuda-graph-max-bs	Integer	Used to set the maximum batch size for CUDA Graph, reducing batch GPU memory pressure during capture and replay.	128
--max-running-requests	Integer	Limit the maximum number of concurrent requests being processed at the same time.	128
--enable-metrics	Command	Collect basic observability metrics. For detailed metric descriptions, see SGLang documentation.	None

After successful application deployment, model startup time varies depending on model size. After the application status changes to Running, view the logs of the corresponding instance on the Application details page. If the log contains the following message: The server is fired up and ready to roll!, the model in the application has started successfully and is accessible.

Step 3: Access the service

After successfully deploying the Kimi-K3 model service, obtain the service domain name and access the service.

Obtain the Kimi-K3 model service domain name.

In the VKE console, go to ServingKit > List of AI applications and find the deployed Kimi-K3 model applications.

Click the → icon in the application section to go to the application details page and view Service domain names.

On a terminal where the curl command is available, execute the following command to access the Kimi-K3 model service.

tip

Replace ${your_service_domain_name} with the service domain name obtained in the previous step.

If no terminal is available, you can log in to the node where the model is deployed by using the ECS Terminal method and use the curl command to access the service. For detailed operations, see ECS Terminal.

curl -X POST http://${your_service_domain_name}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Kimi-K3",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "help me write a random generator in python"}
]
}'

The descriptions of parameters are as follows:

Parameter	Description	Sample value
model	The model to use.	Kimi-K3
messages	The message list of the conversation. role: The role that initiates the message. content: The content of the message.	None

The following figure displays the expected response after you run the command.

Subsequent operations

Enhance service security

You can enhance the security of the Kimi-K3 service by using the API Gateway's API Key.

Create a consumer for the APIG instance. For detailed operations, see Creating consumers.

Generate the Key Auth plugin credential based on the consumer. For detailed operations, see Generating credentials.

Enable the Key Auth plugin: In the left navigation bar, click Plugin Center. On the page that appears, click Key Auth in the Plugin Category section and then click Create Plugin. Specify the parameters of the plugin.

Effective level: Select Routing level, and specify the instance, service, and route to which the plugin belongs.

Authentication configuration: Select header, and specify Authorization as the source, which indicates that the API Key is carried in the Authorization header of the request.

Consumer: Select the consumer created in Step 1 or select all, which indicates that the configuration also applies to newly created consumers.

Keep the default values for other parameters.

After you enable Key Auth plugin authentication, when you access the service by executing the corresponding command, credentials must be included. Otherwise, a message that indicates insufficient permissions will be returned.

curl http://${your_service_domain_name}/v1/chat/completions -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${your_api_key}" \
-d '{
"model": "Kimi-K3",
"messages": [
{
"role": "user",
"content": "help me write a random generator in python"
}
],
"max_tokens": 500,
"temperature": 0.7
}'

Enabling observability

ServingKit AI applications support basic cloud-native observability capabilities by default. You can view the application monitoring dashboard on the AI application details page.

In the VKE console, go to ServingKit > List of AI applications and find the deployed Kimi-K3 model applications.

Click the → icon in the application section to go to the application details page and view the dashboard on the Observability tab.

tip

If you have not enabled the observability capability of the relevant cluster, enable observability based on the prompts in the console. For operations related to cloud-native observability capabilities, see Enabling observability and VKE observability.

Last updated: 2026.08.03 21:49:41