Docs

GPU Compute Service

DeepSeek series models

GPU - Deploying DeepSeek-R1/V3 model

This topic, with two ecs.hpcpni3ln.45xlarge instances as an example, introduces how to deploy and run the DeepSeek-R1 or DeepSeek-V3 model on GPU Compute Service (GPU) instances.

Background

DeepSeek-V3

DeepSeek-V3 is a powerful Mixture-of-Experts (MoE) language model with a total of 671 B parameters. Each token activates 37 B parameters, and DeepSeek-V3 was pre-trained on 14.8 T tokens. Learn more.

DeepSeek-V3-0324

DeepSeek-V3-0324 is a minor version of V3 released by DeepSeek, with 685 B parameters. Compared with the DeepSeek-V3 model, DeepSeek-V3-0324 excels in reasoning, coding, and language capacities. Learn more.

DeepSeek-V3.1

DeepSeek-V3.1 is an inference model post-trained from the DeepSeek-V3.1-Base model. Compared with previous versions of the DeepSeek series, it supports hybrid inference architecture and delivers higher reasoning efficiency and agent capabilities. Learn more.

DeepSeek-V3.1-Terminus

DeepSeek-V3.1-Terminus is an inference model modified based on DeepSeek-V3.1. It fixes issues in DeepSeek-V3.1, such as mixed Chinese and English content and occasional abnormal characters, and further optimizes the performance of Code Agent and Search Agent. Learn more.

DeepSeek-R1

DeepSeek-R1 is an inference model trained and generated based on DeepSeek-V3-Base. Compared with the DeepSeek-V3 model, DeepSeek-R1 excels in complex tasks such as mathematics, coding, and natural language reasoning. It attracts global users for its open-source model, innovative algorithms, and high cost-effectiveness. Learn more.

DeepSeek-R1-0528

DeepSeek-R1-0528 is a minor version of R1 released by DeepSeek, with a parameter scale of 685 B. Compared with the DeepSeek-R1 model, the DeepSeek-R1-0528 model has shown significant improvements in coding and deep reasoning. Learn more.

Software requirements

NVIDIA driver:

GPU driver: a program used to drive NVIDIA GPUs. This topic takes 535.161.08 as an example.

Runtime environment:

SGLang framework: a high-speed serving framework for large language models and visual language models. This topic takes the v0.5.0.rc1 version as an example.

Procedure

Step 1: Prepare the environment

Create a GPU Accelerated instance.

Follow the guidance of Purchasing an instance by following the wizard to create two instances that meet the following conditions:

Configuration item	Recommended
Basic information
Instance type	ecs.hpcpni3ln.45xlarge tip The instance type ecs.hpcpni3ln.45xlarge is currently in Beta. If you need a trial, please submit a ticket or contact your account manager to apply.
Image	Ubuntu 22.04 with GPU Driver 535.161.08 and doca tip If you choose an image of another version, you need to install the GPU driver after creating the instance.
Instance quantity	2 tip The two GPU instances in this topic are "Node 0" and "Node 1" respectively.
Storage	Specification: TSSD TL0 EBS volume Capacity: no less than 1 TiB tip The DeepSeek-R1 and DeepSeek-V3 models used in this topic both have 671 B parameters, and the DeepSeek-R1-0528, DeepSeek-V3-0324, DeepSeek-V3.1, and DeepSeek-V3.1-Terminus models have 685 B parameters. You can adjust the cloud volume capacity based on the parameter scale of the model actually deployed.
Network configuration
Virtual Private Cloud	Two instances need to be in the same VPC.
EIP	Select Elastic IP (EIP).

Install the required software and tools.

Execute the following commands separately on the two instances to install and configure Docker.

sudo apt update
sudo apt install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://mirrors.ivolces.com/docker/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://mirrors.ivolces.com/docker/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

Execute the following commands separately on the two instances to install NVIDIA Container Toolkit.

curl -s https://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/3bf863cc.pub | sudo apt-key add -
cat <<EOF >/etc/apt/sources.list.d/nvidia.list
deb http://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/ /
EOF
apt update
apt install nvidia-container-toolkit -y

Execute the following commands on both instances to configure Docker. This allows the containers to use GPU resources of the instances.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

(Optional) Initialize the local volume.

If the instance type you choose has a local volume provided, we recommend that you execute the following commands to mount the local volume to the directory "/data01" where the model is stored. By doing this, you can improve the model loading speed.

mkfs -t ext4 /dev/nvme0n1
mount -t ext4 /dev/nvme0n1 /data01

tip

Replace /dev/nvme0n1 with the name of the local volume you use.

Replace ext4 with the file system you want to use.

Step 2: Deploy and run the model

Start the Docker container and model services on the instances of the corresponding nodes.

Instance "Node 0"
Instance "Node 1"
Execute the commands based on the region where the instance resides to deploy and run the model.
tip
The following command is used to pull model files through the internet, where the internet bandwidth of the target instance is consumed. For billing details, see Elastic IP.
China (Beijing)

docker run -d --network host --privileged --gpus=all --ipc=host -v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ -v /data01:/data -e MODEL_NAME=DeepSeek-R1 -e MODEL_LENGTH=131072 -e TP=16 -e TOTAL_RANKS=2 -e RANKS=0 -e RANK0_ADDR=192.168.0.2:10240 -e NCCL_MIN_NCHANNELS=24 -e NCCL_IB_QPS_PER_CONNECTION=8 -e TORCHINDUCTOR_FX_GRAPH_CACHE=1 -e TORCHINDUCTOR_AUTOGRAD_CACHE=1 -e TORCHINDUCTOR_CACHE_DIR="/data/cache/torch_compile/" -e SGL_ENABLE_JIT_DEEPGEMM=1 -e CMD_ARGS="--cuda-graph-bs 1 2 4 8 10 16 20 24 28 32 40 48 56 64 72 76 78 80 82 --cuda-graph-max-bs 82 --attention-backend fa3 --speculative-algo NEXTN --speculative-draft /data/models/DeepSeek-R1-NextN --speculative-num-steps 3 --speculative-eagle-topk 2 --speculative-num-draft-tokens 4 --enable-ep-moe --max-running-requests 100 --mem-fraction-static 0.8 --disable-chunked-prefix-cache --disable-radix-cache" ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.0rc1.byted.0.0.5.202508202030

China (Shanghai)

docker run -d --network host --privileged --gpus=all --ipc=host -v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ -v /data01:/data -e MODEL_NAME=DeepSeek-R1 -e MODEL_LENGTH=131072 -e TP=16 -e TOTAL_RANKS=2 -e RANKS=0 -e RANK0_ADDR=192.168.0.2:10240 -e NCCL_MIN_NCHANNELS=24 -e NCCL_IB_QPS_PER_CONNECTION=8 -e TORCHINDUCTOR_FX_GRAPH_CACHE=1 -e TORCHINDUCTOR_AUTOGRAD_CACHE=1 -e TORCHINDUCTOR_CACHE_DIR="/data/cache/torch_compile/" -e SGL_ENABLE_JIT_DEEPGEMM=1 -e CMD_ARGS="--cuda-graph-bs 1 2 4 8 10 16 20 24 28 32 40 48 56 64 72 76 78 80 82 --cuda-graph-max-bs 82 --attention-backend fa3 --speculative-algo NEXTN --speculative-draft /data/models/DeepSeek-R1-NextN --speculative-num-steps 3 --speculative-eagle-topk 2 --speculative-num-draft-tokens 4 --enable-ep-moe --max-running-requests 100 --mem-fraction-static 0.8 --disable-chunked-prefix-cache --disable-radix-cache" ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.0rc1.byted.0.0.5.202508202030
Execute the commands based on the region where the instance resides to deploy and run the model.
tip
The following command is used to pull model files through the internet, where the internet bandwidth of the target instance is consumed. For billing details, see Elastic IP.
China (Beijing)

docker run -d --network host --privileged --gpus=all --ipc=host -v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ -v /data01:/data -e MODEL_NAME=DeepSeek-R1 -e MODEL_LENGTH=131072 -e TP=16 -e TOTAL_RANKS=2 -e RANKS=1 -e RANK0_ADDR=192.168.0.2:10240 -e NCCL_MIN_NCHANNELS=24 -e NCCL_IB_QPS_PER_CONNECTION=8 -e TORCHINDUCTOR_FX_GRAPH_CACHE=1 -e TORCHINDUCTOR_AUTOGRAD_CACHE=1 -e TORCHINDUCTOR_CACHE_DIR="/data/cache/torch_compile/" -e SGL_ENABLE_JIT_DEEPGEMM=1 -e CMD_ARGS="--cuda-graph-bs 1 2 4 8 10 16 20 24 28 32 40 48 56 64 72 76 78 80 82 --cuda-graph-max-bs 82 --attention-backend fa3 --speculative-algo NEXTN --speculative-draft /data/models/DeepSeek-R1-NextN --speculative-num-steps 3 --speculative-eagle-topk 2 --speculative-num-draft-tokens 4 --enable-ep-moe --max-running-requests 100 --mem-fraction-static 0.8 --disable-chunked-prefix-cache --disable-radix-cache" ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.0rc1.byted.0.0.5.202508202030

China (Shanghai)

docker run -d --network host --privileged --gpus=all --ipc=host -v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ -v /data01:/data -e MODEL_NAME=DeepSeek-R1 -e MODEL_LENGTH=131072 -e TP=16 -e TOTAL_RANKS=2 -e RANKS=1 -e RANK0_ADDR=192.168.0.2:10240 -e NCCL_MIN_NCHANNELS=24 -e NCCL_IB_QPS_PER_CONNECTION=8 -e TORCHINDUCTOR_FX_GRAPH_CACHE=1 -e TORCHINDUCTOR_AUTOGRAD_CACHE=1 -e TORCHINDUCTOR_CACHE_DIR="/data/cache/torch_compile/" -e SGL_ENABLE_JIT_DEEPGEMM=1 -e CMD_ARGS="--cuda-graph-bs 1 2 4 8 10 16 20 24 28 32 40 48 56 64 72 76 78 80 82 --cuda-graph-max-bs 82 --attention-backend fa3 --speculative-algo NEXTN --speculative-draft /data/models/DeepSeek-R1-NextN --speculative-num-steps 3 --speculative-eagle-topk 2 --speculative-num-draft-tokens 4 --enable-ep-moe --max-running-requests 100 --mem-fraction-static 0.8 --disable-chunked-prefix-cache --disable-radix-cache" ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.0rc1.byted.0.0.5.202508202030

Parameter descriptions:

In the example, the above commands are used for deploying and starting the DeepSeek-R1 model. If you need to deploy and start other models, replace the argument values of the corresponding environment variables by referring to the following descriptions.

Environment variable	Description	Sample value
MODEL_PATH	The model storage path within the container. The default path is /data/models.	/data/models
MODEL_NAME	The name of the model to be deployed. Value: DeepSeek-R1: Deploy the DeepSeek-R1 model (default). DeepSeek-V3: Deploys the DeepSeek-V3 model. DeepSeek-V3-0324: Deploy the DeepSeek-V3-0324 model. DeepSeek-R1-0528: Deploy the DeepSeek-R1-0528 model. DeepSeek-V3.1: Deploy the DeepSeek-V3.1 model. DeepSeek-V3.1-Terminus: Deploy the DeepSeek-V3.1-Terminus model.	DeepSeek-R1
MODEL_LENGTH	The model maximum length (in tokens). Default value is 131072.	131072
TP	Tensor Parallelism degree. Default value is 16.	16
TOTAL_RANKS	The total number of nodes. Default value is 1.	2
RANKS	Specifies rank of the current node. Default value is 0.	0
RANK0_ADDR	Configures the IP address and port number for accessing the node with rank 0 node, in the format: IP address: port number. IP address: (Recommended) Configure the IP address as the private IP address of node 0 . Port number: the port number for communication between nodes.	192.168.0.2:10240
PORT	The number of the port that the service listens on. Default value is 8000.	8080
CMD_ARGS	Specifies additional parameters to start the SGLang framework. Format: -e CMD_ARGS="<argument argument value>". Example: When you add the --mem-fraction-static, --disable-cuda-graph, and --api-key arguments, append the following parameters: -e CMD_ARGS="--mem-fraction-static 0.95 --disable-cuda-graph --api-key 89LTsDxxxxx". tip When specifying additional parameters for starting the SGLang framework via Shell, you need to use \ to escape the characters $、`、\、! contained in parameter values to prevent the Shell interpreter from treating them as metacharacters. For more SGLang framework startup parameters, refer to Server arguments.	"--mem-fraction-static 0.95"

tip

To improve model inference performance, VolcanoEngine has optimized the community edition of the SGLang v0.5.0rc1 framework as follows:

Tune the fused_moe parameter through the configuration file.

Add the runtime arguments --speculative-num-steps 3 --speculative-eagle-topk 2 --speculative-num-draft-tokens 4 to optimize the MTP (Multi-Token Prediction) module of the DeepSeek model.

When you start the container via command, the system will automatically pull the inference engine image and weight files. Due to the large size of the model files, the installation process may take some time. Please wait patiently.

If you need to download the inference engine and model files manually, refer to the appendix for download instructions.

Execute the following commands separately in the two instances. Then, confirm whether the container and model have started through Docker logs.

View the container ID.

docker ps

View the running log of the corresponding container.

docker logs -f <CONTAINER_ID>

tip

Replace <CONTAINER_ID> with the actual container ID.

Step 3: Call the model

Execute the following commands to call the model. Make sure that the deployed model can perform inference normally.

curl -X POST http://0.0.0.0:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "/data/models/DeepSeek-R1",
"messages": [
{
"role": "user",
"content": "Please prove the Riemann hypothesis."
}
],
"max_tokens": 100,
"temperature": 0.7
}'

Parameter descriptions:

This topic only includes parameters used for verification. For more parameter details, refer to DeepSeek API documentation.

Parameter	Description	Sample value
model	The model in use.	/data/models/DeepSeek-R1
messages	The list of messages in the conversation. role: the role that initiates the message. content: the content of the message.	-
max_tokens	Specifies the maximum number of tokens for the model-generated Completion in one request. Value: an integer between 1 and 8192. Default value: 4096	100
temperature	The sampling temperature. The higher the value (for example 1), the more random the output is. Therefore, a lower temperature value (for example 0.2) makes the output more focused and deterministic. Valid value: between 0and 2.	0.7

FAQ

How to resolve the error "torch.distributed.DistStoreError: Time out after 601 seconds waiting for clients."?

Cause:

The two nodes failed to establish communication within 600 seconds after the model service started.

Solutions:

Make sure that all parameters in the command used to start Docker containers and model services are configured correctly.

Execute the commands used to start Docker containers and model services on the two instances separately as soon as possible to avoid timeouts.

Appendix

You can manually download the VolcanoEngine cached inference engine and model files, or visit the DeepSeek official website to download the relevant content.

Download the inference engine.

Region	Download method
China (Beijing)	docker pull ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.0rc1.byted.0.0.5.202508202030
China (Shanghai)	docker pull ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.0rc1.byted.0.0.5.202508202030

Download the model file.

Execute the following commands to install the Oniond tool.

curl -fsSL https://mirrors.ivolces.com/extra-tools/ubuntu/GPG-KEY-system | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/volc-extra-tools.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/trusted.gpg.d/volc-extra-tools.gpg] http://mirrors.ivolces.com/extra-tools/ubuntu jammy main" > /etc/apt/sources.list.d/volctools.list && apt update && apt install -y onion-ai-data

Download the model file.

oniond download model DeepSeek-V3

tip

Replace DeepSeek-V3 with the model name you need to download. Supported models: DeepSeek-V3, DeepSeek-V3-0324, DeepSeek-V3.1, DeepSeek-V3.1-Terminus, DeepSeek-R1, and DeepSeek-R1-0528.

Last updated: 2025.09.24 15:39:30