Docs

GPU Compute Service

DeepSeek series models

HPC-Single Node Deployment Plan of DeepSeek-R1 using W4A8

This article mainly introduces the optimized plan targeting DeepSeek-R1 launched in collaboration between the cloud platform and NVIDIA, as well as deployment guidelines on GPU Compute Service (GPU) to reduce deployment costs and improve efficiency.

Background information

DeepSeek-R1, with its outstanding deep inference capability, created a sensation in the AI market upon its launch. However, DeepSeek-R1 comprises 671B model arguments, based on FP8 data precision, and weight VRAM consumption approaches 700GB. Even with a single GPU card with 96GB VRAM, it requires 16 cards for multi-card deployment (using 2 ecs.hpcpni3ln.45xlarge instances). The data transmission efficiency among multi-machine inferences also poses significant challenges. How to ensure model performance while improving deployment efficiency and reducing inference costs has become a critical challenge restricting the large-scale deployment and application of DeepSeek-R1.

To address the above challenges, the cloud platform and NVIDIA have deeply collaborated to launch the DeepSeek-R1 accelerated optimization scheme based on the TensorRT-LLM inference framework. Utilizing W4A8 quantization technology has achieved a performance breakthrough by greatly reducing model storage capacity and computational complexity while maximally preserving model precision:

4-bit weight quantization (W4): A mixed-precision dynamic quantization algorithm is adopted, compressing weight storage capacity to about half of its original size while retaining model precision.

8-bit activation quantization (A8): Activation argument data precision adopts FP8 to ensure model inference precision.

Deployment guidelines

By following the steps below, you can efficiently deploy the W4A8 quantized version of DeepSeek-R1 on the GPU Compute Service (GPU), effectively lowering the deployment threshold for the DeepSeek-R1 model. This not only highlights the performance advantages of W4A8 quantization technology but also helps to apply advanced AI models to practical business scenarios.

Step 1: Preparing the environment

Refer to Using the wizard to buy ECS instances to create an HPC-optimized GPU instance. Configurations are as follows:

Specification: ecs.hpcpni3ln.45xlarge

Image: Ubuntu 22.04 with GPU Driver 535.161.08 and doca

Already Associated with an Elastic IP (EIP).

Configure the Docker environment.

Run the following command to view whether Docker is installed.

docker info | grep -i runtime

If Runtime-related information is returned, it indicates that Docker is installed. You can skip the installation and configuration steps and directly execute step f (pull container images and start container image services).

Run the following command to install Docker.

sudo apt update
sudo apt install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://mirrors.ivolces.com/docker/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://mirrors.ivolces.com/docker/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

Run the following commands to install NVIDIA Container Toolkit.

curl -s https://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/3bf863cc.pub | sudo apt-key add -
cat <<EOF >/etc/apt/sources.list.d/nvidia.list
deb http://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/ /
EOF
apt update
apt install nvidia-container-toolkit -y

Run the following command to configure Docker so that containers can utilize the instance's GPU resources.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Run the following commands sequentially to pull container images and start container image services.

docker pull ai-containers-cn-beijing.cr.volces.com/deeplearning/tensorrt-llm:quant_v3
docker run --gpus all -itd --net=host --shm-size=100g --ulimit memlock=-1 --ulimit stack=67108864 --privileged --ipc=host --security-opt seccomp=unconfined --cap-add=ALL -v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ ${IMAGE} /bin/bash

Step 2: Model quantization preparation

Method 1: Directly download the quantized model file via the Oniod tool

Method 2: Manually generate the W4A8 quantized model of DeepSeek-R1

To improve deployment efficiency, the cloud platform has completed model quantization and pre-loaded the quantized models into the Oniond model library. You can quickly download the model files by executing the following commands through the Oniond tool.

Oniond is a self-developed model downloader by the cloud platform. You can use oniond list models to view the currently supported models for retrieval.

Execute the following command to install Oniond.

curl -fsSL https://mirrors.ivolces.com/extra-tools/ubuntu/GPG-KEY-system | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/volc-extra-tools.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/trusted.gpg.d/volc-extra-tools.gpg] http://mirrors.ivolces.com/extra-tools/ubuntu jammy main" > /etc/apt/sources.list.d/volctools.list && apt update && apt install -y onion-ai-data

Download the quantized model file.

oniond download model DeepSeek-R1-W4AFP8

You can also generate the W4A8 quantized model of DeepSeek-R1 through the following steps.

Execute the following command to install Oniond.

curl -fsSL https://mirrors.ivolces.com/extra-tools/ubuntu/GPG-KEY-system | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/volc-extra-tools.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/trusted.gpg.d/volc-extra-tools.gpg] http://mirrors.ivolces.com/extra-tools/ubuntu jammy main" > /etc/apt/sources.list.d/volctools.list && apt update && apt install -y onion-ai-data

Execute the following command to generate activation quantization coefficients using the NVIDIA TensorRT Model Optimizer tool.

oniond download model DeepSeek-R1

PATH_OF_DEEPSEEK_R1=/llm-models/DeepSeek-R1/DeepSeek-R1

# Install ModelOpt
git clone https://github.com/NVIDIA/TensorRT-Model-Optimizer/ && cd modelopt
pip install "nvidia-modelopt[all]" -U --extra-index-url https://pypi.nvidia.com

# Convert model accuracy
python inference/convert.py --hf-ckpt-path $PATH_OF_DEEPSEEK_R1 --save-path ds_r1 --n-experts 256 --model-parallel 8 && cd ..

# Prepare activation file
torchrun --nproc-per-node 8 --master_port=12346 ptq.py --model_path DeepSeek-V3/ds_r1 --config DeepSeek-V3/inference/configs/config_671B.json --quant_cfg FP8_DEFAULT_CFG --output_path ds_r1_fp8_per_tensor_calibration

Execute the following command to complete the weight quantization and integration of the DeepSeek-R1 model.

# /app/tensorrt_llm/model.sh

#!/bin/bash
HF_MODEL_DIR=/path/to/DeepSeek-R1/
OUTPUT_DIR=/workspace/ckpt/

# Act_scales.safetensors is obtained from the above model
ACT_SCALES=/path/to/DeepSeek-R1-W4AFP8/act_scales.safetensors
# Alternatively, files can be obtained from step a, ACT_SCALES=ds_r1_fp8_per_tensor_calibration

if [ ! -d "convert_logs" ]; then
mkdir convert_logs
fi

pids=()
for i in 0 1 2 3 4 5 6 7
do
python examples/quantization/quantize_mixed_precision_moe.py --model_dir $HF_MODEL_DIR --output_dir $OUTPUT_DIR --act_scales $ACT_SCALES --parts 9 --rank $i > convert_logs/log_$i 2>&1 &
pids+=($!)
done

python examples/quantization/quantize_mixed_precision_moe.py --model_dir $HF_MODEL_DIR --output_dir $OUTPUT_DIR --act_scales $ACT_SCALES --parts 9 --rank 8 > convert_logs/log_8 2>&1
pids+=($!)

for pid in ${pids[@]}; do
wait $pid
done

echo "All processes completed!"

Step 3: Performance evaluation and model execution

Aside from quantization and deployment, the performance of the W4A8 quantized model for the DeepSeek-R1 also needs special attention. This article compares the W4A8 quantized version of DeepSeek-R1 against FP8 data precision versions, revealing a performance difference of within 0.5% based on MMLU and MATH-500 benchmarks.

After completing model deployment, you can quickly verify by referencing the following content.

tip

The $CKPT_PATH in the following commands must be replaced with the actual model storage path.

Model availability verification

Execute the following commands to verify the model's availability.

python examples/pytorch/quickstart_advanced.py --model_dir $CKPT_PATH --tp_size 8 --moe_ep_size 8 --moe_tp_size 1

Model accuracy evaluation

This article uses the MMLU test set as an example to explain how to evaluate model accuracy. Please execute the following commands to download the MMLU test dataset and complete the model accuracy evaluation.

# download dataset
wget https://people.eecs.berkeley.edu/~hendrycks/data.tar && tar -xf data.tar
# accuracy evaluation
trtllm-eval --model $CKPT_PATH --backend pytorch --tp_size 8 --ep_size 8 --kv_cache_free_gpu_memory_fraction 0.75 mmlu --dataset_path $PATH_TO_MMLU_DATA

# MMLU weighted average accuracy: 86.88 (14042)

Performance metric evaluation

Additionally, you can complete performance evaluations related to model throughput, TTFT, TPOT, and other metrics by executing the following commands.

# Prepare the dataset
DS_R1_MODEL_PATH=/path/to/DeepSeek-R1
python /app/tensorrt_llm/benchmarks/cpp/prepare_dataset.py \
--stdout --tokenizer ${DS_R1_MODEL_PATH} \
token-norm-dist \
--input-mean 1000 --output-mean 1000 \
--input-stdev0 --output-stdev0 \
--num-requests 2000 > /tmp/synthetic_1000_1000.txt

cat <<EOF > /workspace/extra-llm-api-config.yml
enable_attention_dp: false
pytorch_backend_config:
enable_overlap_scheduler: false
use_cuda_graph: true
cuda_graph_padding_enabled: true
print_iter_log: true
EOF

trtllm-bench --model $CKPT_PATH --model_path $CKPT_PATH throughput --backend pytorch --max_batch_size 128 --max_num_tokens 1127 --dataset /tmp/synthetic_1000_1000.txt --tp8 --ep8 --kv_cache_free_gpu_mem_fraction 0.8 --extra_llm_api_options /workspace/extra-llm-api-config.yml --concurrency 128 --num_requests 640 --streaming --report_json /workspace/logs/mtp/bench_log_2node_bs128_ep8_tp16_conc128_aa_latency.json

Execute the following commands to start the model online task.

trtllm-serve $CKPT_PATH --host localhost --port 8000 --backend pytorch --max_batch_size 128 --max_num_tokens 3627 --tp_size 8 --ep_size 8 --kv_cache_free_gpu_memory_fraction 0.8 --extra_llm_api_options /workspace/extra-llm-api-config.yml

At this point, you have completed the quantization and deployment of the DeepSeek-R1 model. You can use the local curl command to call the API for model inference services.

curl -X POST http://0.0.0.0:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "/path/to/model",
"messages": [
{
"role": "user",
"content": "Please prove the Riemann hypothesis"
}
],
"max_tokens": 100,
"temperature": 0.7
}'

Last updated: 2025.07.08 15:52:38