Docs

GPU Compute Service

GLM series models

GPU - Deploying GLM-5.2 model

This topic describes how to deploy the GLM-5.2 model on GPU instances, using two ecs.hpcpni3ln.45xlarge instances as an example.

Background

GLM-5.2-FP8 is the FP8-quantized version of the latest open-source MoE large model. It delivers a 1M context window, boasting approximately 753B total model parameters and around 40B active parameters during inference. This large-scale model is designed for long-context processing, code generation, complex reasoning, and agent scenarios. Its FP8-quantized version further reduces GPU memory usage for inference, making it more suitable for high-performance deployment and local inference scenarios.

Software requirements

NVIDIA driver:

GPU driver: A program used to drive NVIDIA GPUs. In the example provided in this topic, version 535.261.03 is used.

Runtime environment:

SGLang framework: a high-performance service framework specifically built for Large Language Models (LLMs) and vision-language models. With the collaborative design of the frontend structured generation language and the backend runtime system, the SGLang framework supports efficient interaction with models and ensures controllability of the generation process. In this topic, v0.5.13 is used for demonstration.

Procedure

Step 1: Prepare the environment

Create a GPU Accelerated instance.

Create two instances that meet the following conditions. For more information, see Purchasing instances by following the wizard.

Configuration item	Recommended configuration
Basic information
Instance type	ecs.hpcpni3ln.45xlarge tip The instance type ecs.hpcpni3ln.45xlarge is currently in Beta. If you need a trial, please submit a ticket or contact your account manager to apply.
Image	Ubuntu 22.04. Select "Automatically install GPU driver in the background" to install the GPU driver 535.261.03. tip If you choose an image based on another distribution release, you need to install the GPU driver after creating the instance.
Number of instances	2 tip The two GPU instances created in this topic are "Rank 0" and "Rank 1" respectively.
Storage	Specification: TSSD TL0 volume Capacity: no less than 1,024 GiB tip The GLM-5.2-FP8 model used in this topic has 753 billion parameters. We recommend configuring a volume capacity of no less than 1,024 GiB.
Network configuration
EIP	Select "Assign Elastic IP (EIP)".

Install the required software and tools.

Execute the following commands to install and configure Docker.

sudo apt update
sudo apt install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://mirrors.ivolces.com/docker/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://mirrors.ivolces.com/docker/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

Execute the following commands to install the NVIDIA Container Toolkit.

curl -s https://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/3bf863cc.pub | sudo apt-key add -
cat <<EOF >/etc/apt/sources.list.d/nvidia.list
deb http://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/ /
EOF
apt update
apt install nvidia-container-toolkit -y

Execute the following commands to configure Docker so that containers can use the instance's GPU resources.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

(Optional) Initialize the local volume.

If the instance type you choose has local volume provided, we recommend that you execute the following commands to mount the local volume to the directory "/data00" where the model is stored. By doing this, you can improve the model loading speed.

mkfs -t ext4 /dev/nvme0n1
mount -t ext4 /dev/nvme0n1 /data00

tip

Replace /dev/nvme0n1 with the name of the actual local volume.

Replace ext4 with the file system you want to use.

Step 2: Deploy and run the model

Execute the following command separately on the two instances to install and configure the onion and download the GLM-5.2-FP8 model file.

China (Beijing)
China (Shanghai)

docker run --rm \
-v /data00/:/data00 \
--workdir /data00 \
--entrypoint bash \
ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.13.post1.iaas.dev.202606172008 \
-lc '
set -e
if ! command -v oniond >/dev/null 2>&1; then
apt update
apt install -y ca-certificates curl gnupg gpg
curl -fsSL https://mirrors.ivolces.com/extra-tools/ubuntu/GPG-KEY-system | gpg --dearmor -o /etc/apt/trusted.gpg.d/volc-extra-tools.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/trusted.gpg.d/volc-extra-tools.gpg] http://mirrors.ivolces.com/extra-tools/ubuntu noble main" > /etc/apt/sources.list.d/volctools.list
apt update
apt install -y onion-ai-data
fi
oniond download model GLM-5.2-FP8 --turbo --dir /data00
'

tip

You can also execute the oniond list model command to view the list of available models.

Execute the following commands separately on both GPU instances based on the region of the instance to download the Docker image.

China (Beijing)
China (Shanghai)

docker pull ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.13.post1.iaas.dev.202606172008

Execute the following commands separately on both GPU instances based on the region of the instance to start the Docker container.

tip

Replace /data00 in the command with the actual path where the model is stored.

Replace <rank0_private_ip> in the command with the private IP address of the Rank 0 node instance.

Replace <Port> in the command with the port number for model service communication between the two instances. You can specify any port number (for example, 20000).

China (Beijing)

Rank 0 (primary node)
Rank 1

docker run \
-d \
--gpus all \
--name sglang-glm52 \
--net=host \
--shm-size=100g \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--privileged \
--ipc=host \
--security-opt seccomp=unconfined \
--cap-add=ALL \
-v /data00/:/data00/ \
-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \
-e GLOO_SOCKET_IFNAME=eth0 \
-e MODEL_LENGTH=262144 \
-e NCCL_MIN_NCHANNELS=24 \
-e NCCL_IB_QPS_PER_CONNECTION=8 \
-e CUDA_CACHE_MAXSIZE=2147483648 \
-e SGLANG_ENABLE_SPEC_V2=1 \
--entrypoint sglang \
ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.13.post1.iaas.dev.202606172008 \
serve \
--model-path /data00/GLM-5.2-FP8/ \
--served-model-name glm-5.2 \
--host 0.0.0.0 \
--port 30300 \
--trust-remote-code --tp 16 \
--nnodes 2 \
--node-rank 0 \
--dist-init-addr "<rank0_private_ip>:<Port>" \
--chunked-prefill-size 32768 \
--enable-metrics --enable-cache-report --show-time-cost --reasoning-parser glm45 \
--tool-call-parser glm47 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--cuda-graph-max-bs 8 \
--mem-fraction-static 0.82 \
--enable-metrics-for-all-schedulers

China (Shanghai)

Rank 0（Master）
Rank 1

docker run \
-d \
--gpus all \
--name sglang-glm52 \
--net=host \
--shm-size=100g \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--privileged \
--ipc=host \
--security-opt seccomp=unconfined \
--cap-add=ALL \
-v /data00/:/data00/ \
-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \
-e GLOO_SOCKET_IFNAME=eth0 \
-e MODEL_LENGTH=262144 \
-e NCCL_MIN_NCHANNELS=24 \
-e NCCL_IB_QPS_PER_CONNECTION=8 \
-e CUDA_CACHE_MAXSIZE=2147483648 \
-e SGLANG_ENABLE_SPEC_V2=1 \
--entrypoint sglang \
ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.13.post1.iaas.dev.202606172008 \
serve \
--model-path /data00/GLM-5.2-FP8/ \
--served-model-name glm-5.2 \
--host 0.0.0.0 \
--port 30300 \
--trust-remote-code --tp 16 \
--nnodes 2 \
--node-rank 0 \
--dist-init-addr "<rank0_private_ip>:<Port>" \
--chunked-prefill-size 32768 \
--enable-metrics --enable-cache-report --show-time-cost --reasoning-parser glm45 \
--tool-call-parser glm47 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--cuda-graph-max-bs 8 \
--mem-fraction-static 0.82 \
--enable-metrics-for-all-schedulers

Parameter description

In the example from the previous step, the commands are used to start and deploy the GLM-5.2 model. To deploy and start other models, replace the relevant environment variables and parameter values in the commands according to the parameter descriptions in the following table.

Parameter	Parameter type	Description	Sample value
sglang serve	Startup command	A command entry that indicates starting the model server via SGLang CLI.	-
--model-path	Path/model ID	Specify the LLM to load. Supported: Hugging Face model ID Local model path	/data/models/GLM-5.2-FP8
--tp	Integer	Configures Tensor Parallelism.	16
--host	IP address	Specifies the host address associated with the service.	0.0.0.0
--port	Integer	Specifies the service listening port, which is used for client connection.	30300
--dist-init-addr	IP: Port / DNS: Port	The distributed cluster initialization address used for multi-node communication synchronization. It must be set to the private IP or Service DNS of the Rank 0 node and must be the same across all nodes.	127.0.0:20000
--nnodes	Integer	Specifies the total number of nodes in the distributed cluster.	2
--node-rank	Integer	Specifies the rank of the current node within the cluster, starting from 0.	Rank 0: 0 Rank 1: 1
--chunked-prefill-size	Integer	Controls the chunk granularity for long-context pre-filling, measured in tokens.	32768
--mem-fraction-static	Floating point number	The ratio of static GPU memory allocated to model weight. The remaining memory is reserved as runtime space.	0.82
--cuda-graph-max-bs	Integer	Limits the maximum batch size for CUDA graph capture/replay to reduce peak GPU memory usage.	8

Step 3: Call the model

Execute the following command to enter the Docker container:

tip

Replace <CONTAINER_ID> in the command with the actual container ID.

docker exec -it <CONTAINER_ID> bash

Execute the following command in the container to call the model to verify whether the deployed model can perform inference normally.

curl -X POST http://127.0.0.1:30300/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "glm-5.2",
"messages": [
{
"role": "user",
"content": "Please answer directly: What is 1+1?"
}
],
"max_tokens": 2048,
"temperature": 0
}'

Parameter description

This topic only describes the parameters used for verification.

Parameter	Description	Sample value
model	Indicates the actual model used.	glm-5.2
messages	The list of messages in the conversation. role: the role that initiates the message. content: the content of the message.	-
max_tokens	Specifies the maximum number of tokens in the completion generated by the model for a single request. Valid values: An integer between 1 and 8192. The default value is 4096.	500
temperature	The sampling temperature. Value: Between 0 and 2. The higher the temperature (for example, 1), the more random the output is. A lower value (for example, 0.2) makes the output more focused and deterministic.	0.7

Last updated: 2026.06.25 10:27:49