Docs

GPU Compute Service

GLM series models

GPU - Deploying GLM-5 model

This topic, with two ecs.hpcpni3ln.45xlarge instances as an example, introduces how to deploy the GLM-5 model on GPU Compute Service (GPU) instances.

Background

GLM-5 is a large language model based on the 744B-parameter Mixture-of-Experts (MoE) architecture and a DSA sparse attention mechanism. GLM-5 excels in complex systems engineering, long-horizon agent tasks, programming, and agent capabilities. It supports a 202K ultra-long context window, which significantly enhances its autonomous decision-making and execution capabilities.

Software requirements

NVIDIA driver:

GPU driver: A program used to drive NVIDIA GPUs. In the example provided in this topic, version 535.161.08 is used.

Runtime environment:

SGLang framework: a high-performance service framework specifically built for Large Language Models (LLMs) and vision-language models. With the collaborative design of the frontend structured generation language and the backend runtime system, the SGLang framework supports efficient interaction with models and ensures controllability of the generation process. In this topic, version v0.5.8 is used for demonstration.

Procedure

Step 1: Prepare the environment

Create a GPU Accelerated instance.

Create two instances that meet the following conditions. For more information, see Purchasing instances by following the wizard.

Configuration item	Recommended
Basic information
Instance type	ecs.hpcpni3ln.45xlarge tip The instance type ecs.hpcpni3ln.45xlarge is currently in Beta. If you need a trial, please submit a ticket or contact your account manager to apply.
Image	Ubuntu 22.04 with GPU Driver 535.161.08 and doca tip If you choose an image of another version, you need to install the GPU driver after creating the instance.
Number of instances	2 tip The two GPU instances created in this topic are "Rank 0" and "Rank 1" respectively.
Storage	Specification: TSSD TL0 volume Capacity: no less than 1024 GiB tip The GLM-5 model used in this topic has 744 billion parameters. It is recommended to configure volume capacity of not less than 1024 GiB.
Network configuration
EIP	Select "Assign Elastic IP (EIP)".

Install the required software and tools.

Execute the following commands to install and configure Docker.

sudo apt update
sudo apt install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://mirrors.ivolces.com/docker/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://mirrors.ivolces.com/docker/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

Execute the following commands to install the NVIDIA Container Toolkit.

curl -s https://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/3bf863cc.pub | sudo apt-key add -
cat <<EOF >/etc/apt/sources.list.d/nvidia.list
deb http://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/ /
EOF
apt update
apt install nvidia-container-toolkit -y

Execute the following commands to configure Docker. This allows containers to use GPU resources of instances.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

(Optional) Initialize the local volume.

If the instance type you selected includes a local disk, we recommend mounting it to the /data00 directory where the model is stored to improve model loading speed.

mkfs -t ext4 /dev/nvme0n1
mount -t ext4 /dev/nvme0n1 /data00

tip

Replace /dev/nvme0n1 with the name of the actual local volume.

Replace ext4 with the file system you want to use.

Step 2: Deploy and run the model

Download the model file.

Execute the following command to install and configure onion.

curl -fsSL https://mirrors.ivolces.com/extra-tools/ubuntu/GPG-KEY-system | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/volc-extra-tools.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/trusted.gpg.d/volc-extra-tools.gpg] http://mirrors.ivolces.com/extra-tools/ubuntu jammy main" > /etc/apt/sources.list.d/volctools.list && apt update && apt install -y onion-ai-data

Execute the following command to enter the directory where the model is stored.

cd /data00

Execute the following command to download the GLM-5 model.

oniond download model GLM-5-FP8

tip

You can also execute the oniond list model command to view the list of available models.

Execute the following commands separately on both GPU instances based on the region of the instance to download the Docker image.

Beijing

docker pull ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.8.byted.202602120000

Shanghai

docker pull ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.8.byted.202602120000

Execute the following commands separately on both GPU instances based on the region of the instance to start the Docker container.

Beijing

docker run --gpus all -itd --net=host --shm-size=100g --ulimit memlock=-1 --ulimit stack=67108864 --privileged --ipc=host --security-opt seccomp=unconfined --cap-add=ALL -v /data00/:/data00/ -v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ --entrypoint /bin/bash ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.8.byted.202602120000

Shanghai

docker run --gpus all -itd --net=host --shm-size=100g --ulimit memlock=-1 --ulimit stack=67108864 --privileged --ipc=host --security-opt seccomp=unconfined --cap-add=ALL -v /data00/:/data00/ -v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ --entrypoint /bin/bash ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.8.byted.202602120000

tip

Replace /data00 in the command with the actual path where the model is stored.

Check the Docker logs to confirm whether the container and model have started successfully.

Execute the following command to view the container ID.

docker ps

Execute the following command to view the runtime log of the corresponding container.

docker logs -f <CONTAINER_ID>

tip

Replace <CONTAINER_ID> with the actual container ID.

Start the model service.

Execute the following command to enter the Docker container:

tip

Replace <CONTAINER_ID> in the command with the actual container ID.

docker exec -it <CONTAINER_ID> bash

Execute the following command separately on the two instances to start the GLM-5 model service.

warning

Replace <IP> in the command with the actual private IP address of the Rank 0 node instance.

Replace <Port> with the port number for model service connectivity between the two instances. You can specify the port number based on your needs (for example, 10240).

Rank 0 (primary node)
Rank 1

GLOO_SOCKET_IFNAME=eth0 MODEL_LENGTH=131072 NCCL_MIN_NCHANNELS=24 NCCL_IB_QPS_PER_CONNECTION=8 CUDA_CACHE_MAXSIZE=2147483648 python -m sglang.launch_server --tp 16 --dp 16 --cuda-graph-bs 1 8 16 32 --max-running-requests 32 --enable-dp-attention --host 0.0.0.0 --port 8000 --model-path /data00/GLM-5-FP8/ --dist-init-addr <IP>:<Port> --nnodes 2 --node-rank 0 --enable-metrics --tool-call-parser glm47 --reasoning-parser glm45 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --chunked-prefill-size 16384 --trust-remote-code --mem-fraction-static 0.8 --watchdog-timeout 3000 --disable-custom-all-reduce

Parameter description

In the example from the previous step, the commands are used to start and deploy the GLM-5 model. To deploy and start other models, replace the relevant environment variables and parameter values in the commands according to the parameter descriptions in the following table.

Parameter	Argument Type	Description	Sample value
python -m sglang.launch_server	Startup command	The SGLang model server is started via the Python module. tip This item indicates not a parameter, but the command entry.	-
--model-path	Path/model ID	Specifies the loaded LLM mode. Valid values: Hugging Face model ID. Local model path.	/data00/GLM-5-FP8
--tp	Integer	Configure tensor parallelism. Default value: 16.	16
--dp	Integer	Configure data parallelism. Default value: 8.	16
--enable-dp-attention	Boolean switch	Enables attention computation optimization in data parallel mode.	-
--host	IP address	Specifies the host address associated with the service.	0.0.0.0
--port	Integer	Specifies the number of the port the service listens on for connecting to the client side.	8000
--dist-init-addr	IP: port	Indicates the initialization address of the distributed cluster (the primary node address), which is used for communication synchronization across nodes.	192.168.0.2:10240
--nnodes	Integer	Specifies the total number of nodes in the distributed cluster, which is the number of servers involved in the service.	2
--node-rank	Integer	Specifies the sequence number of the current node in the cluster. The numbering starts from 0.	Rank 0: 0 Rank 1: 1

Step 3: Call the model

Execute the following commands to call the model. Make sure that the deployed model can perform inference normally.

curl -X POST http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "GLM-5-FP8",
"messages": [
{
"role": "user",
"content": "Please introduce the Forbidden City in Beijing."
}
],
"max_tokens": 500,
"temperature": 0.7
}'

Parameter description

This topic only includes arguments used for verification.

Parameter	Description	Sample value
model	Indicates the actual model used.	GLM-5-FP8
messages	The list of messages in the conversation. role: the role that initiates the message. content: the content of the message.	-
max_tokens	Specifies the maximum number of tokens for the model-generated Completion in one request. Value: An integer between 1 and 8192. The default value is 4096.	100
temperature	The sampling temperature. Value: Between 0 and 2. The higher the temperature (for example, 1), the more random the output is. A lower value (for example, 0.2) makes the output more focused and deterministic.	0.7

Last updated: 2026.06.25 10:27:48