GPU云服务器
GPU云服务器
文档指南
请输入
- 文档首页
GPU云服务器GPU云服务器最佳实践AIGC实践Kimi系列模型GPU-部署Kimi-K3模型
复制全文
下载 pdfGPU-部署Kimi-K3模型
文档反馈
问问助手本文以4台ecs.hpcpni3ln.45xlarge实例为例，介绍如何在GPU云服务器上部署Kimi K3模型。
背景信息
Kimi K3是月之暗面（Moonshot AI）发布的开源多模态智能体大模型，也是全球首个突破3T参数门槛的开源模型。模型基于KDA（Kimi Delta Attention）与注意力残差（AttnRes）架构构建，参数规模达2.8T，原生支持视觉理解，并拥有百万Token的超长上下文窗口。Kimi K3专为长周期编程、复杂推理及深度知识工作而设计，在多项权威评测中表现优异。了解更多
软件要求
- NVIDIA驱动：
- GPU驱动：用来驱动NVIDIA GPU的程序，本文以535.261.03为例。
- 运行环境：
- SGLang框架：SGLang框架是一款专为大语言模型（LLMs）及视觉语言模型打造的高性能服务框架，通过协同设计前端结构化生成语言与后端运行时系统，实现与模型的高效交互，并保障了生成过程的可控性。本文以火山引擎提供的SGLang容器镜像为例。
操作步骤
步骤一：准备环境
- 创建GPU计算型实例。
- 请根据通过向导购买实例指引，创建4台符合如下条件的实例：
- 配置项推荐配置基础配置计算规格ecs.hpcpni3ln.45xlarge注意镜像实例数量4说明本文创建的4台GPU实例，分别作为“Rank 0”、“Rank 1”、“Rank 2”、“Rank 3”。存储
- 规格：吞吐型SSD TL0云盘
- 容量：不低于 2048 GiB
注意由于Kimi K3模型文件较大，建议您配置的云盘或本地盘容量不低于 2048 GiB。网络配置私有网络4台实例需处于同一私有网络。公网IP勾选“分配弹性公网IP”。
- 安装依赖的软件、工具。
- 登录目标实例。
- 执行如下命令，安装、配置Docker。
- sudo apt updatesudo apt install ca-certificates curl gnupg lsb-releasesudo mkdir -p /etc/apt/keyringscurl -fsSL https://mirrors.ivolces.com/docker/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpgecho "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://mirrors.ivolces.com/docker/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/nullsudo apt updatesudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
- 执行如下命令，安装NVIDIA Container Toolkit。
- curl -s https://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/3bf863cc.pub | sudo apt-key add -cat <<EOF >/etc/apt/sources.list.d/nvidia.listdeb http://mirrors.ivolces.com/nvidia_all/ubuntu2204/x86_64/ /EOFapt updateapt install nvidia-container-toolkit -y
- 执行如下命令配置Docker，使容器可以使用实例的GPU资源。
- sudo nvidia-ctk runtime configure --runtime=dockersudo systemctl restart docker
- （可选）初始化本地盘。
- 若您选用的实例规格搭配有本地盘，建议您执行如下命令将本地盘挂载到存储模型的目录“/data00”，提升模型加载速度。
- mkfs -t ext4 /dev/nvme0n1mount -t ext4 /dev/nvme0n1 /data00
说明
- 请将/dev/nvme0n1替换为实际本地盘名称。
- 请将ext4替换为您期望使用的文件系统。
步骤二：部署、运行模型
- 登录目标GPU实例。了解更多。
- 根据实例所在地域，在4台实例中分别执行如下命令，下载Kimi K3与Kimi K3 DSpark模型。
- docker run --rm \-v /data00/:/data00 \--workdir /data00 \--entrypoint bash \ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \-lc 'set -eif ! command -v oniond >/dev/null 2>&1; thenapt updateapt install -y ca-certificates curl gnupg gpgcurl -fsSL https://mirrors.ivolces.com/extra-tools/ubuntu/GPG-KEY-system | gpg --dearmor -o /etc/apt/trusted.gpg.d/volc-extra-tools.gpgecho "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/trusted.gpg.d/volc-extra-tools.gpg] http://mirrors.ivolces.com/extra-tools/ubuntu noble main" > /etc/apt/sources.list.d/volctools.listapt updateapt install -y onion-ai-datafioniond download model Kimi-K3 --turbo --dir /data00oniond download model Kimi-K3-DSpark --turbo --dir /data00'docker run --rm \-v /data00/:/data00 \--workdir /data00 \--entrypoint bash \ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \-lc 'set -eif ! command -v oniond >/dev/null 2>&1; thenapt updateapt install -y ca-certificates curl gnupg gpgcurl -fsSL https://mirrors.ivolces.com/extra-tools/ubuntu/GPG-KEY-system | gpg --dearmor -o /etc/apt/trusted.gpg.d/volc-extra-tools.gpgecho "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/trusted.gpg.d/volc-extra-tools.gpg] http://mirrors.ivolces.com/extra-tools/ubuntu noble main" > /etc/apt/sources.list.d/volctools.listapt updateapt install -y onion-ai-datafioniond download model Kimi-K3 --turbo --dir /data00oniond download model Kimi-K3-DSpark --turbo --dir /data00'
- 请根据实例所在地域，在实例中执行对应命令，启动Docker容器。
注意
- 请将/data00替换为模型实际存放目录路径。
- 请将<rank0_private_ip>替换为“Rank 0”实例私网IP地址。
- 请根据您为实例指定的节点名称，执行对应命令。docker run \-d \--gpus all \--name sglang-kimi-k3 \--net=host \--shm-size=100g \--ulimit memlock=-1 \--ulimit stack=67108864 \--privileged \--ipc=host \--security-opt seccomp=unconfined \--cap-add=ALL \-v /data00/:/data00/ \-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \-e GLOO_SOCKET_IFNAME=eth0 \-e NCCL_MIN_NCHANNELS=24 \-e NCCL_IB_QPS_PER_CONNECTION=8 \-e NCCL_CUMEM_ENABLE=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \-e SGLANG_K3_ATTN_RES_MODE=jit \-e SGLANG_MOE_FUSED_GATE_RADIX=1 \--entrypoint sglang \ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \serve \--trust-remote-code \--model-path /data00/Kimi-K3 \--served-model-name kimi-k3 \--tp-size 32 \--nnodes 4 \--node-rank 0 \--dist-init-addr "<rank0_private_ip>:20000" \--ep-size 32 \--moe-runner-backend marlin \--decode-attention-backend flashmla \--mem-fraction-static 0.85 \--mamba-radix-cache-strategy extra_buffer_lazy \--dist-timeout 3600 \--reasoning-parser kimi_k3 \--tool-call-parser kimi_k3 \--mamba-full-memory-ratio 0.8 \--host 0.0.0.0 \--port 8080 \--speculative-algorithm DSPARK \--speculative-draft-model-path /data00/Kimi-K3-DSpark \--speculative-dspark-block-size 7 \--enable-linear-replayssm-spec \--cuda-graph-max-bs 128 \--max-running-requests 128 \--enable-metrics \--enable-cache-report \--show-time-costdocker run \-d \--gpus all \--name sglang-kimi-k3 \--net=host \--shm-size=100g \--ulimit memlock=-1 \--ulimit stack=67108864 \--privileged \--ipc=host \--security-opt seccomp=unconfined \--cap-add=ALL \-v /data00/:/data00/ \-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \-e GLOO_SOCKET_IFNAME=eth0 \-e NCCL_MIN_NCHANNELS=24 \-e NCCL_IB_QPS_PER_CONNECTION=8 \-e NCCL_CUMEM_ENABLE=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \-e SGLANG_K3_ATTN_RES_MODE=jit \-e SGLANG_MOE_FUSED_GATE_RADIX=1 \--entrypoint sglang \ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \serve \--trust-remote-code \--model-path /data00/Kimi-K3 \--served-model-name kimi-k3 \--tp-size 32 \--nnodes 4 \--node-rank 1 \--dist-init-addr "<rank0_private_ip>:20000" \--ep-size 32 \--moe-runner-backend marlin \--decode-attention-backend flashmla \--mem-fraction-static 0.85 \--mamba-radix-cache-strategy extra_buffer_lazy \--dist-timeout 3600 \--reasoning-parser kimi_k3 \--tool-call-parser kimi_k3 \--mamba-full-memory-ratio 0.8 \--host 0.0.0.0 \--port 8080 \--speculative-algorithm DSPARK \--speculative-draft-model-path /data00/Kimi-K3-DSpark \--speculative-dspark-block-size 7 \--enable-linear-replayssm-spec \--cuda-graph-max-bs 128 \--max-running-requests 128 \--enable-metrics \--enable-cache-report \--show-time-costdocker run \-d \--gpus all \--name sglang-kimi-k3 \--net=host \--shm-size=100g \--ulimit memlock=-1 \--ulimit stack=67108864 \--privileged \--ipc=host \--security-opt seccomp=unconfined \--cap-add=ALL \-v /data00/:/data00/ \-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \-e GLOO_SOCKET_IFNAME=eth0 \-e NCCL_MIN_NCHANNELS=24 \-e NCCL_IB_QPS_PER_CONNECTION=8 \-e NCCL_CUMEM_ENABLE=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \-e SGLANG_K3_ATTN_RES_MODE=jit \-e SGLANG_MOE_FUSED_GATE_RADIX=1 \--entrypoint sglang \ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \serve \--trust-remote-code \--model-path /data00/Kimi-K3 \--served-model-name kimi-k3 \--tp-size 32 \--nnodes 4 \--node-rank 2 \--dist-init-addr "<rank0_private_ip>:20000" \--ep-size 32 \--moe-runner-backend marlin \--decode-attention-backend flashmla \--mem-fraction-static 0.85 \--mamba-radix-cache-strategy extra_buffer_lazy \--dist-timeout 3600 \--reasoning-parser kimi_k3 \--tool-call-parser kimi_k3 \--mamba-full-memory-ratio 0.8 \--host 0.0.0.0 \--port 8080 \--speculative-algorithm DSPARK \--speculative-draft-model-path /data00/Kimi-K3-DSpark \--speculative-dspark-block-size 7 \--enable-linear-replayssm-spec \--cuda-graph-max-bs 128 \--max-running-requests 128 \--enable-metrics \--enable-cache-report \--show-time-costdocker run \-d \--gpus all \--name sglang-kimi-k3 \--net=host \--shm-size=100g \--ulimit memlock=-1 \--ulimit stack=67108864 \--privileged \--ipc=host \--security-opt seccomp=unconfined \--cap-add=ALL \-v /data00/:/data00/ \-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \-e GLOO_SOCKET_IFNAME=eth0 \-e NCCL_MIN_NCHANNELS=24 \-e NCCL_IB_QPS_PER_CONNECTION=8 \-e NCCL_CUMEM_ENABLE=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \-e SGLANG_K3_ATTN_RES_MODE=jit \-e SGLANG_MOE_FUSED_GATE_RADIX=1 \--entrypoint sglang \ai-containers-cn-beijing.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \serve \--trust-remote-code \--model-path /data00/Kimi-K3 \--served-model-name kimi-k3 \--tp-size 32 \--nnodes 4 \--node-rank 3 \--dist-init-addr "<rank0_private_ip>:20000" \--ep-size 32 \--moe-runner-backend marlin \--decode-attention-backend flashmla \--mem-fraction-static 0.85 \--mamba-radix-cache-strategy extra_buffer_lazy \--dist-timeout 3600 \--reasoning-parser kimi_k3 \--tool-call-parser kimi_k3 \--mamba-full-memory-ratio 0.8 \--host 0.0.0.0 \--port 8080 \--speculative-algorithm DSPARK \--speculative-draft-model-path /data00/Kimi-K3-DSpark \--speculative-dspark-block-size 7 \--enable-linear-replayssm-spec \--cuda-graph-max-bs 128 \--max-running-requests 128 \--enable-metrics \--enable-cache-report \--show-time-cost请根据您为实例指定的节点名称，执行对应命令。docker run \-d \--gpus all \--name sglang-kimi-k3 \--net=host \--shm-size=100g \--ulimit memlock=-1 \--ulimit stack=67108864 \--privileged \--ipc=host \--security-opt seccomp=unconfined \--cap-add=ALL \-v /data00/:/data00/ \-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \-e GLOO_SOCKET_IFNAME=eth0 \-e NCCL_MIN_NCHANNELS=24 \-e NCCL_IB_QPS_PER_CONNECTION=8 \-e NCCL_CUMEM_ENABLE=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \-e SGLANG_K3_ATTN_RES_MODE=jit \-e SGLANG_MOE_FUSED_GATE_RADIX=1 \--entrypoint sglang \ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \serve \--trust-remote-code \--model-path /data00/Kimi-K3 \--served-model-name kimi-k3 \--tp-size 32 \--nnodes 4 \--node-rank 0 \--dist-init-addr "<rank0_private_ip>:20000" \--ep-size 32 \--moe-runner-backend marlin \--decode-attention-backend flashmla \--mem-fraction-static 0.85 \--mamba-radix-cache-strategy extra_buffer_lazy \--dist-timeout 3600 \--reasoning-parser kimi_k3 \--tool-call-parser kimi_k3 \--mamba-full-memory-ratio 0.8 \--host 0.0.0.0 \--port 8080 \--speculative-algorithm DSPARK \--speculative-draft-model-path /data00/Kimi-K3-DSpark \--speculative-dspark-block-size 7 \--enable-linear-replayssm-spec \--cuda-graph-max-bs 128 \--max-running-requests 128 \--enable-metrics \--enable-cache-report \--show-time-costdocker run \-d \--gpus all \--name sglang-kimi-k3 \--net=host \--shm-size=100g \--ulimit memlock=-1 \--ulimit stack=67108864 \--privileged \--ipc=host \--security-opt seccomp=unconfined \--cap-add=ALL \-v /data00/:/data00/ \-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \-e GLOO_SOCKET_IFNAME=eth0 \-e NCCL_MIN_NCHANNELS=24 \-e NCCL_IB_QPS_PER_CONNECTION=8 \-e NCCL_CUMEM_ENABLE=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \-e SGLANG_K3_ATTN_RES_MODE=jit \-e SGLANG_MOE_FUSED_GATE_RADIX=1 \--entrypoint sglang \ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \serve \--trust-remote-code \--model-path /data00/Kimi-K3 \--served-model-name kimi-k3 \--tp-size 32 \--nnodes 4 \--node-rank 1 \--dist-init-addr "<rank0_private_ip>:20000" \--ep-size 32 \--moe-runner-backend marlin \--decode-attention-backend flashmla \--mem-fraction-static 0.85 \--mamba-radix-cache-strategy extra_buffer_lazy \--dist-timeout 3600 \--reasoning-parser kimi_k3 \--tool-call-parser kimi_k3 \--mamba-full-memory-ratio 0.8 \--host 0.0.0.0 \--port 8080 \--speculative-algorithm DSPARK \--speculative-draft-model-path /data00/Kimi-K3-DSpark \--speculative-dspark-block-size 7 \--enable-linear-replayssm-spec \--cuda-graph-max-bs 128 \--max-running-requests 128 \--enable-metrics \--enable-cache-report \--show-time-costdocker run \-d \--gpus all \--name sglang-kimi-k3 \--net=host \--shm-size=100g \--ulimit memlock=-1 \--ulimit stack=67108864 \--privileged \--ipc=host \--security-opt seccomp=unconfined \--cap-add=ALL \-v /data00/:/data00/ \-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \-e GLOO_SOCKET_IFNAME=eth0 \-e NCCL_MIN_NCHANNELS=24 \-e NCCL_IB_QPS_PER_CONNECTION=8 \-e NCCL_CUMEM_ENABLE=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \-e SGLANG_K3_ATTN_RES_MODE=jit \-e SGLANG_MOE_FUSED_GATE_RADIX=1 \--entrypoint sglang \ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \serve \--trust-remote-code \--model-path /data00/Kimi-K3 \--served-model-name kimi-k3 \--tp-size 32 \--nnodes 4 \--node-rank 2 \--dist-init-addr "<rank0_private_ip>:20000" \--ep-size 32 \--moe-runner-backend marlin \--decode-attention-backend flashmla \--mem-fraction-static 0.85 \--mamba-radix-cache-strategy extra_buffer_lazy \--dist-timeout 3600 \--reasoning-parser kimi_k3 \--tool-call-parser kimi_k3 \--mamba-full-memory-ratio 0.8 \--host 0.0.0.0 \--port 8080 \--speculative-algorithm DSPARK \--speculative-draft-model-path /data00/Kimi-K3-DSpark \--speculative-dspark-block-size 7 \--enable-linear-replayssm-spec \--cuda-graph-max-bs 128 \--max-running-requests 128 \--enable-metrics \--enable-cache-report \--show-time-costdocker run \-d \--gpus all \--name sglang-kimi-k3 \--net=host \--shm-size=100g \--ulimit memlock=-1 \--ulimit stack=67108864 \--privileged \--ipc=host \--security-opt seccomp=unconfined \--cap-add=ALL \-v /data00/:/data00/ \-v /var/run/nvidia-topologyd/:/var/run/nvidia-topologyd/ \-e GLOO_SOCKET_IFNAME=eth0 \-e NCCL_MIN_NCHANNELS=24 \-e NCCL_IB_QPS_PER_CONNECTION=8 \-e NCCL_CUMEM_ENABLE=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \-e SGLANG_K3_ATTN_RES_MODE=jit \-e SGLANG_MOE_FUSED_GATE_RADIX=1 \--entrypoint sglang \ai-containers-cn-shanghai.cr.volces.com/deeplearning/sglang:v0.5.16.iaas.202607280000-kimi-k3-cu130 \serve \--trust-remote-code \--model-path /data00/Kimi-K3 \--served-model-name kimi-k3 \--tp-size 32 \--nnodes 4 \--node-rank 3 \--dist-init-addr "<rank0_private_ip>:20000" \--ep-size 32 \--moe-runner-backend marlin \--decode-attention-backend flashmla \--mem-fraction-static 0.85 \--mamba-radix-cache-strategy extra_buffer_lazy \--dist-timeout 3600 \--reasoning-parser kimi_k3 \--tool-call-parser kimi_k3 \--mamba-full-memory-ratio 0.8 \--host 0.0.0.0 \--port 8080 \--speculative-algorithm DSPARK \--speculative-draft-model-path /data00/Kimi-K3-DSpark \--speculative-dspark-block-size 7 \--enable-linear-replayssm-spec \--cuda-graph-max-bs 128 \--max-running-requests 128 \--enable-metrics \--enable-cache-report \--show-time-cost
- 参数说明：
- 如上命令以部署、启动Kimi K3模型为例，如需部署、启动其他模型，可根据如下说明替换对应环境变量和参数值。
-
- 执行如下命令，通过Docker日志确认容器及模型是否成功启动。
- 查看容器ID。
- docker ps
- 查看对应容器的运行日志。
- docker logs -f <CONTAINER_ID>
说明
请将<CONTAINER_ID>替换为实际容器ID。
步骤三：调用模型
- 执行如下命令调用模型，确保部署的模型可以正常进行推理。
- curl -X POST http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "kimi-k3","messages": [{"role": "user","content": "请直接回答：1+1等于几？"}],"max_tokens": 2048,"temperature": 0}'
- 参数说明：
- 本文仅说明验证使用到的参数信息。
-
-
最近更新时间：2026.08.18 16:44:11
这个页面对您有帮助吗？
有用
有用
无用
无用