- Documentation
Volcengine Kubernetes Engine
Volcengine Kubernetes Engine
FAQs
Node and Node pool FAQs
How do I handle RDMA network connectivity issues for nodes with multiple A100/A800 GPUs?
Node and Node pool FAQs
How do I handle RDMA network connectivity issues for nodes with multiple A100/A800 GPUs?
How do I handle RDMA network connectivity issues for nodes with multiple A100/A800 GPUs?
Issue description
For nodes that use multiple A100/A800 GPUs, the GPUs fail to connect to each other.
Cause analysis
The NVIDIA-Fabric Manager service ensures that multiple GPUs that support NVSwitch (such as A100 and A800) are interconnected through NVSwitch. This guarantees normal network connectivity. If the NVIDIA-Fabric Manager service is not enabled on the node, the connection among multiple GPUs that support NVSwitch cannot be established.
Solution
In scenarios such as creating a node pool or adding new nodes, the NVIDIA-Fabric Manager package can be built into a custom image. In scenarios where nodes already exist, you can directly install NVIDIA-Fabric Manager to resolve this issue.
Solution 1 for scenarios where new nodes are added
- Select the packaged custom image to create a node pool or scale out nodes.
Solution 2 for scenarios where nodes already exist
Last updated: 2026.07.23 14:31:45