ibvwrap.c:106 NCCL WARN Call to ibv_reg_mr failed
proxy.cc:1519 NCCL WARN [Proxy Service 5] Failed to execute operation Connect from rank 5, retcode 2
NCCL WARN socketProgress: Connection closed by remote peer master-0.svc.cluster.local<36167>

Environment configuration

Node model: ecs.hpcpni3ln.45xlarge

NCCL version: 2.18.5

Cause analysis

It might be caused by "the default size set by the command ulimit -l is 64K, which is insufficient for memlock memory in benchmark scenarios". In that case, please refer to the following solution.

Solution

Fix it on new nodes

Fix it on existing nodes

Go to the page "Edit node pool" and select "Advanced settings", then copy the following script to the textbox "InitializeScript" to overwrite all the existing content. Save the changes to take effect.

Warning

The value of the parameter LimitNOFILE in the script needs to be consistent with that of the same parameter in /lib/systemd/system/containerd.service.

sed -i '/LimitNOFILE=10xxx76/a LimitMEMLOCK=infinity\nLimitSTACK=infinity' /lib/systemd/system/containerd.service && systemctl daemon-reload && systemctl restart containerd​

Log in to a node and execute the command vi /lib/systemd/system/containerd.service, then add the configuration LimitMEMLOCK=infinity.

Execute the command sudo systemctl daemon-reload && sudo systemctl restart containerd to reload configuration and restart containerd service.

Last updated: 2026.07.23 14:31:47

Public security network record in Beijing: No. 11010802032137