How to set NCCL_SOCKET_IFNAME

[vainaixr]

I have two computer, when i do ifconfig, then one gives
‘eno1
lo’
2nd computer gives
‘eth0
lo’
What do I specify in nccl_socket_ifname on both nodes, do I specify lo on both?

I receive error like
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled system error, NCCL version 2.4.8
and
connection reset by peer

[sjeaugey]

lo is loopback so you won't be able to use it between two nodes.

Did you try setting NCCL_SOCKET_IFNAME=eno1,eth0 ?

Otherwise you can also write NCCL_SOCKET_IFNAME=eno1 in /etc/nccl.conf on one node and NCCL_SOCKET_IFNAME=eth0 in /etc/nccl.conf on the other node.
If you can't set /etc/nccl.conf, you can also use $HOME/.nccl.conf if not shared between the two nodes.

Finally, if you only have those two interfaces, you should not need to set anything since the default is NCCL_SOCKET_IFNAME=^lo,docker which would pick the first interface which is neither lo nor docker*.

[vainaixr]

it still gives same error, do I need to match nccl version on both computer.

[sjeaugey]

The NCCL version needs to match, but the environment variables might be different (for NCCL_SOCKET_IFNAME at least).

I would start by not setting NCCL_SOCKET_IFNAME at all, but set NCCL_DEBUG=INFO.

You should see lines which look like :

node1:12345:12345 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.1<0>

That will tell you which IP interface NCCL uses on each node. Then you can verify whether those two interfaces can actually communicate with each other (check your firewall, network setup, ...).

[sjeaugey]

Closing old issue. Please re-open if needed.