Nodes Are Unable To Open Outbound Connections Via VPNs

[nisdas]

We have recently updated Prysm to v0.22 from go-libp2p v0.20.3 . However over the course of the release a number of Prysm users running via VPNs mentioned that they were unable to find and connect to any peers. This was only fixed by them falling back to an older prysm release so that they could continue connecting to peers.

After confirming the issue with multiple users, I was able to reproduce this with my own local node when running via a VPN. After looking at all the debug libp2p logs, it became clear that none of the outbound dials were able to reach their remote peer. They all failed this way:

failed to dial 16Uiuabcd:  * [/ip4/xx.xx.xx.xx/tcp/13000] dial tcp4 yy.yy.yy.yy:13000->xx.xx.xx.xx:13000: i/o timeout

I tried updating Prysm to later libp2p releases v0.23.x, however the problem still persisted and Prysm was unable to make any outbound connections if the host machine was running via a VPN. Looking through commits to v0.22 , I found a PR that appeared that it might have touched the dialer #1673 . I reverted the PR on top of v0.22 here
nisdas@3807d6d

and ran prysm with this forked version of libp2p. Prysm was immediately able to find peers and connect with them succesfully. I haven't had time to dig into why exactly it broke for nodes running via VPNs but it does appear that the PR broke it for us.

[marten-seemann]

@nisdas Thank you for digging into this!

@Stebalien (as the author of #1673), any idea what's going on here?

[Stebalien]

• Is the user explicitly listening on specific interfaces instead of 0.0.0.0?
• Is yy.yy.yy.yy the VPN's local IP address?
• I assume the user is listening on 13000?
• Is the user listening on any other ports?

[nisdas]

Is the user explicitly listening on specific interfaces instead of 0.0.0.0?

Yes, most users will not listen in on 0.0.0.0 unless they specifically specify it via a flag. They will bind to a specific network interface instead.

Is yy.yy.yy.yy the VPN's local IP address?

Yes, it is the local ip for the node. The interface that they are binded to.

I assume the user is listening on 13000?

This is correct

Is the user listening on any other ports?

They are not, 13000 is the only port that they listen to.

[MarcoPolo]

@nisdas Do you have a repro steps I could try to follow?

[MarcoPolo]

@nisdas could you please try this patch and see if it fixes it for you: #1902 (you can also update your go.mod to require go-netroute v0.2.1)

[MarcoPolo]

Closed because I merged that PR. But please let me know if this doesn't fix it for you @nisdas and we'll reopen.

[nisdas]

Hey @MarcoPolo , thanks for the PR. I will give it a shot and see if it works on my end

[nisdas]

Hey @MarcoPolo , do you know when the next release will be to include this ?

[p-shahi]

@nisdas we're going to release v0.24.0 this week which will include this fix.

[p-shahi]

@nisdas https://github.com/libp2p/go-libp2p/releases/tag/v0.24.0 is out

[nisdas]

Thanks ! This is part of our next planned release, so hopefully we will stop getting user reports on this issue soon.