How to implement peer discovery using Kademlia (rust-libp2p)

[gdnathan]

I saw bunch of "answers" across the internet, but couldn't find any reliable example.

Here is the step by step process I would like to implement:

Start node A

Start node B by bootstraping Kademlia with node A

node A and B start exchanging messages, or heartbeats (trough gossipsub for example)

I got two issues with this:

-> The first is that despite connecting B to A (A emit RoutingUpdated, A doesn't seems to discover B (it doesn't emit an event). So through gossipsub, B send heartbeat to A, but not the other way around. When facing this issue, both nodes are running on the same local machine (but I use public addresses).

-> The second issue arise when I try to connect two nodes from different machine, either on the same network or not. When doing so, I cannot manage to connect two nodes together. The address A is exposing is responding to the pingcommand, but the telnetcommand timeout. I took down all my firewalls, VPN and everything that could block the request, but nothing changed. Also, as both node were working on WSL, I looked into windows ip configuration and tried every IP I've found, but couldn't get past this issue.

Once both issues are solved, I will make a small repo that expose a reliable example, because that's what I couldn't find online, and I think it would benefit the community.

Here is my behavior définition:
https://pastebin.com/QY9wrp45

My swarm (pretty basic so I paste it here):

let tcp_transport = tcp::async_io::Transport::new(tcp::Config::default().nodelay(true))
.upgrade(upgrade::Version::V1Lazy)
.authenticate(noise::Config::new(&id_keys).expect("signing libp2p-noise static keypair"))
.multiplex(yamux::Config::default())
.timeout(std::time::Duration::from_secs(20))
.boxed();

let behaviour = MyBehaviour::new(local_peer_id.clone(), id_keys)?;

let mut swarm = {
SwarmBuilder::with_async_std_executor(tcp_transport, behaviour, local_peer_id).build()
};
swarm.listen_on("/ip4/0.0.0.0/tcp/0".parse()?)?;

and the swarm execution:
https://pastebin.com/9muEb89L

[thomaseizinger]

The first is that despite connecting B to A (A emit RoutingUpdated, A doesn't seems to discover B (it doesn't emit an event).

We only update the routing table for outbound connections because we cannot be sure that the observed address of another node is actually dialable (e.g. behind NAT). This is why A doesn't discover B in your topology.

The second issue arise when I try to connect two nodes from different machine, either on the same network or not. When doing so, I cannot manage to connect two nodes together. The address A is exposing is responding to the pingcommand, but the telnetcommand timeout. I took down all my firewalls, VPN and everything that could block the request, but nothing changed. Also, as both node were working on WSL, I looked into windows ip configuration and tried every IP I've found, but couldn't get past this issue.

Are the nodes behind NAT? If so, did you forward the ports in your router?

[gdnathan]

I didn't explicitly set NAT, but maybe WSL uses NAT internally. I will try to dig into this and implement port forwarding for wsl and will update this issue.
I tried to connect a node B from WSL to a node A from raw linux, but had the same issue.
But I recently synchronized Bitcoin on WSL and had no issues connecting to other peers and sync, so I don't know🤔
Will investigate!

[thomaseizinger]

But I recently synchronized Bitcoin on WSL and had no issues connecting to other peers and sync, so I don't know🤔

Bitcoin Core will connect to other peers that have open ports and then exchange messages in both directions so you can be behind NAT and still sync it :)

[thomaseizinger]

The same applies to libp2p btw. If one node is behind NAT, it can still make kademlia queries to another node that is reachable on the network.

[gdnathan]

Ok so I tried to implement port forwarding for both my router and my Windows instance.
After a failed attempt, I decided to make it simple by running two nodes on AWS instance.
But still, I couldn't get anywhere..

[thomaseizinger]

Like I said earlier, we will only add a node the routing table that we dialed so in a 2-node setup, the dialed node will never add the other one to the routing table. But there is also no need because you already have a connection!

If the other node advertises a correct external address in identify, you can add it explicitly to the routing table if you want, see add_address.

To do peer discovery using kademlia, you'd call bootstrap in an interval. That will perform a random walk on the DHT and keep your buckets in a healthy state.

That assumes you actually have a network though (each bucket fits max 20 peers).

To discover a peer using kademlia, some node must have dialed it before and put its address into a bucket. Then it get can be returned later by a query to some other node.

Does that make sense?

[gdnathan]

Ok, so I'll try to make some assumptions to see if I got everything you told me correctly.
In my 2 nodes setup, if I bootstrap A with B, A will know about B, but B will not know about A until it dialed it directly. And once A is bootstrapped it will not dial B for no reason, that's why I cannot see any logs on B when A bootstrap, but if A sends a message through gossipsub, B will receive it as expected.
So before I try to use gossipsub to send messages, I need to make sure that all my nodes are connected to at least one peer that knows about another one (so none of them are left alone).

A initial strategy in my head would be to do this:
Bootstrap A with B
A sends a messages that B receive (gossipsub or direct)
use add_address on B with A's address

With this, I believe my bootstrap nodes will know about everyone on the network, and new nodes will only know about bootstrap nodes, but with time and calls to bootstrap, known peers will be more diverse as bootstrapped nodes knows about more peers. Am I right ?

Also, what is the role of Identify in all of these ?

Sorry about all these questions, I'm new to libp2p and have to learn by myself, and it's somewhat pretty complexe with all these topics and interactions 😅

[thomaseizinger]

Sorry about all these questions, I'm new to libp2p and have to learn by myself, and it's somewhat pretty complexe with all these topics and interactions 😅

No worries at all, happy to help!

A couple of things:

1. bootstrap is a "refresh my routing table"-kind of operation. You should actually call it regularly. Also see feat(kad): add config option for bootstrap interval #4730.
2. Try not to think in "order of operations" too much. In a peer-to-peer network, you should be prepared for connections to come and go at any time. Thus, the safer way is to code against events and state machines, rather than which operations you've called already.

Also, what is the role of Identify in all of these ?

3. identify fulfills a couple of roles.

• It has a STUN-like functionality where nodes can figure out what their observed, public address is. This is useful for hole-punching and can be a viable external address if you have port-forwarding configured in your router (or you are not behind NAT in the first place). Whether or not the observed address is actually an external address (and thus dialable by other nodes) depends on many factors. Likely, if multiple different nodes report you the same address, it means that at least you have a stable port mapping (i.e. TCP port-reuse is configured). We recently merged a PR that makes detecting this fairly easy: fix(identify): only report observed address once per connection #4721. It will be part of the next release and makes it trivial to implement a "probabilistic external address" detector where N reports of the same address are considered an external one.
• It also tells you which protocols the other nodes support. This is used by protocols like kademlia to automatically add nodes to the routing table that are kademlia "servers".

Overall, I'd take the following approach:

• Upon start-up of your node, connect to one or more "seed" nodes, also sometimes called bootstrap nodes.
• You should already have an eventloop where you call Swarm::next. As part of that, call kad::Behaviour::bootstrap in a regular interval (Something like 5 minutes, depending on how much churn you have in your network. If your network is very stable, you need to call it less often to have a healthy routing table).
• If your network supports gossipsub, subscribe to the relevant topics.
• Once this eventloop is running, your node is basically "ready" to take commands from the user. What this is depends a lot on your network. In the case of IPFS, it could be something like "get me this particular file". In the case of a blockchain node, it could be "broadcast this transaction to the network".
• Overall though, your node is now running, receiving and forwarding gossipsub messages, serving provider records and discovering new peers through the DHT.

What is important to consider is the role of these individual protocols as part of such an operation.

• Kademlia is a routing protocol. It allows you to figure out where a certain node is (i.e. find its address to connect to it). Using "provider records", you can also find out who has a certain piece of data.
• Gossipsub is a notification protocol. It allows a network to quickly broadcast what the latest state is.

Both of these are complimented by protocols that you perform with a specific peer. Like, once you've learned about the new state, you need to get it from someone. Or once you've learned who has a particular piece of data, connect to them and get it. For this case, request-response is probably a good fit.

Hope this helps!

[jakerumbles]

Thomas, thank you so much for all of this explanation! Very helpful.

I think part of my struggle with understanding Kademlia is that most documentation is focused on being able to find which node has a specific piece of data, which hasn't been very relevant to us in building a blockchain as the idea is that all nodes have the same data more or less. All we needed was a way to store records of other nodes in the network for a decentralized p2p network and then be able to broadcast data with gossipsub, so I guess I can say that we're only using part of Kademlia's intended use.

[thomaseizinger]

In that case, just don't use provider records :)

You can still use random walks to discover more peers of your network and kademlia will serve you just fine with that!

be able to broadcast data with gossipsub

Note that you'll likely have more success if you broadcast notifications about new data via gossipsub, for example, that somebody has mined a new block. Gossiping the actual block will lead to a lot of traffic because it is quite common that a node receives a message multiple times. Once you have the identifier for a block, you can download it from that node via a request-response protocol or something like bitswap.

Bitcoin for example does that too: inv advertises the knowledge of certain items which is very cheap and getdata is then used to actually transfer the data in case the node is interested.