Bootstrap node can't add the connected node into peers using examples/distributed-key-value-store in public network

[ShuochengWang]

I use two Azure servers in public network environments,

• one for bootstrap node, called node A
• one for new-joined node, called node B

The bootstrap node runs examples/distributed-key-value-store example, listening on 3000 port
And the new-joined node runs a modified examples/distributed-key-value-store example, add the address of the bootstrap node, and listening on 3000 port too.

    ......
    swarm.behaviour_mut().kademlia.set_mode(Some(Mode::Server));


    let bootstrap_peer: libp2p::Multiaddr =
        "/ip4/here is my bootstrap_node ip/tcp/3000".parse().unwrap();
    println!("Bootstrap Peer: {:?}", bootstrap_peer);

    use std::str::FromStr;
    // here is my peer_id of my bootstrap node
    let bootstrap_peer_id = libp2p::PeerId::from_str("12D3KooWT1BYJpwNrQpAtF5kYw439PJmabBx8QnhVdUqCTGnhy7p").unwrap();
    println!("PeerId: {:?}", bootstrap_peer_id);

    swarm
        .behaviour_mut()
        .kademlia
        .add_address(&bootstrap_peer_id, bootstrap_peer);


    // Read full lines from stdin
    let mut stdin = io::BufReader::new(io::stdin()).lines().fuse();

    // Listen on all interfaces and whatever port the OS assigns.
    swarm.listen_on("/ip4/0.0.0.0/tcp/3000".parse()?)?;
    ......

And I found that, the new-joined node B can add the bootstrap node A into peers, But the bootstrap node A can't add the new-joined node B into peers. Why? How to fix it?

bootstrap node output

Listening in "/ip4/127.0.0.1/tcp/3000"
Listening in "/ip4/10.0.0.15/tcp/3000"

GET key
Failed to get record: NotFound { key: Key(b"key"), closest_peers: [] }

GET key
Got record "key" "value"

PUT key2 value2
Failed to put record: QuorumFailed { key: Key(b"key2"), success: [], quorum: 1 }

new-joined node output

Listening in "/ip4/127.0.0.1/tcp/3000"
Listening in "/ip4/10.0.0.16/tcp/3000"

GET key
Failed to get record: NotFound { key: Key(b"key"), closest_peers: [PeerId("12D3KooWT1BYJpwNrQpAtF5kYw439PJmabBx8QnhVdUqCTGnhy7p")] }

PUT key value
Successfully put record "key"

GET key2
Got record "key2" "value2"

And if I print all events, I found BehaviourEvent: UnroutablePeer event

peer_id: 12D3KooWMiHQnCFsmoCeeGvYdJ3pmBxq9jdTPqtE8Hwsk4XxBF3y
2024-06-06T10:00:17.612692Z  INFO libp2p_swarm: local_peer_id=12D3KooWMiHQnCFsmoCeeGvYdJ3pmBxq9jdTPqtE8Hwsk4XxBF3y
2024-06-06T10:00:17.614029Z  INFO libp2p_mdns::behaviour::iface: creating instance on iface address address=10.0.0.15
Listening in "/ip4/127.0.0.1/tcp/3000"
Listening in "/ip4/10.0.0.15/tcp/3000"
2024-06-06T10:00:50.027521Z  INFO distributed_key_value_store_example: event: IncomingConnection { connection_id: ConnectionId(1), local_addr: "/ip4/10.0.0.15/tcp/3000", send_back_addr: "/ip4/here is my new-joined node ip/tcp/44470" }
2024-06-06T10:00:50.074444Z  INFO distributed_key_value_store_example: event: ConnectionEstablished { peer_id: PeerId("12D3KooWGqcDokRpdUJgFj8sqysVm6V3UY8KjFBpZWDX6Knz9bMx"), connection_id: ConnectionId(1), endpoint: Listener { local_addr: "/ip4/10.0.0.15/tcp/3000", send_back_addr: "/ip4/here is my new-joined node ip/tcp/44470" }, num_established: 1, concurrent_dial_errors: None, established_in: 46.779269ms }
2024-06-06T10:00:50.074808Z  INFO distributed_key_value_store_example: event: Behaviour(BehaviourEvent: UnroutablePeer { peer: PeerId("12D3KooWGqcDokRpdUJgFj8sqysVm6V3UY8KjFBpZWDX6Knz9bMx") })

2024-06-06T10:00:50.156606Z  INFO distributed_key_value_store_example: event: Behaviour(BehaviourEvent: InboundRequest { request: FindNode { num_closer_peers: 0 } })
2024-06-06T10:00:50.200464Z  INFO distributed_key_value_store_example: event: Behaviour(BehaviourEvent: InboundRequest { request: FindNode { num_closer_peers: 0 } })
2024-06-06T10:00:50.245503Z  INFO distributed_key_value_store_example: event: Behaviour(BehaviourEvent: InboundRequest { request: FindNode { num_closer_peers: 0 } })
2024-06-06T10:00:50.289643Z  INFO distributed_key_value_store_example: event: Behaviour(BehaviourEvent: InboundRequest { request: FindNode { num_closer_peers: 0 } })

[ShuochengWang]

And if I use server B as bootstrap node, server A as new-joined node, the problem is the same
(I have opened the corresponding ports 3000 in my server setting)

[guillaumemichel]

It may be linked to #5349

[dariusc93]

You might want to try adding an external address with Swarm::add_external_address so the nodes would be set into server mode automatically (or maybe change the server mode), assuming that is the actual issue you are having. It might be related to what @guillaumemichel mentioned as well. Any additional logs would be helpful

[Fabijoey]

It may be linked to #5349

I do not think that it is linked to the issue above. I was able to replicate it with my the newest merge.
I think that is the same problem as mentioned here #2673 .

[ShuochengWang]

I solved this problem by using both Swarm::add_external_address (add the real ip and port according to the config file) and identify protocal, and update the event loop

                        SwarmEvent::NewExternalAddrCandidate { address } => {
                            swarm.add_external_address(address);
                        }
                        SwarmEvent::NewExternalAddrOfPeer { peer_id, address } => {
                            swarm.behaviour_mut().kademlia.add_address(&peer_id, address);
                        }
                        SwarmEvent::ConnectionEstablished { peer_id, endpoint, .. } => {
                            if let libp2p::core::ConnectedPoint::Dialer { address, .. } = endpoint {
                                tracing::info!("Add address {:?}", address);
                                swarm.behaviour_mut().kademlia.add_address(&peer_id, address);
                            }
                        }
                        SwarmEvent::Behaviour(BehaviourEvent::Identify(identify::Event::Received {
                            peer_id,
                            info: identify::Info { listen_addrs, observed_addr, protocols, .. },
                        })) => {
                            if protocols.iter().any(|p| *p == kad::PROTOCOL_NAME) {
                                for addr in listen_addrs {
                                    swarm
                                        .behaviour_mut()
                                        .kademlia
                                        .add_address(&peer_id, addr);
                                }
                            }
                        }