Issue with lost packets

[n-lebel]

I’m developing a protocol that requires peers sending data to each other. To achieve that I use libp2p’s tcp::tokio::Transport with the request_response Behaviour. It’s my general understanding that the TCP transport should take care of resending lost packets: however, during unit testing, some peers sometimes get stuck indefinitely waiting for data. After implementing my own resend mechanism based on timeouts, the issue seems to be resolved, but I’m still wondering why TCP isn’t already taking care of that.

Each peer spins up a router which contains a swarm it uses to connect to others, for which the code is below. Peers run within their own tokio::task in unit tests, but obviously on the same machine. That definitely doesn’t fully recapitulate real-world behavior and I guess that's where the main problem lies, but I can't tell what it actually is. Any insight is welcome, thanks!

        let peer_id = PeerId::from(keypair.public());
        let transport = tcp::tokio::Transport::new(tcp::Config::default())
            .upgrade(libp2p::core::upgrade::Version::V1)
            .authenticate(NoiseConfig::new(&keypair).unwrap())
            .multiplex(MplexConfig::new())
            .boxed();

        // Create a Swarm to manage peers and events
        let request_response = request_response::cbor::Behaviour::new(
            [(
                StreamProtocol::new("/my-cbor-protocol"),
                ProtocolSupport::Full,
            )],
            request_response::Config::default(),
        );
        let mut swarm = {
            let mdns = mdns::tokio::Behaviour::new(
                mdns::Config {
                    ttl: Duration::from_secs(network_config.mdns_ttl),
                    query_interval: Duration::from_secs(network_config.mdns_query_interval),
                    enable_ipv6: network_config.mdns_ipv6,
                },
                peer_id,
            )?;
            let behaviour = CustomBehaviour {
                mdns,
                request_response,
            };
            SwarmBuilder::with_tokio_executor(transport, behaviour, peer_id).build()
        };

        // Listen on all interfaces and whatever port the OS assigns
        swarm.listen_on("/ip4/0.0.0.0/tcp/0".parse()?)?;

[mxinden]

Can you post your loop where you poll swarm? In other words, can you post your full code file?

[thomaseizinger]

Does this happen entirely at runtime? Would it perhaps be an option to parameterize the protocol with the two messages types instead? When you say "the protocol-specific logic changes", do you mean source-code changes? i.e. would it be fair to say that you are building yourself a networking library here that is used in multiple projects, but per project the messages stay the same?

Yes it indeed happens at runtime. Essentially a node spins up a single router, and then dynamically adds and removes different types of protocols (which are all conducted between the same peers, hence it making sense to maintain these connections alive in a single router).

What is a router in your domain? Does it contain a Swarm? If you drop a Swarm, the connections will be dropped too.

However if you know of a way to do dynamic multiplexing more idiomatically with libp2p I'd be very interested. I've looked a bit into dynamically modifying the protocol list within the request_response Behaviour, but haven't found a way so far.

Well, entirely dynamic multiplexing is a bit difficult and also not entirely meaningful. For two parties to successfully communicate with each other, they need to agree on what protocol they should speak.

If a node boots up with no knowledge of which protocols it will execute, how is the remote node going to know about these protocols? There must be some common ground here (or out-of-band communication).

Have you considered using pub/sub instead of request-response?

I will definitely look into it, but the nature of the protocols we use require some messages to be only sent between specific peers and not known to the broader network. At any given point each peer knows what they have to send and to whom. There are parts where peers broadcast data to the entire network, but since the other message types require private sending I don't know if pub/sub would yield significant improvements.

It actually entirely depends on how you design your systems. For example, event-based architectures don't have any point-to-point communication even though you could model them as request-response. For example, from a domain perspective, you could say: "When a user orders a product, the shopping cart service needs to tell the warehouse service to ....". But the same problem can also be modeled via events where the individual producers don't know who is going to process their events.

You can apply the same thinking here: Even though individual peers know (from a domain perspective) who they need to send data to, you can also just broadcast it as an event and decouple them from each other. It might work better for your usecase, not sure :)

[n-lebel]

What is a router in your domain? Does it contain a Swarm? If you drop a Swarm, the connections will be dropped too.

Yes it contains the Swarm, that's why it's not supposed to be dropped. Its function is to route messages to their respective protocols through the ad-hoc ProtocolId system we use.

Well, entirely dynamic multiplexing is a bit difficult and also not entirely meaningful. For two parties to successfully communicate with each other, they need to agree on what protocol they should speak.

There's an assumption that nodes can reach consensus on starting protocols outside of this code. They're able to agree upon a ProtocolId and a behaviour (which is always request_response). So even though they don't know that when the Swarm is created, they can all agree dynamically on adding a protocol. My question was mostly whether it's possible to move this dynamic ad-hoc ProtocolId logic directly into the Swarm (perhaps by mutating the protocols list in request_response).

You can apply the same thinking here: Even though individual peers know (from a domain perspective) who they need to send data to, you can also just broadcast it as an event and decouple them from each other. It might work better for your usecase, not sure :)

Interesting, thanks for clarifying, I'll definitely play around with it. I think it would be a great solution to keep in mind if it turns out we have too many peers for full connectivity.

[thomaseizinger]

Well, entirely dynamic multiplexing is a bit difficult and also not entirely meaningful. For two parties to successfully communicate with each other, they need to agree on what protocol they should speak.

There's an assumption that nodes can reach consensus on starting protocols outside of this code. They're able to agree upon a ProtocolId and a behaviour (which is always request_response). So even though they don't know that when the Swarm is created, they can all agree dynamically on adding a protocol. My question was mostly whether it's possible to move this dynamic ad-hoc ProtocolId logic directly into the Swarm (perhaps by mutating the protocols list in request_response).

Technically yes, it could be expressed as a behaviour. I've actually built stuff like this in the past but it turns out to be somewhat broken. The problem is that dynamically registering protocol handlers creates somewhat weird timing constraints for a system. You need to ensure that a node has its protocol handler registered before another one starts talking to it for example.

Typically, you are much better served by designing your system such that you can register your protocol / message handlers once on startup and then just deal with incoming messages as they arrive.

Can you give me a more concrete example of what you are trying to build? This all still sounds very abstract to me which makes it difficult to think of a different way of modelling this.

[n-lebel]

That makes sense. The idea was that since nodes are able to agree beforehand on starting a protocol (and therefore wait until everyone has registered their respective handlers), these types of issues shouldn't arise. I tended to favor this approach because I have a pool of nodes from which specific subsets are chosen to run multiparty computations on demand. It seemed more appropriate to have a structure that just handles the connections and the raw sending/receiving (which I refer to as a Router, since it routes incoming messages to the appropriate MPC protocol), because it made little sense to me to have to reconnect to all the peers every time you wanted to run a protocol.

The other reason for having separate structs is that there are different kinds of protocols that all require network communication (provided by one single Router) as well as protocol-specific higher-level functions (provided by NetworkInterface structs that are specific to each protocol type). Also more practically, there were generic types in my construct which weren't Serialize + Deserialize (but rather arkworks' own CanonicalSerialize + CanonicalDeserialize traits), which kept running into issues with libp2p.

So the logical conclusion for me was to decouple the generic network functionalities (connecting to peers and sending/receiving bytes, essentially) which can be instantiated once on startup, from MPC protocols which need to be instantiated dynamically. Hope that makes sense, happy to walk you through the code more in detail if it doesn't. I've included a short diagram, might make it easier to visualize.

[thomaseizinger]

The recently released stream protocol might be of interest to you: https://github.com/libp2p/rust-libp2p/tree/master/protocols/stream