What is the SwarmBuilder equivalent to applying an upgrade to transport?

[MitchTurner]

Previously we would manually construct the Transport and hand it to our swarm. As part of constructing the Transport, we would call the apply method to add specific "upgrades" to the transport. Like this:

transport
    .authenticate(my_authentication)
    .apply(my_upgrade)
    ...
    .boxed()

But now that we are constructing our Transport with the SwarmBuilder, I don't know how to include that my_upgrade in the transport our Swarm is using once built.

How should I include such an upgrade now?

[thomaseizinger]

At the moment, this is not supported in SwarmBuilder as we considered this too much of a niche usecase to support it in its first version.

Can you explain what your custom upgrade does? With the advent of natively encrypted and multiplexed transports like QUIC, WebRTC and WebTransport, applying a custom upgrade between authentication but prior to the muxer is not possible. Thus, I think we will have to take a step back and look at what the overall feature is that you'd like to solve with this custom upgrade and see, if we can fit that into the model of the SwarmBuilder such that it works with any transport.

[MitchTurner]

Okay.

This is for two blockchain nodes that are connecting. Each upgrade holds a "checksum" of the their chain config + chain id. If the two nodes don't have matching checksums, the connection will fail. This is to prevent nodes with matching network names, but different configurations from communicating.

Here is a link to the old code.

I imagine that this could be implemented at the behavior level as well, but it seems like the developer who designed this decided to do it at a lower level. This admittedly does seem simpler than adding conditional behavior logic to a behavior.

[thomaseizinger]

I think this makes sense. Bitcoin does something similar by prefixing each message with the a magic constant that identifies the network it is running on. I imagine the design was inspired by that.

It is a bit tricky to support because transports like QUIC don't have any "multistream-select" upgrade but only use it on top of new streams that are established. One way of supporting this could be if you embed this information in your protocol names.

For example, instead of having /my-custom-protocol, you'd have /mainnet-e4a5/my-custom-protocol, assuming /my-custom-protocol is a "network-sensitive" protocol. For stuff like ping, identify or any "libp2p" protocol really, it wouldn't strictly be required to do a hard-split.

I'll think a bit more about how else we could solve this as it is a big shift. Note that your current solution only works for TCP connections and even if you don't use the new SwarmBuilder, you will not be able to adopt QUIC, WebRTC or WebTransport.

[MitchTurner]

We are also doing a similar thing but with .authenticate. Also curious if there is no longer support for that with SwarmBuilder.

[thomaseizinger]

I want to stress that it lives alongside it. In other words, a custom authentication will only apply to this particular other_transport.

Meaning any message is sent twice? So if we are hoping to use a custom authentication, the first transport would be bypassing it essentially?

Transports are for a particular technology, like QUIC, TCP with noise+yamux upgrade, WebRTC etc. Which one is picked depends on what the remote uses to connect to you. A custom authentication upgrade only makes sense with TCP because it is not authenticated by default. QUIC has built-in authentication that you can only configure by providing custom TLS certificates and a few other things. Libp2p doesn't expose this though.

What I am saying is: Your design of using a custom authentication to do peer management only works with TCP, not any of the other transport technologies as those do not use composition to achieve an authenticated, multiplexed transport.

A message cannot be "sent twice" because a particular connection is always of a concrete type: TCP, QUIC but not "both". What you are doing with the builder is just composing which transports are possible but an actual connection will always just be one of those.

[thomaseizinger]

QUIC is the superior transport in many ways: Better throughput, lower latency, less RTT for establishing connections, better flow-control, ...
I'd strongly recommend to consider it in the future and thus think about a design that is forwards-compatible with moving your stack to QUIC.

[MitchTurner]

What you are doing with the builder is just composing which transports are possible but an actual connection will always just be one of those.

Right, but under what conditions will it choose the stricter one? The "other_transport" will be the transport we were using before and it has the authentication and the upgrade mentioned at top ^^^. If one is a basic TCP and the other is my fancy TCP, why would it ever use the fancier one?

Can I use the SwarmBuilder and only use with_other_transport and not specify a with_tcp at all? It seems to build, but my tests are still failing so I can't tell if it's janky. I have some tests that pass if I include both, but fail if I only include the with_other_transport.

[thomaseizinger]

If one is a basic TCP and the other is my fancy TCP

Don't do that! :)

You should only ever have one transport that can handle a certain multiaddr. The transport you configure "after" will never be used.

Can I use the SwarmBuilder and only use with_other_transport and not specify a with_tcp at all? It seems to build, but my tests are still failing so I can't tell if it's janky. I have some tests that pass if I include both, but fail if I only include the with_other_transport.

Yes that should work. with_tcp uses the same APIs under the hood! :)