IPFS overlay network

[Stebalien]

Now that we're about to get p2p-circuit support, it would be nice to get an IPFS overlay network. Instead of connecting to all peers directly, nodes (even those not behind NATs) could connect to a subset of their peers through other IPFS nodes. This would:

1. Reduce latencies in some cases as ISPs often pick sub-optimal routes due to peering agreements. That is, the following is sometimes true: latency(a, b) + latency(b, c) + ẟ < latency(a, c).
2. Possibly increase throughput in some cases due to smarter congestion control.
3. Help avoid killing routers by reducing the number of open connections.

[Stebalien]

cc @diasdavid

[ghost]

Agreed 👍 routed overlay networks are a direction definitely worth exploring for libp2p.

The overarching project is ipfs/notes#143 (packet switching & source-routed overlay), but it's useful to start experimenting already on top of the circuit relay. The open research problem here is how to do efficient, resilient, low-latency, high-churn source routing on a cryptographically defined (= flat) keyspace.

A lot of the ideas and notes related to this are unforunately only in my head. Please feel free to pick my brain whenever, I'll also try to dump the most important things into issues. Things that are already pretty good but not yet spec'ed or implemented:

• multigram, a datagram/packet version of multistream and multistream-select
• the packet switch which moves packets based on routing info attached to them

Let's move discussion either to libp2p/libp2p, or ipfs/notes#143

[ghost]

One more thing that looks pretty good but is waiting for someone to implement: Wireguard came up with a very strong datagram-based crypto handshake using the Noise framework, and just recently published a formal proof for it. Apart from the reference kernel-space implementation, there's various userland impls, one of them in go: https://git.zx2c4.com/wireguard-go/

It would be super nice to have this in go-libp2p, and later in js-libp2p too.

[Stebalien]

As a start, I think routing some traffic over a few supernodes makes sense. We could even migrate connections between proxied and non-proxied states on a per-connection basis based on how much traffic we're routing over them. This would, effectively, allow us to keep connections open to many peers that we talk to occasionally without having to keep "real" connections to them.

A full on packet-switching network would be great but a lot more work.

[whyrusleeping]

Would be really cool to have a mode of running ipfs that runs all connections over a given set of relay nodes. I think this is something we should push for sooner rather than later considering the mobile work that open bazaar is doing.

cc @cpacia @drwasho

[drwasho]

Yes please 😸

[yangm97]

I think you're going to/already did move this issue to libp2p, but I'm going to comment anyway: You could use Yggdrasil protocol (and code since it's in go) to achieve this task, as to avoid duplicated effort.
https://github.com/yggdrasil-network/yggdrasil-go

[Mikaela]

There were no replies to the previous comment, but Yggdrasil has released their first version with API support.

Refactoring and API

Around the previous release, it became obvious to us that our codebase was turning into a monolith. We had pretty much all of the necessary behaviour in a single yggdrasil package to run a single node, but this made our codebase inflexible and difficult to maintain and extend. It also meant that Yggdrasil was virtually impossible to integrate into other applications.

Our refactoring efforts in version 0.3.6 mean that our codebase is now easier to manage and to understand. It also includes the first taste of our API! The API makes it possible to take the Yggdrasil core, drop it into your own Go application and use the Yggdrasil network as a fully end-to-end encrypted and distributed transport layer. We’ve also moved all of the IP-specific code into the TUN/TAP module, which means that Yggdrasil’s core now provides a completely protocol-agnostic transport.

Documentation on how to use the API to integrate Yggdrasil into your own applications will follow soon—watch this space! In the meantime, godoc can be used to examine our new API functions.

Please note though that API functions are not yet finalised and may be subject to change in future versions. Yggdrasil is still alpha-grade software at this point so all of the usual warnings apply.

• https://yggdrasil-network.github.io/2019/08/03/release-v0-3-6.html