First-class dat-protocol extensions support

[RangerMauve]

Relating to the working group meeting last month, we should discuss first-class support of extensions for the replication streams in the Hyperdrive and Hypercore APIs in Beaker.

I've surfaced them in hypercore, and there's a PR in the works for hyperdrive. The reasoning is that extensions are very important to data structures building on top of dat for applications like cabal.

I'm hoping to include these new APIs in the dat SDK so that it'll be easier to run Cabal over it, and then make it easier to run cabal cross-platform.

Thoughts @pfrazee?

cc @noffle @cblgh

[pfrazee]

The challenge is that currently hypercore-protocol extensions have to be set by both peers prior to any connection handshake. Beaker doesn't give applications any control of the networking lifecycle of a dat, so we'll need some way to set them up prior to networking.

Might be worth discussing a change to how hypercore-protocol sets up extensions.

[RangerMauve]

I think one avenue that was mentioned was putting the protocol extensions in the hypercore header.

I proposed adding a new field called extensions which would contain a list of extensions to use with this data structure. I'm not sure how the header is going to be used yet, so we might need to wait for that to land first.

One idea I had was to save any extensions used by the client into a set that'd update behind the scenes. If it's the first time an application is loading a hypercore/drive then it's also likely that it'll specify the extensions it wants to use. Else it'll add them on subsequent connections. Especially when it comes to applications like Cabal where you're only really going to use the hypercores from within a Cabal application and therefore will know the extensions right when the constructor is called.

[RangerMauve]

cc @mafintosh @andrewosh

[danimesq]

Other use-case is hypervision live streamming site