Dat: A message-based system on top of raw streams

[aschrijver]

Current design concepts

(NOTE This showcase is part 2 of Positioning, vision and future direction of the Dat Project)

If I it understand it correctly, simply put, with Dat you now have:

    stream exchange --> file exchange --> dat exchange

All by using modules from the ecosystem. AFAIK you currently have a modular decomposition, but you do not yet have a full modular design. or call it framework, if you prefer.

This because transitive dependencies require me to untangle, recompose and add my own glue logic, in order to realign modules in such way that it supports my new cool use case.
But the modularization is not what I wanted to talk about here. That's for a future topic maybe.

Refactoring proposal

Instead consider this: Wouldn't it be better if instead Dat was modelled as follows?

    stream exchange --> message exchange

So as a

decentralized message-based system on top of raw streams

In this design:

• hypercore-protocol defines wire format (nothing changes)
• dat-protocol defines file archive exchange (nothing changes)
• dat-messaging defines allowed messaging formats, data types, core message types, etc.

I think this constitutes a change that requires only little effort, yet comes with tremendous benefit:

• First of all, technology-wise you'll have immediately repositioned yourself according to the 'Broadening vision' objective

• Message-based systems are incredibly modular, decoupled by nature. Actors are only triggered by messages they understand and that are addressed to them

• New applications need only to plug into the - here decentralized - message bus and everything just works. Outdated apps can run in tandem and in peace with their newer versions. Applications can appear, disappear without disruption, etc.

• Want to have Event Sourced application? CQRS? DDD? File exchange? Live video editing? IRC? What have you? Sure, no problem. Define the right message types, pub/sub handling and message processing (with much help from Dat ecosystem)

• You can define typed data models with relationships that correspond to your messaging design and use e.g. GraphQL to help minimize wire traffic and payload sizes

• You would be clearly positioned with regards to IPFS. You would no longer be more or less competing standards. You serve different purposes. And Dat would have the unique selling points now. While any IPFS feature could also be modelled with Dat, the reverse is not true. You are now the more general-purpose technology.

Overhead and effort

The overhead of wrapping data payloads into messages is only very small, depending on the message format you choose. So would code changes to existing modules, I imagine (but I'm not the one to say)

[UPDATE: Both @mafintosh and @joehand have acknowledged the usefulness of this proposal. The next part deals with the actual design of the message layer]

Next part: Design of a message-based abstraction layer on top of hypercore

[benrogmans]

Awesome idea @aschrijver

[aschrijver]

thanks @benrogmans

unfortunately while liked by dat team members it is not high on the priority list for dat, i think :(

[aschrijver]

maybe i'll be looking into vert.x as core tech for this idea..