[Milestone] Waku Network Gen 0

[fryorcraken]

2023 Key Milestones: https://notes.status.im/s/iylE6wdli#

Milestone: https://github.com/waku-org/pm/milestone/1
Priority Tracks: Secure Scalability, Production Readiness, Generating Revenue (Sustainability & Longevity).

Summary

This issue tracks the research and development needed to enable a scalable, shared Waku Network as set out in waku-org/research#3.

Justification

Static sharding is not a fully permissionless solution as someone needs to assign shards to a given application or community.
Moreover, it does not scale well as unused assigned shards cannot easily be recycled and the number of shards that a boostrap fleet can support has a upper limit beyond which peer discovery becomes unreliable.
Finally, opt-in message sign-in is not fully permissionless either and does not provide any capping of bandwidth usage.

For the Waku network to deliver the promises of permissionless participation, censorship-resistance and privacy preservation, it needs:

• To be autoscalable, meaning that no central party has to assign shards. For this we propose an autosharding mechanism that assign messages to a shard depending on their content topic
• To have capped bandwidth usage, to ensure that home connection can fully participate to the Relay network. For this we propose the usage of RLN

Further details in waku-org/research#1.

RAID (Risks, Assumptions, Issues and Dependencies)

• This work builds on existing proven technology (gossipsub sharding, discv5, etc) as well as new technology (RLN, autosharding). While test and simulation will give us confidence, the only way to ensure the success of the network is dogfooding and progressive increase of users and traffic. Meaning that the initial Gen 0 network may not support 1mil users but will be the stepping stone to support such volume.
• This work includes a significant research section, meaning that specific details will need to be worked out as research and development are iterated over.

[fryorcraken]

Weekly Update

• achieved:
  • Critical path work for autosharding done in nwaku, in progress on go-waku
  • Parameters for the Waku Network Gen 0 have been captured in an RFC and use as a basis for simulations and theoretical analysis, removing uncertainty on this milestone around message rates, performance and expected bandwidth usage.
• risks:
  • Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
  • We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
  • js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one joining at end of Oct) so delivery in this client is likely to lag behind other clients.

[fryorcraken]

Note in terms of scope for each client:

What is critical (happy to debate):

• nwaku - relay node
• nwaku - non-relay node, server side
• nwaku - non-relay node , client side (as PoC for implementation reference, does not need to be MVP-mature)
• js-waku - non-relay node, client side, browser
• go-waku - relay node
• go-waku - non-relay node, client side

This is to match the following narrative:

• nwaku as service node and reference implementation
• js-waku as "light client" for the browser
• go-waku for Status, as library that enables both relay and non-relay protocols

With this narrative we can deliver:

• relay network for MVP
• SDKs (JS) to enable builders to build and experiment on Waku
• experimental onboarding of Status Communities on MVP (has caveats)

[fryorcraken]

Weekly Update

• achieved:
  • Further simulation done, with a continued focus on message propagation time and possible improvements.
  • Progress across all client son sharded peer management discovery
  • First PRs merged towards basic distributed store services
• risks:
  • Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
  • We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
  • js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.
  • Uncertainty as to how RLN membership mechanism would hinder application adoption, if memberships need to be distributed or obtained by registration, if staking is necessary to prevent abuse, etc.

[fryorcraken]

Weekly Update

• achieved:
  • Starting internal dogfooding of Waku Network: We have launched the proto-network and it is already possible to run a node in said network.
• risks:
  • Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
  • We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
  • js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.
  • Uncertainty as to how RLN membership mechanism would hinder application adoption, if memberships need to be distributed or obtained by registration, if staking is necessary to prevent abuse, etc.

[fryorcraken]

Weekly Update

• achieved:
  • Started internal dogfooding of proto-network and started to work on documentation.
  • Progressed on capability and sharding discovery.
• risks:
  • Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
  • We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
  • js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.
  • Uncertainty as to how RLN membership mechanism would hinder application adoption, if memberships need to be distributed or obtained by registration, if staking is necessary to prevent abuse, etc.

[fryorcraken]

Weekly Update

• achieved:
  • Internal dogfooding of proto-network continues.
  • Significant progress of autosharding in js-waku: autosharding function implemented, work to integrate in protocols started.
• risks:
  • Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
  • We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
  • js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.
  • Uncertainty as to how RLN membership mechanism would hinder application adoption, if memberships need to be distributed or obtained by registration, if staking is necessary to prevent abuse, etc.

[fryorcraken]

Weekly Update

• achieved:
  • Internal dogfooding of proto-network continues.
  • js-waku work continues.
  • nwaku optimization around peer management are underway.
• risks:
  • Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
  • We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
  • js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.
  • Uncertainty as to how RLN membership mechanism would hinder application adoption, if memberships need to be distributed or obtained by registration, if staking is necessary to prevent abuse, etc. Tracked with #102