A Streamlined e2e Genesis process

[longbowlu]

This is an umbrella task that covers all issues to be addressed for genesis until mainnet (and beyond).

Known issues/ potential improvement:

1. [config] Split GenesisConfiguration and NetworkConfig to contain public info only #1478 : we now remove private key (aka KeyPair) from AuthorityPrivateInfo but it's still in GenesisConfig and NetworkConfig so there's more work to do.
   • The end goal is to make genesis_config a file that can be distributed to all validators, hence no private information (e.g. key_pair or db_path) shall be included.
2. With devnet approaching, we start to spin up "real" networks that involves running validators in a distributed manner. We still reuse the genesis function which is implemented for running all validators locally. As a result, some bespoke behaviors in each setup leak out. For example, [genesis function](

   sui/sui/src/bin/validator.rs

   Line 64 in 6a4a976

   let (network_config, _, _) = genesis(genesis_conf, Some(adddress)).await?;

   ) returns a triplet with key_store and accounts. They shouldn't exist in the case of starting a real distributed network.
   • The end goal is to separate the genesis process for each setup
3. We have some tricks to pre generate a large amount of objects in benchmark, ideally these logic should not blend into prod.
   • The end goal is to separate the genesis process for each setup (same as above)
4. Design & implement testnet/mainnet genesis ceremony

[lxfind]

Relevant issues: #1297

[bmwill]

Here are some thoughts on what the genesis process may look like:

1. We identify the Validators that will comprise the initial validator-set
2. We identify the initial set of objects that we want to be apart of genesis (move stdlib, initial accounts with sui, validator-set, etc)
3. We'll need to have some "ceremony" by which each validator signs the initial set of objects, as a genesis state or initial checkpoint.
4. With signatures from all validators (3f+1) we can form the initial checkpoint which will need to be use to seed each validator's storage as well as the seed for any fullnodes that want to tail the network.

[longbowlu]

thanks @bmwill ! It makes sense. Adding more color:

1. In the ceremony, validators collectively sign the relevant piece of information, generating a genesis certificate
2. In the ceremony, we also generate a genesis config, including the genesis committee info, the genesis certificate from step 1.
3. In the ceremony, each validator runs a genesis process independently, consuming the genesis config from step 2. In this step we get a "genesis blob" (could be a checkpoint). All validators compute and cross validate the hash of the "genesis blob" for integrity. The goal of this step is to have a highly deterministic state agreed by all validators.
4. At a pre-agreed time, validators start the validator binary with the "genesis blob". This will emit a special event (or simply a checkpointing) for fullnodes to ramp up

[lxfind]

@longbowlu @bmwill Could you update the description of this issue to reflect the current plan and missing work?

[bmwill]

Yes I'm actually currently working in designing the flow and should be able to update things shortly

[lxfind]

@bmwill to break this issue down to smaller issues

[bmwill]

Opened #3061 to break out the making of genesis a transaction.

[bmwill]

Quick update on this issue. The current plan of streamlining the genesis ceremony goes a bit like this:

• Use a shared git repository with the validators participating in the ceremony
• implement some tooling such that validators can insert their information into the ceremony, and the create a commit and open a PR to have their info included.
• Once everyone has contributed the tooling can take the git repo and then produce a genesis.blob that everyone can use as their initial state.
• optionally for now (but potentially required later) interactively create the initial checkpoint 0 with an included genesis transaction. This checkpoint would be signed by the initial validator set.

[lxfind]

Need to figure out a stable version to distribute to partners
devnet has roughly 30k certs, its running 0.6.0, and we are using 30-35GB memory after ~6 days
there might still be something using more memory as number of certs go up (or correlated with something else)
should ideally put in memory profiling
set expectations

internal trial run