Bootstrap

[Leo-Besancon]

Potential Problems

• Big bootstrap functional test #4530
• P1 - bootstrap fails on labnet when inject_operations > 0. To test
• P1 - bootstrap fails when keys are too big (Bootstrap fails when we push the ledger to its storage limits #4406)

Improvements

• P2 - Improve bootstrap universe #4540
• P3 - add option on config.toml that sets the max_open_files configuration of rocksdb (default config < 1024)
• P2 - Ensure receiving a non-valid db after bootstraping does not lead to a panic. Instead, we should restart bootstrap from scratch.
  In massa-node > src > main.rs:

   	if !final_state.read().is_db_valid() {
   		// TODO: Bootstrap again instead of panicking
   		panic!("critical: db is not valid after bootstrap");
   	}

Validation

• P1 - ensure that a bootstrapping client can not make a bootstrap server panic by sending invalid data
• P2 - make sure a client cannot deny service by staying connected too long or flooding
• P2 - implement unit tests for those aspects
• P2 - validate bootstrap server/client messages serialization / deserialization
• P3 - calibrate bandwidth limits and timeout values (after ensuring BootstrapMessage serialization limits are working well)
• P2 - calibrate U64VarIntDeserializer max bound for state_new_elements_length_deserializer (same)
• P2 - Limit the number of updates we send during Bootstrap. We keep sending all the changes, but if they are too big, we should send an error to the client so that they can bootstrap again from scratch. (same)
• P1 - test that messages never exceeds limits in size

[litchipi]

If we start a labnet without keeping the ledger, just a normal one without any operations on the branch test_bootstrap_big, the bootstrap process will work but the hashes of final_state will be different.

It seams that some changes I added in the branch test_bootstrap_big is responsible for the hash difference

[litchipi]

I figured out why the final state hash are different when the node generates a brand new ledger, it works now.
However, it's still different when a ledger is generated using massa-ledger-editor

[litchipi]

The hash mismatch occurs when launching a labnet on main with crafted ledger.
After I tried with smaller keys / values in datastore (length 9000), it still does it, so I don't think it's a matter of serialization.
@Leo-Besancon Could it be because of this piece of code we use when crafting the ledger ?
How could I modify the massa-ledger-editor in order to craft a ledger that doesn't require this check bypass ?

EDIT: Disregard this comment, I made a mistake while writing it, see below

[Leo-Besancon]

The hash mismatch occurs when launching a labnet on main with crafted ledger. After I tried with smaller keys / values in datastore (length 9000), it still does it, so I don't think it's a matter of serialization. @Leo-Besancon Could it be because of this piece of code we use when crafting the ledger ? How could I modify the massa-ledger-editor in order to craft a ledger that doesn't require this check bypass ?

Nice research! I think the easiest to ensure that all necessary keys are found would be to start from a working ledger (built normally by the node), and then only add your crafted keys.
This way, the trail_hash, cycle_history and so one would be in your crafted ledger, and it's less likely to cause a problem.

[litchipi]

That's actually what I already do

• A while ago I started a node normally, and copied the storage into a local folder A.
• To craft a new ledger, I copy the content of the folder A to a new folder B
• Using massa-ledger-editor, I add data to the ledger stored in ledger B
• Before I launch the labnet, I copy the folder B to the storage of the massa-node.

(Note that most of what I listed above is automated so I don't forget a step while doing it)

So the whole basis of the ledger comes from an (almost) empty ledger generated by the massa-node.
I only add new entries (new addresses with datastore and bytecode associated), using the massa libraries.
Normally, the only invalid data is the random generated data (datastore keys + value, and the bytecode), the rest is "standard"

[litchipi]

@Leo-Besancon My bad, I made a mistake while writing the message, when using smaller datastore key + value, the hashes DOES MATCH, so it seams that's it's linked to the size of the data inside the ledger.
Either the serialization, transmission, reception or deserialization of the data isn't calibrated for such a size

[Leo-Besancon]

Ok thanks for the clarification! In that case testing the message serialization should be the next step? :)

[litchipi]

Update:

• The Serialization / Deserialization of the BootstrapServerMessage has been tested with limit cases, and it's wokring OK
• When adding to the ledger a specially crafted key in the datastore, and displaying its value after bootstrap, the value is the one expected, but the final_state hash is different after bootstrap (expected). So this test is OK also
• When adding to the ledger a specially crafted bytecode, and displaying it after bootstrap, the value is the one expected, but the final_state hash is different after bootstrap (expected). So this test is OK also

[litchipi]

I'm confused, I just tested again inserting a single data with length 500 and it DOESN'T work anymore...
It also don't work anymore when simply passing an empty ledger, with the keep-ledger option...

From now on, I think I'll have to consider that any ledger crafted using massa-ledger-editor is potentially bad.

[litchipi]

@sydhds Some of the tests you are doing are ticking the boxes on this issue I think