Lazy loading

[RomarQ]

What does it do?

The objective of this PR is to provide a solution for allowing lazy downloading of the state of a live network instead of requiring its state to be downloaded in advance, which may take a long time for chains that have been running for a few years.

What important points reviewers should know?

Building with lazy-loading features enabled:

cargo build --features lazy-loading

New parameters:

--fork-chain-from-rpc <CHAIN_RPC> # Mandatory
--block <BLOCK_FROM_WHICH_TO_START_THE_FORK> # Optional
--fork-state-overrides <STATE_OVERRIDES_JSON_FILE> # Optional 
--runtime-override <WASM_FILE> # Optional 

Example state overrides files:

[
 {
     "pallet": "System",
     "storage": "SelectedCandidates",
     "value": "0x04f24ff3a9cf04c71dbc94d0b566f7a27b94566cac"
 }
]

Running lazy loading:

./target/debug/moonbeam \
--fork-chain-from-rpc https://moonbeam.unitedbloc.com/ \
--fork-state-overrides state_overrides.json  \
--alice \
--no-grandpa \
--reserved-only \
--sealing 6000 \
--unsafe-rpc-external \
--rpc-external \
--rpc-cors all \
--rpc-methods Unsafe \
--force-authoring \
--unsafe-force-node-key-generation

Is there something left for follow-up PRs?

• Add documentation on how to use the lazy loading feature;
• Add lazy loading tests against Moonbeam and Moonriver (done for moonbeam)

What alternative implementations were considered?

Are there relevant PRs or issues in other repositories (Substrate, Polkadot, Frontier, Cumulus)?

What value does it bring to the blockchain users?

This feature enables testing of client code against live chains, which is not possible with Chopsticks. Allowing testing both the client and runtime simultaneously, we can thoroughly verify client-specific components, such as RPCs.

[github-actions]

Coverage Report

@@                 Coverage Diff                 @@
##           master   rq/lazy-loading      +/-   ##
===================================================
- Coverage   80.93%            78.65%   -2.28%     
+ Files         286               294       +8     
+ Lines       81965             84225    +2260     
===================================================
- Hits        66331             66244      -87     
+ Misses      15634             17981    +2347     

Files Changed	Coverage
/client/evm-tracing/src/listeners/call_list.rs	72.55% (-4.84%)	🔽
/client/rpc/finality/src/lib.rs	88.14% (+6.89%)	🔼
/node/cli/src/cli.rs	33.33% (-1.73%)	🔽
/node/cli/src/command.rs	23.13% (+3.90%)	🔼
/node/service/src/lib.rs	61.96% (+0.06%)	🔼
/precompiles/proxy/src/tests.rs	96.68% (-0.09%)	🔽

Coverage generated Wed Sep 11 10:45:02 UTC 2024

[github-actions]

WASM runtime size check:

Compared to target branch

Moonbase runtime: 2188 KB (no changes) ✅

Moonbeam runtime: 2120 KB (no changes) ✅

Moonriver runtime: 2128 KB (no changes) ✅

Compared to latest release (runtime-3102)

Moonbase runtime: 2188 KB (+248 KB compared to latest release) ⚠️

Moonbeam runtime: 2120 KB (+220 KB compared to latest release) ⚠️

Moonriver runtime: 2128 KB (+228 KB compared to latest release) ⚠️

[pLabarta]

Overall it looks good to me. It mostly needs some documentation on the new feature and the tradeoffs of using this particular client for tests, but I know its coming in a future PR.

Also could use a script for setting up the binary as the test suite expects when running it locally. Wrote this one:

#!/bin/bash

# Build lazy-loading client
echo '😴 Building lazy-loading client...'
cargo build --release --features lazy-loading

# Rename
echo '✍️ Renaming lazy-loading client...'
mv target/release/moonbeam target/release/lazy-loading

echo '✅ Lazy loading client ready!'

[gonzamontiel]

The PR looks good, great changes 💯
Maybe we can add this parameter to the description of the PR because it might be needed for testing (I had to add it) —unsafe-force-node-key-generation.

There's a lot of code implementing the substrate client api, I compared to the in_mem.rs to see the actual changes. I paid more attention to backend.rs and call_executor.rs. It looks good to me!

[RomarQ]

The PR looks good, great changes 💯 Maybe we can add this parameter to the description of the PR because it might be needed for testing (I had to add it) —unsafe-force-node-key-generation.

There's a lot of code implementing the substrate client api, I compared to the in_mem.rs to see the actual changes. I paid more attention to backend.rs and call_executor.rs. It looks good to me!

Improved the PR description 👍

changes implemented