[R4R] implement State Verification && Snapshot Commit pipeline #668
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
unclezoro
force-pushed
the
pipeline_commit
branch
from
71705d0 to
c40b1ad
Compare
unclezoro
force-pushed
the
pipeline_commit
branch
21 times, most recently
from
a2ad058 to
4f4f94d
Compare
unclezoro
changed the title
unclezoro
changed the title
unclezoro
force-pushed
the
pipeline_commit
branch
6 times, most recently
from
ecf3134 to
85e5b73
Compare
add metrics reopen trie
unclezoro
force-pushed
the
pipeline_commit
branch
4 times, most recently
from
731474c to
226ff65
Compare
remove prefetcher
unclezoro
force-pushed
the
pipeline_commit
branch
from
226ff65 to
15f5bff
Compare
unclezoro
changed the title
unclezoro
commented
Jan 24, 2022
This was referenced Jan 26, 2022
Merged
unclezoro
added a commit
that referenced
this pull request
Feb 17, 2022
* pipeline commit trie add metrics reopen trie * add unit testcase * resolve keefe's comment * resolve igor's comments * update prefetch remove prefetcher * no need to return error for precacheTransaction * fix lint issue * add some comments * remove useless code * add default option is false * fix diffsync nil point * fix panic on GetProofByHash Co-authored-by: zjubfd <[email protected]>
keefel
pushed a commit
to keefel/bsc
that referenced
this pull request
Jun 6, 2022
…hain#668) * pipeline commit trie add metrics reopen trie * add unit testcase * resolve keefe's comment * resolve igor's comments * update prefetch remove prefetcher * no need to return error for precacheTransaction * fix lint issue * add some comments * remove useless code * add default option is false * fix diffsync nil point * fix panic on GetProofByHash Co-authored-by: zjubfd <[email protected]>
Jemerald83
approved these changes
Feb 25, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The state verification && Snapshot/MPT commit takes a lot of time. While in most cases, the verification can pass, there is no need to wait for the verification result serially. For snapshot/MPT commit, we only persist the bottom diff layer to disk, there is no need to wait for it to finish. We propose an asynchronous state verification and asynchronous snapshot/MPT commit solution.
Spec
The insert blockchain can be split into three-phase:
The three-phase are executed serially.
Phase2 occupies about 15-20% of the time, and in most case, it should succeed.
Phase3 occupies about 10-15% of the time.
State Verification Pipeline
A pipeline workflow is introduced in this PR:
Once Block Process and Snapshot Commit are done, there is no need to wait for State Verification, the blockchain can process the next block. While for the next block, before it starts State Verification, it has to wait until the State Verification of its parent finish.
Persistent Storage Pipeline
Both MPT and Snapshot need to be persisted into storage, the storage can be Memory DB and Disk DB, this PR will introduce the asynchronous persistence of Disk DB without breaking the processing of the following blocks.
Key Points
How to store the Verification Result
Since Snapshot already has a very good layout for the execution result of the block, we try to reuse it by adding a verifiedCh channel as the verification result.
How to Rollback if Verification Failed
There is an auto rollback mechanism when verification is failed.
How to avoid race condition
Introduce a few locks, but ensure it won't bring too much overhead.
Performance
Following is the 24hours comparison of a Pipeline node(10.211.32.131) and a normal node(10.211.31.49)。
Hardware: 24core, 96G, SSD, storage occupation 2.5T
Execution Time
Before:
After:
The execution time is mainly improved by this PR: #704, while in this PR, we further improve it by avoiding the CPU usage burst.
Validation Time
Before:
After:
Commit Time
Before:
After:
CommitTxs Time
Before:
After:
Further Discussion
Merge Multi Difflayer and Do Verification Once
Improve
AccountsIntermediateRootImprove Miner Module