Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

client: add trienode fetcher for snap sync #2623

Merged
merged 66 commits into from
Aug 1, 2023
Merged

client: add trienode fetcher for snap sync #2623

merged 66 commits into from
Aug 1, 2023

Conversation

scorbajio
Copy link
Contributor

No description provided.

@scorbajio scorbajio changed the title [client] Trienode fetcher Trienode fetcher for snap sync Apr 8, 2023
@codecov
Copy link

codecov bot commented Apr 11, 2023 •
edited
Loading

Codecov Report

Merging #2623 (3156232) into master (40c2395) will increase coverage by 0.21%.
The diff coverage is 87.41%.

Additional details and impacted files

Impacted file tree graph

Flag Coverage Δ
block 88.66% <ø> (?)
blockchain 92.58% <ø> (?)
client ?
common 98.64% <ø> (?)
ethash ∅ <ø> (?)
rlp ∅ <ø> (?)
statemanager 85.16% <ø> (?)
trie 89.98% <87.41%> (?)
tx ?
util 86.77% <ø> (?)
vm 79.20% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

scorbajio added 2 commits May 15, 2023 13:28
@scorbajio
parent 569128b
f08c75e 
author Amir <[email protected]> 1680919955 -0700
committer Amir <[email protected]> 1684182403 -0700

Create base for trienode fetcher

Integrate trie node fetcher with account fetcher - Implement task phase

Switch to fetching all nodes starting from root

Keep record of node hashes with requested paths for cross reference check

Add mappings to track path and node requests

Fix bugs

Process node data in store

Queue unkown children of received nodes for fetching

Fetch rest of subtrie before removing active request
@scorbajio
Use Uint8Array instead of Buffer
c6c59d6
@scorbajio scorbajio force-pushed the trie-node-fetcher branch from cfb9404 to b4b58ac Compare May 16, 2023 17:13
@scorbajio scorbajio force-pushed the trie-node-fetcher branch from b4b58ac to c3e3603 Compare May 16, 2023 17:30
scorbajio added 20 commits May 16, 2023 11:22
@scorbajio
Fix bugs and linting issues
a56722e
@scorbajio
Request all nodes on key path for fetching
8b4af25
@scorbajio
Process nodes and request unkown ones
f7ecc1a
@scorbajio
WIP: Debug encoding bugs
b637d94
@scorbajio
Create helper function for converting from hex to keybyte encoding
539c3d6
@scorbajio
Update path with new paths for known child nodes
e85f973
@scorbajio
Append next nibble as a byte to stay consistent with hex encoding
28eb299
@scorbajio
Update debug statements to provide more clear logs
733a8bf
@scorbajio
remove terminator from hex encoded key before appending to path
a99b543
@scorbajio
Debug actual and expected node data discrepancy
c9a925e
@scorbajio
Put fetched node data by account data parsed from leaves
59b824e
@scorbajio
Debug putting accounts from fetched nodes
60e2ebd
@scorbajio
Initialize buffer of correct length
8d8dfa9
@scorbajio
Clean up code
9e3269c
@scorbajio
Rework how the extension node path is calculated
fff609e
@scorbajio
Rename functions and clean up
f39d65a
@scorbajio
Set up checks for storage and code healing
7f19f05
@scorbajio
Queue storage node for fetching from account leaf node
b02425c
@scorbajio
Merge branch 'master' of github.com:ethereumjs/ethereumjs-monorepo in…
6c72924 
…to trie-node-fetcher
@scorbajio
Merge and format paths with new stacked sync path
e22f53a
@scorbajio scorbajio requested a review from g11tech July 24, 2023 20:10
scorbajio and others added 13 commits July 25, 2023 17:45
@scorbajio @acolytec3
Update packages/client/src/sync/fetcher/trienodefetcher.ts
92af328 
Co-authored-by: acolytec3 <[email protected]>
@scorbajio @acolytec3
Update packages/client/src/sync/fetcher/trienodefetcher.ts
f41a6db 
Co-authored-by: acolytec3 <[email protected]>
@scorbajio @acolytec3
Update packages/client/src/sync/fetcher/trienodefetcher.ts
1ea29b4 
Co-authored-by: acolytec3 <[email protected]>
@scorbajio @acolytec3
Update packages/client/src/sync/fetcher/trienodefetcher.ts
1a15291 
Co-authored-by: acolytec3 <[email protected]>
@scorbajio
Use original map to check received data
3524290
@scorbajio
Use MapDB for speedup
92f33f0
@scorbajio
Merge branch 'master' of github.com:ethereumjs/ethereumjs-monorepo in…
c2b099d 
…to trie-node-fetcher
@g11tech
small refac
4caf0c7
@g11tech
Merge remote-tracking branch 'origin/master' into trie-node-fetcher
c8b9b88
@g11tech
fixes
c3c3e95
@g11tech
shift trie instances to using useKeyHashing true
45cc729
@g11tech
Merge remote-tracking branch 'origin/master' into trie-node-fetcher
eace057
@g11tech
Merge remote-tracking branch 'origin/master' into trie-node-fetcher
3156232
g11tech
g11tech approved these changes Aug 1, 2023
View reviewed changes
Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trienode fetcher seems to be working as expected although further testing and overall fetcher intergration work is required which we can undertake in followups

@g11tech g11tech changed the title Trienode fetcher for snap sync client: add trienode fetcher for snap sync Aug 1, 2023
@holgerd77
Copy link
Member

Cool and congrats 👍 🎉, so from my side we could also merge this in.

@g11tech g11tech merged commit e11a93c into ethereumjs:master Aug 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@acolytec3 acolytec3 acolytec3 left review comments

@g11tech g11tech g11tech approved these changes

@jochem-brouwer jochem-brouwer Awaiting requested review from jochem-brouwer

Assignees
No one assigned
Labels
package: client PR state: needs review type: feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@scorbajio @holgerd77 @acolytec3 @g11tech