Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

client: add trienode fetcher for snap sync #2623

Merged
merged 66 commits into from
Aug 1, 2023
Merged
Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
f08c75e
parent 569128b0907a200750abbd7dcd52f08304cb15c9
scorbajio Apr 8, 2023
c6c59d6
Use Uint8Array instead of Buffer
scorbajio May 16, 2023
c3e3603
Merge branch 'master' of github.com:ethereumjs/ethereumjs-monorepo in…
scorbajio May 16, 2023
a56722e
Fix bugs and linting issues
scorbajio May 16, 2023
8b4af25
Request all nodes on key path for fetching
scorbajio May 16, 2023
f7ecc1a
Process nodes and request unkown ones
scorbajio May 23, 2023
b637d94
WIP: Debug encoding bugs
scorbajio Jun 2, 2023
539c3d6
Create helper function for converting from hex to keybyte encoding
scorbajio Jun 6, 2023
e85f973
Update path with new paths for known child nodes
scorbajio Jun 6, 2023
28eb299
Append next nibble as a byte to stay consistent with hex encoding
scorbajio Jun 8, 2023
733a8bf
Update debug statements to provide more clear logs
scorbajio Jun 8, 2023
a99b543
remove terminator from hex encoded key before appending to path
scorbajio Jun 12, 2023
c9a925e
Debug actual and expected node data discrepancy
scorbajio Jun 18, 2023
59b824e
Put fetched node data by account data parsed from leaves
scorbajio Jun 19, 2023
60e2ebd
Debug putting accounts from fetched nodes
scorbajio Jun 21, 2023
8d8dfa9
Initialize buffer of correct length
scorbajio Jun 21, 2023
9e3269c
Clean up code
scorbajio Jun 28, 2023
fff609e
Rework how the extension node path is calculated
scorbajio Jun 29, 2023
f39d65a
Rename functions and clean up
scorbajio Jun 29, 2023
7f19f05
Set up checks for storage and code healing
scorbajio Jun 29, 2023
b02425c
Queue storage node for fetching from account leaf node
scorbajio Jun 29, 2023
6c72924
Merge branch 'master' of github.com:ethereumjs/ethereumjs-monorepo in…
scorbajio Jul 3, 2023
e22f53a
Merge and format paths with new stacked sync path
scorbajio Jul 3, 2023
41ed39f
Complete path merge logic
scorbajio Jul 6, 2023
c911f7a
Store storage node data with account it originates from
scorbajio Jul 9, 2023
7fb0a51
Update how syncPath is being calculated
scorbajio Jul 9, 2023
98a0466
Include originating account node hash in storage node requests
scorbajio Jul 10, 2023
fcace0b
Clean up trienodefetcher.ts
scorbajio Jul 10, 2023
7b3598d
Clean up accountfetcher.ts
scorbajio Jul 10, 2023
5f430e8
Clean up snap sync files
scorbajio Jul 10, 2023
c1cf76a
Add tests for trienodefetcher
scorbajio Jul 10, 2023
a06c645
Merge branch 'master' into trie-node-fetcher
scorbajio Jul 10, 2023
90a7a5d
Add function docstrings and remove duplicated code
scorbajio Jul 11, 2023
4046178
Clean up spacing
scorbajio Jul 11, 2023
f8cce62
Remove getPathTo from encoding helpers
scorbajio Jul 11, 2023
29241ad
Rename functions
scorbajio Jul 12, 2023
fc86b7f
Merge branch 'master' into trie-node-fetcher
scorbajio Jul 12, 2023
131f5e2
Merge branch 'master' into trie-node-fetcher
scorbajio Jul 13, 2023
810e9cd
Test tasks and path request queing functions and helpers
scorbajio Jul 13, 2023
b0d22f1
Process node data collected from Sepolia
scorbajio Jul 13, 2023
44d8cb6
Update test
scorbajio Jul 14, 2023
4796c7c
Merge branch 'master' of github.com:ethereumjs/ethereumjs-monorepo in…
scorbajio Jul 20, 2023
a753a22
Merge branch 'master' of github.com:ethereumjs/ethereumjs-monorepo in…
scorbajio Jul 23, 2023
14df4fe
Move trie node fetcher helper functions into encoding helpers file
scorbajio Jul 23, 2023
ec80297
Use vitest for trie node fetcher tests
scorbajio Jul 23, 2023
3c3bd61
Clean up and update comments
scorbajio Jul 23, 2023
39de2a0
Keep pathStrings in Job
scorbajio Jul 23, 2023
1e340d9
Update tests
scorbajio Jul 23, 2023
21bd78d
Add tests for pathToHexKey from encoding helpers
scorbajio Jul 24, 2023
04a20de
Use push instead of unshift
scorbajio Jul 24, 2023
297764d
Do not start sync in test for finding best peer
scorbajio Jul 24, 2023
fa7e7ae
Update tests
scorbajio Jul 24, 2023
c1d588c
Fix linting issues
scorbajio Jul 24, 2023
92af328
Update packages/client/src/sync/fetcher/trienodefetcher.ts
scorbajio Jul 26, 2023
f41a6db
Update packages/client/src/sync/fetcher/trienodefetcher.ts
scorbajio Jul 26, 2023
1ea29b4
Update packages/client/src/sync/fetcher/trienodefetcher.ts
scorbajio Jul 26, 2023
1a15291
Update packages/client/src/sync/fetcher/trienodefetcher.ts
scorbajio Jul 26, 2023
3524290
Use original map to check received data
scorbajio Jul 26, 2023
92f33f0
Use MapDB for speedup
scorbajio Jul 26, 2023
c2b099d
Merge branch 'master' of github.com:ethereumjs/ethereumjs-monorepo in…
scorbajio Jul 26, 2023
4caf0c7
small refac
g11tech Jul 29, 2023
c8b9b88
Merge remote-tracking branch 'origin/master' into trie-node-fetcher
g11tech Jul 29, 2023
c3c3e95
fixes
g11tech Jul 30, 2023
45cc729
shift trie instances to using useKeyHashing true
g11tech Jul 31, 2023
eace057
Merge remote-tracking branch 'origin/master' into trie-node-fetcher
g11tech Jul 31, 2023
3156232
Merge remote-tracking branch 'origin/master' into trie-node-fetcher
g11tech Aug 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update packages/client/src/sync/fetcher/trienodefetcher.ts
Co-authored-by: acolytec3 <[email protected]>
scorbajio and acolytec3 authored Jul 26, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 1a1529107f4f643a2a020611802734134f817c55
2 changes: 1 addition & 1 deletion packages/client/src/sync/fetcher/trienodefetcher.ts
Original file line number Diff line number Diff line change
@@ -144,7 +144,7 @@ export class TrieNodeFetcher extends Fetcher<JobTask, Uint8Array[], Uint8Array>

try {
// While results are in the same order as requested hashes but there could be gaps/misses in the results
// if the node doesn't has the bytecode. We need an index to move forward through the hashes which are
// if the node doesn't has all of the requested trie nodes. We need an index to move forward through the hashes which are
// absent in the receieved responses
const receivedNodes: Uint8Array[] = []
const requestedNodes = new Set(Array.from(this.requestedNodeToPath.keys()))