client: snap protocol fetchers #2070

scorbajio · 2022-07-23T01:43:42Z

Continues the snap protocol implementation work. Work left to do:

Build fetchers for accounts, storage slots, and bytecode, and just dump it into a mock MPT
Build MPT and successfully run move the chain forward on a normal eth sync
Chart out behavior of non static peer and map next steps

holgerd77 · 2022-07-24T15:19:36Z

You are somewhat off with where you branched off here (lot's of old commits in), can you resubmit in a clean state?

codecov · 2022-07-27T03:32:36Z

Codecov Report

Merging #2070 (a4d991a) into snap-client (95f4bc2) will decrease coverage by 0.02%.
The diff coverage is n/a.

Flag	Coverage Δ
block	`84.01% <ø> (ø)`
blockchain	`80.10% <ø> (ø)`
common	`94.90% <ø> (ø)`
devp2p	`82.36% <ø> (-0.13%)`	⬇️
ethash	`90.81% <ø> (ø)`
evm	`40.97% <ø> (ø)`
statemanager	`84.55% <ø> (ø)`
trie	`80.84% <ø> (ø)`
tx	`92.18% <ø> (ø)`
util	`87.22% <ø> (ø)`
vm	`78.66% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

scorbajio · 2022-08-05T03:21:31Z

@holgerd77 Should I be continuously syncing with master on this branch?

holgerd77 · 2022-08-05T11:09:20Z

@holgerd77 Should I be continuously syncing with master on this branch?

I am not sure if I understand your question, so if you want to continue to work here, you have to bring this to the latest base branch state at some point (of course).

Can you maybe write a bit what's the state of this PR, what is left to do, when/if you would plan to continue.

scorbajio · 2022-08-05T16:20:09Z

@holgerd77 Should I be continuously syncing with master on this branch?

I am not sure if I understand your question, so if you want to continue to work here, you have to bring this to the latest base branch state at some point (of course).

Can you maybe write a bit what's the state of this PR, what is left to do, when/if you would plan to continue.

Thanks for clarifying, some of the changes from #1897 aren't in this PR (like the forceSnapSync option) but do show up in master. Not sure if I missed something while I was branching. Might need to just pull from master to include those changes.

Work that needs to be done:

First, we need to capture data for AccountRange, StorageRanges, ByteCodes, and TrieNodes to get practical visibility into what the returned data for those calls include and how to use them
Create account fetcher
Create storage fetcher
Create byte code fetcher
Create trie node fetcher

My understanding is that there needs to be one fetcher per each get call defined in the snap specs. Open to suggestions on this design plan.

I am planning to continue this work later today and over the weekend. I am currently seeing the following error when connecting ethjs -> geth using the branch I have raised this PR with:


  devp2p:dpt attempt adding peer 192.168.1.158:30303 +0ms
  devp2p:dpt:server:ping send ping to 192.168.1.158:30303 (peerId: -) +0ms
  devp2p:dpt:server:pong received pong from 192.168.1.158:30303 (peerId: de4d2de) +0ms
  devp2p:rlpx connect to 192.168.1.158:30303 (id: de4d2de) +20ms
  devp2p:rlpx connected to 192.168.1.158:30303, handshake waiting.. +5ms
  devp2p:192.168.1.158:rlpx:peer Send auth (EIP8: true) to 192.168.1.158:30303 +0ms
  devp2p:192.168.1.158:rlpx:peer Received ack (EIP8) from 192.168.1.158:30303 +47ms
  devp2p:192.168.1.158:rlpx:peer:HELLO Send HELLO to 192.168.1.158:30303 capabilities=eth66 +0ms
  devp2p:192.168.1.158:rlpx:peer Received header 192.168.1.158:30303 +2ms
  devp2p:192.168.1.158:rlpx:peer Received body 192.168.1.158:30303 80f89705b83d476574682f76312e31302e32312d756e737461626c652d33... +0ms
  devp2p:192.168.1.158:rlpx:peer:HELLO Received HELLO message (code: 0 - 0 = 0) 192.168.1.158:30303 +0ms
  devp2p:192.168.1.158:rlpx:peer:HELLO Received HELLO 192.168.1.158:30303 capabilities=eth66,eth67,snap1 +0ms
  devp2p:rlpx handshake with 192.168.1.158:30303 was successful (peer eip8 ack) +55ms
  devp2p:eth Send STATUS message to 192.168.1.158:30303 (eth66): [V:66, NID:11155111, TD:75749214106569, BestH:90f8874, GenH:25a5cc1, ForkHash: 0xfe3366e7, ForkNext: -] +0ms
  devp2p:eth:STATUS Send STATUS message to 192.168.1.158:30303 (eth66): [V:66, NID:11155111, TD:75749214106569, BestH:90f8874, GenH:25a5cc1, ForkHash: 0xfe3366e7, ForkNext: -] +0ms
  devp2p:192.168.1.158 Send STATUS message to 192.168.1.158:30303 (eth66): [V:66, NID:11155111, TD:75749214106569, BestH:90f8874, GenH:25a5cc1, ForkHash: 0xfe3366e7, ForkNext: -] +0ms
  devp2p:192.168.1.158:rlpx:peer Received header 192.168.1.158:30303 +4ms
  devp2p:192.168.1.158:rlpx:peer Received body 192.168.1.158:30303 105bf05af8594283aa36a7873c656d23029ab0a0d07cce9785d39c0dd240... +2ms
  devp2p:192.168.1.158:rlpx:peer Received ETH subprotocol message (code: 16 - 16 = 0) 192.168.1.158:30303 +0ms
  devp2p:eth Received STATUS message from 192.168.1.158:30303: [V:66, NID:11155111, TD:17000018015853232, BestH:d07cce9, GenH:25a5cc1, ForkHash: 0xfe3366e7, ForkNext: 1735371] +4ms
  devp2p:eth:STATUS Received STATUS message from 192.168.1.158:30303: [V:66, NID:11155111, TD:17000018015853232, BestH:d07cce9, GenH:25a5cc1, ForkHash: 0xfe3366e7, ForkNext: 1735371] +4ms
  devp2p:192.168.1.158 Received STATUS message from 192.168.1.158:30303: [V:66, NID:11155111, TD:17000018015853232, BestH:d07cce9, GenH:25a5cc1, ForkHash: 0xfe3366e7, ForkNext: 1735371] +4ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer Received header 192.168.1.158:30303 +0ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer Received body 192.168.1.158:30303 131244d188c110844d1e354100c783117fff018080 +1ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer Received ETH subprotocol message (code: 19 - 16 = 3) 192.168.1.158:30303 +0ms
  devp2p:eth Received GET_BLOCK_HEADERS message from 192.168.1.158:30303: d188c110844d1e354100c783117fff018080 +2ms
  devp2p:eth:GET_BLOCK_HEADERS Received GET_BLOCK_HEADERS message from 192.168.1.158:30303: d188c110844d1e354100c783117fff018080 +0ms
  devp2p:FIRST_PEER Received GET_BLOCK_HEADERS message from 192.168.1.158:30303: d188c110844d1e354100c783117fff018080 +0ms
  devp2p:eth Send BLOCK_HEADERS message to 192.168.1.158:30303: ca88c110844d1e354100c0 +1ms
  devp2p:eth:BLOCK_HEADERS Send BLOCK_HEADERS message to 192.168.1.158:30303: ca88c110844d1e354100c0 +0ms
  devp2p:FIRST_PEER Send BLOCK_HEADERS message to 192.168.1.158:30303: ca88c110844d1e354100c0 +1ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer Received header 192.168.1.158:30303 +3ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer Received body 192.168.1.158:30303 2732c4f1884b3e439ee1a54e66a02abe954b3cf088f738047d2b6cb32daf... +1ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer Received SNAP subprotocol message (code: 39 - 33 = 6) 192.168.1.158:30303 +0ms
  devp2p:snap Received GET_TRIE_NODES message from 192.168.1.158:30303: f1884b3e439ee1a54e66a02abe954b3cf088f738047d2b6cb32daff3abac... +0ms
  devp2p:snap:GET_TRIE_NODES Received GET_TRIE_NODES message from 192.168.1.158:30303: f1884b3e439ee1a54e66a02abe954b3cf088f738047d2b6cb32daff3abac... +0ms
  devp2p:192.168.1.158 Received GET_TRIE_NODES message from 192.168.1.158:30303: f1884b3e439ee1a54e66a02abe954b3cf088f738047d2b6cb32daff3abac... +0ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer Received header 192.168.1.158:30303 +2ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer Received body 192.168.1.158:30303 01010010 +1ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer:DISCONNECT Received DISCONNECT message (code: 1 - 0 = 1) 192.168.1.158:30303 +0ms
  devp2p:192.168.1.158:FIRST_PEER:rlpx:peer:DISCONNECT:SUBPROTOCOL_ERROR DISCONNECT reason: SUBPROTOCOL_ERROR 192.168.1.158:30303 +0ms

Which I suspect is due to missing changes. @g11tech This is not the expected output for the snap work done so far, right?

scorbajio · 2022-08-05T16:48:15Z

I see so snap-client has been merged and deleted and I need to reraise a new PR to merge with master.

scorbajio · 2022-08-06T04:55:46Z

Work is being continued at #2107 .

g11tech and others added 12 commits July 17, 2022 22:23

a small add for client

198dab7

don't offer snap yet, try only consuming from other peers

9dd6c80

Add snap protocol/methods availability in the peer

a30c214

Snap sync syncronizer stub

3e6fe1a

client: send empty list in case of non-existing headers

a022184

setup snap sender to make request

76708fc

devp2p: fix ETH66 length

e0c51d8

devp2p: update eth65/eth64 protocol length

768e202

client/snapsync: fix GetAccountRange encoding

1d272a7

decode account range response

83a4797

Add other methods of snapsync

c6c65ea

rebase fixes for new master

95f4bc2

scorbajio changed the title ~~Snap client~~ client: snap protocol fetchers Jul 23, 2022

scorbajio added PR state: WIP package: client labels Jul 23, 2022

Create accountfetcher base

025ab98

scorbajio force-pushed the snap-client branch from ae925f1 to 025ab98 Compare July 24, 2022 22:31

Use bigint instead of BN

a4d991a

g11tech force-pushed the snap-client branch from 95f4bc2 to 599652d Compare August 4, 2022 09:02

g11tech mentioned this pull request Aug 4, 2022

Master tracker for snap sync #2100

Closed

19 tasks

g11tech force-pushed the snap-client branch from 50e6a95 to 55e6317 Compare August 4, 2022 10:09

holgerd77 deleted the branch ethereumjs:snap-client August 5, 2022 11:20

holgerd77 closed this Aug 5, 2022

scorbajio deleted the snap-client branch August 6, 2022 00:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client: snap protocol fetchers #2070

client: snap protocol fetchers #2070

scorbajio commented Jul 23, 2022 •

edited

Loading

holgerd77 commented Jul 24, 2022

codecov bot commented Jul 27, 2022 •

edited

Loading

scorbajio commented Aug 5, 2022

holgerd77 commented Aug 5, 2022

scorbajio commented Aug 5, 2022 •

edited

Loading

scorbajio commented Aug 5, 2022

scorbajio commented Aug 6, 2022

client: snap protocol fetchers #2070

client: snap protocol fetchers #2070

Conversation

scorbajio commented Jul 23, 2022 • edited Loading

holgerd77 commented Jul 24, 2022

codecov bot commented Jul 27, 2022 • edited Loading

Codecov Report

scorbajio commented Aug 5, 2022

holgerd77 commented Aug 5, 2022

scorbajio commented Aug 5, 2022 • edited Loading

scorbajio commented Aug 5, 2022

scorbajio commented Aug 6, 2022

scorbajio commented Jul 23, 2022 •

edited

Loading

codecov bot commented Jul 27, 2022 •

edited

Loading

scorbajio commented Aug 5, 2022 •

edited

Loading