Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fetch Headers and Bodies separately #4167 #4632

Merged
merged 125 commits into from
Mar 1, 2024
Merged
Changes from 2 commits
Commits
Show all changes
125 commits
Select commit Hold shift + click to select a range
2a9d399
test
sklppy88 Feb 7, 2024
bc37c35
why does this fix
sklppy88 Feb 10, 2024
f19757d
good
sklppy88 Feb 12, 2024
94bc65d
Readding test
sklppy88 Feb 12, 2024
3dad335
Yarn format
sklppy88 Feb 12, 2024
da70a99
Revert "why does this fix"
sklppy88 Feb 12, 2024
70cea40
fix
sklppy88 Feb 12, 2024
8b505e3
some reorganizing
sklppy88 Feb 12, 2024
b8f442f
more reorg
sklppy88 Feb 12, 2024
7756734
more cleanup
sklppy88 Feb 12, 2024
ae39116
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 12, 2024
f0362d7
formatting and fixing imports
sklppy88 Feb 12, 2024
bc551e6
Merge remote-tracking branch 'origin/ek/fix/align-block-structs-with-…
sklppy88 Feb 12, 2024
aef101f
more
sklppy88 Feb 12, 2024
6d3a980
test
sklppy88 Feb 12, 2024
f277bbc
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 12, 2024
ba62405
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 13, 2024
c8b343f
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 13, 2024
90cadf3
initial
sklppy88 Feb 13, 2024
7099ff6
Comments
sklppy88 Feb 13, 2024
58eae75
more organizing
sklppy88 Feb 13, 2024
a3db555
yarn format
sklppy88 Feb 13, 2024
f55a7a6
comments
sklppy88 Feb 13, 2024
107cccc
address comments
sklppy88 Feb 13, 2024
841f0c9
initial
sklppy88 Feb 13, 2024
9574cf8
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 13, 2024
2f69022
Merge branch 'ek/fix/align-block-structs-with-yp' into ek/fix/fetch-h…
sklppy88 Feb 13, 2024
acd0dab
next
sklppy88 Feb 13, 2024
1861272
modify a bit
sklppy88 Feb 14, 2024
9ce8d61
Merge branch 'ek/fix/align-block-structs-with-yp' into ek/fix/fetch-h…
sklppy88 Feb 14, 2024
81d2a7e
kinda working
sklppy88 Feb 14, 2024
ac78531
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 14, 2024
785c16d
Fix formatting and addressing comments
sklppy88 Feb 14, 2024
8fb5976
Yarn format
sklppy88 Feb 14, 2024
8b76426
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 14, 2024
d0414f8
test
sklppy88 Feb 14, 2024
e377156
test
sklppy88 Feb 14, 2024
7b2a632
Address comments
sklppy88 Feb 15, 2024
46444b5
Fix
sklppy88 Feb 15, 2024
3bbd1f7
Fix formatting
sklppy88 Feb 15, 2024
92c49ab
Addressing comments
sklppy88 Feb 15, 2024
a3e738e
Addressing comments
sklppy88 Feb 15, 2024
29e064d
Address comments
sklppy88 Feb 16, 2024
8a0d4bb
Formatting fix
sklppy88 Feb 16, 2024
47580cb
Fix test
sklppy88 Feb 16, 2024
243280f
Merge branch 'ek/fix/align-block-structs-with-yp' into ek/fix/fetch-h…
sklppy88 Feb 16, 2024
1a7cc41
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 16, 2024
62957da
next
sklppy88 Feb 16, 2024
14bbaa5
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 16, 2024
3e8aac3
Address comments
sklppy88 Feb 16, 2024
db9f3ea
Merge remote-tracking branch 'origin/ek/fix/align-block-structs-with-…
sklppy88 Feb 16, 2024
645b8f2
Address comments
sklppy88 Feb 16, 2024
1869a34
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 16, 2024
16cde68
fixing broken tests
sklppy88 Feb 17, 2024
9b211b2
yarn fmt
sklppy88 Feb 17, 2024
0a459a9
Merge branch 'ek/fix/align-block-structs-with-yp' into ek/fix/fetch-h…
sklppy88 Feb 17, 2024
cbb14e2
test
sklppy88 Feb 17, 2024
e5d83bb
Fixing tests
sklppy88 Feb 17, 2024
bc10ec1
Merge branch 'ek/fix/align-block-structs-with-yp' into ek/fix/fetch-h…
sklppy88 Feb 17, 2024
f87e5b3
fix formatting
sklppy88 Feb 17, 2024
8ae3931
asdf
sklppy88 Feb 17, 2024
c1d9e46
Merge branch 'ek/fix/align-block-structs-with-yp' into ek/fix/fetch-h…
sklppy88 Feb 17, 2024
198c068
Merge branch 'master' into ek/fix/align-block-structs-with-yp
sklppy88 Feb 17, 2024
d798290
Address comments
sklppy88 Feb 17, 2024
b64d79b
test
sklppy88 Feb 17, 2024
2e624c9
Merge branch 'ek/fix/align-block-structs-with-yp' into ek/fix/fetch-h…
sklppy88 Feb 17, 2024
56331da
formatting fix
sklppy88 Feb 17, 2024
555e468
yarn fmt
sklppy88 Feb 17, 2024
cc5a1f2
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 22, 2024
72bd82d
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 22, 2024
51b9a4b
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 23, 2024
90ccc41
asdf
sklppy88 Feb 27, 2024
6e47010
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 27, 2024
66f241f
fix
sklppy88 Feb 27, 2024
2b92022
asdf
sklppy88 Feb 27, 2024
1afb545
asdf
sklppy88 Feb 27, 2024
47a6e09
asdf
sklppy88 Feb 27, 2024
1c2b9c8
fix typing
sklppy88 Feb 27, 2024
62393cc
asdf
sklppy88 Feb 27, 2024
4f7573b
sdf
sklppy88 Feb 27, 2024
7cd83de
asdf
sklppy88 Feb 28, 2024
0cb3ad9
yarn fmt and fix
sklppy88 Feb 28, 2024
8d81aa7
working
sklppy88 Feb 28, 2024
6f73e5b
edit
sklppy88 Feb 28, 2024
84bcd01
fix archiver tests
sklppy88 Feb 28, 2024
4be5cf2
add block body add/retrieve test
sklppy88 Feb 28, 2024
bb201ce
yarn fmt
sklppy88 Feb 28, 2024
b694441
fix
sklppy88 Feb 28, 2024
7036dfa
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 28, 2024
fb0bf04
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 28, 2024
e30f97c
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 28, 2024
c196348
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 28, 2024
5fde53a
address comments 1
sklppy88 Feb 28, 2024
a520e8f
Merge remote-tracking branch 'origin/ek/fix/fetch-headers-and-bodies-…
sklppy88 Feb 28, 2024
d283bb3
Addressing commments
sklppy88 Feb 28, 2024
c2bd595
Addressing comments
sklppy88 Feb 28, 2024
a16fd06
address
sklppy88 Feb 28, 2024
7365473
addressing
sklppy88 Feb 28, 2024
97748ba
lint
sklppy88 Feb 28, 2024
c5940a0
comments
sklppy88 Feb 29, 2024
ce9b7b8
Update yarn-project/archiver/src/archiver/archiver_store.ts
sklppy88 Feb 29, 2024
130b3ec
Update yarn-project/archiver/src/archiver/kv_archiver_store/kv_archiv…
sklppy88 Feb 29, 2024
80d7cad
Update yarn-project/archiver/src/archiver/data_retrieval.ts
sklppy88 Feb 29, 2024
c65cc38
Merge remote-tracking branch 'origin/ek/fix/fetch-headers-and-bodies-…
sklppy88 Feb 29, 2024
8f9483a
comments
sklppy88 Feb 29, 2024
997de35
fix
sklppy88 Feb 29, 2024
2534514
more fixes
sklppy88 Feb 29, 2024
89724bd
comments
sklppy88 Feb 29, 2024
db4e687
comments
sklppy88 Feb 29, 2024
8a1b830
asdf
sklppy88 Feb 29, 2024
4f6fbc6
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 29, 2024
2379f7d
fix
sklppy88 Feb 29, 2024
31e4543
yarn fmt
sklppy88 Feb 29, 2024
2c70e95
remove
sklppy88 Feb 29, 2024
ff5bf52
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 29, 2024
733ec31
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Feb 29, 2024
0562ac0
stale comment
sklppy88 Feb 29, 2024
7e1eaeb
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Mar 1, 2024
6ce30f2
Address comments
sklppy88 Mar 1, 2024
ac91763
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Mar 1, 2024
c2712ff
Merge branch 'master' into ek/fix/fetch-headers-and-bodies-separately
sklppy88 Mar 1, 2024
b997964
fix
sklppy88 Mar 1, 2024
bfa273d
Merge remote-tracking branch 'origin/ek/fix/fetch-headers-and-bodies-…
sklppy88 Mar 1, 2024
80e881b
address comments
sklppy88 Mar 1, 2024
3b6f99e
comments
sklppy88 Mar 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions yarn-project/pxe/src/synchronizer/synchronizer.test.ts
Original file line number Diff line number Diff line change
@@ -41,7 +41,6 @@ describe('Synchronizer', () => {
it('sets header from latest block', async () => {
const block = L2Block.random(1, 4);
aztecNode.getLogs.mockResolvedValueOnce([block.body.encryptedLogs]).mockResolvedValue([block.body.unencryptedLogs]);
block.body.txEffects.forEach(txEffect => delete txEffect.logs);
aztecNode.getBlocks.mockResolvedValue([block]);

await synchronizer.work();
@@ -65,7 +64,7 @@ describe('Synchronizer', () => {
aztecNode.getLogs
.mockResolvedValueOnce([block1.body.encryptedLogs])
.mockResolvedValue([block1.body.unencryptedLogs]);
block1.body.txEffects.forEach(txEffect => delete txEffect.logs);

aztecNode.getBlocks.mockResolvedValue([block1]);

await synchronizer.work();
@@ -76,7 +75,6 @@ describe('Synchronizer', () => {
// But they should change when we process block with height 5
const block5 = L2Block.random(5, 4);

block5.body.txEffects.forEach(txEffect => delete txEffect.logs);
aztecNode.getBlocks.mockResolvedValue([block5]);

await synchronizer.work();
@@ -98,9 +96,6 @@ describe('Synchronizer', () => {
.mockResolvedValueOnce([blocks[0].body.encryptedLogs])
.mockResolvedValueOnce([blocks[1].body.encryptedLogs]);

blocks[0].body.txEffects.forEach(txEffect => delete txEffect.logs);
blocks[1].body.txEffects.forEach(txEffect => delete txEffect.logs);

aztecNode.getBlocks
// called by synchronizer.work, we are testing fromFields in this first call
.mockResolvedValueOnce([
15 changes: 2 additions & 13 deletions yarn-project/pxe/src/synchronizer/synchronizer.ts
Original file line number Diff line number Diff line change
@@ -174,25 +174,14 @@ export class Synchronizer {
}

try {
let encryptedLogs = await this.node.getLogs(from, limit, LogType.ENCRYPTED);
if (!encryptedLogs.length) {
// This should never happen because this function should only be called when the note processor is lagging
// behind main sync.
throw new Error('No encrypted logs in processor catch up mode');
}

// Note: If less than `limit` encrypted logs is returned, then we fetch only that number of blocks.
const blocks = await this.node.getBlocks(from, encryptedLogs.length);
const blocks = await this.node.getBlocks(from, limit);
if (!blocks.length) {
// This should never happen because this function should only be called when the note processor is lagging
// behind main sync.
throw new Error('No blocks in processor catch up mode');
}

if (blocks.length !== encryptedLogs.length) {
// "Trim" the encrypted logs to match the number of blocks.
encryptedLogs = encryptedLogs.slice(0, blocks.length);
}
const encryptedLogs = blocks.flatMap(block => block.body.encryptedLogs);

const blockContexts = blocks.map(block => new L2BlockContext(block));