Add chunking to da sync #239
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Have you tested this pre-merge? It probably still works as it should if it compiles post-merge. |
|
Yes, I've tested it |
|
Just fix the merge conflict then? |
|
And make sure it compiles. |
|
The current compatible branch is broken so I wouldn't be able to release it |
|
No but you would be able to merge it |
MartinOndejka
force-pushed
the
MartinOndejka/add-chunking-to-da-sync
branch
from
ea32e08 to
95d69f7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When syncing the da node, do it in chunks of diffs. Currently the whole da layer is ~100MB so loading the all of the diffs to memory at once might will get out of hands in future. Doing all at the some time also timeouts the tcp connection currently, so it's not possible to do it.
Syncing currently takes ~10 minutes, so I've removed the syncing of da node out of sequencer into the cli command, so that it wouldn't block the sequencer from starting.
It takes that long even if the da node is missing only few diffs, this is because you need ledger openings to post a diff and to get ledger openings you need to sync the whole ledger from beginning. This is suboptimal, we might need to store some history of ledger checkpoints somewhere.