Add continuity script to pre-migration #347
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alecps
force-pushed
the
alecps/alwaysCheckDb
branch
from
a00cdca to
4cf83d3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds the runDBCheck process to the beginning of the pre-migration, and also adds a
startparameter so that the continuity check can start from the last migrated ancient block in the newDB. This allows us to attach the continuity script to the beginning of the pre-migration without incurring extra delays during a full migration, as the majority of blocks will have already been pre-migrated and will not be checked for continuity.The last ancient block in the newDB is chosen as the starting point because the non-ancient block data is modified during a full migration and only copied during a pre-migration. Depending on whether previous full or pre-migrations have been run, the non ancient data may have been modified in various ways since the last continuity check was performed. Non-ancient data is also only 90000 blocks. For these reasons, it makes sense to always run the continuity script on the non-ancient data as it only takes a few seconds.
In local testing using a baklava datadir the continuity script only added about 17s to the pre-migration when run a second time. This is mostly due to time spent opening and closing the dbs. While this is slow considering that most blocks have already been checked, it's much faster than running the whole continuity script again. This approach aims to minimize code changes to the migration script and keep things simple given our timeline.
Tested locally on baklava datadir