Ensure quick-repair commit on shutdown #897
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Traditionally, redb has stored its allocator state in the region headers and region tracker page, which are only valid after a clean shutdown. Now that we also have the allocator state table (which works even after a crash), it would be nice to use that exclusively. This is the first step in that transition: making sure that after a clean shutdown, the allocator state is saved to both places. Then, once enough time has passed, we can remove all support for reading and writing the old allocator state. Even after the transition, any redb version will be able to open any database. If the database has been opened at least once during the transition period, it'll open cleanly; otherwise it'll require repair, but it'll always work.
cberner
approved these changes
Nov 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Traditionally, redb has stored its allocator state in the region headers and region tracker page, which are only valid after a clean shutdown. Now that we also have the allocator state table (which works even after a crash), it would be nice to use that exclusively.
This is the first step in that transition: making sure that after a clean shutdown, the allocator state is saved to both places. Then, once enough time has passed, we can remove all support for reading and writing the old allocator state.
Even after the transition, any redb version will be able to open any database. If the database has been opened at least once during the transition period, it'll open cleanly; otherwise it'll require repair, but it'll always work.
Note that this means we're saving the allocator state twice on shutdown (unless the user's last commit already had quick-repair enabled), which costs maybe 50 ms for a small database or a few hundred ms for a large one. (But large databases probably want to use quick-repair anyway, in which case there's no cost at all.) It's most noticeable when running the tests, which already spend a substantial fraction of their time writing out allocator state, since they're constantly opening and closing databases without writing much actual data.
Most of this cost will go away in the future when we get rid of the old allocator state, but not all. At the moment, writing the allocator state table is a bit slower than writing the old allocator state, mostly because of inefficiencies in how redb handles large pages. I haven't been worrying about this, since I feel like the performance is totally acceptable already -- but if you're interested, let me know and I can tell you all about what could be done to make it faster. None of this involves breaking compatibility, so these improvements could be done at any point.