-
Notifications
You must be signed in to change notification settings - Fork 479
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ingest: assign sequence number to excises and make EFOS infallible #3210
Labels
Comments
This change would allow us to drop |
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jan 23, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait for all relevant keys in their key bounds to be flushed, just in case an excise came in before that happened. This change avoids the need to do that by assigning a seqnum to every excise (which will be useful when we do cockroachdb#2676), and by flushing any memtables that overlap with the protected ranges of any EFOS that has a sequence number below the excise. The reduction in waiting is desirable as actual conflicts between excises and EventuallyFileOnlySnapshots are rare, while users of EventuallyFileOnlySnapshots were previously stuck waiting or stuck doing excessive flushes at times when no such excises were happening. Fixes cockroachdb#3210.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jan 23, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait for all relevant keys in their key bounds to be flushed, just in case an excise came in before that happened. This change avoids the need to do that by assigning a seqnum to every excise (which will be useful when we do cockroachdb#2676), and by flushing any memtables that overlap with the protected ranges of any EFOS that has a sequence number below the excise. The reduction in waiting is desirable as actual conflicts between excises and EventuallyFileOnlySnapshots are rare, while users of EventuallyFileOnlySnapshots were previously stuck waiting or stuck doing excessive flushes at times when no such excises were happening. Fixes cockroachdb#3210.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jan 31, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait for all relevant keys in their key bounds to be flushed, just in case an excise came in before that happened. This change avoids the need to do that by assigning a seqnum to every excise (which will be useful when we do cockroachdb#2676), and by flushing any memtables that overlap with the protected ranges of any EFOS that has a sequence number below the excise. The reduction in waiting is desirable as actual conflicts between excises and EventuallyFileOnlySnapshots are rare, while users of EventuallyFileOnlySnapshots were previously stuck waiting or stuck doing excessive flushes at times when no such excises were happening. Fixes cockroachdb#3210.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jan 31, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait for all relevant keys in their key bounds to be flushed, just in case an excise came in before that happened. This change avoids the need to do that by assigning a seqnum to every excise (which will be useful when we do cockroachdb#2676), and by flushing any memtables that overlap with the protected ranges of any EFOS that has a sequence number below the excise. The reduction in waiting is desirable as actual conflicts between excises and EventuallyFileOnlySnapshots are rare, while users of EventuallyFileOnlySnapshots were previously stuck waiting or stuck doing excessive flushes at times when no such excises were happening. Fixes cockroachdb#3210.
itsbilal
added a commit
that referenced
this issue
Jan 31, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait for all relevant keys in their key bounds to be flushed, just in case an excise came in before that happened. This change avoids the need to do that by assigning a seqnum to every excise (which will be useful when we do #2676), and by flushing any memtables that overlap with the protected ranges of any EFOS that has a sequence number below the excise. The reduction in waiting is desirable as actual conflicts between excises and EventuallyFileOnlySnapshots are rare, while users of EventuallyFileOnlySnapshots were previously stuck waiting or stuck doing excessive flushes at times when no such excises were happening. Fixes #3210.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, excises as part of an
IngestAndExcise
call "always win" unless they overlap with a memtable at the time of excise, in which case the excise waits for the memtable to be flushed and then the excise is applied. If we assign a sequence number for excises, we can optionally add them to the flushable queue (see #2676), but alternatively an assigned sequence number also allows the excise to wait on any EFOS with snapshot sequence numbers below the excise to transition to a file-only snapshot before the excise is applied.This issue is about introducing the seqnum assignment for excises and the linear search for any overlapping EFOS if there are any. #2676 is about the bigger change to move excises into the flushable queue, similar to flushable ingests.
The text was updated successfully, but these errors were encountered: