Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingest: assign sequence number to excises and make EFOS infallible #3210

Closed
itsbilal opened this issue Jan 9, 2024 · 1 comment · Fixed by #3241
Closed

ingest: assign sequence number to excises and make EFOS infallible #3210

itsbilal opened this issue Jan 9, 2024 · 1 comment · Fixed by #3241
Assignees
Labels

Comments

@itsbilal
Copy link
Contributor

itsbilal commented Jan 9, 2024

Currently, excises as part of an IngestAndExcise call "always win" unless they overlap with a memtable at the time of excise, in which case the excise waits for the memtable to be flushed and then the excise is applied. If we assign a sequence number for excises, we can optionally add them to the flushable queue (see #2676), but alternatively an assigned sequence number also allows the excise to wait on any EFOS with snapshot sequence numbers below the excise to transition to a file-only snapshot before the excise is applied.

This issue is about introducing the seqnum assignment for excises and the linear search for any overlapping EFOS if there are any. #2676 is about the bigger change to move excises into the flushable queue, similar to flushable ingests.

@itsbilal itsbilal added C-enhancement New feature or request A-storage labels Jan 9, 2024
@itsbilal
Copy link
Contributor Author

itsbilal commented Jan 9, 2024

This change would allow us to drop WaitForFileOnly as a method on EventuallyFileOnlySnapshot, allowing consistency checks in crdb to not need to wait at all before proceeding.

itsbilal added a commit to itsbilal/pebble that referenced this issue Jan 23, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait
for all relevant keys in their key bounds to be flushed, just in case
an excise came in before that happened. This change avoids the need
to do that by assigning a seqnum to every excise (which will be
useful when we do cockroachdb#2676), and by flushing any memtables that
overlap with the protected ranges of any EFOS that has a sequence
number below the excise.

The reduction in waiting is desirable as actual conflicts between
excises and EventuallyFileOnlySnapshots are rare, while users of
EventuallyFileOnlySnapshots were previously stuck waiting or stuck
doing excessive flushes at times when no such excises were happening.

Fixes cockroachdb#3210.
itsbilal added a commit to itsbilal/pebble that referenced this issue Jan 23, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait
for all relevant keys in their key bounds to be flushed, just in case
an excise came in before that happened. This change avoids the need
to do that by assigning a seqnum to every excise (which will be
useful when we do cockroachdb#2676), and by flushing any memtables that
overlap with the protected ranges of any EFOS that has a sequence
number below the excise.

The reduction in waiting is desirable as actual conflicts between
excises and EventuallyFileOnlySnapshots are rare, while users of
EventuallyFileOnlySnapshots were previously stuck waiting or stuck
doing excessive flushes at times when no such excises were happening.

Fixes cockroachdb#3210.
itsbilal added a commit to itsbilal/pebble that referenced this issue Jan 31, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait
for all relevant keys in their key bounds to be flushed, just in case
an excise came in before that happened. This change avoids the need
to do that by assigning a seqnum to every excise (which will be
useful when we do cockroachdb#2676), and by flushing any memtables that
overlap with the protected ranges of any EFOS that has a sequence
number below the excise.

The reduction in waiting is desirable as actual conflicts between
excises and EventuallyFileOnlySnapshots are rare, while users of
EventuallyFileOnlySnapshots were previously stuck waiting or stuck
doing excessive flushes at times when no such excises were happening.

Fixes cockroachdb#3210.
itsbilal added a commit to itsbilal/pebble that referenced this issue Jan 31, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait
for all relevant keys in their key bounds to be flushed, just in case
an excise came in before that happened. This change avoids the need
to do that by assigning a seqnum to every excise (which will be
useful when we do cockroachdb#2676), and by flushing any memtables that
overlap with the protected ranges of any EFOS that has a sequence
number below the excise.

The reduction in waiting is desirable as actual conflicts between
excises and EventuallyFileOnlySnapshots are rare, while users of
EventuallyFileOnlySnapshots were previously stuck waiting or stuck
doing excessive flushes at times when no such excises were happening.

Fixes cockroachdb#3210.
itsbilal added a commit that referenced this issue Jan 31, 2024
Previously, we had to make EventuallyFileOnlySnapshots (EFOS) wait
for all relevant keys in their key bounds to be flushed, just in case
an excise came in before that happened. This change avoids the need
to do that by assigning a seqnum to every excise (which will be
useful when we do #2676), and by flushing any memtables that
overlap with the protected ranges of any EFOS that has a sequence
number below the excise.

The reduction in waiting is desirable as actual conflicts between
excises and EventuallyFileOnlySnapshots are rare, while users of
EventuallyFileOnlySnapshots were previously stuck waiting or stuck
doing excessive flushes at times when no such excises were happening.

Fixes #3210.
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant