-
Notifications
You must be signed in to change notification settings - Fork 471
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
compaction: integrate range keys #1686
Comments
A note that came up in cockroachdb/cockroach#82041 — Compactions and flushes involving inputs containing range keys should calculate a separate |
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jun 7, 2022
This change implements compaction of sstables containing range keys, as well as allows memtables with non-empty rangeKeySkl skiplists to write correctly to sstable range key blocks in flushes. Range key elision and snapshot striping is implement in a keyspan.Transform passed to keyspan.MergingIter inside the compaction iter. Otherwise, behaviour is kept as close to that of range deletions as possible, to improve confidence in correctness. Fixes cockroachdb#1686.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jun 8, 2022
This change implements compaction of sstables containing range keys, as well as allows memtables with non-empty rangeKeySkl skiplists to write correctly to sstable range key blocks in flushes. Range key elision and snapshot striping is implement in a keyspan.Transform passed to keyspan.MergingIter inside the compaction iter. Otherwise, behaviour is kept as close to that of range deletions as possible, to improve confidence in correctness. Fixes cockroachdb#1686.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jun 8, 2022
This change implements compaction of sstables containing range keys, as well as allows memtables with non-empty rangeKeySkl skiplists to write correctly to sstable range key blocks in flushes. Range key elision and snapshot striping is implement in a keyspan.Transform passed to keyspan.MergingIter inside the compaction iter. Otherwise, behaviour is kept as close to that of range deletions as possible, to improve confidence in correctness. Fixes cockroachdb#1686.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jun 9, 2022
This change implements compaction of sstables containing range keys, as well as allows memtables with non-empty rangeKeySkl skiplists to write correctly to sstable range key blocks in flushes. Range key elision and snapshot striping is implement in a keyspan.Transform passed to keyspan.MergingIter inside the compaction iter. Otherwise, behaviour is kept as close to that of range deletions as possible, to improve confidence in correctness. Fixes cockroachdb#1686.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jun 10, 2022
This change implements compaction of sstables containing range keys, as well as allows memtables with non-empty rangeKeySkl skiplists to write correctly to sstable range key blocks in flushes. Range key elision and snapshot striping is implement in a keyspan.Transform passed to keyspan.MergingIter inside the compaction iter. Otherwise, behaviour is kept as close to that of range deletions as possible, to improve confidence in correctness. Fixes cockroachdb#1686.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Jun 10, 2022
This change implements compaction of sstables containing range keys, as well as allows memtables with non-empty rangeKeySkl skiplists to write correctly to sstable range key blocks in flushes. Range key elision and snapshot striping is implement in a keyspan.Transform passed to keyspan.MergingIter inside the compaction iter. Otherwise, behaviour is kept as close to that of range deletions as possible, to improve confidence in correctness. Fixes cockroachdb#1686.
itsbilal
added a commit
that referenced
this issue
Jun 10, 2022
This change implements compaction of sstables containing range keys, as well as allows memtables with non-empty rangeKeySkl skiplists to write correctly to sstable range key blocks in flushes. Range key elision and snapshot striping is implement in a keyspan.Transform passed to keyspan.MergingIter inside the compaction iter. Otherwise, behaviour is kept as close to that of range deletions as possible, to improve confidence in correctness. Fixes #1686.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Range keys (#1339) need to be read and written during compactions. Like range deletions, range keys exist separately from point keys and are surfaced through the
keyspan.FragmentIterator
interface. Here's a sketch of the anticipated integration.keyspan.LevelIter
is created for each input level that contains at least 1 file with range keys.keyspan.LevelIter
s are wrapped in akeyspan.MergingIter
. TheMergingIter
is configured with a special keyspan.Transform. TheTransform
is configured with the slice of open snapshots. It partitions aSpan
'sKeys
into snapshot stripes, calling rangekey.Coalesce on theKeys
of each stripe. We may want to refactorrangekey.Coalesce
's interface to only take an input[]Keys
slice and output*[]Keys
pointer. Therangekey.Coalesce
calls performs shadowing, dropping unnecessary keys. ThisTransform
is also likely where we should examine in-use key ranges (eg,elideRangeTombstone
) to determine whether or not to dropRangeKeyUnset
andRangeKeyDelete
keys that exist in the last snapshot stripe.MergingIter
is then wrapped in akeyspan.DefragmentingIter
configured with theDefragmentInternal
defragmenting method. The defragmenting iterator will ensure the fragments persisted to disk are only as fine-grained as required.keyspan.DefragmentingIter
is wrapped with akeyspan.InterleavingIter
to interleave range keys into the main compaction loop. This is similar to what was done for range deletions in compaction: use InterleavingIter to interleave rangedels #1659.RANGEKEYSET
,RANGEKEYUNSET
orRANGEKEYDELETE
, it retrieves the span from the interleaving iterator and adds it to akeyspan.Fragmenter
(separate fromc.rangeDelFrag
). All the truncation logic mirrors range deletions. Range keys should be added to the output sstable using Writer.AddRangeKey (not theWriter.RangeKey{Set,Unset,Delete}
methods). Future work may consider removing thekeyspan.Fragmenter
, but I think it should be deferred for now, and we should track as close to possible to the current handling of range deletions.The text was updated successfully, but these errors were encountered: