Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compaction: integrate range keys #1686

Closed
jbowens opened this issue May 6, 2022 · 1 comment · Fixed by #1738
Closed

compaction: integrate range keys #1686

jbowens opened this issue May 6, 2022 · 1 comment · Fixed by #1738
Assignees

Comments

@jbowens
Copy link
Collaborator

jbowens commented May 6, 2022

Range keys (#1339) need to be read and written during compactions. Like range deletions, range keys exist separately from point keys and are surfaced through the keyspan.FragmentIterator interface. Here's a sketch of the anticipated integration.

  1. When constructing a compaction input iterator, a keyspan.LevelIter is created for each input level that contains at least 1 file with range keys.
  2. The individual levels' range key keyspan.LevelIters are wrapped in a keyspan.MergingIter. The MergingIter is configured with a special keyspan.Transform. The Transform is configured with the slice of open snapshots. It partitions a Span's Keys into snapshot stripes, calling rangekey.Coalesce on the Keys of each stripe. We may want to refactor rangekey.Coalesce's interface to only take an input []Keys slice and output *[]Keys pointer. The rangekey.Coalesce calls performs shadowing, dropping unnecessary keys. This Transform is also likely where we should examine in-use key ranges (eg, elideRangeTombstone) to determine whether or not to drop RangeKeyUnset and RangeKeyDelete keys that exist in the last snapshot stripe.
  3. The MergingIter is then wrapped in a keyspan.DefragmentingIter configured with the DefragmentInternal defragmenting method. The defragmenting iterator will ensure the fragments persisted to disk are only as fine-grained as required.
  4. Finally, the keyspan.DefragmentingIter is wrapped with a keyspan.InterleavingIter to interleave range keys into the main compaction loop. This is similar to what was done for range deletions in compaction: use InterleavingIter to interleave rangedels #1659.
  5. When the main compaction loop encounters a RANGEKEYSET, RANGEKEYUNSET or RANGEKEYDELETE, it retrieves the span from the interleaving iterator and adds it to a keyspan.Fragmenter (separate from c.rangeDelFrag). All the truncation logic mirrors range deletions. Range keys should be added to the output sstable using Writer.AddRangeKey (not the Writer.RangeKey{Set,Unset,Delete} methods). Future work may consider removing the keyspan.Fragmenter, but I think it should be deferred for now, and we should track as close to possible to the current handling of range deletions.
@jbowens
Copy link
Collaborator Author

jbowens commented Jun 6, 2022

A note that came up in cockroachdb/cockroach#82041 — Compactions and flushes involving inputs containing range keys should calculate a separate inUseKeyRanges using the range-key bounds of sstables. That will ensure we're able to drop ineffective RANGEKEYDELs much sooner.

itsbilal added a commit to itsbilal/pebble that referenced this issue Jun 7, 2022
This change implements compaction of sstables containing
range keys, as well as allows memtables with non-empty
rangeKeySkl skiplists to write correctly to sstable range
key blocks in flushes. Range key elision and snapshot
striping is implement in a keyspan.Transform passed to
keyspan.MergingIter inside the compaction iter. Otherwise,
behaviour is kept as close to that of range deletions
as possible, to improve confidence in correctness.

Fixes cockroachdb#1686.
itsbilal added a commit to itsbilal/pebble that referenced this issue Jun 8, 2022
This change implements compaction of sstables containing
range keys, as well as allows memtables with non-empty
rangeKeySkl skiplists to write correctly to sstable range
key blocks in flushes. Range key elision and snapshot
striping is implement in a keyspan.Transform passed to
keyspan.MergingIter inside the compaction iter. Otherwise,
behaviour is kept as close to that of range deletions
as possible, to improve confidence in correctness.

Fixes cockroachdb#1686.
itsbilal added a commit to itsbilal/pebble that referenced this issue Jun 8, 2022
This change implements compaction of sstables containing
range keys, as well as allows memtables with non-empty
rangeKeySkl skiplists to write correctly to sstable range
key blocks in flushes. Range key elision and snapshot
striping is implement in a keyspan.Transform passed to
keyspan.MergingIter inside the compaction iter. Otherwise,
behaviour is kept as close to that of range deletions
as possible, to improve confidence in correctness.

Fixes cockroachdb#1686.
itsbilal added a commit to itsbilal/pebble that referenced this issue Jun 9, 2022
This change implements compaction of sstables containing
range keys, as well as allows memtables with non-empty
rangeKeySkl skiplists to write correctly to sstable range
key blocks in flushes. Range key elision and snapshot
striping is implement in a keyspan.Transform passed to
keyspan.MergingIter inside the compaction iter. Otherwise,
behaviour is kept as close to that of range deletions
as possible, to improve confidence in correctness.

Fixes cockroachdb#1686.
itsbilal added a commit to itsbilal/pebble that referenced this issue Jun 10, 2022
This change implements compaction of sstables containing
range keys, as well as allows memtables with non-empty
rangeKeySkl skiplists to write correctly to sstable range
key blocks in flushes. Range key elision and snapshot
striping is implement in a keyspan.Transform passed to
keyspan.MergingIter inside the compaction iter. Otherwise,
behaviour is kept as close to that of range deletions
as possible, to improve confidence in correctness.

Fixes cockroachdb#1686.
itsbilal added a commit to itsbilal/pebble that referenced this issue Jun 10, 2022
This change implements compaction of sstables containing
range keys, as well as allows memtables with non-empty
rangeKeySkl skiplists to write correctly to sstable range
key blocks in flushes. Range key elision and snapshot
striping is implement in a keyspan.Transform passed to
keyspan.MergingIter inside the compaction iter. Otherwise,
behaviour is kept as close to that of range deletions
as possible, to improve confidence in correctness.

Fixes cockroachdb#1686.
itsbilal added a commit that referenced this issue Jun 10, 2022
This change implements compaction of sstables containing
range keys, as well as allows memtables with non-empty
rangeKeySkl skiplists to write correctly to sstable range
key blocks in flushes. Range key elision and snapshot
striping is implement in a keyspan.Transform passed to
keyspan.MergingIter inside the compaction iter. Otherwise,
behaviour is kept as close to that of range deletions
as possible, to improve confidence in correctness.

Fixes #1686.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants