Align compaction output file boundaries to the next level ones #10655

jay-zhuang · 2022-09-09T03:10:26Z

Try to align the compaction output file boundaries to the next level ones
(grandparent level), to reduce the level compaction write-amplification.

In level compaction, there are "wasted" data at the beginning and end of the
output level files. Align the file boundary can avoid such "wasted" compaction.
With this PR, it tries to align the non-bottommost level file boundaries to its
next level ones. It may cut file when the file size is large enough (at least
50% of target_file_size) and not too large (2x target_file_size).

db_bench shows about 12.56% compaction reduction:

TEST_TMPDIR=/data/dbbench2 ./db_bench --benchmarks=fillrandom,readrandom -max_background_jobs=12 -num=400000000 -target_file_size_base=33554432

# baseline:
Flush(GB): cumulative 25.882, interval 7.216
Cumulative compaction: 285.90 GB write, 162.36 MB/s write, 269.68 GB read, 153.15 MB/s read, 2926.7 seconds

# with this change:
Flush(GB): cumulative 25.882, interval 7.753
Cumulative compaction: 249.97 GB write, 141.96 MB/s write, 233.74 GB read, 132.74 MB/s read, 2534.9 seconds

The compaction simulator shows a similar result (14% with 100G random data).
As a side effect, with this PR, the SST file size can exceed the
target_file_size, but is capped at 2x target_file_size. And there will be
smaller files. Here are file size statistics when loading 100GB with the target
file size 32MB:

          baseline      this_PR
count  1.656000e+03  1.705000e+03
mean   3.116062e+07  3.028076e+07
std    7.145242e+06  8.046139e+06

The feature is enabled by default, to revert to the old behavior disable it
with AdvancedColumnFamilyOptions.level_compaction_dynamic_file_size = false

Also includes #1963 to cut file before skippable grandparent file. Which is for
use case like user adding 2 or more non-overlapping data range at the same
time, it can reduce the overlapping of 2 datasets in the lower levels.

facebook-github-bot · 2022-09-15T19:08:02Z

@jay-zhuang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

cbi42

Thanks for the PR, the grandparent logic is very readable. Added some comments/questions. Will look at tests next.

db/compaction/compaction.cc

db/compaction/compaction_outputs.cc

db/compaction/compaction_outputs.h

db/compaction/compaction_outputs.cc

facebook-github-bot · 2022-09-23T19:42:54Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-09-23T19:44:25Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-09-23T21:00:33Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-09-23T23:04:00Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-09-24T01:08:02Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

siying · 2022-09-26T20:16:34Z

db/compaction/compaction_outputs.cc

-  const Slice& internal_key = c_iter.key();
+  // no need to have complicated cut logic for non level compaction
+  if (compaction_->immutable_options()->compaction_style !=
+      kCompactionStyleLevel) {


In the ongoing project of incremental universal compaction project, we rely on original line 134 to 139 to align with next level to reduce recomputing in those incremental runs. Can we keep the logic for universal compaction too?

Okay, I'm removing it.

siying · 2022-09-26T20:20:21Z

db/compaction/compaction_outputs.cc

+  // etc.
+  const Slice& internal_key = c_iter.key();
+  const uint64_t previous_overlapped_bytes = grandparent_overlapped_bytes_;
+  size_t switched_num = UpdateGrandparentBoundaryInfo(internal_key);


Perhaps pick a better variable name. Perhaps more straight forward ones like num_parent_boundaries_in_gap or something like that.

Updated to num_grandparent_boundaries_crossed.

siying · 2022-09-26T20:22:07Z

db/compaction/compaction_outputs.cc

+    // without any overlap, it may likely happen.
+    // More details, check PR #1963
+    const size_t skippable_file_switched_num =
+        size_t{3} - being_grandparent_gap_;


I'm a little bit confused how 3 - bool means here. Is there a way to write the code to be more intuitive?

Updated to

const size_t num_skippable_boundaries_crossed = being_grandparent_gap_ ? 2 : 3;

also consistent with the num_grandparent_boundaries_crossed.

siying · 2022-09-26T20:25:24Z

db/compaction/compaction_outputs.cc

+        current_output_file_size_ >
+            (50 +
+             std::min(grandparent_boundary_switched_num_ * 5, size_t{40})) *
+                compaction_->target_output_file_size() / 100) {


Is there a potential chance of overflow, considering user can set target file size to arbitrary level.

right, it would be better to do the / 100 first. Updated.

facebook-github-bot · 2022-09-26T23:55:27Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-09-27T03:23:00Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-09-27T03:53:31Z

@jay-zhuang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

siying

The general approach looks good and I will defer to @cbi42 for a thorough review.

siying · 2022-09-27T16:34:26Z

db/compaction/compaction_outputs.cc

+    // More details, check PR #1963
+    const size_t num_skippable_boundaries_crossed =
+        being_grandparent_gap_ ? 2 : 3;
+    if (compaction_->immutable_options()->level_compaction_dynamic_file_size &&


Here we probably would check leveled compaction too before dynamic level, as in universal compaction, users can still set level_compaction_dynamic_file_size=true although the expectation is that it is no-op. This check can probably be combined with line 236.

db/compaction/compaction_outputs.cc

db/db_range_del_test.cc

db/db_test2.cc

db/compaction/compaction_job_test.cc

db/compaction/compaction.cc

mdcallag · 2022-09-28T15:12:36Z

This change seems like it can have a big impact but the commit message content is too brief. Please include:

a few more sentences about what was changed
more detail on the perf test
a description of whether or not this behavior depends on options. From the files I see there is a new option, that option should be explained in the commit message

facebook-github-bot · 2022-09-29T00:25:21Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

cbi42

LTGM once we add more detail to the commit message as Mark suggested above.

facebook-github-bot · 2022-09-30T00:28:52Z

@jay-zhuang has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-09-30T00:29:46Z

@jay-zhuang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

ajkr

Nice PR, thanks!

ajkr · 2022-09-30T20:03:16Z

db/compaction/compaction_outputs.cc

+    if (compaction_->immutable_options()->compaction_style ==
+            kCompactionStyleLevel &&
+        compaction_->immutable_options()->level_compaction_dynamic_file_size &&
+        num_grandparent_boundaries_crossed >=
+            num_skippable_boundaries_crossed &&
+        grandparent_overlapped_bytes_ - previous_overlapped_bytes >
+            compaction_->target_output_file_size() / 8) {


Should there be a minimum on current_output_file_size_?

ajkr · 2022-09-30T20:10:22Z

db/compaction/compaction_outputs.cc

+    if (compaction_->immutable_options()->compaction_style ==
+            kCompactionStyleLevel &&
+        compaction_->immutable_options()->level_compaction_dynamic_file_size &&
+        current_output_file_size_ >
+            ((compaction_->target_output_file_size() + 99) / 100) *
+                (50 + std::min(grandparent_boundary_switched_num_ * 5,
+                               size_t{40}))) {
+      return true;
+    }


Should the threshold be higher when being_grandparent_gap_ == true? It feels less urgent in that case.

Summary: The option is introduced in #10655 to allow reverting to old behavior. The option is enabled by default and there has not been a need to disable it. Remove it for 9.0 release. Also fixed and improved a few unit tests that depended on setting this option to false. Pull Request resolved: #12325 Test Plan: existing tests. Reviewed By: hx235 Differential Revision: D53369430 Pulled By: cbi42 fbshipit-source-id: 0ec2440ca8d88db7f7211c581542c7581bd4d3de

facebook-github-bot added the CLA Signed label Sep 9, 2022

jay-zhuang requested review from siying, ajkr and cbi42 September 9, 2022 03:12

jay-zhuang force-pushed the comp_precut2 branch from 5209a1b to cba67dc Compare September 15, 2022 18:44

jay-zhuang changed the base branch from zjay_will_del_comp_precut to main September 15, 2022 18:45

jay-zhuang marked this pull request as ready for review September 15, 2022 18:45

cbi42 reviewed Sep 22, 2022

View reviewed changes

db/compaction/compaction.cc Outdated Show resolved Hide resolved

db/compaction/compaction_outputs.cc Outdated Show resolved Hide resolved

db/compaction/compaction_outputs.cc Show resolved Hide resolved

db/compaction/compaction_outputs.h Show resolved Hide resolved

siying reviewed Sep 22, 2022

View reviewed changes

db/compaction/compaction_outputs.cc Outdated Show resolved Hide resolved

db/compaction/compaction_outputs.cc Outdated Show resolved Hide resolved

jay-zhuang force-pushed the comp_precut2 branch from 9fafbb2 to 96a6cba Compare September 23, 2022 19:42

siying reviewed Sep 26, 2022

View reviewed changes

jay-zhuang force-pushed the comp_precut2 branch from ec293d4 to ec10214 Compare September 26, 2022 23:55

jay-zhuang force-pushed the comp_precut2 branch from ec10214 to 8955eef Compare September 27, 2022 03:22

siying reviewed Sep 27, 2022

View reviewed changes

cbi42 reviewed Sep 28, 2022

View reviewed changes

jay-zhuang force-pushed the comp_precut2 branch from 8955eef to 67773a8 Compare September 29, 2022 00:25

cbi42 approved these changes Sep 29, 2022

View reviewed changes

jay-zhuang changed the title ~~Improves compaction output file cut to reduce WA~~ Align compaction output file boundaries to the next level ones Sep 30, 2022

Cut compaction output file aligned with grandparents

e5286ad

jay-zhuang added 9 commits September 29, 2022 17:28

Add level_compaction_dynamic_file_size option to opt-out the feature

169f1e8

Update a typo

3de9642

Fix build

75559b0

Fix test

5e6c52a

Fix another test

d0af93f

a few review feedbacks update

4610c30

Fix a test

180fd2e

Address the review feedbacks

dfc4204

Add HISTORY

c5798a5

jay-zhuang force-pushed the comp_precut2 branch from 67773a8 to c5798a5 Compare September 30, 2022 00:28

facebook-github-bot closed this in f3cc666 Sep 30, 2022

ajkr reviewed Sep 30, 2022

View reviewed changes

ajkr mentioned this pull request Nov 1, 2022

Compaction to cut output file if it can avoid to overlap a grandparent file #1963

Closed

tabokie mentioned this pull request Aug 14, 2023

cherry-pick compaction enhancements tikv/rocksdb#345

Closed

cbi42 mentioned this pull request Feb 2, 2024

Remove deprecated option level_compaction_dynamic_file_size #12325

Closed

cbi42 mentioned this pull request Aug 23, 2024

Appearance of files much larger than target_file_size_base option #12956

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Align compaction output file boundaries to the next level ones #10655

Align compaction output file boundaries to the next level ones #10655

jay-zhuang commented Sep 9, 2022 •

edited

Loading

facebook-github-bot commented Sep 15, 2022

cbi42 left a comment •

edited

Loading

facebook-github-bot commented Sep 23, 2022

facebook-github-bot commented Sep 23, 2022

facebook-github-bot commented Sep 23, 2022

facebook-github-bot commented Sep 23, 2022

facebook-github-bot commented Sep 24, 2022

siying Sep 26, 2022

jay-zhuang Sep 26, 2022

siying Sep 26, 2022

jay-zhuang Sep 26, 2022

siying Sep 26, 2022

jay-zhuang Sep 26, 2022

siying Sep 26, 2022

jay-zhuang Sep 26, 2022

facebook-github-bot commented Sep 26, 2022

facebook-github-bot commented Sep 27, 2022

facebook-github-bot commented Sep 27, 2022

siying left a comment

siying Sep 27, 2022

mdcallag commented Sep 28, 2022

facebook-github-bot commented Sep 29, 2022

cbi42 left a comment

facebook-github-bot commented Sep 30, 2022

facebook-github-bot commented Sep 30, 2022

ajkr left a comment

ajkr Sep 30, 2022

ajkr Sep 30, 2022

Align compaction output file boundaries to the next level ones #10655

Align compaction output file boundaries to the next level ones #10655

Conversation

jay-zhuang commented Sep 9, 2022 • edited Loading

facebook-github-bot commented Sep 15, 2022

cbi42 left a comment • edited Loading

Choose a reason for hiding this comment

facebook-github-bot commented Sep 23, 2022

facebook-github-bot commented Sep 23, 2022

facebook-github-bot commented Sep 23, 2022

facebook-github-bot commented Sep 23, 2022

facebook-github-bot commented Sep 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Sep 26, 2022

facebook-github-bot commented Sep 27, 2022

facebook-github-bot commented Sep 27, 2022

siying left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdcallag commented Sep 28, 2022

facebook-github-bot commented Sep 29, 2022

cbi42 left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Sep 30, 2022

facebook-github-bot commented Sep 30, 2022

ajkr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jay-zhuang commented Sep 9, 2022 •

edited

Loading

cbi42 left a comment •

edited

Loading