[BREAKING] opt(compactions): Improve compaction performance #1574
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manishrjain
changed the title
…ction priority adjustment.
…score exceeds 1.0.
![codelingo[bot]](https://avatars.githubusercontent.com/in/14339?s=60&v=4){kind=small}
levels.go
Outdated
| // concurrently, only iterating over the provided key range, generating tables. | ||
| // This speeds up the compaction significantly. | ||
| func (s *levelsController) subcompact(it y.Iterator, kr keyRange, cd compactDef, | ||
| inflightBuilders *y.Throttle, res chan *table.Table) { |
There was a problem hiding this comment.
Returned channels or channel arguments should generally have a direction.
| @@ -66,7 +66,7 @@ var ( | |||
| vlogMaxEntries uint32 | |||
| loadBloomsOnOpen bool | |||
| detectConflicts bool | |||
| compression bool | |||
| zstdComp bool | |||
There was a problem hiding this comment.
Avoid global variables to improve readability and reduce complexity
NamanJain8
pushed a commit
that referenced
this pull request
Nov 5, 2020
Implement multiple ideas for speeding up compactions: 1. Dynamic Level Sizes: https://rocksdb.org/blog/2015/07/23/dynamic-level.html 2. L0 to L0 compactions: https://rocksdb.org/blog/2017/06/26/17-level-based-changes.html 3. Sub Compactions: Split up one compaction into multiple sub-compactions using key ranges, which can be run concurrently. 4. If a table being generated at Li overlaps with >= 10 tables at Li+1, finish the table. This helps avoid big overlaps and expensive compactions later. 5. Update compaction priority based on the priority of the next level prioritizing compactions of lower levels over upper levels, resulting in an always healthy LSM tree structure. With these changes, we can load 1B entries (160GB of data) into Badger (without the Stream framework) in 1h25m at 31 MB/s. This is a significant improvement over current master. Co-authored-by: Ibrahim Jarif <[email protected]> fix(tests): Writebatch, Stream, Vlog tests (#1577) This PR fixes the following issues/tests - Deadlock in writes batch - Use atomic to set value of `writebatch.error` - Vlog Truncate test - Fix issues with empty memtables - Test options - Set memtable size. - Compaction tests - Acquire lock before updating level tables - Vlog Write - Truncate the file size if the transaction cannot fit in vlog size - TestPersistLFDiscardStats - Set numLevelZeroTables=1 to force compaction. This PR also fixes the failing bank test by adding an index cache to the bank test.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement multiple ideas for speeding up compactions:
With these changes, we can load 1B entries (160GB of data) into Badger (without the Stream framework) in 1h25m at 31 MB/s. This is a significant improvement over current master.
This change is