Fix memory spike when compacting overwritten points #6567
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If a large series contains a point that is overwritten, the compactor
would load the whole series into RAM during a full compaction and deduplicate
the points. If the series was large, it could cause very large RAM spikes and OOMs.
This change reworks the compactor to merge blocks more incrementally
similar to the fix done in #6556. It also replaces the deduplication method with
a sorted merge approache used in #6556.
To reproduce the issue, I wrote 25M points to a single series and then overwrote the first point. When a full compaction runs, RSS spikes up quite drastically.
There was also a bug discovered in the newly introduced
*Values.Merge
funcs where the values being merged could end up unsorted. This is now fixed.Fixes #6557
Some before and after stats:
Before
After
After compaction completed, RSS decreased from 3.1GB to 32MB and compaction time decreased from 49s to 8s.