Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory spike when compacting overwritten points #6567

Merged
merged 1 commit into from
May 6, 2016
Merged

Conversation

jwilder
Copy link
Contributor

@jwilder jwilder commented May 6, 2016

  • Rebased/mergable
  • Tests pass
  • CHANGELOG.md updated

If a large series contains a point that is overwritten, the compactor
would load the whole series into RAM during a full compaction and deduplicate
the points. If the series was large, it could cause very large RAM spikes and OOMs.

This change reworks the compactor to merge blocks more incrementally
similar to the fix done in #6556. It also replaces the deduplication method with
a sorted merge approache used in #6556.

To reproduce the issue, I wrote 25M points to a single series and then overwrote the first point. When a full compaction runs, RSS spikes up quite drastically.

There was also a bug discovered in the newly introduced *Values.Merge funcs where the values being merged could end up unsorted. This is now fixed.

Fixes #6557

Some before and after stats:

Before

$ ps -o rss,vsz $(pgrep influxd)
   RSS      VSZ
3148776 145316856
[tsm1] 2016/05/05 21:53:49 compacted full 3 files into 1 files in 48.80575863s

After

$ ps -o rss,vsz $(pgrep influxd)
   RSS      VSZ
 32584 145292860

[tsm1] 2016/05/05 21:48:00 compacted full 4 files into 1 files in 8.040986536s

After compaction completed, RSS decreased from 3.1GB to 32MB and compaction time decreased from 49s to 8s.

If a large series contains a point that is overwritten, the compactor
would load the whole series into RAM during a full compaction.  If
the series was large, it could cause very large RAM spikes and OOMs.

The change reworks the compactor to merge blocks more incrementally
similar to the fix done in #6556.
@jwilder
Copy link
Contributor Author

jwilder commented May 6, 2016

@benbjohnson @e-dard

@benbjohnson
Copy link
Contributor

👍

@jwilder jwilder merged commit b4f922d into master May 6, 2016
@jwilder jwilder deleted the jw-tsm-values branch May 6, 2016 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Overwriting points on large series can cause memory spikes during compactions
2 participants