Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix TSM tmp file lingering on disk #8416

Merged
merged 2 commits into from
May 23, 2017
Merged

Fix TSM tmp file lingering on disk #8416

merged 2 commits into from
May 23, 2017
+13 −12

Conversation

jwilder
Copy link
Contributor

@jwilder jwilder commented May 22, 2017

Required for all non-trivial PRs
  • Rebased/mergable
  • Tests pass
  • CHANGELOG.md updated
  • Sign CLA (if not already signed)

This fixes some cases where TSM tmp files could remain on disk as well as tmp files being deleted incorrectly causing errors in the logs and compactions to be aborted. If it occurs frequently enough, space is not reclaimed until a restart occurs.

  • Running a delete or drop would disable compactions and also run cleanup which would delete all tmp files. Since snapshot compactions are not disabled, any in progress one will fail causing Error adding new TSM files from snapshot: rename XXX.tsm.tmp XXX.tsm: no such file or directory #8387. This could also cause failures where shard snapshots are used such as backups.
  • There are some cases where a failed compaction would not remove its tmp file. Previously, compactions could get scheduled for the same files and if both deleted the tmp file neither compaction would succeed and they would get stuck. The recent changes to the compaction planning prevent this and remove failed tmp files due to compaction failures.

Fixes #8387

jwilder added 2 commits May 22, 2017 14:51
@jwilder
Fix TSM tmp files leaking
1833475 
TMP files could leak when compactions failed for various reasons. They
were also being deleted inadvertently when compactions were disabled causing
other errors to be reported in the logs.
@jwilder
Update changelog
31d2309
@jwilder jwilder added this to the 1.3.0 milestone May 22, 2017
@jwilder jwilder requested a review from benbjohnson May 22, 2017 21:01
@jwilder jwilder added the review label May 22, 2017
@jwilder jwilder merged commit 5619946 into master May 23, 2017
@jwilder jwilder deleted the jw-tsm-tmp branch May 23, 2017 16:12
@jwilder jwilder removed the review label May 23, 2017
@jwilder jwilder mentioned this pull request May 23, 2017
Closed
@jwilder jwilder mentioned this pull request Jun 26, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@benbjohnson benbjohnson benbjohnson approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.3.0
Development

Successfully merging this pull request may close these issues.

Error adding new TSM files from snapshot: rename XXX.tsm.tmp XXX.tsm: no such file or directory
2 participants
@jwilder @benbjohnson