Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: discard excessive errors #22379

Merged
merged 2 commits into from
Sep 3, 2021
Merged

fix: discard excessive errors #22379

merged 2 commits into from
Sep 3, 2021

Conversation

davidby-influx
Copy link
Contributor

The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328

@davidby-influx davidby-influx self-assigned this Sep 2, 2021
@davidby-influx
fix: discard excessive errors
c4b80d5 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328
@davidby-influx davidby-influx linked an issue Sep 2, 2021 that may be closed by this pull request
(*tsmBatchKeyIterator) should not store too many errors. #22328
Closed
gwossum
gwossum reviewed Sep 2, 2021
View reviewed changes
tsdb/engine/tsm1/compact.go Outdated Show resolved Hide resolved
gwossum
gwossum approved these changes Sep 3, 2021
View reviewed changes
Copy link
Member

@gwossum gwossum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Glad you tested that the last error message was the proper error overflow message.

@davidby-influx davidby-influx merged commit e53f75e into master-1.x Sep 3, 2021
@davidby-influx davidby-influx deleted the DSB_append_error branch September 3, 2021 16:11
davidby-influx added a commit that referenced this pull request Sep 3, 2021
@davidby-influx
fix: discard excessive errors (#22379)
86d37de 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328

(cherry picked from commit e53f75e)
davidby-influx added a commit that referenced this pull request Sep 3, 2021
@davidby-influx
fix: discard excessive errors (#22379) (#22389)
25e21c9 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328

(cherry picked from commit e53f75e)

Closes #22380
davidby-influx added a commit that referenced this pull request Sep 3, 2021
@davidby-influx
fix: discard excessive errors (#22379)
7677a71 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328

(cherry picked from commit e53f75e)
davidby-influx added a commit that referenced this pull request Sep 3, 2021
@davidby-influx
fix: discard excessive errors (#22379)
e513576 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328

(cherry picked from commit e53f75e)
davidby-influx added a commit that referenced this pull request Sep 3, 2021
@davidby-influx
fix: discard excessive errors (#22379)
9a0a4f3 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328

(cherry picked from commit e53f75e)
davidby-influx added a commit that referenced this pull request Sep 3, 2021
@davidby-influx
fix: discard excessive errors (#22379) (#22391)
7ad612b 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328

(cherry picked from commit e53f75e)

closes #22381
davidby-influx added a commit that referenced this pull request Sep 3, 2021
@davidby-influx
fix: discard excessive errors (#22379) (#22397)
c0dd3d3 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes #22328

(cherry picked from commit e53f75e)

closes #22382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gwossum gwossum gwossum approved these changes

Assignees

@davidby-influx davidby-influx

Labels
1.x area/error handling area/tsm kind/bug team/edge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

(*tsmBatchKeyIterator) should not store too many errors.
2 participants
@davidby-influx @gwossum