changefeedccl: improve performance of gzip, and add zstd compression #88635
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
shermanCRL
changed the title
HonoreDB
approved these changes
Sep 23, 2022
There was a problem hiding this comment.
Reviewed 6 of 6 files at r1, 4 of 4 files at r2, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @stevendanna)
miretskiy
force-pushed
the
gzip
branch
5 times, most recently
from
5a9c603 to
569a1c4
Compare
stevendanna
approved these changes
Sep 26, 2022
pkg/ccl/changefeedccl/compresion.go
Outdated
| case sinkCompressionZstd: | ||
| return zstd.NewWriter(dest, zstd.WithEncoderLevel(zstd.SpeedFastest)) | ||
| default: | ||
| return nil, errors.AssertionFailedf("unsupported encoder algorithm %q", algo) |
There was a problem hiding this comment.
Perhaps?
Suggested change
| return nil, errors.AssertionFailedf("unsupported encoder algorithm %q", algo) | |
| return nil, errors.AssertionFailedf("unsupported compression algorithm %q", algo) |
|
cc @nollenr |
miretskiy
force-pushed
the
gzip
branch
3 times, most recently
from
116c20f to
c3dd28b
Compare
shermanCRL
reviewed
Sep 27, 2022
pkg/ccl/changefeedccl/compresion.go
Outdated
| @@ -0,0 +1,70 @@ | |||
| // Copyright 2022 The Cockroach Authors. | |||
There was a problem hiding this comment.
Might want to correct the spelling on the file name.
There was a problem hiding this comment.
Ohh.... Ooops... Or .. maybe it was compres'd?
Expand the set of supported compression algorithms in changefeed. A faster implementation of gzip algorithm is avaible, and is used by default. The gzip algorithm implementation can be reverted to Go standard gzip implementation via the `changefeed.fast_gzip.enabled` setting. In addition, add support for compression files with zstd. Release notes (enterprise change): Changefeed can emit files compressed with zstd algorithm -- which provides good compression, and is much faster than gzip. In addition, a new, faster implementation of gzip is used by default.
|
bors r+ |
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build succeeded: |
|
Backported to v22.2.1 via #91002 |
miretskiy
pushed a commit
to miretskiy/cockroach
that referenced
this pull request
Jul 13, 2023
Ensure resources acquired by cloud storage files are released when the sink is closed. As of cockroachdb#88635, cloud storage uses faster implementation of gzip compression algorithm (along with zstd). This new implementation is sufficiently different from the standard gzip implementation in that it requires the compression codec to be closed, even when the caller is terminating. Failure to do so results in the memory as well as the goroutine leakage. This resource leakage may become sufficiently noticable if the changefeed experiences many repeated errors. This PR modifies Close() call to make sure that the underlying compression codecs are also closed (Note: we rely on the high level logic in distSQL to ensure that the processor gets orderly shut down, and the shutdown code calls Close() method; However, there is still exists a possiblity that the shutdown might not be orderly, and in those cases resource leakage may still occur. This possiblity will need to be revisited in the follow on PR). Fixes cockroachdb#106774 Release note (enterprise change): Fix an issue where the changefeeds emitting to cloud sink with compression may experience resource leakage (memory and go routines) when experiencing transient errors.
craig bot
pushed a commit
that referenced
this pull request
Jul 14, 2023
106786: util/log: allow custom crash report tags r=pjtatlow a=pjtatlow Today it can be difficult to trace back a sentry event to the CC cluster where it originated, especially for serverless clusters. This change enables a new environment variable (COCKROACH_CRASH_REPORT_TAGS), which allows the database operator to provide additional information that will be included in the sentry event. Release Note: None 106795: changefeedccl: Cleanup resources when closing file r=miretskiy a=miretskiy Ensure resources acquired by cloud storage files are released when the sink is closed. As of #88635, cloud storage uses faster implementation of gzip compression algorithm (along with zstd). This new implementation is sufficiently different from the standard gzip implementation in that it requires the compression codec to be closed, even when the caller is terminating. Failure to do so results in the memory as well as the goroutine leakage. This resource leakage may become sufficiently noticable if the changefeed experiences many repeated errors. This PR modifies Close() call to make sure that the underlying compression codecs are also closed (Note: we rely on the high level logic in distSQL to ensure that the processor gets orderly shut down, and the shutdown code calls Close() method; However, there is still exists a possiblity that the shutdown might not be orderly, and in those cases resource leakage may still occur. This possiblity will need to be revisited in the follow on PR). Fixes #106774 Release note (enterprise change): Fix an issue where the changefeeds emitting to cloud sink with compression may experience resource leakage (memory and go routines) when experiencing transient errors. 106827: insights: fix flakey TestInsightsIntegrationForContention r=j82w a=j82w The test is flakey sometimes because there was a minimum time check the contention duration. The problem is other parts of the crdb could be slow which caused the contention time to be less than expected. The test now only checks if the value is greater than 0 and is less than 1 minute. waiting_txn_fingerprint_id should always have a value. Fixed the check to make sure it's not the default value and renamed variables to match the value. Fixes: #106622 Release note: None Epic: none 106840: schemachanger: Fix CREATE SEQUENCE OWNED BY failure r=Xiang-Gu a=Xiang-Gu Previously, stmts like `CREATE SEQUENCE s OWNED BY col` where table name is missing will fail with an internal error. This commit fix this. Fix #106838 Release note: None Co-authored-by: PJ Tatlow <[email protected]> Co-authored-by: Yevgeniy Miretskiy <[email protected]> Co-authored-by: j82w <[email protected]> Co-authored-by: Xiang Gu <[email protected]>
miretskiy
pushed a commit
that referenced
this pull request
Jul 14, 2023
Ensure resources acquired by cloud storage files are released when the sink is closed. As of #88635, cloud storage uses faster implementation of gzip compression algorithm (along with zstd). This new implementation is sufficiently different from the standard gzip implementation in that it requires the compression codec to be closed, even when the caller is terminating. Failure to do so results in the memory as well as the goroutine leakage. This resource leakage may become sufficiently noticable if the changefeed experiences many repeated errors. This PR modifies Close() call to make sure that the underlying compression codecs are also closed (Note: we rely on the high level logic in distSQL to ensure that the processor gets orderly shut down, and the shutdown code calls Close() method; However, there is still exists a possiblity that the shutdown might not be orderly, and in those cases resource leakage may still occur. This possiblity will need to be revisited in the follow on PR). Fixes #106774 Release note (enterprise change): Fix an issue where the changefeeds emitting to cloud sink with compression may experience resource leakage (memory and go routines) when experiencing transient errors.
miretskiy
pushed a commit
that referenced
this pull request
Jul 14, 2023
Ensure resources acquired by cloud storage files are released when the sink is closed. As of #88635, cloud storage uses faster implementation of gzip compression algorithm (along with zstd). This new implementation is sufficiently different from the standard gzip implementation in that it requires the compression codec to be closed, even when the caller is terminating. Failure to do so results in the memory as well as the goroutine leakage. This resource leakage may become sufficiently noticable if the changefeed experiences many repeated errors. This PR modifies Close() call to make sure that the underlying compression codecs are also closed (Note: we rely on the high level logic in distSQL to ensure that the processor gets orderly shut down, and the shutdown code calls Close() method; However, there is still exists a possiblity that the shutdown might not be orderly, and in those cases resource leakage may still occur. This possiblity will need to be revisited in the follow on PR). Fixes #106774 Release note (enterprise change): Fix an issue where the changefeeds emitting to cloud sink with compression may experience resource leakage (memory and go routines) when experiencing transient errors.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Expand the set of supported compression algorithms in changefeed.
A faster implementation of gzip algorithm is avaible, and is used
by default. The gzip algorithm implementation can be reverted
to Go standard gzip implementation via the
changefeed.fast_gzip.enabledsetting.In addition, add support for compression files with zstd.
Addresses #88585
Release notes (enterprise change): Changefeed can emit files compressed
with zstd algorithm -- which provides good compression, and is much
faster than gzip. In addition, a new, faster implementation of
gzip is used by default.