changefeedccl: Cleanup resources when closing file #106795
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ensure resources acquired by cloud storage files are released when the sink is closed. As of cockroachdb#88635, cloud storage uses faster implementation of gzip compression algorithm (along with zstd). This new implementation is sufficiently different from the standard gzip implementation in that it requires the compression codec to be closed, even when the caller is terminating. Failure to do so results in the memory as well as the goroutine leakage. This resource leakage may become sufficiently noticable if the changefeed experiences many repeated errors. This PR modifies Close() call to make sure that the underlying compression codecs are also closed (Note: we rely on the high level logic in distSQL to ensure that the processor gets orderly shut down, and the shutdown code calls Close() method; However, there is still exists a possiblity that the shutdown might not be orderly, and in those cases resource leakage may still occur. This possiblity will need to be revisited in the follow on PR). Fixes cockroachdb#106774 Release note (enterprise change): Fix an issue where the changefeeds emitting to cloud sink with compression may experience resource leakage (memory and go routines) when experiencing transient errors.
miretskiy
added
backport-22.1.x
backport-23.1.x
|
This change is |
Proactively close compression codecs if an error occurs when writing to cloud storage file. While we always call Close on the sink object, this PR ensures that even if this doesn't happen, the resources used by compression code are released as soon as an error (during write) occurs. Informs cockroachdb#106774 Release note: None
jayshrivastava
approved these changes
Jul 14, 2023
There was a problem hiding this comment.
Reviewable status:
complete! 1 of 0 LGTMs obtained (waiting on @dt and @HonoreDB)
|
bors r+ |
|
Build succeeded: |
This was referenced Jul 14, 2023
|
Encountered an error creating backports. Some common things that can go wrong:
You might need to create your backport manually using the backport tool. error creating merge commit from 8d0a804 to blathers/backport-release-22.1-106795: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict [] you may need to manually resolve merge conflicts with the backport tool. Backport to branch 22.1.x failed. See errors above. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
22.1 didn't have this functionality; so ignore backport failure |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ensure resources acquired by cloud storage files are released when the sink is closed.
As of #88635, cloud storage uses faster implementation of gzip compression algorithm (along with zstd). This new implementation is sufficiently different from the standard gzip implementation in that it requires the compression codec to be closed, even when the caller is terminating. Failure to do so results in the memory as well as the goroutine leakage.
This resource leakage may become sufficiently noticable if the changefeed experiences many repeated errors.
This PR modifies Close() call to make sure that the underlying compression codecs are also closed (Note: we rely on the high level logic in distSQL to ensure that the processor gets orderly shut down, and the shutdown code calls Close() method; However, there is still exists a possiblity that the shutdown might not be orderly, and in those cases resource leakage may still occur. This possiblity will need to be revisited in the follow on PR).
Fixes #106774
Release note (enterprise change): Fix an issue where the changefeeds emitting to cloud sink with compression may experience resource leakage (memory and go routines) when experiencing transient errors.