Implement streaming multipart zstd compression. #1413
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
obi1kenobi
force-pushed
the
pg/streaming-multipart-zstd
branch
from
5f8ee11 to
f3729d4
Compare
obi1kenobi
force-pushed
the
pg/streaming-multipart-zstd
branch
from
f3729d4 to
a2bd2a7
Compare
agola11
approved these changes
Jan 15, 2025
obi1kenobi
added a commit
that referenced
this pull request
Jan 16, 2025
Builds upon #1413 and uses that functionality in the PyO3 crate. This is a major breaking change in the API, so make it be part of a v0.2.0-rc1 release.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uses a hand-written streaming multipart writer that can wrap a zstd compressor, so that we try to not keep
QueuedRunvalues in memory any longer than strictly necessary.A "TODO" for tomorrow's revamp of the error types is to switch from unbounded to bounded channels and either use the
QueueFullerror variant or just block until the channel frees up. @agola11 do you have a preference one way or the other? This governs what happens in the rare instance if e.g. the server is much too slow at ingesting data and traces are piling up on the client. We can either drop data, or force the client to slow down, and this is a product-level decision.It's a good idea to use bounded channels over unbounded ones, even if the bound is huge and shouldn't ever be hit. Bounded channels are both more efficient and can more gracefully handle excessive load. Unbounded channels grow until OOM and then kill the program, which is a worse experience.