fix: Content-Encoding: gzip
along with Transfer-Encoding: chunked
sometimes terminates early
#1608
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The issue
When
GZIPInputStream
completes processing an individual member it will callInputStream#available()
to determine if there is more stream to try and process. If the call to
available()
returns 0GZIPInputStream
will determine it has processed the entirety of the underlying stream. This isspurious, as
InputStream#available()
is allowed to return 0 if it would require blocking in orderfor more bytes to be available. When
GZIPInputStream
is reading from aTransfer-Encoding: chunked
response, if the chunk boundary happens to align closely enough to the member boundary
GZIPInputStream
won't consume the whole response.The fix
Add new
OptimisticAvailabilityInputStream
, which provides an optimistic "estimate" of the number ofavailable()
bytes in the underlying stream. When instantiating aGZIPInputStream
for a response,automatically decorate the provided
InputStream
with anOptimisticAvailabilityInputStream
.Verification
This scenario isn't unique to processing of chunked responses, and can be replicated reliably using
a
java.io.SequenceInputStream
with two underlyingjava.io.ByteArrayInputStream
. SeeGzipSupportTest.java for a reproduction.
The need for this class has been verified for the following JVMs: