Gzip decoder returns too early while consuming TCP fragmented chuncked body #2381
Comments
|
Seems like I've found out a reason. reqwest/src/async_impl/decoder.rs Lines 468 to 472 in 8c7f338 #[cfg(feature = "gzip")]
DecoderType::Gzip => Poll::Ready(Ok(Inner::Gzip(Box::pin(FramedRead::new(
{
let mut d = GzipDecoder::new(StreamReader::new(_body));
d.multiple_members(true);
d
},
BytesCodec::new(),
))))),And now it works as expected! Should all other decoders be created with this option enabled? I think yes. Should this option be enabled only if
But response processing hangs up when neither And I'm not sure how multiple-resource bodies should be processed. https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages#body_2 |
…nstar#2484) When a response body is being decompressed, and the length wasn't known, but was using chunked transfer-encoding, the remaining `0\r\n\r\n` was not consumed. That would leave the connection in a state that could be not be reused, and so the pool had to discard it. This fix makes sure the remaining end chunk is consumed, improving the amount of pooled connections that can be reused. Closes seanmonstar#2381
Hello. I believe a found a bug. It doesn't make HTTP response incorrect but it effectively disables connection reusing.
I drew a diagram for such case.
As you can see Client (with activated
gzipfeature) sends request to server. Server responds with chunked gzipped body, but body somehow is fragmented at TCP level (see diagram) and those fragments are divided by some time (milliseconds at least, so you can'tread()them together with one call). But first fragment contains whole gzipped bytes.Gzip decoder returns (and, as consequence, Response::text() call too) immediately after decoding last encoded byte, i.e. it doesn't wait for full response body arrival. And then Response is dropped (because Response::text() consumes Response). And it prevents connection from being put back in pool and being reused (because connection itself is still waiting for bytes from server).
I created minimal reproduction https://github.com/Andrey36652/reqwest-chunked-bug where client shoud make 3 consecutive requests to the server that responds with fragmented body (there is a 4 seconds delay between the fragments, so it should be ~12 seconds in total). But when I run it, all 3 requests are resolved immediately as you can see from timestamps (and each uses new connection).
If I disable gzip deconding in Client (uncomment line in
src/client.rs) then behaviour is correct - each request waits until full body will be consumed and returns only after that moment.I don't know if an error is in
reqwestor inasync-compressioncrate, and I'm afraid my knowledge isn't enough to find a solution.The text was updated successfully, but these errors were encountered: