Fix chance of incomplete read in partial response stream reading #778
Conversation
|
Doesn't this "undo" the streaming feature? |
|
I see. Perhaps there's something else needed to prevent the write from finishing too early if more content is expected? |
Codecov Report
@@ Coverage Diff @@
## master #778 +/- ##
==========================================
- Coverage 77.47% 76.60% -0.88%
==========================================
Files 38 38
Lines 2562 2573 +11
==========================================
- Hits 1985 1971 -14
- Misses 577 602 +25
Continue to review full report at Codecov.
|
I think I just updated the fix to allow for streaming? Could someone run the CI please 🙏🏻 |
IanButterworth
force-pushed
the
ib/fix_partial_response_stream
branch
from
21c1505 to
c0ac2f3
Compare
IanButterworth
force-pushed
the
ib/fix_partial_response_stream
branch
from
0d698cf to
76b6fcd
Compare
|
Tests passing on my fork IanButterworth#1 |
| @@ -303,12 +303,23 @@ function Base.unsafe_read(http::Stream, p::Ptr{UInt8}, n::UInt) | |||
| nothing | |||
| end | |||
|
|
|||
| Base.readbytes!(http::Stream, buf::Base.BufferStream, n=bytesavailable(http)) = Base.readbytes!(http, buf.buffer, n) | |||
There was a problem hiding this comment.
Why did you have to special case BufferStream and IOBuffer? Will this fail with other IO now?
…ing (JuliaWeb#778)" This reverts commit 80815e3.
…ing (JuliaWeb#778)" This reverts commit 80815e3.
…ing (JuliaWeb#778)" This reverts commit 80815e3.
This appears to fix a bug seen over in JuliaCloud/AWS.jl#515 where a 206 partial request response has the correct Content-Length in the header but HTTP.jl returns fewer bytes with no error.
The bug is seen most frequently when running many
@asyncHTTP.requests each requesting a dataRangeon slower internet connections.The problem appears to be that a simple
write(response_stream, http)bypassed thentoreadhandling here that ensures successive reads were done if the promised content length wasn't fully retrieved the first time, which allows for the buffer to populate slowly on slower connections, so it needs to bewrite(response_stream, read(http))instead