Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue using Accept-Encoding: gzip and GzipDecompressorStream on chunked requests #859

Closed
Pangoraw opened this issue Jun 19, 2022 · 2 comments · Fixed by #861
Closed

Issue using Accept-Encoding: gzip and GzipDecompressorStream on chunked requests #859

Pangoraw opened this issue Jun 19, 2022 · 2 comments · Fixed by #861

Comments

@Pangoraw
Copy link

Pangoraw commented Jun 19, 2022 •
edited
Loading

While trying the new 1.0 release, I am having an issue making a HTTP request to a website returning a gzip encoded payload. Since the Accept-Encoding: gzip is now set automatically this can lead to failures in the default case. To reproduce:

julia> HTTP.get("https://github.com") # or with explicit headers:
julia> HTTP.get("https://github.com"; headers = Dict("Accept-Encoding" => "gzip"))
┌ Warning: Reading one byte at a time from HTTP.Stream is inefficient.
│ Use: io = BufferedInputStream(http::HTTP.Stream) instead.
│ See: https://github.com/BioJulia/BufferedStreams.jl
└ @ HTTP.Streams ~/Projects/HTTP.jl/src/Streams.jl:241
ERROR: HTTP.Exceptions.RequestError(HTTP.Messages.Request:
"""
GET / HTTP/1.1
Accept-Encoding: gzip
Host: github.com
Accept: */*
User-Agent: HTTP.jl/1.7.3
Content-Length: 0

""", ErrorException("zlib error: <no message> (code: -5)"))
Complete stack trace 
julia> HTTP.get("https://github.com"; headers = Dict("Accept-Encoding" => "gzip"))
┌ Warning: Reading one byte at a time from HTTP.Stream is inefficient.
│ Use: io = BufferedInputStream(http::HTTP.Stream) instead.
│ See: https://github.com/BioJulia/BufferedStreams.jl
└ @ HTTP.Streams ~/Projects/HTTP.jl/src/Streams.jl:241
ERROR: HTTP.Exceptions.RequestError(HTTP.Messages.Request:
"""
GET / HTTP/1.1
Accept-Encoding: gzip
Host: github.com
Accept: */*
User-Agent: HTTP.jl/1.7.3
Content-Length: 0

""", ErrorException("zlib error: <no message> (code: -5)"))
Stacktrace:
  [1] (::HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.ConnectionRequest ~/Projects/HTTP.jl/src/clientlayers/ConnectionRequest.jl:111
  [2] (::HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; canonicalize_headers::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.CanonicalizeRequest ~/Projects/HTTP.jl/src/clientlayers/CanonicalizeRequest.jl:17
  [3] (::Base.var"#76#78"{Base.var"#76#77#79"{ExponentialBackOff, HTTP.RetryRequest.var"#3#6"{Int64, HTTP.Messages.Request, Base.RefValue{Int64}}, HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(args::HTTP.Messages.Request; kwargs::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ Base ./error.jl:294
  [4] (::HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_non_idempotent::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.RetryRequest ~/Projects/HTTP.jl/src/clientlayers/RetryRequest.jl:59
  [5] (::HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.CookieRequest ~/Projects/HTTP.jl/src/clientlayers/CookieRequest.jl:42
  [6] (::HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; detect_content_type::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.ContentTypeDetection ~/Projects/HTTP.jl/src/clientlayers/ContentTypeRequest.jl:23
  [7] (::HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; basicauth::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.BasicAuthRequest ~/Projects/HTTP.jl/src/clientlayers/BasicAuthRequest.jl:22
  [8] (::HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Bool, kw::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Int64}}})
    @ HTTP.DefaultHeadersRequest ~/Projects/HTTP.jl/src/clientlayers/DefaultHeadersRequest.jl:47
  [9] (::HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Int64}}})
    @ HTTP.RedirectRequest ~/Projects/HTTP.jl/src/clientlayers/RedirectRequest.jl:25
 [10] #1#2
    @ ~/Projects/HTTP.jl/src/clientlayers/DebugRequest.jl:22 [inlined]
 [11] (::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::Vector{UInt8}; response_stream::Nothing, http_version::VersionNumber, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ HTTP.MessageRequest ~/Projects/HTTP.jl/src/clientlayers/MessageRequest.jl:17
 [12] #1
    @ ~/Projects/HTTP.jl/src/clientlayers/MessageRequest.jl:16 [inlined]
 [13] request(stack::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}}, method::String, url::String, h::Dict{String, String}, b::Vector{UInt8}, q::Nothing; headers::Dict{String, String}, body::Vector{UInt8}, query::Nothing, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ HTTP ~/Projects/HTTP.jl/src/HTTP.jl:391
 [14] request(stack::Function, method::String, url::String, h::Dict{String, String}, b::Vector{UInt8}, q::Nothing)
    @ HTTP ~/Projects/HTTP.jl/src/HTTP.jl:391
 [15] #request#16
    @ ~/Projects/HTTP.jl/src/HTTP.jl:272 [inlined]
 [16] #get#30
    @ ~/Projects/HTTP.jl/src/HTTP.jl:423 [inlined]
 [17] top-level scope
    @ REPL[2]:1

caused by: zlib error: <no message> (code: -5)
Stacktrace:
  [1] changemode!(stream::TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompressor, HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}}, newmode::Symbol)
    @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:717
  [2] callprocess(stream::TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompressor, HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}}, inbuf::TranscodingStreams.Buffer, outbuf::TranscodingStreams.Buffer)
    @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:649
  [3] fillbuffer(stream::TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompressor, HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}}; eager::Bool)
    @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:577
  [4] fillbuffer
    @ ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:564 [inlined]
  [5] eof(stream::TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompressor, HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}})
    @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:188
  [6] readbytes!(stream::TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompressor, HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}}, b::Vector{UInt8}, nb::Int64)
    @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:371
  [7] read(s::TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompressor, HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}}, nb::Int64)
    @ Base ./io.jl:965
  [8] read
    @ ./io.jl:964 [inlined]
  [9] readbody(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}, res::HTTP.Messages.Response, decompress::Bool)
    @ HTTP.StreamRequest ~/Projects/HTTP.jl/src/clientlayers/StreamRequest.jl:108
 [10] macro expansion
    @ ~/Projects/HTTP.jl/src/clientlayers/StreamRequest.jl:47 [inlined]
 [11] macro expansion
    @ ./task.jl:399 [inlined]
 [12] streamlayer(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}; iofunction::Nothing, decompress::Bool, kw::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Int64}}})
    @ HTTP.StreamRequest ~/Projects/HTTP.jl/src/clientlayers/StreamRequest.jl:34
 [13] (::HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}})(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}; status_exception::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.ExceptionRequest ~/Projects/HTTP.jl/src/clientlayers/ExceptionRequest.jl:14
 [14] (::HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}; readtimeout::Int64, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.TimeoutRequest ~/Projects/HTTP.jl/src/clientlayers/TimeoutRequest.jl:17
 [15] (::HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.ConnectionRequest ~/Projects/HTTP.jl/src/clientlayers/ConnectionRequest.jl:101
 [16] (::HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; canonicalize_headers::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.CanonicalizeRequest ~/Projects/HTTP.jl/src/clientlayers/CanonicalizeRequest.jl:17
 [17] (::Base.var"#76#78"{Base.var"#76#77#79"{ExponentialBackOff, HTTP.RetryRequest.var"#3#6"{Int64, HTTP.Messages.Request, Base.RefValue{Int64}}, HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(args::HTTP.Messages.Request; kwargs::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ Base ./error.jl:294
 [18] (::HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_non_idempotent::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.RetryRequest ~/Projects/HTTP.jl/src/clientlayers/RetryRequest.jl:59
 [19] (::HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.CookieRequest ~/Projects/HTTP.jl/src/clientlayers/CookieRequest.jl:42
 [20] (::HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; detect_content_type::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.ContentTypeDetection ~/Projects/HTTP.jl/src/clientlayers/ContentTypeRequest.jl:23
 [21] (::HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; basicauth::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
    @ HTTP.BasicAuthRequest ~/Projects/HTTP.jl/src/clientlayers/BasicAuthRequest.jl:22
 [22] (::HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Bool, kw::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Int64}}})
    @ HTTP.DefaultHeadersRequest ~/Projects/HTTP.jl/src/clientlayers/DefaultHeadersRequest.jl:47
 [23] (::HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Int64}}})
    @ HTTP.RedirectRequest ~/Projects/HTTP.jl/src/clientlayers/RedirectRequest.jl:25
 [24] #1#2
    @ ~/Projects/HTTP.jl/src/clientlayers/DebugRequest.jl:22 [inlined]
 [25] (::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::Vector{UInt8}; response_stream::Nothing, http_version::VersionNumber, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ HTTP.MessageRequest ~/Projects/HTTP.jl/src/clientlayers/MessageRequest.jl:17
 [26] #1
    @ ~/Projects/HTTP.jl/src/clientlayers/MessageRequest.jl:16 [inlined]
 [27] request(stack::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}}, method::String, url::String, h::Dict{String, String}, b::Vector{UInt8}, q::Nothing; headers::Dict{String, String}, body::Vector{UInt8}, query::Nothing, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ HTTP ~/Projects/HTTP.jl/src/HTTP.jl:391
 [28] request(stack::Function, method::String, url::String, h::Dict{String, String}, b::Vector{UInt8}, q::Nothing)
    @ HTTP ~/Projects/HTTP.jl/src/HTTP.jl:391
 [29] #request#16
    @ ~/Projects/HTTP.jl/src/HTTP.jl:272 [inlined]
 [30] #get#30
    @ ~/Projects/HTTP.jl/src/HTTP.jl:423 [inlined]
 [31] top-level scope
    @ REPL[2]:1

I believe the error is caused by the combination of HTTP.Stream and GZipDecompressorStream (note the warning that reading one byte from the stream is inneficient) because reading the encoded buffer and then decoding it does not fail:

julia> r = HTTP.get("https://github.com"; headers = Dict("Accept-Encoding" => "gzip"), decompress = false); # disable decompressing

julia> HTTP.decode(r, "gzip") |> String # decompress after the stream as been read
"\n\n\n\n\n\n<!DOCTYPE html>\n<html lang=\"en\"  data-a11y-animated-images=\"system\">\n  <head>\n    <meta charset=\"utf-8\">\n  <link rel=\"dns-prefetch\" href=\"https://github.githubassets.com\">\n  <link rel=\"dns-prefetch\" href=\"https://avatars.githubusercontent.com\">\n  <link rel=\"dns-prefetch\" href=\"https://github-cloud.s3.amazonaws.com\">\n  <link rel=\"dns-prefetch\" href=\"https://user-images.githubusercontent.com/\">\n  <link rel=\"preconnect\" href=\"https://github.githubassets.com\"" ⋯ 209836 bytes ⋯ "25h-7.5z\"></path>\n</svg>\n      <svg aria-hidden=\"true\" height=\"16\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" data-view-component=\"true\" class=\"octicon octicon-check js-clipboard-check-icon color-fg-success d-none m-2\">\n    <path fill-rule=\"evenodd\" d=\"M13.78 4.22a.75.75 0 010 1.06l-7.25 7.25a.75.75 0 01-1.06 0L2.22 9.28a.75.75 0 011.06-1.06L6 10.94l6.72-6.72a.75.75 0 011.06 0z\"></path>\n</svg>\n    </clipboard-copy>\n  </div>\n</template>\n\n\n\n\n  </body>\n</html>\n\n"

Please specify the following versions when submitting a bug report:

  • Julia 1.7.3
  • HTTP.jl 1.0.1 (master)
  • MbedTLS.jl 1.0.3
@Pangoraw
Copy link
Author

Pangoraw commented Jun 19, 2022 •
edited
Loading

Using ENV["JULIA_DEBUG"] = "TranscodingStreams" and comparing the debug output for both types of streams (HTTP.Stream and IOBuffer) I saw that the HTTP.Stream version does not stop when the :end state is reached thus getting this zlib error.

With HTTP.Stream 
┌ Debug: called process()
│   code = :ok
│   input_size = 1370
│   output_size = 16384
│   input_delta = 1370
│   output_delta = 6474
└ @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:638
┌ Debug: called process()
│   code = :end
│   input_size = 143
│   output_size = 16384
│   input_delta = 143
│   output_delta = 603
└ @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:638
┌ Debug: called process()
│   code = :error
│   input_size = 0
│   output_size = 16384
│   input_delta = 0
│   output_delta = 0
└ @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/stream.jl:638
With IOBuffer 
┌ Debug: called process()
│   code = :ok
│   input_size = 8996
│   output_size = 36059
│   input_delta = 5375
│   output_delta = 36059
└ @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/transcode.jl:91
┌ Debug: called process()
│   code = :end
│   input_size = 3621
│   output_size = 36059
│   input_delta = 3621
│   output_delta = 30478
└ @ TranscodingStreams ~/.julia/packages/TranscodingStreams/IVlnc/src/transcode.jl:91

The problem is that eof is returning false when the decoding is done (7 bytes are remaining in the stream.io::Connection.buffer), causing the decoding to start again with error Z_BUF_ERROR because this is not a valid zlib buffer. I found that adding stop_on_end=true when creating the TranscodingStream solves the issue but it may be that other chunks are also present in the request and won't be read:

diff --git a/src/clientlayers/StreamRequest.jl b/src/clientlayers/StreamRequest.jl
index e4ea889..df5633b 100644
--- a/src/clientlayers/StreamRequest.jl
+++ b/src/clientlayers/StreamRequest.jl
@@ -103,7 +103,7 @@ writechunk(stream, body::Union{Dict, NamedTuple}) = writebodystream(stream, body
 writechunk(stream, body) = write(stream, body)

 function readbody(stream::Stream, res::Response, decompress)
-    readstream = decompress && header(res, "Content-Encoding") == "gzip" ? GzipDecompressorStream(stream) : stream
+    readstream = decompress && header(res, "Content-Encoding") == "gzip" ? GzipDecompressorStream(stream; stop_on_end=true) : stream
     if isbytes(res.body)
         res.body = read(readstream)
     elseif !isredirect(stream) && !retryable(stream)

@Pangoraw Pangoraw changed the title Issue using Accept-Encoding: gzip and GZipDecompressorStream Issue using Accept-Encoding: gzip and GzipDecompressorStream on chunked requests Jun 19, 2022
fredrikekre added a commit that referenced this issue Jun 20, 2022
@fredrikekre
Fix some issues with automatic gzip decompression
4d27e76 
This plugs a buffer stream in between the decompressor stream and the
"user" stream. This make sure that (i) the correct number of bytes is
read from the http stream and thus decoded (fixes #859) and (ii) that we
can read the http stream in chunks instead of byte-by-byte (the previous
code even warns about this usage).

Fixes #859.
@fredrikekre fredrikekre mentioned this issue Jun 20, 2022
Merged
fredrikekre added a commit that referenced this issue Jun 20, 2022
@fredrikekre
Fix some issues with automatic gzip decompression
99d3177 
This plugs a buffer stream in between the decompressor stream and the
"user" stream. This make sure that (i) the correct number of bytes is
read from the http stream and thus decoded (fixes #859) and (ii) that we
can read the http stream in chunks instead of byte-by-byte (the previous
code even warns about this usage).

Fixes #859.
quinnj pushed a commit that referenced this issue Jun 20, 2022
@fredrikekre
Fix some issues with automatic gzip decompression (#861)
23fc381 
This plugs a buffer stream in between the decompressor stream and the
"user" stream. This make sure that (i) the correct number of bytes is
read from the http stream and thus decoded (fixes #859) and (ii) that we
can read the http stream in chunks instead of byte-by-byte (the previous
code even warns about this usage).

Fixes #859.
@quinnj
Copy link
Member

quinnj commented Jun 22, 2022

Thanks for the detailed investigation @Pangoraw! @fredrikekre kindly put up a PR using a BufferedStream approach that seems to work well. Thanks for reporting the issue and helping out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix some issues with automatic gzip decompression JuliaWeb/HTTP.jl
2 participants
@quinnj @Pangoraw