Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EOFError when calling S3.list_objects_v2 #903

Closed
arnaudh opened this issue Aug 10, 2022 · 4 comments
Closed

EOFError when calling S3.list_objects_v2 #903

arnaudh opened this issue Aug 10, 2022 · 4 comments

Comments

@arnaudh
Copy link

arnaudh commented Aug 10, 2022 •
edited
Loading

We (at Invenia) have bumped a few times recently into EOFError errors when calling S3.list_objects_v2 which uses HTTP.jl under the hood:

Stacktrace: 
HTTP.Exceptions.RequestError(HTTP.Messages.Request:
  """
  GET /some-invenia-bucket HTTP/1.1
  User-Agent: AWS.jl/1.0.0
  Host: s3.us-east-1.amazonaws.com
  x-amz-date: 20220722T012325Z
  x-amz-content-sha256: xxx
  Content-MD5: xxx
  x-amz-security-token: xxx
  Authorization: xxx
  Accept: */*
  Content-Length: 0
  Accept-Encoding: gzip
  
  """, EOFError())
  Stacktrace:
    [1] (::HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
      @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/ConnectionRequest.jl:107
    [2] (::HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; canonicalize_headers::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
      @ HTTP.CanonicalizeRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/CanonicalizeRequest.jl:17
    [3] (::HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_non_idempotent::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
      @ HTTP.RetryRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/RetryRequest.jl:25
    [4] (::HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, NTuple{4, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :retry), Tuple{Nothing, Bool, Int64, Bool}}})
      @ HTTP.CookieRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/CookieRequest.jl:42
    [5] (::HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; detect_content_type::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, NTuple{4, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :retry), Tuple{Nothing, Bool, Int64, Bool}}})
      @ HTTP.ContentTypeDetection ~/.julia/packages/HTTP/XJG1J/src/clientlayers/ContentTypeRequest.jl:23
    [6] (::HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; basicauth::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, NTuple{4, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :retry), Tuple{Nothing, Bool, Int64, Bool}}})
      @ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/BasicAuthRequest.jl:22
    [7] (::HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Bool, kw::Base.Iterators.Pairs{Symbol, Integer, Tuple{Symbol, Symbol}, NamedTuple{(:verbose, :retry), Tuple{Int64, Bool}}})
      @ HTTP.DefaultHeadersRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/DefaultHeadersRequest.jl:47
    [8] (::HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Base.BufferStream, kw::Base.Iterators.Pairs{Symbol, Integer, Tuple{Symbol, Symbol}, NamedTuple{(:verbose, :retry), Tuple{Int64, Bool}}})
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/RedirectRequest.jl:17
    [9] #1#2
      @ ~/.julia/packages/HTTP/XJG1J/src/clientlayers/DebugRequest.jl:22 [inlined]
   [10] (::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; response_stream::Base.BufferStream, http_version::VersionNumber, kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol, Symbol}, NamedTuple{(:redirect, :retry), Tuple{Bool, Bool}}})
      @ HTTP.MessageRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/MessageRequest.jl:17
   [11] request(stack::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}}, method::String, url::URIs.URI, h::Vector{Pair{SubString{String}, SubString{String}}}, b::String, q::Nothing; headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String, query::Nothing, kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:redirect, :retry, :response_stream), Tuple{Bool, Bool, Base.BufferStream}}})
      @ HTTP ~/.julia/packages/HTTP/XJG1J/src/HTTP.jl:412
   [12] #request#16
      @ ~/.julia/packages/HTTP/XJG1J/src/HTTP.jl:272 [inlined]
   [13] macro expansion
      @ ~/.julia/packages/Mocking/MsKoy/src/mock.jl:26 [inlined]
   [14] (::AWS.var"#40#42"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}})()
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:220
   [15] (::Base.var"#70#72"{Base.var"#70#71#73"{AWS.AWSExponentialBackoff, AWS.var"#41#43", AWS.var"#40#42"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})(; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./error.jl:301
   [16] (::Base.var"#70#72"{Base.var"#70#71#73"{AWS.AWSExponentialBackoff, AWS.var"#41#43", AWS.var"#40#42"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})()
      @ Base ./error.jl:284
   [17] _http_request(http_backend::AWS.HTTPBackend, request::AWS.Request, response_stream::IOBuffer)
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:245
   [18] macro expansion
      @ ~/.julia/packages/Mocking/MsKoy/src/mock.jl:26 [inlined]
   [19] (::AWS.var"#33#36"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}})()
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:129
   [20] (::AWS.var"#34#38"{AWS.var"#33#36"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer})()
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:144
   [21] (::Base.var"#70#72"{Base.var"#70#71#73"{AWS.AWSExponentialBackoff, AWS.var"#35#39"{AWS.AWSConfig, Vector{String}, Vector{String}, Int64}, AWS.var"#34#38"{AWS.var"#33#36"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer}}})(; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./error.jl:288
   [22] (::Base.var"#70#72"{Base.var"#70#71#73"{AWS.AWSExponentialBackoff, AWS.var"#35#39"{AWS.AWSConfig, Vector{String}, Vector{String}, Int64}, AWS.var"#34#38"{AWS.var"#33#36"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer}}})()
      @ Base ./error.jl:284
   [23] submit_request(aws::AWS.AWSConfig, request::AWS.Request; return_headers::Nothing)
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:195
   [24] (::AWS.RestXMLService)(request_method::String, request_uri::String, args::Dict{String, Any}; aws_config::AWS.AWSConfig, feature_set::AWS.FeatureSet)
      @ AWS ~/.julia/packages/AWS/CsqNq/src/AWS.jl:284
   [25] #list_objects_v2#126
      @ ~/.julia/packages/AWS/CsqNq/src/services/s3.jl:3626 [inlined]
   [26] _s3_exists_dir(aws::AWS.AWSConfig, bucket::SubString{String}, path::String)
      @ AWSS3 ~/.julia/packages/AWSS3/qMCp5/src/AWSS3.jl:267
   [27] s3_exists_unversioned
      @ ~/.julia/packages/AWSS3/qMCp5/src/AWSS3.jl:307 [inlined]
   [28] #s3_exists#13
      @ ~/.julia/packages/AWSS3/qMCp5/src/AWSS3.jl:322 [inlined]
   [29] exists(fp::S3Path{Nothing})
      @ AWSS3 ~/.julia/packages/AWSS3/qMCp5/src/s3path.jl:235
   [30] isdir(fp::S3Path{Nothing})
      @ AWSS3 ~/.julia/packages/AWSS3/qMCp5/src/s3path.jl:245
   [... invenia code ...]
  
  caused by: EOFError: read end of file
  Stacktrace:
    [1] read_to_buffer(c::HTTP.ConnectionPool.Connection, sizehint::Int64)
      @ HTTP.ConnectionPool ~/.julia/packages/HTTP/XJG1J/src/ConnectionPool.jl:194
    [2] readuntil(c::HTTP.ConnectionPool.Connection, f::Function, sizehint::Int64)
      @ HTTP.ConnectionPool ~/.julia/packages/HTTP/XJG1J/src/ConnectionPool.jl:0
    [3] readuntil
      @ ~/.julia/packages/HTTP/XJG1J/src/ConnectionPool.jl:211 [inlined]
    [4] readheaders
      @ ~/.julia/packages/HTTP/XJG1J/src/Messages.jl:511 [inlined]
    [5] startread(http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection})
      @ HTTP.Streams ~/.julia/packages/HTTP/XJG1J/src/Streams.jl:154
    [6] macro expansion
      @ ~/.julia/packages/HTTP/XJG1J/src/clientlayers/StreamRequest.jl:47 [inlined]
    [7] macro expansion
      @ ./task.jl:387 [inlined]
    [8] streamlayer(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}; iofunction::Nothing, decompress::Bool, kw::Base.Iterators.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Int64}}})
      @ HTTP.StreamRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/StreamRequest.jl:35
    [9] (::HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}})(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}; status_exception::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
      @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/ExceptionRequest.jl:14
   [10] (::HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Connection}; readtimeout::Int64, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
      @ HTTP.TimeoutRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/TimeoutRequest.jl:17
   [11] (::HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
      @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/ConnectionRequest.jl:103
   [12] (::HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; canonicalize_headers::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
      @ HTTP.CanonicalizeRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/CanonicalizeRequest.jl:17
   [13] (::HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_non_idempotent::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose), Tuple{Nothing, Bool, Int64}}})
      @ HTTP.RetryRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/RetryRequest.jl:25
   [14] (::HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, NTuple{4, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :retry), Tuple{Nothing, Bool, Int64, Bool}}})
      @ HTTP.CookieRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/CookieRequest.jl:42
   [15] (::HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; detect_content_type::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, NTuple{4, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :retry), Tuple{Nothing, Bool, Int64, Bool}}})
      @ HTTP.ContentTypeDetection ~/.julia/packages/HTTP/XJG1J/src/clientlayers/ContentTypeRequest.jl:23
   [16] (::HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; basicauth::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Integer}, NTuple{4, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :retry), Tuple{Nothing, Bool, Int64, Bool}}})
      @ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/BasicAuthRequest.jl:22
   [17] (::HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Bool, kw::Base.Iterators.Pairs{Symbol, Integer, Tuple{Symbol, Symbol}, NamedTuple{(:verbose, :retry), Tuple{Int64, Bool}}})
      @ HTTP.DefaultHeadersRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/DefaultHeadersRequest.jl:47
   [18] (::HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Base.BufferStream, kw::Base.Iterators.Pairs{Symbol, Integer, Tuple{Symbol, Symbol}, NamedTuple{(:verbose, :retry), Tuple{Int64, Bool}}})
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/RedirectRequest.jl:17
   [19] #1#2
      @ ~/.julia/packages/HTTP/XJG1J/src/clientlayers/DebugRequest.jl:22 [inlined]
   [20] (::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; response_stream::Base.BufferStream, http_version::VersionNumber, kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol, Symbol}, NamedTuple{(:redirect, :retry), Tuple{Bool, Bool}}})
      @ HTTP.MessageRequest ~/.julia/packages/HTTP/XJG1J/src/clientlayers/MessageRequest.jl:17
   [21] request(stack::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#4"{HTTP.ConnectionRequest.var"#1#2#5"{HTTP.TimeoutRequest.var"#1#4"{HTTP.TimeoutRequest.var"#1#2#5"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}}, method::String, url::URIs.URI, h::Vector{Pair{SubString{String}, SubString{String}}}, b::String, q::Nothing; headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String, query::Nothing, kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:redirect, :retry, :response_stream), Tuple{Bool, Bool, Base.BufferStream}}})
      @ HTTP ~/.julia/packages/HTTP/XJG1J/src/HTTP.jl:412
   [22] #request#16
      @ ~/.julia/packages/HTTP/XJG1J/src/HTTP.jl:272 [inlined]
   [23] macro expansion
      @ ~/.julia/packages/Mocking/MsKoy/src/mock.jl:26 [inlined]
   [24] (::AWS.var"#40#42"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}})()
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:220
   [25] (::Base.var"#70#72"{Base.var"#70#71#73"{AWS.AWSExponentialBackoff, AWS.var"#41#43", AWS.var"#40#42"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})(; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./error.jl:301
   [26] (::Base.var"#70#72"{Base.var"#70#71#73"{AWS.AWSExponentialBackoff, AWS.var"#41#43", AWS.var"#40#42"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})()
      @ Base ./error.jl:284
   [27] _http_request(http_backend::AWS.HTTPBackend, request::AWS.Request, response_stream::IOBuffer)
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:245
   [28] macro expansion
      @ ~/.julia/packages/Mocking/MsKoy/src/mock.jl:26 [inlined]
   [29] (::AWS.var"#33#36"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}})()
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:129
   [30] (::AWS.var"#34#38"{AWS.var"#33#36"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer})()
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:144
   [31] (::Base.var"#70#72"{Base.var"#70#71#73"{AWS.AWSExponentialBackoff, AWS.var"#35#39"{AWS.AWSConfig, Vector{String}, Vector{String}, Int64}, AWS.var"#34#38"{AWS.var"#33#36"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer}}})(; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base ./error.jl:288
   [32] (::Base.var"#70#72"{Base.var"#70#71#73"{AWS.AWSExponentialBackoff, AWS.var"#35#39"{AWS.AWSConfig, Vector{String}, Vector{String}, Int64}, AWS.var"#34#38"{AWS.var"#33#36"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer}}})()
      @ Base ./error.jl:284
   [33] submit_request(aws::AWS.AWSConfig, request::AWS.Request; return_headers::Nothing)
      @ AWS ~/.julia/packages/AWS/CsqNq/src/utilities/request.jl:195
   [34] (::AWS.RestXMLService)(request_method::String, request_uri::String, args::Dict{String, Any}; aws_config::AWS.AWSConfig, feature_set::AWS.FeatureSet)
      @ AWS ~/.julia/packages/AWS/CsqNq/src/AWS.jl:284
   [35] #list_objects_v2#126
      @ ~/.julia/packages/AWS/CsqNq/src/services/s3.jl:3626 [inlined]
   [36] _s3_exists_dir(aws::AWS.AWSConfig, bucket::SubString{String}, path::String)
      @ AWSS3 ~/.julia/packages/AWSS3/qMCp5/src/AWSS3.jl:267
   [37] s3_exists_unversioned
      @ ~/.julia/packages/AWSS3/qMCp5/src/AWSS3.jl:307 [inlined]
   [38] #s3_exists#13
      @ ~/.julia/packages/AWSS3/qMCp5/src/AWSS3.jl:322 [inlined]
   [39] exists(fp::S3Path{Nothing})
      @ AWSS3 ~/.julia/packages/AWSS3/qMCp5/src/s3path.jl:235
   [40] isdir(fp::S3Path{Nothing})
      @ AWSS3 ~/.julia/packages/AWSS3/qMCp5/src/s3path.jl:245
   [... invenia code ...]

This happens only very intermittently (hundreds of requests otherwise seem to go through without errors).
Let me know if there is anything I can do to help nail down / reproduce better.

Versions used:

  • Julia 1.6.1
  • HTTP.jl 1.2.0 (we have only bumped into this on HTTP 1.2.0, not before)
  • MbedTLS.jl 1.1.1

Seems like it could be related to this recent work? issue #875 and fix #885
@quinnj
Copy link
Member

quinnj commented Aug 10, 2022

Thanks for the report; this sounds familiar to JuliaLang/MbedTLS.jl#247. We just made new releases for MbedTLS.jl and HTTP.jl (v1.1.3 & v1.2.1, respectively, in the last day or so), so it would be great to know if those new releases, which included fixes hopefully for these kinds of things, help.

@arnaudh
Copy link
Author

arnaudh commented Aug 10, 2022

Ok, will upgrade those dependencies and keep an eye out for the issue, thanks.

@svilupp svilupp mentioned this issue Aug 15, 2022
Merged
@quinnj
Copy link
Member

quinnj commented Aug 18, 2022

@arnaudh any update from your end here?

@arnaudh
Copy link
Author

arnaudh commented Aug 19, 2022 •
edited
Loading

Haven't seen the issue since upgrading the dependencies, though haven't tested extensively. I'll close the issue for now, and re-open if it comes up again.

Thank you!

@arnaudh arnaudh closed this as completed Aug 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arnaudh @quinnj