Requests sent twice

[NikEyX]

Julia 1.5.3
HTTP 0.9.1

In very rare circumstances the HTTP lib seems to send the same request TWICE using
r = HTTP.request("GET", url; verbose=0, connection_limit=500, pipeline_limit=0, readtimeout=60)

This happens maybe every 10,000 requests. Note that I am running this on spawned threads and that the double request is coming from the same thread.

I gonna try to create a reproduceable example but until then would anyone have an idea what the cause of that could be?

[quinnj]

There are default retry options:

HTTP.jl/src/HTTP.jl

Line 117 in 337a372

- `retries = 4`, number of times to retry.

. If a request fails for some retry-able reason, then the request will automatically be retried. You can prevent this by passing retry=false or retries=0.

[NikEyX]

Ok I will try that and reopen if the problem resurfaces. Thanks Jacob

[NikEyX]

Hmm, I seem to be getting a bunch of these now:

IOError(Base.IOError("stream is closed or unusable", 0) 

I am not sure why it would show "stream is closed".

Currently I am using:

r = HTTP.request("GET", url; verbose=3, connection_limit=500, pipeline_limit=0, readtimeout=60 * 60, retry=false, redirect=false)

I wonder if this may be somehow related to the pipeline_limit=0?

So what's the right fix for this? And is there any particular reason that this seems to happen on the server side?

Note: I also updated to HTTP 0.9.2 because I've seen there were some commits on those lines, but that didn't seem to fix it.

[NikEyX]

Ok, so this seems to happen notably when HTTP requests are done in quick succession, i.e.:

r = HTTP.request("GET", url; verbose=3, connection_limit=500, pipeline_limit=0, readtimeout=60 * 60, retry=false, redirect=false)
r = HTTP.request("GET", url; verbose=3, connection_limit=500, pipeline_limit=0, readtimeout=60 * 60, retry=false, redirect=false)
r = HTTP.request("GET", url; verbose=3, connection_limit=500, pipeline_limit=0, readtimeout=60 * 60, retry=false, redirect=false)

will ALWAYS result in the IOError(Base.IOError("stream is closed or unusable", 0) during request(...)) error.
However, if I add sleep(1) in between it will work without problem. So it seems almost like the HTTP lib is trying to reuse the connection despite the pipeline_limit=0 suggesting that it shouldn't do that?

[c42f]

I think pipeline_limit isn't respected, due to the race conditions described in #517 (comment)

[fonsp]

Pipelines have been removed for clients, #783 , which might fix this issue. Could you try this again with:

julia> ]
pkg> add HTTP#master 

[quinnj]

Any remaining issues here should be resolved in HTTP#master