Differences between open("GET", url) do io and request("GET", url; response_stream=io)?

[dave7895]

• Julia 1.7.0
• HTTP.jl 0.9.17
• MbedTLS.jl 1.0.3
  I am using HTTP.jl to stream events as they are coming in. When using HTTP.open I can manipulate the data and do what I want but my problem is that errors don't propagate and I can't see them which hurts debugging.
  So my next approach was to try and use a BufferStream or an IOBuffer and use an asynchronously started while-loop to process the data. But now the problem is that the stream doesn't seem to get the response written to it.
  According to the debug log create by verbose=3 there is data coming from the remote, but the loop

io = IOBuffer() #Base.BufferStream()
@async while !eof(io)
    println("start of while loop")
    bytes = read(io, String)
    println("bytes: $bytes")
end

doesn't print anything, not even the message that its starting.
Quite possibly it's my approach that is at fault here, not very experienced with both HTTP or buffers in general.

[fredrikekre]

Note that

julia> io = IOBuffer();

julia> eof(io)
true

so with IOBuffer the while-condition would be false from the start.

Note also that read(io, String) will read until eof, so that will block until the stream is closed.

[dave7895]

Oh, well these two issues could very well explain why it didn't work for me.
So I do need readavailable or readline to read anything from a stream that's not yet closed?
Already very thankful for these insights, going to test as soon as I can, and then close the issue or come back if I am still not able to work it out.

[dave7895]

Now I have a different but I think connected problem. Again I had an implementation using HTTP.open that was working, while the attempt with the BufferStream fails. I broke it down to a minimal working example:
Prerequisites
JSON: v0.21.2
HTTP: v0.9.17

using HTTP
using JSON
header = Dict("Authorization" => "Bearer $(ENV["LICHESS_TOKEN"])")
eventUrl = "https://lichess.org/api/stream/event"

works as expected:

HTTP.open("GET", eventUrl, header) do http
    while !eof(http)
        s = String(readavailable(http))
        println(s)
        (length(s) < 5 || isnothing(findfirst('{', s))) && continue
        state = JSON.parse(s)
        if state["type"] == "challenge"
            challengeId = state["challenge"]["id"]
            declineUrl = "https://lichess.org/api/challenge/$challengeId/decline"
            HTTP.request("POST", declineUrl, header)
        end
    end
end

Output with open

{"type":"challenge","challenge":{"id":"ddSsTreP","url":"https://lichess.org/ddSsTreP","status":"created","challenger":{"id":"dave7895","name":"dave7895","title":null,"rating":1117,"online":true,"lag":4},"destUser":{"id":"davidsguterbot","name":"DavidsGuterBot","title":"BOT","rating":858,"provisional":true,"online":true},"variant":{"key":"standard","name":"Standard","short":"Std"},"rated":false,"speed":"blitz","timeControl":{"type":"clock","limit":300,"increment":0,"show":"5+0"},"color":"random","perf":{"icon":"","name":"Blitz"}},"compat":{"bot":true,"board":true}}

DEBUG: 2021-12-20T23:03:18.498 41c5 ➡️   "80798384324797112105479910497108108101110103101471001008311584114101804710010199108105110101327284848047494649131065117116104111114105122971161051111105832661019711410111432108105112959010256501216751846810511210776112574952107781011310721111151165832108105991041011151154611111410313106599991011121165832424742131085115101114456510310111011658327284848046106108474946554648131067111110116101110116457610111010311610458324813101310" (write)
DEBUG: 2021-12-20T23:03:18.573 41c5 ⬅️   "HTTP/1.1 200 OK\r\n" (readuntil)
DEBUG:                                  "Server: nginx\r\n"
DEBUG:                                  "Date: Mon, 20 Dec 2021 22:03:20 GMT\r\n"
DEBUG:                                  "Content-Type: application/json\r\n"
DEBUG:                                  "Content-Length: 11\r\n"
DEBUG:                                  "Connection: keep-alive\r\n"
DEBUG:                                  "Vary: Origin\r\n"
DEBUG:                                  "X-OAuth-Scopes: preference:read, preference:write, challenge:read, challenge:write, challenge:bulk, puzzle:read, team:read, team:write, follow:write, msg:write, bot:play\r\n"
DEBUG:                                  "X-Accepted-OAuth-Scopes: challenge:write, bot:play, board:play\r\n"
DEBUG:                                  "Access-Control-Allow-Origin: *\r\n"
DEBUG:                                  "Access-Control-Allow-Headers: Origin, Authorization, If-Modified-Since, Cache-Control\r\n"
DEBUG:                                  "Access-Control-Allow-Methods: OPTIONS, GET, POST, DELETE\r\n"
DEBUG:                                  "Strict-Transport-Security: max-age=63072000; includeSubDomains; preload\r\n"
DEBUG:                                  "Expect-CT: max-age=31536000, enforce\r\n"
DEBUG:                                  "X-Frame-Options: DENY\r\n"
DEBUG:                                  "Permissions-Policy: interest-cohort=()\r\n"
DEBUG:                                  "\r\n"
DEBUG: 2021-12-20T23:03:18.573 41c5 ⬅️   "{\"ok\":true}" (unsafe_read)


{"type":"challengeDeclined","challenge":{"id":"ddSsTreP","url":"https://lichess.org/ddSsTreP","status":"declined","challenger":{"id":"dave7895","name":"dave7895","title":null,"rating":1117,"online":true,"lag":4},"destUser":{"id":"davidsguterbot","name":"DavidsGuterBot","title":"BOT","rating":858,"provisional":true,"online":true},"variant":{"key":"standard","name":"Standard","short":"Std"},"rated":false,"speed":"blitz","timeControl":{"type":"clock","limit":300,"increment":0,"show":"5+0"},"color":"random","perf":{"icon":"","name":"Blitz"},"declineReason":"I'm not accepting challenges at the moment."}} 

does not work, hangs and doesn't decline:

buf = Base.BufferStream()
@async while !eof(buf)
    s = String(readavailable(buf))
    println(s)
    (length(s) < 5 || isnothing(findfirst('{', s))) && continue
    state = JSON.parse(s)
    if state["type"] == "challenge"
        challengeId = state["challenge"]["id"]
        declineUrl = "https://lichess.org/api/challenge/$challengeId/decline"
        HTTP.request("POST", declineUrl, header)
    end
end
HTTP.request("GET", eventUrl, header, response_stream = buf)

Output when using StreamBuffer

{"type":"challenge","challenge":{"id":"7R3wcwH8","url":"https://lichess.org/7R3wcwH8","status":"created","challenger":{"id":"dave7895","name":"dave7895","title":null,"rating":1117,"online":true,"lag":4},"destUser":{"id":"davidsguterbot","name":"DavidsGuterBot","title":"BOT","rating":858,"provisional":true,"online":true},"variant":{"key":"standard","name":"Standard","short":"Std"},"rated":false,"speed":"blitz","timeControl":{"type":"clock","limit":300,"increment":0,"show":"5+0"},"color":"random","perf":{"icon":"","name":"Blitz"}},"compat":{"bot":true,"board":true}}

DEBUG: 2021-12-20T23:13:13.496 5e03 ➡️   "80798384324797112105479910497108108101110103101475582511199911972564710010199108105110101327284848047494649131065117116104111114105122971161051111105832661019711410111432108105112959010256501216751846810511210776112574952107781011310721111151165832108105991041011151154611111410313106599991011121165832424742131085115101114456510310111011658327284848046106108474946554648131067111110116101110116457610111010311610458324813101310" (write)

Even after waiting a full minute there wasn't anything happening anymore.

[quinnj]

@dave7895, I believe I'm seeing the same behavior for both these examples when running Julia 1.8 and HTTP#master. Please let me know if you're still seeing something off/different/hanging.