clarification on cookie types and behavior

[Crghilardi]

Julia 1.5.3
HTTP.jl 0.9.5
MbedTLS.jl 1.0.3

I am working on a script that submits gets some credentials and submits a webform in a persistent session. I'm writing this away from the computer where it happened so actual error and version details are a little fuzzy. I think this is an "understanding the docs" rather than technical.

I am able to successfully pass the first part and generate the session tokens using the cookies flag like:

HTTP.get(someurl.com, headers = ["somekey" => "somevalue"]; cookies = true)

I can also view the returned cookies using HTTP.CookiesRequest.default_cookiejar and now need to be able to pass them back in my request. I thought from reading the docstrings I could use either the cookies or cookiejar flag to pass them back in directly such as:

HTTP.post(someurl.com, headers = ["somekey" => "somevalue"]; cookiejar = default_cookiejar)

but that errors with default_cookiejar not found.

My second attempt with copy pasting the tokens into a Dict fails to return 200 (usually with some kind of EOF error?).

HTTP.post(someurl.com, headers = ["somekey" => "somevalue"]; cookies = Dict("cookie1" => "cookie2","somekey" => "somevalue","cookie3" => "somevalue"))

Long story short, I was able to get a successful return after reading this discourse post and modifying my script to pass the hardcoded cookie strings in the header array like:

HTTP.post(someurl.com, headers = ["somekey" => "somevalue","cookie1" => "cookie2","somekey" => "somevalue","cookie3" => "somevalue"])

Happy to take a stab at a PR with some guidance. I guess my questions so far include:

1. Is the ; cookies = Dict(...) syntax still valid or should everything just go straight into headers?
2. I don't think the CookiesRequest.default_cookiejar syntax is mentioned anywhere in docstrings, where is a good place for that be added/clarified?
3. Any other clarifying points about cookie mechanisms floating around?

[quinnj]

Thanks for opening an issue; it's certainly something we should improve on in the docs (thanks for the example PR that already helps a little at least!).

My thoughts:

• The normal usage should just be cookies=true on all requests; once you make a request w/ cookies=true and the response to that request has a Set-Cookie, those get stored in the HTTP.default_cookiejar[Threads.threadid()]; if a subsequent request matches the domain of stored cookies and they haven't expired, they will be included in the next request automatically (in your example, for the HTTP.post, you should just be able to do cookies=true again)
• HTTP.default_cookiejar isn't really meant to be a public interface; it just happens to be what we named the default cookie jar. What you can do is make your own cookie jar and pass it like cookiejar = Dict{String, Set{Cookie}}(); HTTP.get(...; cookiejar=cookiejar); if the request returns a Set-Cookie header, the cookies will be parsed and put in the passed cookiejar
• I'd be open to defining an official HTTP.cookiejar() function that retrieved the default cookie jar if people think that would be helpful and "less scary"
• I'm not exactly sure why passing the cookies as a Dict failed in your case, I'd have ot see an extended example and we could diagnose what is going wrong exactly; it might be tricky to get things to match up for the right domain or something. In any case, you might have better luck passing in a Cookie header with the desired values in the headers argument instead. So with all of that, yes, it might be best if we deprecate passing Dict for cookies if it's going to be not that helpful for most cases

[Crghilardi]

Trying again this morning after a cup of coffee (rather than 4:30 on a Friday), I was wrong and can confirm that the cookie behavior works as you describe. Where I think I was going wrong previously was I was also moving around how I was passing in credentials as a HTTP.Form at the same time.

#intial get
HTTP.post(someurl.com, headers = ["somekey" => "somevalue"]; cookies = true)

#post with the added cookie flag works 
HTTP.post(someurl.com, headers = ["somekey" => "somevalue", "username" => "username",
        "pass" => "password"]; cookies = true)

#alternatively, post with passing a Dict with hardcoded cookie values (copy pasted from `HTTP.CookieRequest.default_cookiejar`) also works
HTTP.post(someurl.com, headers = ["somekey" => "somevalue", "username" => "username",
        "pass" => "password"]; cookies = Dict("cookie1" => "cookie2","somekey" => "somevalue","cookie3" => "somevalue"))

To your bullet points, I do think some kind of convenience function/struct field to be able to view cookies would be nice. I was running sanity checks in python on the side and could view cookies with a simple response.cookies call. I can also expand on the example docs now that I understand a little better.

I think the better title for this issue is probably more about HTTP.Form behavior and how to use that.

previously I was creating a dictionary and wrapping it like

payload = Dict("username" => "username",
        "pass" => "password")

#does not work 
HTTP.post(someurl.com, headers = ["somekey" => "somevalue"], HTTP.Form(payload); cookies = true)

#returns ERROR: IOError(EOFError() during request(someurl.com))

#getting rid of the HTTP.Form and moving form payload key/values to header array worked for me, same call as above
HTTP.post(someurl.com, headers = ["somekey" => "somevalue", "username" => "username",
        "pass" => "password" ]; cookies = true)

probably a better question is how should `HTTP.Form` be used in requests?

[quinnj]

Going to close this for now as a more broad documentation update is planned and coming soon. Also as a quick follow up, in #836, there are some cookie improvements, in particular, we'll be able to call HTTP.cookies(r::Union{Request, Response}) to get back the cookies from a request or response.