Fix binary and chunked HTTP downloads. Add diffs from @anod221.

[HHHartmann]

Fixes #2405.

• This PR is for the dev branch rather than for master.
• This PR is compliant with the other contributing guidelines as well (if not, please describe why).
• I have thoroughly tested my contribution.

Checking the sources that @anod221 gave I created these diffs.
They look ok but I did not have a chance to test them yet.

They also fix an issue with chunked encoding replies which I also have not tested yet.

[wtarreau]

Hi!
I've just tested it now with the same method as in #3036, and it works fine, it now properly reports all downloaded bytes and does not stop at \0 anymore:

> http.get("http://10.9.0.22:8080/lfs.img","range: bytes=0-1023\r\n",function(code,data,headers) print(code,headers["content-length"],#data) end)
> 206   1024    1024

A large response still causes a panic though if I omit the range:

> http.get("http://10.9.0.22:8080/lfs.img","",function(code,data,headers) print(code,headers["content-length"],#data) end)
> HTTP client: Response too long (8761)
PANIC: unprotected error in call to Lua API (stdin:1: attempt to index a nil value)

but my understanding is that it was more or less expected anyway, and nothing prevents us from downloading large images 1kB at a time using range requests.

[HHHartmann]

@wtarreau thats great news, Thaks for testing this.
I was pretty sure I wrote the message I am writing now before. Don't know why it is'nt shown here.

In your example ending up in a PANIC I suspect that the PANIC is either from accessing headers or data parameters to the callback which will be nil in case of an error.
The message seems to be printed by the c code. (haven't checked in the code though)

Do you have a chance to also check the chunked transfer part?

[wtarreau]

On Sat, Apr 25, 2020 at 12:29:59PM -0700, Gregor Hartmann wrote: I was pretty sure I wrote the message I am writing now before. Don't know why it is shown here.

It has happened to me from time to time to lose comments when posting, most likely due to errors when sending.

Do you have a chance to also check the chunked transfer part?

Ah I feel ashamed, that's mainly what I wanted to test and simply forgot! I'll do it. Willy

[marcelstoer]

@wtarreau any news?

[wtarreau]

Hi Marcel. Seems like it's the second time I have to tell you I feel ashamed. I knew I had something to do last week-end and didn't remember what, given how long my todo list was and that I forgot to note it :-/ Now I've written it on my todo-list hoping I'll do it this week-end. Sorry again :-/

[marcelstoer]

No worries. Don't be too hard on yourself. There are many other developers in our community - including yours truly - who could have tested this as well.

[wtarreau]

Hi Marcel,

so the test is conclusive, it works. I also verified that prepending zeroes in front of the chunk sizes worked as expected:

server$ printf "HTTP/1.1 200 OK\r\nTransfer-encoding: chunked\r\nConnection: close\r\n\r\n1\r\nA\r\n02\r\nBC\r\n003\r\nDEF\r\n0\r\n\r\n" | nc6 --half-close -lp8181
GET /foo HTTP/1.1
Host: 10.9.0.22:8181
Connection: close
User-Agent: ESP8266

> http.get("http://10.9.0.22:8181/foo",nil,function(code,data,headers) print(code,#data,data) end)
200   6       ABCDEF

I had met an issue with panics when the server is not listening that caused me some gray hair, until I figured that it was my "print(#data)" which used to trigger the panic when data is nil, so everything's OK here in fact :-)

Thus for me the patch is both valid and useful, it will be very welcome!

[marcelstoer]

Thanks Willy!