Make it easy to set TCP_NODELAY on server sockets
#629
Comments
|
As an aside, with these socket options set, we totally blow |
|
Thinking about this a little bit more, we don't just want an easy way to set An alternative way of doing this would be to set |
|
Okay, I did a few more experiments (and learned more about In this case, I think what this basically boils down to is that in This gives us a natural segmentation of TCP packets, and serves to essentially "flush" the TCP buffer once the user callback is done constructing their HTTP response. You can simulate this right now, with the following: using HTTP, Sockets
function serve(host, port)
num_requests = 1
@info("Listening on $(host):$(port)")
HTTP.listen(host, port) do http
Sockets.nagle(http.stream.c.io, true)
HTTP.setstatus(http, 200)
# I'm using this just to disable chunked encoding, which inflates payload size
# dramatically when writing a single character at a time. ;)
HTTP.setheader(http, "Content-Length" => "12")
HTTP.startwrite(http)
for c in "Hello, Julia"
write(http, string(c))
end
Sockets.nagle(http.stream.c.io, false)
end
end
serve("0.0.0.0", 8000)Testing with If we were to simply disable Nagle for the entire callback, as I had originally proposed, we would get something like this: And while our latency remains sub-millisecond, our throughput drops to only 1706.5 requests per second. |
|
This is some fantastic investigative work. Happy to support any changes we need in HTTP.jl to improve things. Happy to jump on a zoom call or anything if we want to do a jam session and talk about the best way to thread changes through. |
|
What do you think about just putting: Sockets.nagle(http.stream.c.io, false)
Sockets.nagle(http.stream.c.io, true)Within |
|
Yeah, that sounds fine to me; make a PR? |
|
@staticfloat, sorry for the slow follow up. I'd really hate to lose all the great investigative work you originally did here; I've put up a PR with my understanding of what would be needed to get the benefits you proposed? It seems like there are a few different places we could put this code, but what do you think about my idea? |
In my quest to improve PkgServer reliability, I noticed that my current favorite load testing tool k6 shows a reliable
40msdelay whenever dealing withHTTP.jl.The HTTP.jl server code:
The
k6configuration:The results of running
k6against the server show a consistent40msdelay in receiving the first byte of data from the server:The roundness of
40mstickled my brain, and I remembered an article I read aboutTCP_NODELAY. Grepping through Julia's source code, I discovered thatSockets.nagle()was the way to toggle this. Finding the underlying socket object took a bit more grepping throughHTTP.jl, but eventually I found it inhttp.stream.c.io. This seems a little "internalsy", which is why I opened this issue, but the results are dramatic. Uncommenting thatSockets.nagle()call in the Julia code above, we get:So a nice, almost 100x speedup (
2106.5requests served per second, as opposed to22.6). Now, of course, HTTP.jl is able to multiplex connections, so if we increase the number of VU's in k6 to get parallelized requests going, we can push the throughput up, but not the latency. Using 100 VU's and the default configuration, we can get an average45.3mslatency with2193.2requests served per second, however addingTCP_NODELAYback on drops the latency to8.6msand gets us a whopping11467.9requests served per second.There is a point (1000 VUs) at which we can completely overwhelm the CPU and the added work of setting
Sockets.nagle()actually hurts us more than it helps; at that point theTCP_NODELAYversion gets a lower throughput and higher latency than the default version, but at that point the CPUs on my test machine are screaming for relief, so I'm okay with it.I tried out some other tools and some of them did not show this issue. I discovered it had to do with the behavior of
k6keeping the socket open, for connection reuse. Turning that off (forcing it to use a new socket for each connection attempt) also got rid of the latency, but it has a significant impact on total throughput.The end result of this issue is to ask: is there a nice,
HTTP.jl-approved way for me to alter the socket objects created by theaccept()call inHTTP.listen()? I don't want to rely on the path ofhttp.stream.c.ioas it seems kind of internals-y and could change in a future release. Perhaps some kind of callback could be issued (similar totcpisvalid, which I could abuse for this purpose to mutate the socket) to allow me to modify the socket options?The text was updated successfully, but these errors were encountered: