SO_REUSEPORT support

[chantra]

Just dropping this here. I tested SO_REUSEPORT on go-dns using:

https://gist.github.com/chantra/dfab011267be22989b25134537963e5c

To test, I patched reflect with https://gist.github.com/chantra/abc8edf0c4cdce2e5aaf8c0d5473dd79

Perf test

GOPATH=~/go go run reflect/reflect.go  -soreuseport 0 -cpu 8

~88k

GOPATH=~/go go run reflect/reflect.go  -soreuseport 0 -cpu 4

~92k

GOPATH=~/go go run reflect/reflect.go  -soreuseport 4 -cpu 8

~225k

GOPATH=~/go go run reflect/reflect.go  -soreuseport 8 -cpu 8

~221k

GOPATH=~/go go run reflect/reflect.go  -soreuseport 4 -cpu 4

~150k

GOPATH=~/go go run reflect/reflect.go  -soreuseport 8 -cpu 4

~150k

So there is ways to squeeze more out of the service but there is some fine tuning that needs to be done between GOMAXPROCS and the number of sockets.

known issue

The empty address is not supported, e.g passing :5353 as an IP/Port to bind to would fail and report error:

address family not supported by protocol

see libp2p/go-reuseport#7

[ztheory]

golang 1.11 is slated to support setting socket options before listening or dialing, which includes so_reuseport and tcp_fastopen. Might be better to just wait it out and create a PR when it's natively supported.

https://go-review.googlesource.com/c/go/+/72810

[chantra]

Yeah will be much better. This is not a PR, I just meant to share some findings of the benefits you could expect from using SO_REUSEPORT.

[fantuz]

Interesting, @chantra
Always a good technique to look at pipelining, both in UDP and TCP ;)

Weren't there counter-arguments in reusing ports on highly-loaded DNS servers ? i.e. magic number overlap, TCP starvation, spoofing, ordering vs holding, etc ?

As you outline, balance might be when all processors take part into orchestration, and there is a limit where is not so advantageous anymore (load matters, test data too).

I believe it must deal with time_wait and how aggressively or mildly the conneciton is detected as idle, for TCPs. less clear how it acts on UDP ..

Anyway I can see that your code implement both UDP and TCP, could you split the perf results accordingly ? just interested at this since some time.

bonnes vacances,
Max

[chantra]

Weren't there counter-arguments in reusing ports on highly-loaded DNS servers ? i.e. magic number overlap, TCP starvation, spoofing, ordering vs holding, etc ?

I think you are mixing this up with tw_tcp_reuse and tw_tcp_recycle. See https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux
On the contrary, SO_REUSEPORT allows you to have multiple listening sockets versus having all threads trying to read from the same socket.

As you outline, balance might be when all processors take part into orchestration, and there is a limit where is not so advantageous anymore (load matters, test data too).
I would tend to believe the bottleneck is lock contention or something, but I did not perform any profiling.

Anyway I can see that your code implement both UDP and TCP, could you split the perf results accordingly ? just interested at this since some time.

This was purely UDP payload performed from the local host.

[chantra]

it is actually dominated by syscall.Syscall which is just below syscall.sendmsg ...

[miekg]

This is pretty neat. I might be worth checking out how this can fit in CoreDNS.

Go does this now automatically I believe:

if *cpu != 0 {
		runtime.GOMAXPROCS(*cpu)
	}

So basically NUMCPU is the amount of listeners we want/need.

[miekg]

I wonder how much of #565 can be attributed to less contention - or maybe we should do both...

[chantra]

Quickly checked #565 and it does perform better in the "reflect" case only when I use a very small amount of workers (5-10) which bring the QPS in the 8/8 case to 250k vs 225k.

[miekg]

[ Quoting <[email protected]> in "Re: [miekg/dns] SO_REUSEPORT suppor..." ]

Quickly checked #565 and it does perform better in the "reflect" case only when I use a very small amount of workers (5-10) which bring the QPS in the 8/8 case to 250k vs 225k.

Thanks. Yes, my big dislike of that PR was the magic worker number... Just using NUMCPU seems a much better why of dealing with this. /Miek

…

-- Miek Gieben

[miekg]

I'm trying to replace this, but don't see the impressive increase in qps that you've observed...

[fantuz]

Incoming connections and datagrams are distributed to the server sockets using a hash based on the 4-tuple of the connection—that is, the peer IP address and port plus the local IP address and port. This means, for example, that if a client uses the same socket to send a series of datagrams to the server port, then those datagrams will all be directed to the same receiving server (as long as it continues to exist) [https://lwn.net/Articles/542629/]

HI @miekg how is your test performed ? Is source port randomized ? is server multihomed ?

[chantra]

With go 1.11beta2 out, I played a bit with the new API to set SO_REUSEPORT.

chantra@e80149d

is an example. With this diff applied, reflect can be used as described above, given that https://gist.github.com/chantra/abc8edf0c4cdce2e5aaf8c0d5473dd79 was included in miekg/exdns@417eb53#diff-362257c9400bf5663ef5ff8016619500

The current patch is not great, I believe at least we may need to put a default function which is a noop and have a linux version that do set SO_REUSEPORT.

Anyway, feel free to play with it.

[chantra]

golang/go#26771 is tracking the availability of syscall.SO_REUSEPORT constant

[chantra]

ok, so unix.SO_REUSEPORT it will be as apparently syscall only get new constant when required by standard library.

[ztheory]

@chantra What're your thoughts about this issue since Golang 1.11 now supports setting these socket options, and you no longer have to include a 3rd-party library? Not only is SO_REUSEPORT going to be helpful, but also TCP_FASTOPEN for DNS-over-TLS/HTTPS support.

Ref: https://golang.org/doc/go1.11#net

[chantra]

@ztheory see #654 (comment)

[miekg]

Is anyone in a position to create/rework this code into a proper PR? Support wise we need to check if this features is available as we want to support 1.10 and 1.11 (last two releases)