integrate iperfv3 benchmark

[avsm]

In the same spirit as #478, I've implemented the IPerfv3 protocol over in https://github.com/avsm/ocaml-iperf, which is convenient for testing the relative performance of different backends in the face of parallel connections.

I'll create a PR here to integrate it for OS-specific tests (I want a uring vs posix benchmark) once the #473 PR is merged so that I can exec the iperf client subprocess.

In the meanwhile, if anyone just wants to manually call iperf against an Eio-based server, there's a iperfv3-server executable in the https://github.com/avsm/ocaml-iperf that works.

% iperf3-darwin -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 56629 connected to 127.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd          RTT
[  5]   0.00-1.00   sec  6.83 GBytes  58.7 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   1.00-2.00   sec  7.09 GBytes  60.9 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   2.00-3.00   sec  7.11 GBytes  61.1 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   3.00-4.00   sec  7.09 GBytes  60.9 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   4.00-5.00   sec  7.09 GBytes  60.9 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   5.00-6.00   sec  7.13 GBytes  61.2 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   6.00-7.00   sec  7.10 GBytes  61.0 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   7.00-8.00   sec  7.09 GBytes  60.9 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   8.00-9.00   sec  7.06 GBytes  60.7 Gbits/sec    0   4.00 MBytes   1ms     
[  5]   9.00-10.00  sec  7.12 GBytes  61.2 Gbits/sec    0   4.00 MBytes   1ms     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  70.7 GBytes  60.7 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  receiver

[talex5]

Just tried it here. I'm not familiar with the details of iperf3, but it looks like Eio is faster than iperf3's native C server?? I get:

• 72.1 Gbits/sec for iperf3 -s
• 76.7 Gbits/sec for dune exec -- ./src-eio/server.exe
• 74.9 Gbits/sec for EIO_BACKEND=posix dune exec -- ./src-eio/server.exe

[avsm]

The eio server is half as fast on macOS for me:

• 105 Gbits/sec for iperf3-darwin -s
• 61.8 Gbits/sec for dune exec -- ./src-eio/server.exe (the posix backend)

Need to dtruss it to find out if the syscall profile is similar, but can't right now as that needs System Integrity Protection disabled.

[avsm]

On Linux, similar slight slowdown on the iperf server:

• 66.4 Gbits/sec for iperf3 -s
• 69.2 Gbits/sec for EIO_BACKEND=io-uring dune exec -- ./src-eio/server.exe
• 68.3 Gbits/sec for EIO_BACKEND=posix dune exec -- ./src-eio/server.exe

The strace for the iperf server shows it doing a sequence of pselect6 and read syscalls.

pselect6(6, [3 4 5], [], NULL, {tv_sec=0, tv_nsec=503758000}, NULL) = 1 (in [5], left {tv_sec=0, tv_nsec=503757477})
read(5, "\344\36\233\366\227\356\377<~!\231[\3\7\7\367\210^\302\347\333QG\274y\315&dqQ\22<"..., 131072) = 131072
pselect6(6, [3 4 5], [], NULL, {tv_sec=0, tv_nsec=503717000}, NULL) = 1 (in [5], left {tv_sec=0, tv_nsec=503716487})
read(5, "\344\36\233\366\227\356\377<~!\231[\3\7\7\367\210^\302\347\333QG\274y\315&dqQ\22<"..., 131072) = 131072
pselect6(6, [3 4 5], [], NULL, {tv_sec=0, tv_nsec=503681000}, NULL) = 1 (in [5], left {tv_sec=0, tv_nsec=503680394})

The io_uring server has the overly sequential (just one IO consumed per syscall):

io_uring_enter(3, 1, 0, 0, NULL, 8)     = 1
io_uring_enter(3, 1, 0, 0, NULL, 8)     = 1
io_uring_enter(3, 1, 0, 0, NULL, 8)     = 1
io_uring_enter(3, 1, 0, 0, NULL, 8)     = 1

and our POSIX backend is not bothering to select and going straight for a readv:

readv(7, [{iov_base="(ow\23p\3776\320\240Z\"\310\362ki\271\247L\305\t\351\16\203l\257%\221\373^\312]{"..., iov_len=131072}], 1) = 131072
readv(7, [{iov_base="(ow\23p\3776\320\240Z\"\310\362ki\271\247L\305\t\351\16\203l\257%\221\373^\312]{"..., iov_len=131072}], 1) = 131072
readv(7, [{iov_base="(ow\23p\3776\320\240Z\"\310\362ki\271\247L\305\t\351\16\203l\257%\221\373^\312]{"..., iov_len=131072}], 1) = 131072
readv(7, [{iov_base="(ow\23p\3776\320\240Z\"\310\362ki\271\247L\305\t\351\16\203l\257%\221\373^\312]{"..., iov_len=131072}], 1) = 131072
readv(7, [{iov_base="(ow\23p\3776\320\240Z\"\310\362ki\271\247L\305\t\351\16\203l\257%\221\373^\312]{"..., iov_len=131072}], 1) = 131072

So we could probably speed up the uring version further by submitting more IO per io_uring_enter, and speed up the POSIX version slightly by using read instead of readv since it's just a single iovec.

[avsm]

A parallel test with 20 connections (iperf3 -c 127.0.0.1 -P 20):

• 3.30 Gbits/sec for iperf3 -s
• 3.21 Gbits/sec for EIO_BACKEND=io-uring dune exec -- ./src-eio/server.exe
• 3.43 Gbits/sec for EIO_BACKEND=posix dune exec -- ./src-eio/server.exe

We should be doing a little better here for uring:

io_uring_enter(3, 3, 0, 0, NULL, 8)     = 3
io_uring_enter(3, 3, 0, 0, NULL, 8)     = 3
io_uring_enter(3, 4, 0, 0, NULL, 8)     = 4
io_uring_enter(3, 3, 0, 0, NULL, 8)     = 3
io_uring_enter(3, 3, 0, 0, NULL, 8)     = 3
io_uring_enter(3, 3, 0, 0, NULL, 8)     = 3

It's got 3 or 4 parallel submits per uring. Unfortunately it's hard to trace what's actually on the ring -- we really need a way to dump the ring contents itself as a debug aid.

The posix backend is blasting away with readv and only a few ppolls:

ppoll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=3, events=POLLIN}, {fd=-1}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, 
events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14,
 events=POLLIN}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}, {fd=17, events=POLLIN}, {fd=18, events=POLLIN}, {fd=19, events=POLLIN}, {fd=2
0, events=POLLIN}, {fd=21, events=POLLIN}, {fd=22, events=POLLIN}, {fd=23, events=POLLIN}, {fd=24, events=POLLIN}, {fd=25, events=POLLIN}, {fd
=26, events=POLLIN}], 27, NULL, NULL, 8) = 2 ([{fd=7, revents=POLLIN}, {fd=8, revents=POLLIN}])
readv(7, [{iov_base="H`3\r\275[\10\247\332W\2567$Y\204g\4r\225\335\325>B\224\277\26\23R\357\36\276\236"..., iov_len=131072}], 1) = 43776
readv(8, [{iov_base="\350#\n\345\22\232)\31\254f\252\370\355\372\345s{WT\343\301\300\334\223\33E8\207\264@\227\255"..., iov_len=131072}], 1) =
 43776
readv(7, [{iov_base="\363o\244\245\275\277\3550\177\33\376@\341\25\251h\2lv\327\254\266\302\30$\r\320\311\255\20\273\t"..., iov_len=87296}], 1
) = 43776
readv(8, [{iov_base="q+\216\254\37L\"\1\251:\35\255\f\331\256\30\302qU,a\336\260g\315x\233\274\257>3\260"..., iov_len=87296}], 1) = 43776
readv(7, [{iov_base="D\f\366\331\364g-\203p\334\313\2429\\\227\3774E\0\333aU-\n?\277]J\245\ns4"..., iov_len=43520}], 1) = 43520
readv(8, [{iov_base="\306='\245v\17/\30\32q\357 _\324\rP\36 N]B\3039}#\v\264\357\314\243\233\250"..., iov_len=43520}], 1) = 43520
readv(7, [{iov_base="H`3\r\275[\10\247\332W\2567$Y\204g\4r\225\335\325>B\224\277\26\23R\357\36\276\236"..., iov_len=131072}], 1) = 131072
readv(8, [{iov_base="\350#\n\345\22\232)\31\254f\252\370\355\372\345s{WT\343\301\300\334\223\33E8\207\264@\227\255"..., iov_len=131072}], 1) =
 131072
readv(7, [{iov_base=0x7f9f8367c010, iov_len=131072}], 1) = -1 EAGAIN (Resource temporarily unavailable)
readv(8, [{iov_base=0x7f9f8365b010, iov_len=131072}], 1) = -1 EAGAIN (Resource temporarily unavailable)
ppoll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=3, events=POLLIN}, {fd=-1}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, 
events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14,
 events=POLLIN}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}, {fd=17, events=POLLIN}, {fd=18, events=POLLIN}, {fd=19, events=POLLIN}, {fd=2
0, events=POLLIN}, {fd=21, events=POLLIN}, {fd=22, events=POLLIN}, {fd=23, events=POLLIN}, {fd=24, events=POLLIN}, {fd=25, events=POLLIN}, {fd
=26, events=POLLIN}], 27, NULL, NULL, 8) = 19 ([{fd=7, revents=POLLIN}, {fd=9, revents=POLLIN}, {fd=10, revents=POLLIN}, {fd=11, revents=POLLI
N}, {fd=12, revents=POLLIN}, {fd=13, revents=POLLIN}, {fd=14, revents=POLLIN}, {fd=15, revents=POLLIN}, {fd=16, revents=POLLIN}, {fd=17, reven
ts=POLLIN}, {fd=18, revents=POLLIN}, {fd=19, revents=POLLIN}, {fd=20, revents=POLLIN}, {fd=21, revents=POLLIN}, {fd=22, revents=POLLIN}, {fd=2
3, revents=POLLIN}, {fd=24, revents=POLLIN}, {fd=25, revents=POLLIN}, {fd=26, revents=POLLIN}])
readv(7, [{iov_base="H`3\r\275[\10\247\332W\2567$Y\204g\4r\225\335\325>B\224\277\26\23R\357\36\276\236"..., iov_len=131072}], 1) = 131072
readv(9, [{iov_base="\262\277\23O\335(\217\6\335\31\232\244\374\354\2641$5L{#\250)\32\21\10 \311\206\214w\356"..., iov_len=131072}], 1) = 4377
6
readv(10, [{iov_base="\224\37.{-\210\372\241\24\351\233\312\211vv\21\267o\317\301/6K\245\375~;\304\341\200:L"..., iov_len=131072}], 1) = 43776
readv(11, [{iov_base="X\24\253\361U\243\260Ua\257p\316\n\363\36%-2\375\20{\344>o\200\320\261\240Qm\313\360"..., iov_len=131072}], 1) = 43776
readv(12, [{iov_base="q\222\1\24\31\233Nu\232\353\260\312S\266Z\"\337\345\351dsRLN\366\26Q\315\257\207\221\213"..., iov_len=131072}], 1) = 437