feat(perf): add (provision, build, run) tooling #163

mxinden · 2023-04-04T14:24:04Z

Continued in #184.

See `perf/README.md`.

mxinden · 2023-04-06T19:46:02Z

Test run between server in us-west-1 and client in us-east-1:

Note that this is still using t2.micro machines, in other words these numbers are not representative.

➜  perf git:(perf-terraform) ✗ ssh ec2-user@$(terraform output -raw client_public_ip) sudo docker run --tty --rm --entrypoint perf-client mxinden/libp2p-perf --server-address /ip4/$(terraform output -raw server_public_ip)/tcp/4001

[2023-04-06T18:34:17Z INFO  perf_client] Start benchmark: round-trip-time latency
[2023-04-06T18:34:48Z INFO  perf_client] Finished: 56 pings in 30.3030s
[2023-04-06T18:34:48Z INFO  perf_client] - 0.5399 s median
[2023-04-06T18:34:48Z INFO  perf_client] - 0.5499 s 95th percentile
    
[2023-04-06T18:34:48Z INFO  perf_client] Start benchmark: single connection single channel throughput
[2023-04-06T18:35:15Z INFO  perf_client] Finished: sent 10.00 MiB in 13.56 s and received 10.00 MiB in 13.66 s
[2023-04-06T18:35:15Z INFO  perf_client] - 5.90 MiBit/s up
[2023-04-06T18:35:15Z INFO  perf_client] - 5.86 MiBit/s down
    
[2023-04-06T18:35:15Z INFO  perf_client] Start benchmark: single connection parallel requests per second
[2023-04-06T18:35:16Z INFO  perf_client] Finished: sent 1000 1 bytes requests with 1 bytes response each within 0.40 s
[2023-04-06T18:35:16Z INFO  perf_client] - 2493.58 req/s
    
[2023-04-06T18:35:16Z INFO  perf_client] Start benchmark: sequential connections with single request per second
[2023-04-06T18:35:47Z INFO  perf_client] Finished: established 39 connections with one 1 bytes request and one 1 bytes response within 30.81 s
[2023-04-06T18:35:47Z INFO  perf_client] - 0.2488 s 95th percentile connection establishment
[2023-04-06T18:35:47Z INFO  perf_client] - 0.8114 s 95th percentile connection establishment + one request

{"benchmarks":[{"comparisons":[],"name":"Single Connection throughput – Upload","results":[{"implementation":"rust-libp2p","result":773013.3502842287,"transportStack":"TODO","version":"TODO"}],"unit":"bits/s"},{"comparisons":[],"name":"Single Connection 1 byte round trip latency 95th percentile","results":[{"implementation":"rust-libp2p","result":0.811350517,"transportStack":"TODO","version":"TODO"}],"unit":"s"}]}

mxinden · 2023-04-08T18:45:11Z

Recent commits add a runner implementation, running a Rust Perf docker image on the AWS client once with TCP, once with QUIC:

➜  runner git:(perf-terraform) npm run start -- --client-public-ip $(terraform output -raw -state ../terraform/terraform.tfstate client_public_ip) --server-public-ip $(terraform output -raw -state ../terraform/terraform.tfstate server_public_ip)

> [email protected] start
> node ./dist/index.js --client-public-ip xxx --server-public-ip xxx

[2023-04-08T18:42:55Z INFO  perf_client] Start benchmark: custom
[2023-04-08T18:43:02Z INFO  perf_client] Finished: Established 10 connections uploading 1 and download 1 bytes each
[2023-04-08T18:43:07Z INFO  perf_client] Start benchmark: custom
[2023-04-08T18:43:09Z INFO  perf_client] Finished: Established 10 connections uploading 1 and download 1 bytes each
{
  "benchmarks": [
    {
      "name": "Single Connection 1 byte round trip latency",
      "unit": "s",
      "results": [
        {
          "result": [
            0.541189669,
            0.543760197,
            0.554063168,
            0.544573805,
            0.535035136,
            0.538494264,
            0.539836617,
            0.536488084,
            0.538661487,
            0.542860167
          ],
          "implementation": "",
          "version": "rust-master",
          "transportStack": "tcp"
        },
        {
          "result": [
            0.122755473,
            0.120762102,
            0.122967167,
            0.120803698,
            0.124939367,
            0.123426268,
            0.121066374,
            0.120640572,
            0.121048847,
            0.123058823
          ],
          "implementation": "",
          "version": "rust-master",
          "transportStack": "quic-v1"
        }
      ],
      "comparisons": []
    }
  ]
}

mxinden · 2023-04-11T15:12:25Z

Status Update:

Terraform provision script
- is feature complete
- needs to move away from t2.micro to large machine types, ideally with its own network card.
Runner
- Has base functionality
- Requires binaries to support following flags
```
  --server-address <SERVER_ADDRESS>  
  --upload-bytes <UPLOAD_BYTES>      
  --download-bytes <DOWNLOAD_BYTES>  
  --n-times <N_TIMES>                
```
- Does not yet support --parallel-connections flag. I suggest we deprioritize this in favor of first adding the other languages.
- Needs minor touches to support JS and Go
- Runs download, upload and connection establishment latency benchmark
- Prints agreed schema (with one minor caveat, instead of percentiles, include all datapoints)
Binaries
- Rust supports minimal work and is integrated in the above runner
- Go Perf protocol go-libp2p#2161
- JS feat(@libp2p/protocol-perf): Implement perf protocol js-libp2p#1604

MarcoPolo · 2023-04-11T17:31:34Z

Very cool!

perf/runner/src/index.ts

MarcoPolo · 2023-04-11T17:36:55Z

Where is the rust implementation?

mxinden · 2023-04-11T17:59:59Z

Where is the rust implementation?

Rust implementation is here:

libp2p/rust-libp2p#3646

MarcoPolo · 2023-04-12T04:29:48Z

Pushed a Go implementation, could you try it out?

I'm curious if we'll see the 2023/04/12 04:27:38 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details. message. That would be a problem.

- Makes emited values self-describing. - Ensures accounting of connection establishment. - Allows differentiation in subsequent data analysis.

BigLep · 2023-05-19T21:11:41Z

@mxinden:

Move to TLS only (for now). Easier to compare with HTTPs and no additional security protocol negotiation.

Agreed we should have something very comparable to HTTPs, but can we also add other security protocol negotiation so we can see how much that impacts performance?

Use TLS only instead of both TLS and Noise. Removes the additional multistream-select security protocol negotiation. Thus makes it easier to compare with TCP+TLS+HTTP/2

Makes the JSON easier to read.

mxinden

@MarcoPolo @marten-seemann as discussed before I added patches to the go-libp2p perf implementation to not wait for the libp2p identify exchange when (1) establishing the connection and (2) opening the perf stream.

There is likely a more go-libp2p idiomatic way of doing this. Feedback welcome.

mxinden · 2023-05-24T06:54:22Z

perf/impl/go-libp2p/v0.27/main.go

+	start := time.Now()
+	h.Peerstore().AddAddrs(serverInfo.ID, serverInfo.Addrs, peerstore.TempAddrTTL)
+	// Use h.Network().DialPeer() instead of h.Connect to skip waiting for
+	// identify protocol to finish.
+	_, err = h.Network().DialPeer(context.Background(), serverInfo.ID)
+	if err != nil {
+		panic(err)
+	}
+	connectionEstablished := time.Since(start)


Skipping the identify wait in Connect.

@marten-seemann @MarcoPolo @sukunrt any idea why the connection establishment with the code above takes 3 network round-trips? I would expect it to take 2 RTT.

My assumption:

1 RTT for TCP.

I only enable TLS, thus no security protocol negotiation, more specifically optimistic 0RTT multistream-select.

1 RTT for TLS. (Assuming TLS 1.3 is the default.)

Using default muxers, which should be only Yamux, thus no muxer negotiation, more specifically optimistic 0RTT multistream-select.

Not waiting for the additional identify exchange.

go-libp2p will spend 1RTT negotiating the security protocol.
go-libp2p doesn't do 0RTT select for security protocol. It does 0RTT select for other protocols when it knows the peer supports the given protocol otherwise it'll still do the 1RTT select.

go-libp2p will spend 1RTT negotiating the security protocol.

Given that the client only supports one security protocol, i.e. TLS, shouldn't it be using the lazy mechanism, in other words optimistically use TLS @sukunrt?

https://github.com/multiformats/go-multistream/blob/master/lazyClient.go

I think we can change this. But currently this is not what's happening.
https://github.com/libp2p/go-libp2p/blob/master/p2p/net/upgrader/upgrader.go#L319-L330
In security negotiation we don't use lazyClient.
It uses this function which does wait for the reply. https://github.com/multiformats/go-multistream/blob/master/client.go#L117

Thank you @sukunrt. I forgot that go-libp2p supports the multistream-select simultaneous-open extension and thus can not do an optimistic multistream-select negotiation when negotiating a single protocol (here TLS) only.

The extension is not needed in libp2p's hole punching as roles (i.e. dialer and listener) are derived from the roles on the relayed connection, see DCUtR protocol section. As far as I can tell the simultaneous-open extension is only needed for the case where two nodes randomly connect via TCP at the same time.

In my eyes the probability for two nodes to randomly, i.e. not coordinated via DCUtR, connect via TCP at the same time is low and only leads to a connection failure. Thus we don't support it in rust-libp2p.

@sukunrt @MarcoPolo @marten-seemann what do you think of removing the simultaneous-open extension in go-libp2p and thus enabling the optimistic security protocol negotiation when negotiation a single protocol only? This would allow go-libp2p to be able to compete with vanilla TCP+TLS+HTTP when it comes to connection establishment latency. In other words both go-lib2p with TLS+Yamux and vanilla TCP+TLS+HTTP would take 3 RTT (2 RTT connection establishment + 1 RTT request/response).

I don’t think we should make code changes just to look better in benchmarks. In practice, most configurations (IPFS and Lotus) use TLS and Noise.

@vyzo had numbers on simultaneous opens happening in the wild. Would you mind sharing them here?

FWIW, in nim-libp2p, we don't bind when we are public, which avoids uncoordinated simultaneous opens in the wild

We don't have enough data yet to see if it causes issues, but so far the only scenario where it doesn't work is if someone made a manual port mapping to a different port than the internal one, and didn't specify it to libp2p. In this case, we can't discover our public port, and we think we are private (since we don't bind)

Thank you for the input everyone.

I documented the above as well as past discussions in libp2p/go-libp2p#2330. Let's continue the discussion over there in case there is interest.

mxinden · 2023-05-24T06:54:48Z

perf/impl/go-libp2p/v0.27/perf.go

+	// Use ps.Host.Network().NewStream() instead of ps.Host.NewStream() to
+	// skip waiting for identify protocol to finish.
+	s, err := ps.Host.Network().NewStream(network.WithNoDial(ctx, "already dialed"), p)
+	if err != nil {
+		return 0, 0, err
+	}
+	s.SetProtocol(ID)
+	lzcon := msmux.NewMSSelect(s, ID)
+	s = &streamWrapper{
+		Stream: s,
+		rw:     lzcon,
+	}


Skipping the identify wait in NewStream.

mxinden · 2023-05-24T07:21:28Z

The above brings us one step closer to reaching https latency performance. The vanilla Go https implementation needs 3RTT, both rust-libp2p and with the above now also go-libp2p with TCP need 4 RTT.

See https://observablehq.com/@mxinden-workspace/libp2p-perf#cell-621.

mxinden · 2023-05-25T00:14:35Z

@mxinden:

Move to TLS only (for now). Easier to compare with HTTPs and no additional security protocol negotiation.

Agreed we should have something very comparable to HTTPs, but can we also add other security protocol negotiation so we can see how much that impacts performance?

We can. I suggest doing so in future iterations. Added to the list of potential next steps in the pull request description.

mxinden · 2023-05-26T02:06:34Z

I changed the base of the pull request to the perf branch on the upstream libp2p/test-plans repository. Will ease collaboration with @galargh on mxinden#2.

Let's continue on #184.

MarcoPolo and others added 9 commits March 16, 2023 12:20

WIP readme with thoughts

a26aada

Add example data

ac3bba3

Fix example data

dcd44bf

Extend example data

cd66695

Add dashboard link

b75cb3b

Update readme

343b2e5

Update readme

3e45f54

Remove wrong schema file

8d94c83

feat: add basic provision script for client and server perf

22bb337

See `perf/README.md`.

mxinden requested review from MarcoPolo and removed request for MarcoPolo April 4, 2023 14:24

mxinden added 2 commits April 6, 2023 19:56

Check in .terraform.lock.hcl

2d36ba1

Spin up server in us-west-1 and client in us-east-1

7fa9f82

mxinden added 4 commits April 8, 2023 20:15

feat: introduce runner

9361069

fix(.gitignore): don't include public keys

7358746

Minor refactorings

8545b01

Small addition to perf readme

2abcf7b

mxinden changed the title ~~feat: add basic provision script for client and server perf~~ feat: add provision script and runner Apr 8, 2023

mxinden changed the title ~~feat: add provision script and runner~~ feat(perf): add provision script and runner Apr 8, 2023

Introduce download and upload benchmarks

4f2d7f9

MarcoPolo reviewed Apr 11, 2023

View reviewed changes

perf/runner/src/index.ts Outdated Show resolved Hide resolved

MarcoPolo reviewed Apr 11, 2023

View reviewed changes

perf/runner/src/index.ts Outdated Show resolved Hide resolved

Add Go impl

f0e98f6

mxinden added 3 commits May 19, 2023 18:23

Change implementation output format

c5de2c7

- Makes emited values self-describing. - Ensures accounting of connection establishment. - Allows differentiation in subsequent data analysis.

Update benchmark-results.json

b1408bd

Fix field names

ce94834

mxinden added 8 commits May 23, 2023 12:58

fix(go-libp2p): use tls only

388886f

Use TLS only instead of both TLS and Noise. Removes the additional multistream-select security protocol negotiation. Thus makes it easier to compare with TCP+TLS+HTTP/2

List results last

23ed486

Makes the JSON easier to read.

fix(runner): trust ssh host key

fd8a878

fix(go-libp2p): don't wait for identify round trip

e1a49da

fix(quic-go): update to latest upstream

89302ef

Udate benchmark results json

c05e6d6

fix(go-libp2p): don't wait for identify when dialing

9ad921e

Add comments

71fde9d

mxinden commented May 24, 2023

View reviewed changes

mxinden added 3 commits May 24, 2023 17:36

Parse --server-address in client mode

28c590b

Log building implementations on servers

d2d1c2b

fix(go-libp2p): use NewWithoutDefaults to disable Noise

f6b968a

Commit latest results

2cca4a0

This was referenced May 25, 2023

Future-proof Benchmarking ⏱ #63

Open

Automate performance benchmarking #183

Closed

mxinden changed the title ~~feat(perf): add performance benchmarking~~ feat(perf): add tooling (provision, build, run) May 25, 2023

mxinden changed the title ~~feat(perf): add tooling (provision, build, run)~~ feat(perf): add (provision, build, run) tooling May 25, 2023

mxinden changed the base branch from master to perf May 26, 2023 02:00

mxinden marked this pull request as ready for review May 26, 2023 02:00

mxinden merged commit e77a6d5 into libp2p:perf May 26, 2023

mxinden mentioned this pull request May 26, 2023

feat(perf): add (automation, provision, build, run) tooling #184

Merged

8 tasks

This was referenced Jun 4, 2023

Consider dropping multistream-select sim open libp2p/go-libp2p#2330

Closed

feat(@libp2p/protocol-perf): Implement perf protocol libp2p/js-libp2p#1604

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(perf): add (provision, build, run) tooling #163

feat(perf): add (provision, build, run) tooling #163

mxinden commented Apr 4, 2023 •

edited

Loading

mxinden commented Apr 6, 2023 •

edited

Loading

mxinden commented Apr 8, 2023 •

edited

Loading

mxinden commented Apr 11, 2023

MarcoPolo commented Apr 11, 2023

MarcoPolo commented Apr 11, 2023

mxinden commented Apr 11, 2023

MarcoPolo commented Apr 12, 2023

BigLep commented May 19, 2023

mxinden left a comment •

edited

Loading

mxinden May 24, 2023

mxinden May 25, 2023

sukunrt May 26, 2023 •

edited

Loading

mxinden May 26, 2023

sukunrt May 26, 2023

mxinden May 26, 2023

marten-seemann May 26, 2023

Menduist May 26, 2023

mxinden Jun 4, 2023

mxinden May 24, 2023

mxinden commented May 24, 2023 •

edited

Loading

mxinden commented May 25, 2023

mxinden commented May 26, 2023

feat(perf): add (provision, build, run) tooling #163

feat(perf): add (provision, build, run) tooling #163

Conversation

mxinden commented Apr 4, 2023 • edited Loading

mxinden commented Apr 6, 2023 • edited Loading

mxinden commented Apr 8, 2023 • edited Loading

mxinden commented Apr 11, 2023

MarcoPolo commented Apr 11, 2023

MarcoPolo commented Apr 11, 2023

mxinden commented Apr 11, 2023

MarcoPolo commented Apr 12, 2023

BigLep commented May 19, 2023

mxinden left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sukunrt May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden commented May 24, 2023 • edited Loading

mxinden commented May 25, 2023

mxinden commented May 26, 2023

mxinden commented Apr 4, 2023 •

edited

Loading

mxinden commented Apr 6, 2023 •

edited

Loading

mxinden commented Apr 8, 2023 •

edited

Loading

mxinden left a comment •

edited

Loading

sukunrt May 26, 2023 •

edited

Loading

mxinden commented May 24, 2023 •

edited

Loading