Connectivity Test: Add latency measurement

[darox]

This PR adds netperf based latency tests that can be triggered by running:

cilium connectivity test --perf --perf-latency --perf-samples 5
ℹ️  Monitor aggregation detected, will skip some flow validation steps
⚠️  Each zone only has a single node - could impact the performance test results
⌛ [arn:aws:eks:eu-central-1:679388779924:cluster/test-test-dario] Waiting for deployment cilium-test/perf-client to become ready...
⌛ [arn:aws:eks:eu-central-1:679388779924:cluster/test-test-dario] Waiting for deployment cilium-test/perf-server to become ready...
⌛ [arn:aws:eks:eu-central-1:679388779924:cluster/test-test-dario] Waiting for deployment cilium-test/perf-client-other-node to become ready...
⌛ [arn:aws:eks:eu-central-1:679388779924:cluster/test-test-dario] Waiting for CiliumEndpoint for pod cilium-test/perf-client-567c9764b5-ghqt9 to appear...
⌛ [arn:aws:eks:eu-central-1:679388779924:cluster/test-test-dario] Waiting for CiliumEndpoint for pod cilium-test/perf-client-other-node-f98d8784-dmsgx to appear...
⌛ [arn:aws:eks:eu-central-1:679388779924:cluster/test-test-dario] Waiting for CiliumEndpoint for pod cilium-test/perf-server-bb67f66b5-vzxsk to appear...
🔭 Enabling Hubble telescope...
⚠️  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4245: connect: connection refused"
ℹ️  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ️  Cilium version: 1.13.6
🏃 Running tests...
[=] Test [network-perf]
..

🔥 Latency Test Summary: 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 Scenario        | Pod                                                | Test            | Num Samples     | Duration        | Min             | Mean            | Max             | P50             | P90             | P99            
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 pod-net         | perf-client-567c9764b5-ghqt9                       | TCP_RR          | 5               | 10s             | 22.80        μs | 29.64        μs | 4088.20      μs | 29.00        μs | 30.00        μs | 41.80        μs
📋 pod-net         | perf-client-other-node-f98d8784-dmsgx              | TCP_RR          | 5               | 10s             | 478.40       μs | 501.04       μs | 6432.00      μs | 496.60       μs | 509.00       μs | 553.20       μs
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

✅ All 1 tests (2 actions) successful, 0 tests skipped, 0 scenarios skipped.

[learnitall]

One suggestion here is to add the ability to see percentile results in addition to the average. Having 50, 90, and 99th percentiles can be really helpful when trying to understand the average.

[darox]

One suggestion here is to add the ability to see percentile results in addition to the average. Having 50, 90, and 99th percentiles can be really helpful when trying to understand the average.

Implemented

[learnitall]

Great! Thank you for this. I have one nit, but otherwise LGTM.

Also, just a note, averaging percentiles can sometimes get you into trouble (see https://www.circonus.com/2018/11/the-problem-with-percentiles-aggregation-brings-aggravation/). However, I think that for this use case it's totally valid. The percentiles you are averaging together are samples from the same test, so the distributions should be the same.

[derailed]

@darox Nice work!

[darox]

@darox Nice work!

Thank you. I implemented all of your proposals.

[christarazi]

LGTM. It might make sense at some point to consider collapsing all of the --perf-* flags into one --perf flag that accepts enums such as --perf=latency or --perf=net. The former runs latency tests and the latter only runs the base "network" tests. We could have --perf=all to run all perf-related tests. This would reduce the need to keep adding more and more flags. Not required for this PR though.

[darox]

LGTM. It might make sense at some point to consider collapsing all of the --perf-* flags into one --perf flag that accepts enums such as --perf=latency or --perf=net. The former runs latency tests and the latter only runs the base "network" tests. We could have --perf=all to run all perf-related tests. This would reduce the need to keep adding more and more flags. Not required for this PR though.

Thank you for the feedback. Absolutely, this is something I wanna work on after this PR is merged.

[tklauser]

LGTM. One non-blocking nit regarding summary headers for latency tests vs. existing performance tests. Can be addressed as a follow-up. Thanks.

[darox]

LGTM. One non-blocking nit regarding summary headers for latency tests vs. existing performance tests. Can be addressed as a follow-up. Thanks.

Awesome catch. I didn't see that.