From 4e2bf8f9bca978208e0fed81f6a1ec7cfae7488b Mon Sep 17 00:00:00 2001 From: chad Date: Mon, 14 Aug 2023 15:02:48 -0500 Subject: [PATCH 1/4] feat(perf): Added new CLI flags for iterations + updated docs --- perf/README.md | 70 ++++++++++++++++-------- perf/runner/src/benchmark-result-type.ts | 4 +- perf/runner/src/index.ts | 16 ++++-- 3 files changed, 60 insertions(+), 30 deletions(-) diff --git a/perf/README.md b/perf/README.md index f063facb7..7b9072261 100644 --- a/perf/README.md +++ b/perf/README.md @@ -15,24 +15,49 @@ Benchmark results can be visualized with https://observablehq.com/@libp2p-worksp 3. Wait for action run to finish and to push a commit to your branch. 4. Visualize results on https://observablehq.com/@libp2p-workspace/performance-dashboard. -## Running via local machine +## Running with Terraform on AWS manually +### Pre-requisites + +- Terraform 1.5.5 or later +- Node.js 18 or later +- [an AWS IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) +## ### Provision infrastructure -1. Save your public SSH key as the file `./terraform/modules/short_lived/files/perf.pub`; or generate a new key pair with `make ssh-keygen` and add it to your SSH agent with `make ssh-add`. -2. `cd terraform/configs/local` -3. `terraform init` -4. `terraform apply` -5. `CLIENT_IP=$(terraform output -raw client_ip)` -6. `SERVER_IP=$(terraform output -raw server_ip)` +1. Ensure you ssh agent is running, you can check that via +```bash + eval "$(ssh-agent -s)" +``` +2. Save your public SSH key as the file `./terraform/modules/short_lived/files/perf.pub`; or generate a new key pair with `make ssh-keygen` and add it to your SSH agent with `make ssh-add`. +3. Change to the terraform local directory ```cd terraform/configs/local``` and initalize terraform ```terraform init``` +4. `terraform apply` A terraform plan will be generated, type `yes` to continue. -To preview the plan, you can run `terraform plan` +5. Once that is finished, you can get the public IP addresses of the client and server with `terraform output client_ip` and `terraform output server_ip`. Set these to the environment variables via: + 1. `CLIENT_IP=$(terraform output -raw client_ip)` + 2. `SERVER_IP=$(terraform output -raw server_ip)` + +**Notes** +- You may need to reset the infrastructure if you encounter any errors, you can do that by running `terraform destroy` and then `terraform apply`. +- While running terraform you may encounter the following error: + ```bash + Error: collecting instance settings: reading EC2 Launch Template versions: couldn't find resource + │ + │ with module.short_lived_server[0].aws_instance.perf, + │ on ../../modules/short_lived/main.tf line 15, in resource "aws_instance" "perf": + │ 15: resource "aws_instance" "perf" { + ``` + If you set *TF_VAR* [`long_lived_enabled`](./terraform/configs/local/terraform.tf#L42) env variable to default to **true** terraform should spin up the long-lived resources that are required for the short-lived resources to be created. + +- On some occassions the instances may run out of space, this [is a known issue](https://github.com/libp2p/test-plans/pull/249) which requires [the volume to be expanded](https://github.com/libp2p/test-plans/pull/249/files#diff-28d5c648894177b493aed0b03f92e21ff4d926ce824f50499080a33161d0c0baR111-R119) +- It's best to destroy the infrastructure after you're done with your testing, you can do that by running `terraform destroy`. -### Build and run implementations +### Build and run libp2p implementations -_WARNING_: Running the perf tests might take a while. +Given you have provisioned your infrastructure, you can now build and run the libp2p implementations on the AWS instances. 1. `cd runner` -2. `npm ci` -3. `npm run start -- --client-public-ip $CLIENT_IP --server-public-ip $SERVER_IP` +2. Install the depencies required for the runner `npm ci` +3. The amount of iterations that perf will run your implementation defaults to 10, but you can pass in another value `npm run start -- --client-public-ip $CLIENT_IP --server-public-ip $SERVER_IP --iterations 50` ### Deprovision infrastructure @@ -41,27 +66,28 @@ _WARNING_: Running the perf tests might take a while. ## Adding a new implementation or a new version -1. Add implementation to `impl/`. - - For a new implementation, create a folder `impl//` +1. Add the implementation to new subdirectory in [`impl/*`](./impl/). + - For a new implementation, create a folder `impl//` e.g. `go-libp2p` - For a new version of an existing implementation, create a folder `impl//`. - In that folder include a `Makefile` that builds an executable and stores it next to the `Makefile` under the name `perf`. - Requirements for the executable: - - Running as a libp2p-perf server - - Command line flags - - `--run-server` + - Running as a libp2p-perf server: + - The perf server must not exit as it will be closed by the test runner. + - The executable must accept the command flag `--run-server` which indicates it's running as server. - Running as a libp2p-perf client + - Given that perf is a client driven set of benchmarks, the performance will be measured by the client. - Input via command line - - `--server-ip-address` - - `--transport` (see `runner/versions.ts` for possible variants) + - `--server-address` + - `--transport` (see [`runner/versions.ts`](./runner/src/versions.ts#L7-L43) for possible variants) - `--upload-bytes` number of bytes to upload per stream. - `--download-bytes` number of bytes to download per stream. - Output - - Logging MUST go to stderr. - - Measurement output is printed to stdout as JSON in the form of: + - Logging MUST go to `stderr`. + - Measurement output is printed to **stdout** as JSON in the form of: ```json {"latency": 0.246442851} ``` Note that the measurement includes the time to (1) establish the connection, (2) upload the bytes and (3) download the bytes. -2. For a new implementation, in `impl/Makefile` include your implementation in the `all` target. -3. For a new version, reference version in `runner/src/versions.ts`. +2. For a new implementation, in [`impl/Makefile` include your implementation in the `all` target.](./impl/Makefile#L7) +3. For a new version, reference version in [`runner/src/versions.ts`](./runner/src/versions.ts#L7-L43). diff --git a/perf/runner/src/benchmark-result-type.ts b/perf/runner/src/benchmark-result-type.ts index 34d1088ef..7645ecb16 100644 --- a/perf/runner/src/benchmark-result-type.ts +++ b/perf/runner/src/benchmark-result-type.ts @@ -36,9 +36,7 @@ export type Result = { }; export type ResultValue = { - connectionEstablishedSeconds: number, - uploadSeconds: number, - downloadSeconds: number, + latency: number }; export type Comparison = { diff --git a/perf/runner/src/index.ts b/perf/runner/src/index.ts index 1f6020331..3e78103e4 100644 --- a/perf/runner/src/index.ts +++ b/perf/runner/src/index.ts @@ -4,7 +4,7 @@ import yargs from 'yargs'; import fs from 'fs'; import { BenchmarkResults, Benchmark, Result, IperfResults, PingResults, ResultValue } from './benchmark-result-type'; -async function main(clientPublicIP: string, serverPublicIP: string) { +async function main(clientPublicIP: string, serverPublicIP: string, iterations: number) { const pings = runPing(clientPublicIP, serverPublicIP); const iperf = runIPerf(clientPublicIP, serverPublicIP); @@ -19,7 +19,7 @@ async function main(clientPublicIP: string, serverPublicIP: string) { uploadBytes: 100 << 20, downloadBytes: 0, unit: "bit/s", - iterations: 10, + iterations, }), runBenchmarkAcrossVersions({ name: "Single Connection throughput – Download 100 MiB", @@ -28,7 +28,7 @@ async function main(clientPublicIP: string, serverPublicIP: string) { uploadBytes: 0, downloadBytes: 100 << 20, unit: "bit/s", - iterations: 10, + iterations, }), runBenchmarkAcrossVersions({ name: "Connection establishment + 1 byte round trip latencies", @@ -37,7 +37,7 @@ async function main(clientPublicIP: string, serverPublicIP: string) { uploadBytes: 1, downloadBytes: 1, unit: "s", - iterations: 100, + iterations: iterations * 10, }), ]; @@ -232,8 +232,14 @@ const argv = yargs demandOption: true, description: 'Server public IP address', }, + 'iterations': { + type: 'number', + default: 10, + description: 'Number of iterations to run', + demandOption: false, + } }) .command('help', 'Print usage information', yargs.help) .parseSync(); -main(argv['client-public-ip'] as string, argv['server-public-ip'] as string); +main(argv['client-public-ip'] as string, argv['server-public-ip'] as string, argv['iterations'] as number); From f30e858d5f32acaedee6586ff01aa42e81901f56 Mon Sep 17 00:00:00 2001 From: chad Date: Sun, 20 Aug 2023 14:22:46 -0500 Subject: [PATCH 2/4] docs: update README instructions --- perf/README.md | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/perf/README.md b/perf/README.md index 7b9072261..7bd600eb0 100644 --- a/perf/README.md +++ b/perf/README.md @@ -17,7 +17,7 @@ Benchmark results can be visualized with https://observablehq.com/@libp2p-worksp ## Running with Terraform on AWS manually -### Pre-requisites +### Prerequisites - Terraform 1.5.5 or later - Node.js 18 or later @@ -25,16 +25,12 @@ Benchmark results can be visualized with https://observablehq.com/@libp2p-worksp ## ### Provision infrastructure -1. Ensure you ssh agent is running, you can check that via -```bash - eval "$(ssh-agent -s)" -``` -2. Save your public SSH key as the file `./terraform/modules/short_lived/files/perf.pub`; or generate a new key pair with `make ssh-keygen` and add it to your SSH agent with `make ssh-add`. -3. Change to the terraform local directory ```cd terraform/configs/local``` and initalize terraform ```terraform init``` -4. `terraform apply` A terraform plan will be generated, type `yes` to continue. -To preview the plan, you can run `terraform plan` -5. Once that is finished, you can get the public IP addresses of the client and server with `terraform output client_ip` and `terraform output server_ip`. Set these to the environment variables via: - 1. `CLIENT_IP=$(terraform output -raw client_ip)` - 2. `SERVER_IP=$(terraform output -raw server_ip)` +1. Save your public SSH key as the file `./terraform/modules/short_lived/files/perf.pub`; or generate a new key pair with `make ssh-keygen` and add it to your SSH agent with `make ssh-add`. +2. ```cd terraform/configs/local``` +3. ```terraform init``` +4. ```terraform apply``` +5. ```CLIENT_IP=$(terraform output -raw client_ip)``` +6. ```SERVER_IP=$(terraform output -raw server_ip)`` **Notes** - You may need to reset the infrastructure if you encounter any errors, you can do that by running `terraform destroy` and then `terraform apply`. @@ -46,9 +42,8 @@ Benchmark results can be visualized with https://observablehq.com/@libp2p-worksp │ on ../../modules/short_lived/main.tf line 15, in resource "aws_instance" "perf": │ 15: resource "aws_instance" "perf" { ``` - If you set *TF_VAR* [`long_lived_enabled`](./terraform/configs/local/terraform.tf#L42) env variable to default to **true** terraform should spin up the long-lived resources that are required for the short-lived resources to be created. +- If you set *TF_VAR* [`long_lived_enabled`](./terraform/configs/local/terraform.tf#L42) env variable to default to **true** terraform should spin up the long-lived resources that are required for the short-lived resources to be created. -- On some occassions the instances may run out of space, this [is a known issue](https://github.com/libp2p/test-plans/pull/249) which requires [the volume to be expanded](https://github.com/libp2p/test-plans/pull/249/files#diff-28d5c648894177b493aed0b03f92e21ff4d926ce824f50499080a33161d0c0baR111-R119) - It's best to destroy the infrastructure after you're done with your testing, you can do that by running `terraform destroy`. ### Build and run libp2p implementations @@ -56,8 +51,9 @@ Benchmark results can be visualized with https://observablehq.com/@libp2p-worksp Given you have provisioned your infrastructure, you can now build and run the libp2p implementations on the AWS instances. 1. `cd runner` -2. Install the depencies required for the runner `npm ci` -3. The amount of iterations that perf will run your implementation defaults to 10, but you can pass in another value `npm run start -- --client-public-ip $CLIENT_IP --server-public-ip $SERVER_IP --iterations 50` +2. `npm ci` +3. `npm run start -- --client-public-ip $CLIENT_IP --server-public-ip $SERVER_IP --iterations 50` + * Note: The default number of iterations that perf will run is 10; desired iterations can be set with the `--iterations ` option. ### Deprovision infrastructure From 46560930d09dda834126c56c437816639fcd855c Mon Sep 17 00:00:00 2001 From: chad Date: Sun, 20 Aug 2023 14:26:02 -0500 Subject: [PATCH 3/4] chore: revert infrastructure instructions --- perf/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/perf/README.md b/perf/README.md index 7bd600eb0..3ce86a001 100644 --- a/perf/README.md +++ b/perf/README.md @@ -22,15 +22,16 @@ Benchmark results can be visualized with https://observablehq.com/@libp2p-worksp - Terraform 1.5.5 or later - Node.js 18 or later - [an AWS IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) -## + + ### Provision infrastructure 1. Save your public SSH key as the file `./terraform/modules/short_lived/files/perf.pub`; or generate a new key pair with `make ssh-keygen` and add it to your SSH agent with `make ssh-add`. -2. ```cd terraform/configs/local``` -3. ```terraform init``` -4. ```terraform apply``` -5. ```CLIENT_IP=$(terraform output -raw client_ip)``` -6. ```SERVER_IP=$(terraform output -raw server_ip)`` +2. `cd terraform/configs/local` +3. `terraform init` +4. `terraform apply` +5. `CLIENT_IP=$(terraform output -raw client_ip)` +6. `SERVER_IP=$(terraform output -raw server_ip)` **Notes** - You may need to reset the infrastructure if you encounter any errors, you can do that by running `terraform destroy` and then `terraform apply`. From c8152ab308d751b10656c8b4efef0f1d73149f33 Mon Sep 17 00:00:00 2001 From: Chad Nehemiah Date: Tue, 22 Aug 2023 09:31:28 -0500 Subject: [PATCH 4/4] docs: update perf README to exclude iterations flag in default command Co-authored-by: Prithvi Shahi --- perf/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf/README.md b/perf/README.md index 3ce86a001..57d3ed68f 100644 --- a/perf/README.md +++ b/perf/README.md @@ -53,7 +53,7 @@ Given you have provisioned your infrastructure, you can now build and run the li 1. `cd runner` 2. `npm ci` -3. `npm run start -- --client-public-ip $CLIENT_IP --server-public-ip $SERVER_IP --iterations 50` +3. `npm run start -- --client-public-ip $CLIENT_IP --server-public-ip $SERVER_IP` * Note: The default number of iterations that perf will run is 10; desired iterations can be set with the `--iterations ` option. ### Deprovision infrastructure