[Perf framework] Use Proxy tool to mock/replay the requests (#16518)

* test-proxy in options

* recordingClient from recorder-new

* getRecordingClient public method

* blobServiceClient uses getRecordingClient

* not browser

* checkpoint

* add login steps in getting started

* checkpoint

* NaN bug fix

* remove temp-location note

* logs and minor tweaks - some bug at playback

* remove logs

* update options

* rushx format

* fix sendRequest

* removing unneeded details

* getting started and removing console.log

* improve type readability

* do not redeclare static PerfStressTest.recorder

* rushx format

* simplified tsconfig

* create project

* create project

* simple and batch tests

* core-v2 prototype with addPolicy

* readem disclaimer

* Address Mike's feedback

* rename recorder to testProxyHttpClient

* rename

* fix build failures

* rushx format

* fix test

* readme and getting started

* pnpm-lock file

* rename policy

* swap v1 and v2

* bug fix

* update getting started

* update getting started

* rushx format

* changelog

* Scott says no need to login anymore

* Add workflow in comments

* description

* pnpm-lock file

* RecordingStateManager

* URLBuilder -> URL

* rushx format

* readme

* lock file

* Update sdk/test-utils/perfstress/src/options.ts

Co-authored-by: Mike Harder <[email protected]>

* configureClientOptionsCoreV1 & configureClient

* update types

* comments

* getting started

* testProxyClient is not set, please make sure the client/options are configured properly.

* if (!request.headers.get("x-recording-upstream-base-uri"))  set upstream uri

* Add undici

* Investigate hanging docker or image (#33)

* getting started

* testProxyClient is not set, please make sure the client/options are configured properly.

* if (!request.headers.get("x-recording-upstream-base-uri"))  set upstream uri

* Add undici

* checkpoint

* make core-v2 client identical to core-v1 except for sendReq

* update error message

* formatting

* TestProxyHttpClientV1 depends on V2

* rushx format

* Mike's final minor feedback

* For corev1, extend TestProxyHttpClient instead of DefaultHttpClient

* update tests file

* no instaceof checks

* move to http.request

* Jeff's feedback

* CachedProxyClients

* keep clients on the test class

* bad merge conflict resolution

* Update sdk/test-utils/perfstress/GettingStarted.md

* final minor feedback

* remove CachedProxyClients wrapper

Co-authored-by: Jose Manuel Heredia Hidalgo <[email protected]>
Co-authored-by: Mike Harder <[email protected]>

sdk/storage/perf-tests/storage-blob/test/storageTest.spec.ts

@@ -27,7 +27,7 @@ export abstract class StorageBlobTest<TOptions> extends PerfStressTest<TOptions>
       getValueInConnString(connectionString, "AccountName"),
       getValueInConnString(connectionString, "AccountKey")
     );
-    this.blobServiceClient = BlobServiceClient.fromConnectionString(connectionString);
+    this.blobServiceClient = BlobServiceClient.fromConnectionString(connectionString, this.configureClientOptionsCoreV1({}));
     this.containerClient = this.blobServiceClient.getContainerClient(StorageBlobTest.containerName);
   }
 

sdk/storage/perf-tests/storage-blob/tsconfig.json

@@ -1,7 +1,6 @@
 {
   "extends": "../../../../tsconfig.package",
   "compilerOptions": {
-    "target": "es5",
     "module": "commonjs",
     "declarationDir": "./typings/latest",
     "outDir": "./dist-esm",

sdk/tables/perf-tests/data-tables/README.md

@@ -6,9 +6,9 @@
 2. Copy the `sample.env` file and name it as `.env`.
 3. Create a Storage or CosmosDB account and populate the `.env` file with the relevant credentials.
 4. Refer to the [Storage](https://docs.microsoft.com/azure/azure-resource-manager/management/azure-subscription-service-limits#storage-limits) or [CosmosDB](https://docs.microsoft.com/azure/cosmos-db/concepts-limits) rate limits and then run the tests as follows
-     - `npm run perf-test:node -- CreateSimpleEntityTest`
-     - `npm run perf-test:node -- CreateSimpleEntityBatchTest`
-     - `npm run perf-test:node -- CreateComplexEntityTest`
-     - `npm run perf-test:node -- CreateComplexEntityBatchTest`
-     - `npm run perf-test:node -- ListSimpleEntitiesTest`
-     - `npm run perf-test:node -- ListComplexEntitiesTest`
+   - `npm run perf-test:node -- CreateSimpleEntityTest`
+   - `npm run perf-test:node -- CreateSimpleEntityBatchTest`
+   - `npm run perf-test:node -- CreateComplexEntityTest`
+   - `npm run perf-test:node -- CreateComplexEntityBatchTest`
+   - `npm run perf-test:node -- ListSimpleEntitiesTest`
+   - `npm run perf-test:node -- ListComplexEntitiesTest`

sdk/tables/perf-tests/data-tables/test/tables.spec.ts

@@ -9,7 +9,7 @@ export abstract class TablesTest<TOptions = Record<string, unknown>> extends Per
   constructor(tableName: string) {
     super();
     const connectionString = getEnvVar("SAS_CONNECTION_STRING");
-    this.client = TableClient.fromConnectionString(connectionString, tableName);
+    this.client = this.configureClient(TableClient.fromConnectionString(connectionString, tableName));
   }
 
   public async globalSetup() {

sdk/test-utils/perfstress/CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## 1.0.0 (Unreleased)
 
+### 2021-08-05
+
+- Adds test-proxy tool support to the perf framework. With this, the tests can avoid service throttling by hitting the test-proxy instead to get the recorded responses.
+  [#16518](https://github.com/Azure/azure-sdk-for-js/pull/16518)
+
+### 2021-07-26
+
+- Average number of requests so far was reported as NaN when the lastMillisecondsElapsed=0.
+  Fixed in [#16583](https://github.com/Azure/azure-sdk-for-js/pull/16583)
+
 ### 2021-07-14
 
 - Removed the run method in the `PerfStressTest` class as we only deal with the async methods when it comes to performance.

sdk/test-utils/perfstress/GettingStarted.md

@@ -14,6 +14,7 @@
   - [Command to run](#command-to-run)
   - [Adding Readme/Instructions](#adding-readme/instructions)
   - [Testing an older track 2 version](#testing-an-older-track-2-version)
+- [Using Proxy Tool](#using-proxy-tool)
 
 ## [Setting up the project](#setting-up-the-project)
 
@@ -261,3 +262,47 @@ Example: Currently `@azure/<service-sdk>` is at 12.4.0 on master and you want to
 - Navigate to `sdk\storage\perf-tests\<service-sdk>`
 - `rush build -t perf-test-<service-sdk>`
 - Run the tests as suggested before, example `npm run perf-test:node -- TestClassName --warmup 2 --duration 7 --iterations 2 --parallel 50`
+
+## [Using Proxy Tool](#using-proxy-tool)
+
+### Using the testProxy option
+
+To be able to leverage the powers of playing back the requests using the test proxy, add the following to your code.
+
+      ```ts
+      /// Core V1 SDKs - For services depending on core-http
+      /// Call this.configureClientOptionsCoreV1 method on your client options
+      this.blobServiceClient = BlobServiceClient.fromConnectionString(connectionString, this.configureClientOptionsCoreV1({}));
+
+      /// Core V2 SDKs - For services depending on core-rest-pipeline
+      /// this.configureClient call to modify your client
+      this.client = this.configureClient(TableClient.fromConnectionString(connectionString, tableName));
+
+      // Not all core-v1 SDKs allow passing httpClient option.
+      // Not all core-v2 SDKs allow adding policies via pipeline option.
+      // Please reach out if your service doesn't support.
+      ```
+
+### Running the proxy server
+
+Run this command
+
+- `docker run -p 5000:5000 azsdkengsys.azurecr.io/engsys/ubuntu_testproxy_server:latest`
+
+Reference: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy#via-docker-image
+
+To use the proxy-tool in your test pass this option in cli `--test-proxy http://localhost:5000`(Make sure the port is same as what you have used to run the `docker run` command).
+
+Sample command(using storage-blob perf tests as example (Core-v1)!)
+
+> npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --test-proxy http://localhost:5000
+
+> npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2 --test-proxy http://localhost:5000
+
+Sample command(using data-tables perf tests as example (Core-v2)!)
+
+> npm run perf-test:node -- ListComplexEntitiesTest --duration 7 --iterations 2 --parallel 2 --test-proxy http://localhost:5000
+
+> npm run perf-test:node -- ListComplexEntitiesTest --duration 7 --iterations 2 --parallel 2
+
+**Using proxy-tool** part is still under construction. Please reach out to the owners/team if you face issues.

sdk/test-utils/perfstress/README.md

@@ -9,12 +9,37 @@ Link to the wiki - [Writing-Performance-Tests](https://github.com/Azure/azure-sd
 ## KeyConcepts
 
 - A **PerfStressTest** test is a test that will be executed repeatedly to show both the performance of the program, and how it behaves under stress.
-- Tests can have both a synchronous method called `run`, and an asynchronous method called `runAsync`. By default, `runAsync` will be the only method executed. If the command line parameter `--sync` is passed, only the `run` method will be executed instead.
+- Tests have an asynchronous method called `runAsync` which is executed based on the duration, iterations, and parallel options provided for the perf test. More about options below.
 - A **PerfStressOption** is a command line parameter. We use `minimist` to parse them appropriately, and then to consolidate them in a dictionary of options that is called `PerfStressOptionDictionary<string>`. The dictionary class accepts a union type of strings that defines the options that are allowed by each test.
 - Some default options are parsed by the PerfStress program. Their longer names are: `help`, `no-cleanups`, `parallel`, `duration`, `warmup`, `iterations`, `no-cleanup` and `milliseconds-to-log`.
 - PerfStress tests are executed as many times as possible until the `duration` parameter is specified. This process may repeat as many `iterations` are given. Before each iteration, tests might be called for a period of time up to `warmup`, to adjust to possible runtime optimizations. In each iteration, as many as `parallel` instances of the same test are called without waiting for each other, letting the event loop decide which one is prioritized (it's not true parallelism, but it's an approximation that aligns with the design in other languages, we might improve it over time).
 - Each test can have a `globalSetup` method, which is called once before the process begins, a `globalCleanup` method, which is called once after the process finishes.
 - Each test can have a `setup` method, which is called as many times as test instances are created (up to `parallel`), and help specify local state for each test instance. A `cleanup` method is also optional, called the same amount of times, but after finishing running the tests.
+- `test-proxy` url option - this option can be leveraged to avoid hitting throttling scenarios while testing the services. This option lets the requests go through the proxy server based on the url provided, we run runAsync method once in record mode to save the requests and responses in memory and then a ton of times in playback. Workflow with the test-proxy below.
+
+## Workflow with test proxy
+
+Steps below constitute the workflow of a typical perf test.
+
+- test resources are setup
+  - hitting the live service
+- then start record
+  - making a request to the proxy server to start recording
+  - proxy server gives a recording id, we'll use this id to save the actual requests and responses
+- run the runAsync once
+  - proxy-server saves all the requests and responses in memory
+- stop record
+  - making a request to the proxy server to stop recording
+- start playback
+  - making a request to the proxy server to start playback
+  - we use the same recording-id that we used in the record mode since that's the only way proxy-server knows what requests are supposed to be played back
+  - As a response, we get a new recording-id, which will be used for future playback requests
+- run runAsync again
+  - based on the duration, iterations, and parallel options provided for the perf test
+  - all the requests in the runAsync method are played back since we have already recorded them before
+- when the runAsync loops end, stop playback
+  - making a request to the proxy server to stop playing back
+- delete the live resources that we have created before
 
 ## Examples
 

sdk/test-utils/perfstress/package.json

@@ -62,6 +62,7 @@
   "dependencies": {
     "@azure/abort-controller": "^1.0.0",
     "@azure/core-http": "^2.0.0",
+    "@azure/core-rest-pipeline": "^1.1.0",
     "tslib": "^2.2.0",
     "node-fetch": "^2.6.0",
     "minimist": "~1.2.5",

sdk/test-utils/perfstress/src/options.ts

@@ -65,6 +65,7 @@ export interface DefaultPerfStressOptions {
   iterations: number;
   "no-cleanup": boolean;
   "milliseconds-to-log": number;
+  "test-proxy": string;
 }
 
 /**
@@ -98,6 +99,10 @@ export const defaultPerfStressOptions: PerfStressOptionDictionary<DefaultPerfStr
   "no-cleanup": {
     description: "Disables test cleanup"
   },
+  "test-proxy": {
+    description: "URI of TestProxy server",
+    defaultValue: undefined
+  },
   "milliseconds-to-log": {
     description: "Log frequency in milliseconds",
     shortName: "mtl",

sdk/test-utils/perfstress/src/program.ts

@@ -10,6 +10,7 @@ import {
   DefaultPerfStressOptions
 } from "./options";
 import { PerfStressParallel } from "./parallel";
+import { TestProxyHttpClientV1, TestProxyHttpClient } from "./testProxyHttpClient";
 
 export type TestType = "";
 
@@ -101,13 +102,17 @@ export class PerfStressProgram {
     const secondsPerOperation = 1 / operationsPerSecond;
     const weightedAverage = totalOperations / operationsPerSecond;
     console.log(
-      `Completed ${totalOperations.toLocaleString(undefined, { maximumFractionDigits: 0 })} ` +
+      `Completed ${totalOperations.toLocaleString(undefined, {
+        maximumFractionDigits: 0
+      })} ` +
         `operations in a weighted-average of ` +
         `${weightedAverage.toLocaleString(undefined, {
           maximumFractionDigits: 2,
           minimumFractionDigits: 2
         })}s ` +
-        `(${operationsPerSecond.toLocaleString(undefined, { maximumFractionDigits: 2 })} ops/s, ` +
+        `(${operationsPerSecond.toLocaleString(undefined, {
+          maximumFractionDigits: 2
+        })} ops/s, ` +
         `${secondsPerOperation.toLocaleString(undefined, {
           maximumFractionDigits: 3,
           minimumFractionDigits: 3
@@ -281,6 +286,10 @@ export class PerfStressProgram {
       }
     }
 
+    if (this.tests[0].parsedOptions["test-proxy"].value) {
+      await this.recordAndStartPlayback(this.tests[0]);
+    }
+
     if (Number(options.warmup.value) > 0) {
       await this.runTest(0, Number(options.warmup.value), "warmup");
     }
@@ -290,6 +299,10 @@ export class PerfStressProgram {
       await this.runTest(i, Number(options.duration.value), "test");
     }
 
+    if (this.tests[0].parsedOptions["test-proxy"].value) {
+      await this.stopPlayback(this.tests[0]);
+    }
+
     if (!options["no-cleanup"].value && this.tests[0].cleanup) {
       console.log(
         `=== Calling cleanup() for the ${this.parallelNumber} instantiated ${this.testName} tests ===`
@@ -308,4 +321,63 @@ export class PerfStressProgram {
       }
     }
   }
+
+  /**
+   * This method records the requests-responses and lets the proxy-server know when to playback.
+   * We run runAsync once in record mode to save the requests and responses in memory and then a ton of times in playback.
+   *
+   * ## Workflow of the perf test
+   * - test resources are setup
+   *   - hitting the live service
+   * - then start record
+   *   - making a request to the proxy server to start recording
+   *   - proxy server gives a recording id, we'll use this id to save the actual requests and responses
+   * - run the runAsync once
+   *   - proxy-server saves all the requests and responses in memory
+   * - stop record
+   *   - making a request to the proxy server to stop recording
+   * - start playback
+   *   - making a request to the proxy server to start playback
+   *   - we use the same recording-id that we used in the record mode since that's the only way proxy-server knows what requests are supposed to be played back
+   *   - as a response, we get a new recording-id, which will be used for future playback requests
+   * - run runAsync again
+   *   - based on the duration, iterations, and parallel options provided for the perf test
+   *   - all the requests in the runAsync method are played back since we have already recorded them before
+   * - when the runAsync loops end, stop playback
+   *   - making a request to the proxy server to stop playing back
+   * - delete the live resources that we have created before
+   */
+  private async recordAndStartPlayback(test: PerfStressTest) {
+    // If test-proxy,
+    // => then start record
+    // => run the runAsync
+    // => stop record
+    // => start playback
+    let recorder: TestProxyHttpClientV1 | TestProxyHttpClient;
+    if (test.testProxyHttpClient) {
+      recorder = test.testProxyHttpClient;
+    } else if (test.testProxyHttpClientV1) {
+      recorder = test.testProxyHttpClientV1;
+    } else {
+      throw new Error(
+        "testProxyClient is not set, please make sure the client/options are configured properly."
+      );
+    }
+
+    await recorder.startRecording();
+    recorder._mode = "record";
+    await test.runAsync!();
+
+    await recorder.stopRecording();
+    await recorder.startPlayback();
+    recorder._mode = "playback";
+  }
+
+  private async stopPlayback(test: PerfStressTest) {
+    if (test.testProxyHttpClient) {
+      await test.testProxyHttpClient.stopPlayback();
+    } else if (test.testProxyHttpClientV1) {
+      await test.testProxyHttpClientV1.stopPlayback();
+    }
+  }
 }