[Perf Framework] Restructuring the perf test folders - Proposal 2 #13770

HarshaNalluru · 2021-02-12T21:31:11Z

Description

To allow testing the older versions of track-2 SDKs and for smoother automation with the pipelines, we are moving away from the in-place perf tests to independent projects under the service folders. This change doesn't affect how the perf tests are written.

The main change here is that the track 2 perf tests are made into a project maintained by "rush" as opposed to in-place tests.
Track 1 tests are not managed by "rush", it is an npm project.

Old

sdk/storage/storage-blob/test/perfstress/track-1/ - npm project (but imports the perf framework)
sdk/storage/storage-blob/test/perfstress/track-2/ - depends on src code like the regular tests

New

sdk/storage/perf-tests/storage-blob-track-1/ - Independent npm project (but imports the perf framework)
sdk/storage/perf-tests/storage-blob/ - New project maintained through "rush" (depends on storage-blob on master)

One key change is that the rush property - "projectFolderMaxDepth" is being changed from 3 to 4 to allow creating new projects at 4 levels down from the root.

The following walk-through for executing perf tests uses storage-blob as an example. (Check #13740 for proposal 1 - old.)

Testing Track 1

rush update
Navigate to sdk\storage\perf-tests\storage-blob-track-1
Run npm run setup (Builds the perf package and installs it for track-1 perf tests)
Add .env file as suggested in the readme
Run the tests as suggested by readme, example npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2

Testing Track 2 - unpublished(current master)

rush update
Navigate to sdk\storage\perf-tests\storage-blob
rush build -t perf-test-storage-blob
Add .env file as suggested in the readme
Run the tests as suggested by readme, example npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2

Testing Track 2 - older versions

Example: To test 12.2.0
- Update "@azure/storage-blob" version in package.json to 12.2.0
- Add a new exception in common\config\rush\common-versions.json under allowedAlternativeVersions
  - "@azure/storage-blob": [..., "12.2.0"]
rush update (generates a new pnpm-lock file)
Navigate to sdk\storage\perf-tests\storage-blob
rush build -t perf-test-storage-blob
Add .env file as suggested in the readme
Run the tests as suggested by readme, example npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2

…to harshan/perf-restructuring

xirzec

Overall looks great, I like the move a lot. 👍

Left some small comments/suggestions.

common/tools/perf-tests-track-1-setup.js

xirzec · 2021-02-18T19:17:11Z

sdk/storage/perf-tests/storage-blob-track-1/README.md

+### Guide
+
+1. Navigate to `sdk\storage\perf-tests\storage-blob-track-1`
+2. Do `rush update`.


should one rebuild storage-blob itself as well? e.g. rush rebuild -t @azure/storage-blob ?

This is track-1(Version 10), we don't maintain it in our repo.
So, building storage-blob is not needed for track-1.

npm run setup in the next line, builds the test-utils-perfstress project, and installs it here.

(Also, track-1 perf tests are not maintained by rush, they are independent npm projects.)

sdk/storage/perf-tests/storage-blob-track-1/package.json

sdk/storage/perf-tests/storage-blob-track-1/test/upload.spec.ts

xirzec · 2021-02-18T19:19:42Z

sdk/storage/perf-tests/storage-blob/tsconfig.json

+{
+  "extends": "../../../../tsconfig.package",
+  "compilerOptions": {
+    "target": "es5",


target: es5 but lib: es5, es6, es7, esnext feels like a huge mismatch.

I just copied the tsconfig that we have for storage-blob SDK.

Looks like I need "lib": ["dom", "esnext"], removed the rest.

xirzec · 2021-02-18T19:20:06Z

sdk/storage/perf-tests/storage-blob/tsconfig.json

+    "outDir": "./dist-esm",
+    "lib": ["dom", "es5", "es6", "es7", "esnext"]
+  },
+  "compileOnSave": true,


does this work with rush?

Same, copied the tsconfig for storage-blob SDK.

compileOnSave is working when tried editing TS files in VSCode.

witemple-msft · 2021-02-18T19:33:19Z

rush.json

@@ -632,6 +632,11 @@
      "projectFolder": "sdk/digitaltwins/digital-twins-core",
      "versionPolicyName": "client"
    },
+    {
+      "packageName": "@azure/perf-test-storage-blob",


If we're moving forward with the approach of using little micro-packages that are rush-managed, can we put the "perf-test" in the namespace part of the name? I'm just thinking out loud that it will be useful for scripting/bulk processing if these are @azure-perf-test/storage-blob than otherwise. It'll make it easiest to query package names for specific kinds of packages, since we will eventually have hundreds of packages as we add more services and more perf tests. Maybe we could rename dev-tool, eslint-plugin-azure-sdk, test-utils-recorder, and the other internal tools to use an @azure-tools namespace as well.

Just something to consider.

Very good idea. I love @azure-perf-test/storage-blob.

A couple of things came up while making this change.
rush strips out the namespace name and saves it as this.

Not that it doesn't work, but something to consider. storage-blob.tgz and storage-blob-1.tgz can be confusing.

And I can no longer build storage-blob with rush build -t storage-blob, we must use rush build -t @azure/storage-blob to build storage-blob SDK which would break the Dev Exp.

@azure-tests/perf-storage-blob?

I like that one too. Feels like a hack, but I don't have any objections.
Going with @azure-tests/perf-storage-blob.

(The constraint here is that the package name after stripping out the namespace has to be unique.)

HarshaNalluru · 2021-02-18T21:20:50Z

/azp run js - storage-blob - tests

azure-pipelines · 2021-02-18T21:21:02Z

Azure Pipelines successfully started running 1 pipeline(s).

jeremymeng

Looks good!

sdk/storage/perf-tests/storage-blob-track-1/README.md

jeremymeng · 2021-02-18T21:59:01Z

rush.json

@@ -632,6 +632,11 @@
      "projectFolder": "sdk/digitaltwins/digital-twins-core",
      "versionPolicyName": "client"
    },
+    {
+      "packageName": "@azure/perf-test-storage-blob",


@azure-tests/perf-storage-blob?

…to harshan/perf-restructuring-proposal-2

EmmaZhu

It looks much clearer to manage test packages in this way.
LGTM

ghost · 2021-02-23T19:22:00Z

Hello @HarshaNalluru!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

…talake perf tests to the latest model (#14075) ### Description Makes changes to `storage-file-share` and `storage-file-datalake` based on #13770 and #14024 ### Old - `sdk/storage/storage-file-share/test/perfstress/track-1/` - npm project (but imports the perf framework) - `sdk/storage/storage-file-share/test/perfstress/track-2/` - depends on src code like the regular tests ### New - `sdk/storage/perf-tests/storage-file-share-track-1/` - Independent npm project (but imports the perf framework) - `sdk/storage/perf-tests/storage-file-share/` - New project maintained through "rush" (depends on storage-file-share on master) The following walk-through for executing perf tests uses storage-file-share as an example. (Check #13740 for proposal 1 - old.) ## Testing Track 1 - `rush update` - Navigate to `sdk\storage\perf-tests\storage-file-share-track-1` - Run `npm run setup` (Builds the perf package and installs it for track-1 perf tests) - Add .env file as suggested in the readme - Run the tests as suggested by readme, example `npm run perf-test:node -- StorageFileShareDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2` ## Testing Track 2 - unpublished(current master) - `rush update` - Navigate to `sdk\storage\perf-tests\storage-file-share` - `rush build -t perf-test-storage-file-share` - Add .env file as suggested in the readme - Run the tests as suggested by readme, example `npm run perf-test:node -- StorageFileShareDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2` ## Testing Track 2 - older versions - Example: To test 12.2.0 - Update `"@azure/storage-file-share"` version in `package.json` to `12.2.0` - Add a new exception in `common\config\rush\common-versions.json` under `allowedAlternativeVersions` - `"@azure/storage-file-share": [..., "12.2.0"]` - `rush update` (generates a new pnpm-lock file) - Navigate to `sdk\storage\perf-tests\storage-file-share` - `rush build -t perf-test-storage-file-share` - Add .env file as suggested in the readme - Run the tests as suggested by readme, example `npm run perf-test:node -- StorageFileShareDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2`

Communication cli directive config checkin (Azure#13770) * communication-cli-directive-config-checkin * typo * typo

HarshaNalluru added 26 commits February 10, 2021 13:27

Move to the "perf-tests" folder, a sibling to the SDK folders

588d9c5

Set "projectFolderMaxDepth": 4

84e1fa3

Add perf-test entry for blob in rush.json

dbd4b3d

package.json

10f9147

formatting

9770bbd

pnpm-lock file

380e3e6

.env

8ff7aa6

import blob package

271af7d

tsconfig

9ef71fe

pnpm-lock

f54772b

Update readme and package.json

898f7c1

tsconfig

b24dcd4

formatting

54d847b

utils and import updates

657555e

rush and pnpm-lock

de2163a

package.json - track 1 and 2

35735ff

tsconfig

d295cb9

compile error

594d679

sample.env

8ed74d9

readme and test updates

57d2284

common versions

e631d82

Merge branch 'master' of https://github.com/Azure/azure-sdk-for-js in…

631d34f

…to harshan/perf-restructuring

pnpm-lock file

a8bfb92

remove track 1 from rush

18cd827

@azure/test-utils-perfstress

6a92549

setup.js

f16d983

HarshaNalluru requested review from bterlson, EmmaZhu, jeremymeng and jiacfan as code owners February 12, 2021 21:31

HarshaNalluru requested review from sarangan12 and sadasant February 18, 2021 19:15

xirzec approved these changes Feb 18, 2021

View reviewed changes

witemple-msft reviewed Feb 18, 2021

View reviewed changes

HarshaNalluru added 4 commits February 18, 2021 12:10

do not depend on core-http - trrack 1

f8dcaba

Use path.resolve to resolve the path

242c2e1

Description and copyright header

76ec805

formatting

dee6937

jeremymeng reviewed Feb 18, 2021

View reviewed changes

HarshaNalluru added 5 commits February 18, 2021 15:41

change package name to @azure-tests/perf-storage-blob

1d917f1

pnpm-lock

3ef5ffc

.env file - Jeremy's suggestion

e9c5ffb

Merge branch 'master' of https://github.com/Azure/azure-sdk-for-js in…

bb526b7

…to harshan/perf-restructuring-proposal-2

"@azure/core-https": "1.0.0-beta.2",

3979cf2

mikeharder approved these changes Feb 19, 2021

View reviewed changes

HarshaNalluru mentioned this pull request Feb 19, 2021

[Event Hub] Ignore test/perf/track-1 when linting #13865

Merged

EmmaZhu approved these changes Feb 23, 2021

View reviewed changes

HarshaNalluru added the auto-merge label Feb 23, 2021

ghost merged commit 91496d9 into Azure:master Feb 23, 2021

HarshaNalluru mentioned this pull request Feb 23, 2021

[Perf Tests] Migrate the older perf test projects to new model #13944

Closed

5 tasks

HarshaNalluru deleted the harshan/perf-restructuring-proposal-2 branch February 23, 2021 20:30

HarshaNalluru mentioned this pull request Feb 23, 2021

[Perf Tests] Write-up on starting a new perf project for new service/SDK #13946

Closed

HarshaNalluru mentioned this pull request Mar 3, 2021

[Perf Tests] [Storage] Porting storage-file-share and storage-file-datalake perf tests to the latest model #14075

Merged

HarshaNalluru mentioned this pull request Mar 5, 2021

[Perf Framework] Restructuring #13739

Closed

HarshaNalluru mentioned this pull request Sep 8, 2021

[Perf Tests] Migrate the older perf test projects to new model - event based SDKs #17383

Merged

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Perf Framework] Restructuring the perf test folders - Proposal 2 #13770

[Perf Framework] Restructuring the perf test folders - Proposal 2 #13770

HarshaNalluru commented Feb 12, 2021 •

edited

Loading

xirzec left a comment

xirzec Feb 18, 2021

HarshaNalluru Feb 18, 2021

xirzec Feb 18, 2021

HarshaNalluru Feb 18, 2021

HarshaNalluru Feb 19, 2021

xirzec Feb 18, 2021

HarshaNalluru Feb 18, 2021

HarshaNalluru Feb 19, 2021

witemple-msft Feb 18, 2021

HarshaNalluru Feb 18, 2021 •

edited

Loading

HarshaNalluru Feb 18, 2021

jeremymeng Feb 18, 2021

HarshaNalluru Feb 18, 2021 •

edited

Loading

HarshaNalluru commented Feb 18, 2021

azure-pipelines bot commented Feb 18, 2021

jeremymeng left a comment

jeremymeng Feb 18, 2021

EmmaZhu left a comment

ghost commented Feb 23, 2021

[Perf Framework] Restructuring the perf test folders - Proposal 2 #13770

[Perf Framework] Restructuring the perf test folders - Proposal 2 #13770

Conversation

HarshaNalluru commented Feb 12, 2021 • edited Loading

Description

Old

New

Testing Track 1

Testing Track 2 - unpublished(current master)

Testing Track 2 - older versions

xirzec left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HarshaNalluru Feb 18, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HarshaNalluru Feb 18, 2021 • edited Loading

Choose a reason for hiding this comment

HarshaNalluru commented Feb 18, 2021

azure-pipelines bot commented Feb 18, 2021

jeremymeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EmmaZhu left a comment

Choose a reason for hiding this comment

ghost commented Feb 23, 2021

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

HarshaNalluru commented Feb 12, 2021 •

edited

Loading

HarshaNalluru Feb 18, 2021 •

edited

Loading

HarshaNalluru Feb 18, 2021 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.