[Recorder] Unified recorder prototyping with storage-queue/data-table…

…s SDKs (#15826) * update rush.json * lock file * package.json, rollup and tsconfig * src file * test * add recorder.stop call * set http * set protocol to http * core-rest-pipeline draft * storage test * keep only the storage test * lock file * update package.json * recorder-new * lock file * undefined to null * remove target es5 * add a guide with starter steps * testing-recorder-new * lock file * types - recorder-new * import urlBuilder from core-http * disclaimer in the guide * comment out the test in recorder-new * testing recorder-new setup and test * rest-pipeline 1.1.0 * lock file * "versionPolicyName": "test" and sdkType * karma.conf * fix package.json * update test to take sas url * omit readme checks for testing-recorder-new, recorder-new * lock file from master * lock file and readmes * remove TEST_MODE variable * update readme with temp-location * resources update * skip runnign in ci * update package.json * remove console.logs and fix queue name * fix browser mappings * index.browser.ts and console.logs * remove .olg from clean command * test file with logs * update readme * update readme to reflect additional environment variables that must be set * login steps * lock file * test file * Copying the recordings saved in the container * remove console logs * "@azure/data-tables": "^12.1.2" * dependencies * rename test file * core-v2 recorder first draft * core-v2 node test works * karma-conf fix * uri -> url * update tests * refactor core-v1 and core-v2 recorder clients * refactor common code between core-v1 core-v2 and node and browser * renames and underscore removals * typings -> types * address feedback * RecordingStateManager * lock file * lock file * lock file from master * lock file * recorder-new package test skipped * delete commented test file * Update sdk/test-utils/testing-recorder-new/README.md * empty test file * lock file * add link descriptions * Daniel's feedback * Update sdk/test-utils/recorder-new/README.md Co-authored-by: Scott Beddall <[email protected]> * Scott's feedback * Addressing Will's feedback * Add tslib * RecordingState * lock file * currentState * lock file * docker cp * remove lib from tsconfig * more feedback * utils file and base tests * update error message * initial set of tests * Add copyright headers * No need of the if checks * both Test_Modes * Append ${testMode} mode: * karma conf and tests * Daniel's new found love - npm run clean move from prebuild to build * Added many many comments for Daniel 🐱‍👤 * lock file * package renames Co-authored-by: scbedd <[email protected]>
Azure · Aug 26, 2021 · 85437db · 85437db
1 parent 1f667c4
commit 85437db
Show file tree

Hide file tree

Showing 26 changed files with 1,440 additions and 8 deletions.
diff --git a/common/config/rush/pnpm-lock.yaml b/common/config/rush/pnpm-lock.yaml
diff --git a/eng/.docsettings.yml b/eng/.docsettings.yml
@@ -22,8 +22,8 @@ omitted_paths:
   - sdk/servicefabric/servicefabric/README.md
   - sdk/storage/storage-datalake/README.md
   - sdk/storage/storage-internal-avro/*
-  - sdk/test-utils/perfstress/README.md
-  - sdk/test-utils/multi-version/README.md
+  - sdk/test-utils/*/README.md
+
 language: js
 root_check_enabled: True
 required_readme_sections:

diff --git a/rush.json b/rush.json
@@ -686,6 +686,11 @@
       "projectFolder": "sdk/template/template",
       "versionPolicyName": "client"
     },
+    {
+      "packageName": "@azure-tools/testing-recorder-new",
+      "projectFolder": "sdk/test-utils/testing-recorder-new",
+      "versionPolicyName": "test"
+    },
     {
       "packageName": "@azure/eslint-plugin-azure-sdk",
       "projectFolder": "common/tools/eslint-plugin-azure-sdk",
@@ -696,6 +701,11 @@
       "projectFolder": "sdk/test-utils/recorder",
       "versionPolicyName": "utility"
     },
+    {
+      "packageName": "@azure-tools/test-recorder-new",
+      "projectFolder": "sdk/test-utils/recorder-new",
+      "versionPolicyName": "utility"
+    },
     {
       "packageName": "@azure/test-utils-perfstress",
       "projectFolder": "sdk/test-utils/perfstress",

diff --git a/sdk/eventhub/eventhubs-checkpointstore-table/package.json b/sdk/eventhub/eventhubs-checkpointstore-table/package.json
@@ -61,7 +61,7 @@
   "dependencies": {
     "@azure/abort-controller": "^1.0.0",
     "@azure/event-hubs": "^5.6.0",
-    "@azure/data-tables": "^12.1.1",
+    "@azure/data-tables": "^12.1.2",
     "@azure/logger": "^1.0.0",
     "tslib": "^2.2.0"
   },

diff --git a/sdk/tables/perf-tests/data-tables/package.json b/sdk/tables/perf-tests/data-tables/package.json
@@ -8,7 +8,7 @@
   "author": "",
   "license": "ISC",
   "dependencies": {
-    "@azure/data-tables": "^12.1.1",
+    "@azure/data-tables": "^12.1.2",
     "@azure/test-utils-perfstress": "^1.0.0",
     "dotenv": "^8.2.0",
     "uuid": "^8.3.0"

diff --git a/sdk/test-utils/recorder-new/README.md b/sdk/test-utils/recorder-new/README.md
@@ -0,0 +1,38 @@
+## Azure @azure-tools/test-recorder-new library for JavaScript
+
+This is an experimental tool to record and playback the tests in the JS repo by leveraging the unified out-of-process test proxy server. This library is still under construction.
+Feature work is being tracked at [#15829](https://github.com/Azure/azure-sdk-for-js/issues/15829)
+
+## Resources
+
+- [Azure SDK Tools Test Proxy](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy)
+- [Using Test Proxy with docker container](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/docker#build-and-run)
+
+## Running the proxy server
+
+Run this command
+
+> `docker run -v temp-location:/etc/testproxy -p 5001:5001 -p 5000:5000 azsdkengsys.azurecr.io/engsys/testproxy-lin:latest`
+
+(Eventually, recorder will trigger this for you!)
+
+[Note: Update `temp-location` in the command to your desired location.]
+
+If the above command doesn't work directly, try [Troubleshooting Access to Public Container Registry](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/docker#troubleshooting-access-to-public-container-registry).
+
+Reference: [Using Test Proxy with docker container](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/docker#build-and-run)
+
+## Run the tests using recorder-new at `test-utils/testing-recorder-new`
+
+- Navigate to the test-utils\testing-recorder-new folder
+- Run `rush update && rush build -t .`
+- Run `rushx test:node`
+- Run `rushx test:browser`
+
+## Copying the recordings saved in the container
+
+For some reason, the volume mapping did not work for you, copy the recordings manually.
+
+- `docker cp <container_id>:/etc/testproxy/ temp-location`
+
+  [This will be fixed eventually [#Issue-17138](https://github.com/Azure/azure-sdk-for-js/issues/17138)]
diff --git a/sdk/test-utils/recorder-new/karma.conf.js b/sdk/test-utils/recorder-new/karma.conf.js
@@ -0,0 +1,127 @@
+// https://github.com/karma-runner/karma-chrome-launcher
+process.env.CHROME_BIN = require("puppeteer").executablePath();
+require("dotenv").config({ path: "../.env" });
+
+module.exports = function(config) {
+  config.set({
+    // base path that will be used to resolve all patterns (eg. files, exclude)
+    basePath: "./",
+
+    // frameworks to use
+    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
+    frameworks: ["mocha"],
+
+    plugins: [
+      "karma-mocha",
+      "karma-mocha-reporter",
+      "karma-chrome-launcher",
+      "karma-edge-launcher",
+      "karma-firefox-launcher",
+      "karma-ie-launcher",
+      "karma-env-preprocessor",
+      "karma-coverage",
+      "karma-sourcemap-loader",
+      "karma-junit-reporter",
+      "karma-json-preprocessor"
+    ],
+
+    // list of files / patterns to load in the browser
+    files: [
+      "dist-test/index.browser.js",
+      { pattern: "dist-test/index.browser.js.map", type: "html", included: false, served: true }
+    ],
+
+    // list of files / patterns to exclude
+    exclude: [],
+
+    // preprocess matching files before serving them to the browser
+    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
+    preprocessors: {
+      "**/*.js": ["sourcemap", "env"]
+      // IMPORTANT: COMMENT following line if you want to debug in your browsers!!
+      // Preprocess source file to calculate code coverage, however this will make source file unreadable
+      // "dist-test/index.browser.js": ["coverage"]
+    },
+
+    // inject following environment values into browser testing with window.__env__
+    // environment values MUST be exported or set with same console running "karma start"
+    // https://www.npmjs.com/package/karma-env-preprocessor
+    // envPreprocessor: [],
+
+    // test results reporter to use
+    // possible values: 'dots', 'progress'
+    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
+    reporters: ["mocha", "coverage", "junit"],
+
+    coverageReporter: {
+      // specify a common output directory
+      dir: "coverage-browser/",
+      reporters: [
+        { type: "json", subdir: ".", file: "coverage.json" },
+        { type: "lcovonly", subdir: ".", file: "lcov.info" },
+        { type: "html", subdir: "html" },
+        { type: "cobertura", subdir: ".", file: "cobertura-coverage.xml" }
+      ]
+    },
+
+    junitReporter: {
+      outputDir: "", // results will be saved as $outputDir/$browserName.xml
+      outputFile: "test-results.browser.xml", // if included, results will be saved as $outputDir/$browserName/$outputFile
+      suite: "", // suite will become the package name attribute in xml testsuite element
+      useBrowserName: false, // add browser name to report and classes names
+      nameFormatter: undefined, // function (browser, result) to customize the name attribute in xml testcase element
+      classNameFormatter: undefined, // function (browser, result) to customize the classname attribute in xml testcase element
+      properties: {} // key value pair of properties to add to the <properties> section of the report
+    },
+
+    // web server port
+    port: 9328,
+
+    // enable / disable colors in the output (reporters and logs)
+    colors: true,
+
+    // level of logging
+    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
+    logLevel: config.LOG_INFO,
+
+    // enable / disable watching file and executing tests whenever any file changes
+    autoWatch: false,
+
+    // start these browsers
+    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
+    // 'ChromeHeadless', 'Chrome', 'Firefox', 'Edge', 'IE'
+    // --no-sandbox allows our tests to run in Linux without having to change the system.
+    // --disable-web-security allows us to authenticate from the browser without having to write tests using interactive auth, which would be far more complex.
+    browsers: ["ChromeHeadlessNoSandbox"],
+    customLaunchers: {
+      ChromeHeadlessNoSandbox: {
+        base: "ChromeHeadless",
+        flags: ["--no-sandbox", "--disable-web-security"]
+      }
+    },
+
+    // Continuous Integration mode
+    // if true, Karma captures browsers, runs the tests and exits
+    singleRun: false,
+
+    // Concurrency level
+    // how many browser should be started simultaneous
+    concurrency: 1,
+
+    browserNoActivityTimeout: 600000,
+    browserDisconnectTimeout: 10000,
+    browserDisconnectTolerance: 3,
+    browserConsoleLogOptions: {
+      // We would usually hide the logs from the tests, but we don't need to do this inside of the recorder package because we are not recording the tests.
+      // // terminal: process.env.TEST_MODE !== "record"
+    },
+
+    client: {
+      mocha: {
+        // change Karma's debug.html to the mocha web reporter
+        reporter: "html",
+        timeout: "600000"
+      }
+    }
+  });
+};