diff --git a/documentation/test-scenario/how-to/QuickStart.md b/documentation/api-scenario/how-to/QuickStart.md similarity index 68% rename from documentation/test-scenario/how-to/QuickStart.md rename to documentation/api-scenario/how-to/QuickStart.md index 1057dcfa8f40..c16a579ed61c 100644 --- a/documentation/test-scenario/how-to/QuickStart.md +++ b/documentation/api-scenario/how-to/QuickStart.md @@ -14,14 +14,15 @@ ```sh npm install -g oav@latest ``` + ### OAV Features + - Very easy to use and run. - Support postman collection format. Debug easily. - Request response validation. `oav` implement a powerful validation algorithm and help developer to detect service issue in the early phase. -- Validation result report. After each run test scenario, developer will get a validation report which contains detect issue in api test. +- Validation result report. After each run API scenario, developer will get a validation report which contains detect issue in api test. - Integrate everywhere. Easily integrate with azure-pipeline, cloud-test. - ## Create AAD app To run API test, first please prepare an AAD app which is used for provisioning Azure resource. Please grant subscription contributor permission to this AAD app. @@ -30,24 +31,24 @@ For how to create AAD app, please follow this doc https://docs.microsoft.com/en- ## Authoring steps -We will write test scenario file for SignalR service as an example. +We will write API scenario file for SignalR service as an example. -#### 1. Write your first test scenario file +#### 1. Write your first API scenario file -First, create a folder `scenarios` under the api version folder. All test scenario files under the `scenarios` folder should bind with the api version. +First, create a folder `scenarios` under the api version folder. All API scenario files under the `scenarios` folder should bind with the api version. ![folder-structure](./folder-structure.png) -Now write your basic test scenario. For more detail about test scenario file format, please refer to -[Test Scenario Definition Reference](../references/TestDefinitionReference.md). +Now write your basic API scenario. For more detail about API scenario file format, please refer to +[API Scenario Definition Reference](../references/ApiScenarioDefinition.md). ```yaml -# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/documentation/test-scenario/references/v1.0/schema.json +# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/documentation/api-scenario/references/v1.1/schema.json -contentVersion: 1.0.0 scope: ResourceGroup -testScenarios: - - description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate +scenarios: + - scenario: quickStart + description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate steps: - step: SignalR_CreateOrUpdate exampleFile: ../examples/SignalR_CreateOrUpdate.json @@ -57,7 +58,7 @@ testScenarios: #### 2. create your env file -The `env.json` file contains required test scenario variables such as, subscriptionId, AAD applicationId, AAD applicationSecret. +The `env.json` file contains required API scenario variables such as, subscriptionId, AAD applicationId, AAD applicationSecret. ```json { @@ -77,9 +78,9 @@ oav run /home/user/azure-rest-api-specs/specification/signalr/resource-manager/M #### 4. Debug with postman -Sometimes the command `oav run` may fail due to non 2xx HTTP status code. Now you need to debug the test scenario with postman. +Sometimes the command `oav run` may fail due to non 2xx HTTP status code. Now you need to debug the API scenario with postman. -When run `run`, it automatically generate postman collection and postman env in `generated////` folder. Here is the generated file folder structure. The `collection.json` and `env.json` is generated postman collection file and environment file. `202105120922-5c3x5` is current runId. For each run command it will generated unique runId. +When run `run`, it automatically generate postman collection and postman env in `generated////` folder. Here is the generated file folder structure. The `collection.json` and `env.json` is generated postman collection file and environment file. `202105120922-5c3x5` is current runId. For each run command it will generated unique runId. ``` generated @@ -104,7 +105,7 @@ After you import postman collection, you will get such requests. Now you could d #### 5. manual update example value -After debug with postman, you need to rewrite back all the updated values and run `oav run -e ` again. The result should be successful. +After debug with postman, you need to rewrite back all the updated values and run `oav run -e ` again. The result should be successful. ## Feedback diff --git a/documentation/test-scenario/how-to/testScenarioWithARMTemplate.md b/documentation/api-scenario/how-to/apiScenarioWithARMTemplate.md similarity index 87% rename from documentation/test-scenario/how-to/testScenarioWithARMTemplate.md rename to documentation/api-scenario/how-to/apiScenarioWithARMTemplate.md index 7f6524b7c359..ab7dd60bb69d 100644 --- a/documentation/test-scenario/how-to/testScenarioWithARMTemplate.md +++ b/documentation/api-scenario/how-to/apiScenarioWithARMTemplate.md @@ -1,4 +1,4 @@ -# Test scenario integrate with armTemplate +# API scenario integrate with armTemplate ## Background @@ -13,7 +13,7 @@ Here is an example about `generate unique resource name for signalR service` #### Generate unique resource name -We use `armTemplate output` to overwrite `resourceName` variable and following `createResource` step will use this variable. Below is generate unique name armTemplate. This armTemplate output `resourceName` variables, so test scenario following step will using the output variable. +We use `armTemplate output` to overwrite `resourceName` variable and following `createResource` step will use this variable. Below is generate unique name armTemplate. This armTemplate output `resourceName` variables, so API scenario following step will using the output variable. ```json { @@ -38,7 +38,7 @@ We use `armTemplate output` to overwrite `resourceName` variable and following ` } ``` -After we have this armTemplate, we could define current test scenario file. We defined `resourceName` variable globally. `./generate_unique_string.json` is armTemplate. +After we have this armTemplate, we could define current API scenario file. We defined `resourceName` variable globally. `./generate_unique_string.json` is armTemplate. `SignalR_CreateOrUpdate.json` @@ -80,8 +80,9 @@ After we have this armTemplate, we could define current test scenario file. We d scope: ResourceGroup variables: resourceName: "" -testScenarios: - - description: Microsoft.SignalRService/signalR CRUD +scenarios: + - scenario: quickStart + description: Microsoft.SignalRService/signalR CRUD steps: - step: Generate_Unique_string armTemplateDeployment: ./generate_unique_string.json diff --git a/documentation/test-scenario/how-to/armTemplate.png b/documentation/api-scenario/how-to/armTemplate.png similarity index 100% rename from documentation/test-scenario/how-to/armTemplate.png rename to documentation/api-scenario/how-to/armTemplate.png diff --git a/documentation/test-scenario/how-to/folder-structure.png b/documentation/api-scenario/how-to/folder-structure.png similarity index 100% rename from documentation/test-scenario/how-to/folder-structure.png rename to documentation/api-scenario/how-to/folder-structure.png diff --git a/documentation/test-scenario/how-to/genTestScenario.gif b/documentation/api-scenario/how-to/genTestScenario.gif similarity index 100% rename from documentation/test-scenario/how-to/genTestScenario.gif rename to documentation/api-scenario/how-to/genTestScenario.gif diff --git a/documentation/api-scenario/how-to/generateABasicApiScenario.md b/documentation/api-scenario/how-to/generateABasicApiScenario.md new file mode 100644 index 000000000000..d042b0a936d3 --- /dev/null +++ b/documentation/api-scenario/how-to/generateABasicApiScenario.md @@ -0,0 +1,57 @@ +# Generate a basic API scenario file + +## Prerequisite + +We use `oav` tools to generate basic API scenario. `oav` analyze swagger file and use swagger example as API scenario steps. So first, you need to install the latest oav. + +## Introduction + +`oav` support rule based API scenario file generation. We use this command to generate API scenario file. + +`oav generate-static-api-scenario --readme --tag --rules ` + +- readme: swagger readme file. +- tag: which tag to generate. oav will analyze swagger file under the tag and generate API scenario. +- rules: Currently support two types. `resource-put-delete`, `operations-list`. Default: `resource-put-delete` + - `resource-put-delete`: generate resource put and delete API scenario. + - `operations-list`: generate operations list API scenario. `operations-list` is the simplest API which must be defined in swagger. + +Example: + +![](./genTestScenario.gif) + +This command will load and analyze swagger and generate a basic API scenario file (`resource-put-delete`). + +Result: the output contains two files + +- scenarios/signalR.yaml: The API scenario file. +- readme.test.md: The entry for SDK test generation + +The generated API scenario file: The generated API scenario file contains two steps. Create signalR and delete it. It's a basic API scenario and developer can add more step based on the basic API scenario file. + +``` +scope: ResourceGroup +testScenarios: + - description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate + steps: + - step: SignalR_CreateOrUpdate + exampleFile: ../examples/SignalR_CreateOrUpdate.json + - step: SignalR_Delete + exampleFile: ../examples/SignalR_Delete.json +``` + +If you pass rule option `operations-list`, you will get such API scenario file. + +``` +scope: ResourceGroup +testScenarios: + - description: operationsList + steps: + - step: operationsList + exampleFile: ../examples/Operations_List.json + +``` + +## Reference + +- [oav](https://github.com/Azure/oav/tree/develop) diff --git a/documentation/test-scenario/how-to/import-postman-collection.png b/documentation/api-scenario/how-to/import-postman-collection.png similarity index 100% rename from documentation/test-scenario/how-to/import-postman-collection.png rename to documentation/api-scenario/how-to/import-postman-collection.png diff --git a/documentation/test-scenario/how-to/postman-collection-signalr.PNG b/documentation/api-scenario/how-to/postman-collection-signalr.PNG similarity index 100% rename from documentation/test-scenario/how-to/postman-collection-signalr.PNG rename to documentation/api-scenario/how-to/postman-collection-signalr.PNG diff --git a/documentation/test-scenario/how-to/runApiTest.gif b/documentation/api-scenario/how-to/runApiTest.gif similarity index 100% rename from documentation/test-scenario/how-to/runApiTest.gif rename to documentation/api-scenario/how-to/runApiTest.gif diff --git a/documentation/api-scenario/readme.md b/documentation/api-scenario/readme.md new file mode 100644 index 000000000000..2e1ec0a84514 --- /dev/null +++ b/documentation/api-scenario/readme.md @@ -0,0 +1,29 @@ +# API Scenario Documentation + +API Scenario is a YAML file defining RESTful API usage scenarios of your service with a series of API calls. API scenario can be used for service functional test, API quality validation and SDK/CLIs test generation. + +_**Caution**: This project is in early preview phase, hence breaking changes should be expected._ + +## Features + +- Simple to use: Intuitive step definition based on Swagger examples and raw REST call. +- ARM Template integration: Support creating external Azure resources with ARM Template and executing Azure Powershell or Azure CLI scripts with ARM Template deployment script. +- Implementation independent: [oav](https://github.com/Azure/oav) is the default API scenario runner, and more runners will be supported, like SDKs in different languages. + +### Demo gif + +![demo](./how-to/runApiTest.gif) + +## Quick start + +- [Example: Write and run your first API scenario file](./how-to/QuickStart.md) +- [Example: Generate a basic API scenario file](./how-to/generateABasicApiScenario.md) +- [Example: use armTemplate to generate unique resourceName](./how-to/apiScenarioWithARMTemplate.md) +- [API scenario file sample](../samplefiles/Microsoft.YourServiceName/stable/YYYY-MM-DD/scenarios/quickStart.yaml) + +## References + +- [API Scenario Definition Reference](./references/ApiScenarioDefinition.md) +- [API Scenario Variable Definition Reference](./references/Variables.md) +- [API Scenario Runner Reference](./references/Runner.md) +- [API Scenario JSON Schema](./references/v1.1/schema.json) diff --git a/documentation/test-scenario/references/TestDefinitionReference.md b/documentation/api-scenario/references/ApiScenarioDefinition.md similarity index 66% rename from documentation/test-scenario/references/TestDefinitionReference.md rename to documentation/api-scenario/references/ApiScenarioDefinition.md index 731d3127ccc6..be96bddb58b6 100644 --- a/documentation/test-scenario/references/TestDefinitionReference.md +++ b/documentation/api-scenario/references/ApiScenarioDefinition.md @@ -1,22 +1,21 @@ -# Test Definition Reference +# API Scenario Definition Reference -## Test Definition File +## API Scenario Definition File -See [Test Definition File Schema](./v1.0/schema.json#L1) +See [API Scenario Definition File Schema](./v1.1/schema.json#L1) File should be in format of yaml. **Example:** + ```yaml scope: ResourceGroup -requiredVariables: - - subscriptionId variables: publicIpAddressName: pubipdns prepareSteps: - step: prepare_resources armTemplateDeployment: ./dep-something.json -testScenarios: +scenarios: - description: test_network_public_ip steps: - step: Create_publicIPAddresses_pubipdns @@ -28,12 +27,13 @@ testScenarios: ``` **Fields:** + - **scope** - **Type:** Required, Enum - **Enum:** ResourceGroup - Now only "ResourceGroup" is supported. - - **ResourceGroup:** All of the following test scenario and steps should be under some resourceGroup. It means: - - The consumer (test scenario runner or anything consumes test scenario) SHOULD maintain the resource group itself. Usually it requires user to input the subscriptionId/location, then it creates the resource group before test running, and deletes the resource group after running + - **ResourceGroup:** All of the following API scenario and steps should be under some resourceGroup. It means: + - The consumer (API scenario runner or anything consumes API scenario) SHOULD maintain the resource group itself. Usually it requires user to input the subscriptionId/location, then it creates the resource group before test running, and deletes the resource group after running - The consumer SHOULD set the following variables: - **subscriptionId** - **resourceGroupName** @@ -42,25 +42,23 @@ testScenarios: - **variables** - **Type:** Optional, Map of strings - See [Variables](./Variables.md) -- **requiredVariables** - - **Type:** Optional, Array of string - - Variables that must be defined by user. By default, **subscriptionId** and **location** are required. - **prepareSteps** - - **Type:** Optional, Array of [Test Step](#test-step) - - Steps that should run before every test scenario steps. -- **testScenarios** - - **Type:** Required, Array of [Test Scenario](#test-scenario) + - **Type:** Optional, Array of [Step](#step) + - Steps that should run before every API scenario steps. +- **scenarios** + - **Type:** Required, Array of [Scenario](#scenario) -## Test Scenario +## Scenario -See [Test Scenario Schema](./v1.0/schema.json#L331). +See [Scenario Schema](./v1.1/schema.json#L83). -It defines one test scenario that could go through on its own. +It defines one API scenario that could go through on its own. **Example:** + ```yaml description: test_network_public_ip -shareTestScope: true +shareScope: true steps: - step: Create_publicIPAddresses_pubipdns resourceName: publicIPAddresses_pubipdns @@ -71,37 +69,39 @@ variables: ``` **Fields:** + - **description** - **Type:** Required, String - - Description for this test scenario. -- **shareTestScope** + - Description for this API scenario. +- **shareScope** - **Type:** Optional, Boolean or String - **Default:** true - - Describe how the testScope (ResourceGroup if scope is ResourceGroup) could be shared with other tests. If it's true or it's the same string setting for different test scenario, then they share the same test scope, which means: - - These tests will run under the same test scope (e.g. ResourceGroup). They may launch in parallel. - - **prepareSteps** will only run once in the testScope. The variables will be shared. - - By default all the test scenario in one test definition file will be launched in the same test scope. If shareTestScope is false then it will not share anything with other test scenarios in the same file. + - Describe how the scope (ResourceGroup if scope is ResourceGroup) could be shared with other tests. If it's true or it's the same string setting for different API scenario, then they share the same scope, which means: + - These tests will run under the same scope (e.g. ResourceGroup). They may launch in parallel. + - **prepareSteps** will only run once in the scope. The variables will be shared. + - By default all the API scenario in one definition file will be launched in the same scope. If shareScope is false then it will not share anything with other API scenarios in the same file. - **variables** - **Type:** Optional, Map of strings - See [Variables](./Variables.md) - **steps** - - **Type:** Required, Array of [Test Step](#test-step) - - Steps in this test scenario + - **Type:** Required, Array of [Step](#step) + - Steps in this API scenario -## Test Step +## Step -See [Test Step Schema](./v1.0/schema.json#L50). +See [Step Schema](./v1.1/schema.json#L114). -Defines one test step in test scenario. +Defines one step in API scenario. Should be one of the following: -- [Test Step](#test-step) -- [Test Step ARM Template Deployment](#test-step-arm-template-deployment) -- [Test Step Rest Call](#test-step-rest-call) - - [Rest Call](#rest-call) - - [Rest Call by ResourceName Tracking and Update](#rest-call-by-resourcename-tracking-and-update) + +- [Step REST Call](#step-rest-call) + - [REST Call](#rest-call) + - [REST Call by ResourceName Tracking and Update](#rest-call-by-resourcename-tracking-and-update) +- [Step ARM Template Deployment](#step-arm-template-deployment) All of the above definitions share the following fields: + - **variables** - **Type:** Optional, Map of Strings - See [Variables](./Variables.md) @@ -109,30 +109,28 @@ All of the above definitions share the following fields: - **Type:** Required, String - Step name. Must be unique in the same file. -## Test Step ARM Template Deployment +## Step ARM Template Deployment -See [Test Step ARM Template Deployment Schema](./v1.0/schema.json#L78). +See [Step ARM Template Deployment Schema](./v1.1/schema.json#L247). -Step to deploy ARM template to the test scope. Template parameters and outputs will also interact with variables automatically, see [Variables](./Variables.md). +Step to deploy ARM template to the scope. Template parameters and outputs will also interact with variables automatically, see [Variables](./Variables.md). **Example:** + ```yaml step: prepare_resources armTemplateDeployment: ./dep-storage-account.json -armTemplateParameters: ./dep-storage-account-params.json ``` **Fields:** + - **armTemplateDeployment** - **Type:** Required, String - Path to ARM template json file. See [ARM Template](https://docs.microsoft.com/azure/templates/). -- **armTemplateParameters** - - **Type:** Optional, String - - Path to ARM template parameter file. See [ARM Template Parameter File](https://docs.microsoft.com/azure/azure-resource-manager/templates/parameter-files). -## Test Step Rest Call +## Step REST Call -See [Test Step Rest Call Schema](./v1.0/schema.json#L97) +See [Step REST Call Schema](./v1.1/schema.json#L205) Step to run a swagger operation defined rest call. This may not be just one http call. @@ -144,11 +142,13 @@ Step to run a swagger operation defined rest call. This may not be just one http Rest call step could be defined either by an example file, or by resourceName tracking and update. Rest call will have computed **requestParameter** and **responseExpected** after parsing and loading: -- **requestParameter** -### Rest Call +- **requestParameter** + +### REST Call **Example:** + ```yaml step: Create_publicIPAddresses_pubipdns resourceName: publicIPAddresses_pubipdns @@ -158,6 +158,7 @@ statusCode: 200 ``` **Fields:** + - **exampleFile** - **Type:** Optional, String - Path to example file. Should be in format of "x-ms-example" files. @@ -184,6 +185,7 @@ statusCode: 200 ### Rest Call by ResourceName Tracking and Update **Example** + ```yaml - step: Create_publicIPAddresses_pubipdns resourceName: publicIPAddresses_pubipdns @@ -194,13 +196,14 @@ statusCode: 200 - step: Update_publicIPAddresses resourceName: publicIPAddresses_pubipdns resourceUpdate: - - replace: /properties/location - value: westus + - replace: /properties/location + value: westus ``` -Different steps with the same resourceName will be tracked by the test scenario. It knows that you are trying to update the same resource. You can use the first request with example to specify the request and resource id, then the following step with the same resourceName will use the same resource id to update the resource. For the +Different steps with the same resourceName will be tracked by the API scenario. It knows that you are trying to update the same resource. You can use the first request with example to specify the request and resource id, then the following step with the same resourceName will use the same resource id to update the resource. For the **Fields:** + - **resourceName** - **Type:** Required, String - The user-defined resource name of the resource to be tracked. It's only used as a name of that resource and do not need to be same as the actual resource name. @@ -218,36 +221,41 @@ resourceUpdate will help to automate compute the request body and the expected r ### JsonPatchOp -JsonPatchOp is used to define the update operation on json. You could add, remove, replace, move, copy and merge on json path. +JsonPatchOp is used to define the update operation on json. You could add, remove, replace, move, copy and merge on json path. All the json path used in JsonPatchOp is in format of [JsonPointer](https://datatracker.ietf.org/doc/html/rfc6901). - - [JsonPatchOp](#jsonpatchop) - - [JsonPatchOpAdd](#jsonpatchopadd) - - [JsonPatchOpRemove](#jsonpatchopremove) - - [JsonPatchOpReplace](#jsonpatchopreplace) - - [JsonPatchOpMove](#jsonpatchopmove) - - [JsonPatchOpCopy](#jsonpatchopcopy) - - [JsonPatchOpMerge](#jsonpatchopmerge) +- [JsonPatchOp](#jsonpatchop) + - [JsonPatchOpAdd](#jsonpatchopadd) + - [JsonPatchOpRemove](#jsonpatchopremove) + - [JsonPatchOpReplace](#jsonpatchopreplace) + - [JsonPatchOpCopy](#jsonpatchopcopy) + - [JsonPatchOpMove](#jsonpatchopmove) + - [JsonPatchOpTest](#jsonpatchoptest) + #### JsonPatchOpAdd **Example** + ```yaml add: /properties/items value: 1 ``` **Fields:** + - **add** - **Type:** Required, JsonPointer - **value** - **Type:** Required, Any Add json property at specified path. + 1. If any segment of path does not exist, then it will be created. 2. If any value already exists on the path, then it will be overwritten. 3. If the parent of the destination is array, then the value will be inserted at the specified index. **Example of add** + ``` apply: - add: /properties/location @@ -274,19 +282,23 @@ result: #### JsonPatchOpRemove **Example** + ```yaml remove: /properties/items/1 ``` **Fields:** + - **remove** - **Type:** Required, JsonPointer Remove element at specified path. + 1. If any segment of path does not exist, then error will be thrown. 2. If parent of the specified path is array, then the element will be removed from the array. **Example of remove** + ``` apply: - remove: /properties/items @@ -311,22 +323,26 @@ result: #### JsonPatchOpReplace **Example** + ```yaml replace: /properties/items value: 1 ``` **Fields:** + - **replace** - **Type:** Required, JsonPointer - **value** - **Type:** Required, Any Replace json property at specified path. + 1. If any segment of path does not exist, error will be thrown. 2. If any value already exists on the path, then it will be overwritten. **Example of replace** + ``` apply: - replace: /properties/location @@ -339,93 +355,98 @@ result: - { "properties": { "location": "eastus" } } } ``` -#### JsonPatchOpMove +#### JsonPatchOpCopy **Example** + ```yaml -move: /properties/items -path: /properties/items2 +copy: /properties/items2 +from: /properties/items ``` **Fields:** -- **move** + +- **copy** - **Type:** Required, JsonPointer -- **path** +- **from** - **Type:** Required, JsonPointer -Move json property at specified path to another path. It works as a combination of remove followed by add. Array index is also supported and works as add/remove does. +Copy json property from specified path to another path. Array index is also supported and works as add/remove does. + +**Example of copy** -**Example of move** ``` apply: -- move: /properties/items - path: /properties/items2 +- copy: /properties/items2 + from: /properties/items on data: - { "properties": { "items": [1, 2, 3] } } result: -- { "properties": { "items2": [1, 2, 3] } } +- { "properties": { "items": [1, 2, 3] }, "items2": [1, 2, 3] } } ``` -#### JsonPatchOpCopy + +#### JsonPatchOpMove **Example** + ```yaml -copy: /properties/items -path: /properties/items2 +move: /properties/items2 +from: /properties/items ``` **Fields:** -- **copy** + +- **move** - **Type:** Required, JsonPointer -- **path** +- **from** - **Type:** Required, JsonPointer -Copy json property at specified path to another path. Array index is also supported and works as add/remove does. +Move json property from specified path to another path. It works as a combination of remove followed by add. Array index is also supported and works as add/remove does. + +**Example of move** -**Example of copy** ``` apply: -- copy: /properties/items - path: /properties/items2 +- move: /properties/items2 + from: /properties/items on data: - { "properties": { "items": [1, 2, 3] } } result: -- { "properties": { "items": [1, 2, 3] }, "items2": [1, 2, 3] } } +- { "properties": { "items2": [1, 2, 3] } } ``` -#### JsonPatchOpMerge + +#### JsonPatchOpTest **Example** + ```yaml -merge: /properties/item -value: - a: 1 - b: 2 +test: /properties/item +value: a ``` **Fields:** -- **merge** + +- **test** - **Type:** Required, JsonPointer - **value** - **Type:** Required, Object -Merge values into the object at specified path. -1. Property value at the specified path must be an object. -2. Properties with same key will be overwritten. +Test that a value at the target location is equal to a specified value. + +**Example of test** -**Example of merge** ``` apply: -- merge: /properties - value: - a: 1 - b: 2 +- test: /properties/a + value: 1 on data: -- { "properties": { "b": 0, "c": 0} } +- { "properties": { "a": 0, "b": 1} } result: -- { "properties": { "a": 1, "b": 2, "c": 0 } } -``` \ No newline at end of file +- throws error +``` diff --git a/documentation/test-scenario/references/ErrorCodeReference.md b/documentation/api-scenario/references/ErrorCodeReference.md similarity index 100% rename from documentation/test-scenario/references/ErrorCodeReference.md rename to documentation/api-scenario/references/ErrorCodeReference.md diff --git a/documentation/api-scenario/references/Runner.md b/documentation/api-scenario/references/Runner.md new file mode 100644 index 000000000000..1fddd6ff8330 --- /dev/null +++ b/documentation/api-scenario/references/Runner.md @@ -0,0 +1,130 @@ +# Runner Behavior + +This document explains the expected behavior of runner. The word "runner" here references to any customer that consume the API scenario, including the runner that send out requests defined by API scenario, code generator that generate code that executes steps defined by API scenario, and any other consumer that need to understand the content of API scenario. + +## Load API Scenario via OAV + +You could load the API scenario file via oav. It would be resolved as a simple object. + +```typescript +const readmeMd: string = + "/home/username/azure-rest-api-specs/specification/containerservice/resource-manager/readme.md"; +const argv = { + ["try-require"]: "readme.test.md", + tag: "package-2020-12" +}; + +// Get input-file config in readme.md +const autorestConfig = await getAutorestConfig(argv, readmeMd); +const swaggerFilePaths: string[] = autorestConfig["input-file"]; +const fileRoot = dirname(readmeMd); + +console.log("input-file:"); +console.log(swaggerFilePaths); + +// Create the loader from OAV +const loader = ApiScenarioLoader.create({ + useJsonParser: false, + checkUnderFileRoot: false, + fileRoot, + swaggerFilePaths +}); + +// Load the API scenario file. File list could also be specified in readme.test.md +const scenarioDef = await loader.load( + "Microsoft.ContainerService/stable/2020-12-01/scenarios/containerService.yaml" +); + +console.log(scenarioDef.scenarios[0].steps); + +// Setup initial variable env +const env = new VariableEnv(); +env.setBatch({ + subscriptionId: "__your_subs_id_", + location: "westus", + SSH_PUBLIC_KEY: "__public_key_ssh__" +}); + +// Reference runner implementation in OAV. You need to implement your own runner. +const runner = new ApiScenarioRunner({ + jsonLoader: loader.jsonLoader, + env, + client: new ApiScenarioRestClient(getDefaultAzureCredential(), {}) +}); + +try { + for (const scenario of scenarioDef.scenarios) { + await runner.executeScenario(scenario); + } +} catch (e) { + console.log(e.message, e.stack); +} finally { + console.timeLog("TestLoad"); + await runner.cleanAllScope(); +} +``` + +After the API scenario is loaded, the step will be slightly different from the file content. Every REST step will have the following resolved fields: + +- requestParameters + - Type: `object`, map of resolved parameter name and value. +- responseExpected + - Type: `any` + - The expected response body from the request. + +## Procedure of runner + +### Input + +Let's assume the following things as input of runner: + +- API scenario definition that loaded via OAV. +- The scenario id of the API scenario. API scenario definition file could contains multiple API scenarios, runner need to run one of them. +- Extra environment variables that required by API scenario, defined in `requiredVariables`. + +### Scope + +It's the `scope` field defined at top level of API scenario file. Now only `ResourceGroup` is supported, it means that the API scenario will: + +- Run under specified resource group. +- Runner would manage the resource group, it could create the resource group (defined by variable `subscriptionId`, `resourceGroup`, `location`) or use predefined resource group, it could also delete the resource group after the API scenario is done. Runner itself is responsible for managing the resource group, the behavior is not defined by this spec. +- Runner would run all the arm template deployment under the specified resource group. + +The scope is a convention, however it would not be enforced by API scenario. User could override the variable `resourceGroup` in any step to run that step in another resource group for example. + +### Variables + +See [Variables](./Variables.md) for variable spec. The runner must follow the variable definition in API scenario. Runner do not need to care about the variable conventions as it's already resolved by OAV. The runner must: + +- Load variables layer by layer as defined in the variable spec. +- Resolve variables like `$(variableName)` step by step in: + - requestParameter + - responseExpected (if it's used by runner) + - armTemplate payload + +### Procedure + +- Load definition via OAV, load required variables' value (runner need to specify how to load it). +- Manage the scope, runner could create/reuse the scope as user defined in input. +- Run top level prepareSteps if it has not run. it's a list of steps defined in API scenario. +- Pass the variables from prepareSteps to the following main steps. +- For each steps defined in the API scenario array: + - If `type` field is `restCall`: + - Replace variables in `requestParameters`. + - Fill the request via parameter definition in swagger and parameter value in `requestParameters`. + - Send out the request. + - If the request is long running request, runner need to poll for the response. + - If the response's long running poll's final call is operation status, and the step itself is resource PUT/PATCH/DELETE, runner could run another GET against the resource to check. For DELETE, the final GET is expected to return 404. For PUT/PATCH, the final GET is expected to return 200, and it should represent the final response of the step. + - Check if the response status code is the same as the expected `statusCode` field defined in step. Optional. + - Check if the response body is the same as the expected `responseExpected` field defined in step. Optional. + - If `outputVariables` is defined, runner need to extract and define the variable from specified path in response body. + - If `type` field is `armTemplateDeployment`: + - Use the convention to replace arm template parameters if the parameter name matches the variable name and the parameter type is string. + - Send arm template deployment request under the resource group. + - Wait for the deployment to finish. + - Get the output variables from the deployment. Define the variables from the output variables. + - Else `type` is unsupported in runner. + +### Compare the response with expectedResponse + +It's hard for service team to make sure every field in expectedResponse is the same as the response, so here API scenario suggest to compare properties that are not `readOnly` and are not `x-ms-secret`. The detail should be defined by the runner, not this spec. diff --git a/documentation/test-scenario/references/Variables.md b/documentation/api-scenario/references/Variables.md similarity index 66% rename from documentation/test-scenario/references/Variables.md rename to documentation/api-scenario/references/Variables.md index 6c9329e57fc9..ebd04efc0372 100644 --- a/documentation/test-scenario/references/Variables.md +++ b/documentation/api-scenario/references/Variables.md @@ -1,36 +1,40 @@ -# Variables in test scenario +# Variables in API scenario ## Variable definition and replacement -Variables could be defined in different level of test scenario: -- Test Definition level variable definition -- Test Scenario level variable definition -- Test Step level variable definition +Variables could be defined in different level of API scenario: -Variable could be referenced by `$(variableName)`. Currently variable type must be string. +- `runtime`: Variables specified at runtime +- `global`: API scenario definition level variable definition +- `scope`: Scope level variable +- `scenario`: API scenario level variable definition +- `step`: Step level variable definition -For example, in the following test scenario: +Variable could be referenced by `$(variableName)`. Currently variable type must be string. + +For example, in the following API scenario: ```yaml variables: resourceName: level-1 -test-scenarios: -- definition: Create some resource - variables: - resourceName: level-2 - steps: - - step: Create resource +scenarios: + - definition: Create some resource variables: - resourceName: level-3 - exampleFile: ../examples/ResourceCreate.json + resourceName: level-2 + steps: + - step: Create resource + variables: + resourceName: level-3 + exampleFile: ../examples/ResourceCreate.json ``` if in `../examples/ResourceCreate.json` we have `$(resourceName)` in some string, it would be replaced with `level-3`. -Variables could also be defined on test running. For example you could set `subscriptionId` or `resourceGroupName` on the global scope. How to set global env is based on the test scenario consumer. +Variables could also be defined on test running. For example you could set `subscriptionId` or `resourceGroupName` on the global scope. How to set global env is based on the API scenario consumer. Variables could be replaced recursively. For example if we have the following variables: + ```yaml variables: resourceName: abc @@ -43,11 +47,13 @@ Variable resolving is limited to at most 100 times for certain string. ## Convention: parameter name in example -In one rest call step, if we have any parameter, for example it's named `param`, in the step requestParameters, with value `paramValue`, and test scenario has variable `param`, then by default, test scenario loader will: +In one rest call step, if we have any parameter, for example it's named `param`, in the step requestParameters, with value `paramValue`, and API scenario has variable `param`, then by default, API scenario loader will: + - replace the parameter value of `param` to `$(param)` so that it will reference the variable value. - replace `paramValue` to `$(param)` in every string in request body (in requestParameter) and in responseExpected. For example, for the following step with loaded requestParameter and responseExpected + ```yaml requestParameters: subscriptionId: 00000000-0000-0000-0000-000000000000 @@ -88,9 +94,10 @@ With this convention, you could control most of the parameters with variables. ## Convention: location -In one rest call step, if we have variable `location` (exact match) in the test scenario, and we have `location` as top level property defined in request body (`requestParameters[bodyParamName]`) and response body (responseExpected), then the top level location property will be replaced with variable value of location. +In one rest call step, if we have variable `location` (exact match) in the API scenario, and we have `location` as top level property defined in request body (`requestParameters[bodyParamName]`) and response body (responseExpected), then the top level location property will be replaced with variable value of location. For example, + ```yaml requestParameters: parameters: @@ -115,7 +122,7 @@ responseExpected: ## Convention: Arm Template Deployment -When you deploy arm template in test scenario, you could define template parameters and outputs. By default if the parameter name matches the variable exists and the parameter type is string, then the parameter value would use the variable value. If the template has output which is string type, the variables will be set with output values. +When you deploy arm template in API scenario, you could define template parameters and outputs. By default if the parameter name matches the variable exists and the parameter type is string, then the parameter value would use the variable value. If the template has output which is string type, the variables will be set with output values. For example, given the following arm template: @@ -123,18 +130,17 @@ For example, given the following arm template: { "parameters": { "userName": { - "type": "string" + "type": "string" } }, - "resources": [ - ], + "resources": [], "outputs": { "nameResult": { - "type": "string", - "value": "[concat('prefix/', parameters['userName'])]" + "type": "string", + "value": "[concat('prefix/', parameters['userName'])]" } } } ``` -If we have variable `userName` defined with `abc`, then we will have variable `nameResult` defined with value `prefix/abc` so that following steps in the test scenario could use variable `nameResult`. +If we have variable `userName` defined with `abc`, then we will have variable `nameResult` defined with value `prefix/abc` so that following steps in the API scenario could use variable `nameResult`. diff --git a/documentation/test-scenario/references/v1.0/schema.json b/documentation/api-scenario/references/v1.1/schema.json similarity index 55% rename from documentation/test-scenario/references/v1.0/schema.json rename to documentation/api-scenario/references/v1.1/schema.json index 8d31455e0eb5..9efa827cffe5 100644 --- a/documentation/test-scenario/references/v1.0/schema.json +++ b/documentation/api-scenario/references/v1.1/schema.json @@ -1,11 +1,6 @@ { - "$schema": "http://json-schema.org/draft-04/schema#", + "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", - "allOf": [ - { - "$ref": "#/definitions/VariableScope" - } - ], "properties": { "scope": { "type": "string", @@ -13,134 +8,172 @@ "ResourceGroup" ] }, - "requiredVariables": { + "variables": { + "$ref": "#/definitions/Variables" + }, + "prepareSteps": { "type": "array", + "description": "Prepare steps before executing scenarios", "items": { - "type": "string" + "$ref": "#/definitions/Step" } }, - "prepareSteps": { + "scenarios": { "type": "array", + "description": "API scenarios", "items": { - "$ref": "#/definitions/TestStep" - } + "$ref": "#/definitions/Scenario" + }, + "minItems": 1 }, - "testScenarios": { + "cleanUpSteps": { "type": "array", + "description": "Clean up steps after executing scenarios", "items": { - "$ref": "#/definitions/TestScenario" + "$ref": "#/definitions/Step" } } }, "required": [ - "testScenarios" + "scenarios" ], + "additionalProperties": false, "definitions": { "Name": { "type": "string", - "pattern": "^[a-zA-Z0-9_-]+$" + "pattern": "^[A-Za-z_][A-Za-z0-9_-]*$" }, "JsonPointer": { "type": "string", - "description": "String syntax for identifying a specific value within JSON document", + "description": "JSON Pointer described by RFC 6901, e.g. /foo/bar", "pattern": "^(/(([^/~])|(~[01]))*)*$" }, - "VariableScope": { + "Variables": { + "type": "object", + "propertyNames": { + "$ref": "#/definitions/Name" + }, + "additionalProperties": { + "oneOf": [ + { + "type": "string", + "description": "Default value of the variable" + }, + { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "string", + "secureString" + ], + "default": "string" + }, + "defaultValue": { + "type": "string", + "description": "Default value of the variable" + } + }, + "additionalProperties": false + } + ] + } + }, + "Scenario": { "type": "object", "properties": { + "scenario": { + "$ref": "#/definitions/Name", + "description": "Name of the scenario" + }, + "description": { + "type": "string", + "description": "A long description of the scenario" + }, "variables": { - "type": "object", - "additionalProperties": true + "$ref": "#/definitions/Variables" + }, + "shareScope": { + "type": "boolean", + "description": "Whether to share the scope and prepareSteps with other scenarios", + "default": true + }, + "steps": { + "type": "array", + "items": { + "$ref": "#/definitions/Step" + } } - } + }, + "required": [ + "steps" + ], + "additionalProperties": false }, - "TestStep": { + "Step": { "oneOf": [ { - "$ref": "#/definitions/TestStepRestCall" + "$ref": "#/definitions/StepRestCall" }, { - "$ref": "#/definitions/TestStepArmTemplateDeployment" + "$ref": "#/definitions/StepRestOperation" }, { - "$ref": "#/definitions/TestStepRawCall" + "$ref": "#/definitions/StepArmTemplateDeployment" + }, + { + "$ref": "#/definitions/StepRawCall" } ] }, - "TestStepBase": { + "StepBase": { "properties": { "step": { - "$ref": "#/definitions/Name" + "$ref": "#/definitions/Name", + "description": "Name of the step" }, - "type": { - "type": "string" + "description": { + "type": "string", + "description": "A long description of the step" + }, + "variables": { + "$ref": "#/definitions/Variables" }, "outputVariables": { "type": "object", + "propertyNames": { + "$ref": "#/definitions/Name" + }, "additionalProperties": { "type": "object", "properties": { + "type": { + "type": "string", + "enum": [ + "string", + "secureString" + ], + "default": "string" + }, "fromResponse": { "type": "string" } } } } - }, - "required":[ - "step" - ] - }, - "TestStepArmTemplateDeployment": { - "type": "object", - "allOf": [ - { - "$ref": "#/definitions/TestStepBase" - } - ], - "properties": { - "type": { - "type": "string", - "enum": [ - "armTemplate" - ] - }, - "armTemplateDeployment": { - "type": "string" - }, - "armTemplateParameters": { - "type": "string" - } - }, - "required": [ - "armTemplateDeployment" - ] + } }, - "TestStepRestCall": { - "type": "object", + "StepRestBase": { "allOf": [ { - "$ref": "#/definitions/VariableScope" - }, - { - "$ref": "#/definitions/TestStepBase" + "$ref": "#/definitions/StepBase" } ], "properties": { - "type": { - "type": "string", - "enum": [ - "exampleFile" - ] - }, - "resourceName": { - "type": "string" - }, - "exampleFile": { - "type": "string" - }, "resourceUpdate": { "type": "array", + "description": "Update resource properties in body for both request and expected response", "items": { "$ref": "#/definitions/JsonPatchOp" }, @@ -148,41 +181,93 @@ }, "requestUpdate": { "type": "array", + "description": "Update request parameters", "items": { "$ref": "#/definitions/JsonPatchOp" - } + }, + "minItems": 1 }, "responseUpdate": { "type": "array", + "description": "Update expected response", "items": { "$ref": "#/definitions/JsonPatchOp" - } - }, - "operationId": { - "type": "string" + }, + "minItems": 1 }, "statusCode": { - "type": "number" + "type": "integer", + "description": "Expected response code", + "default": 200 } } }, - "TestStepRawCall": { + "StepRestCall": { "type": "object", "allOf": [ { - "$ref": "#/definitions/VariableScope" + "$ref": "#/definitions/StepRestBase" + } + ], + "properties": { + "exampleFile": { + "type": "string" }, + "resourceName": { + "$ref": "#/definitions/Name", + "description": "Name a resource for tracking" + } + }, + "required": [ + "exampleFile" + ] + }, + "StepRestOperation": { + "type": "object", + "allOf": [ { - "$ref": "#/definitions/TestStepBase" + "$ref": "#/definitions/StepRestBase" } ], "properties": { - "type": { + "operationId": { "type": "string", - "enum": [ - "rawCall" - ] + "description": "The operationId to perform on a tracking resource" }, + "resourceName": { + "$ref": "#/definitions/Name", + "description": "Reference a tracking resource" + } + }, + "required": [ + "operationId", + "resourceName" + ] + }, + "StepArmTemplateDeployment": { + "type": "object", + "allOf": [ + { + "$ref": "#/definitions/StepBase" + } + ], + "properties": { + "armTemplateDeployment": { + "type": "string" + } + }, + "required": [ + "armTemplateDeployment" + ] + }, + "StepRawCall": { + "type": "object", + "allOf": [ + { + "$ref": "#/definitions/StepBase" + } + ], + "properties": { "method": { "type": "string", "enum": [ @@ -204,15 +289,13 @@ "type": "string" } }, - "requestBody": { - "type": "string" - }, + "requestBody": {}, "statusCode": { - "type": "number" + "type": "integer", + "description": "Expected response code", + "default": 200 }, - "responseExpected": { - "type": "string" - } + "expectedResponse": {} }, "required": [ "method", @@ -223,6 +306,7 @@ }, "JsonPatchOp": { "type": "object", + "description": "Change a JSON document in a format described by RFC 6902", "oneOf": [ { "$ref": "#/definitions/JsonPatchOpAdd" @@ -240,7 +324,7 @@ "$ref": "#/definitions/JsonPatchOpMove" }, { - "$ref": "#/definitions/JsonPatchOpMerge" + "$ref": "#/definitions/JsonPatchOpTest" } ] }, @@ -288,13 +372,13 @@ "type": "object", "required": [ "copy", - "path" + "from" ], "properties": { "copy": { "$ref": "#/definitions/JsonPointer" }, - "path": { + "from": { "$ref": "#/definitions/JsonPointer" } }, @@ -304,71 +388,31 @@ "type": "object", "required": [ "move", - "path" + "from" ], "properties": { "move": { "$ref": "#/definitions/JsonPointer" }, - "path": { + "from": { "$ref": "#/definitions/JsonPointer" } }, "additionalProperties": false }, - "JsonPatchOpMerge": { + "JsonPatchOpTest": { "type": "object", "required": [ - "merge", + "test", "value" ], "properties": { - "merge": { + "test": { "$ref": "#/definitions/JsonPointer" }, - "value": { - "type": "object", - "additionalProperties": true - } + "value": {} }, "additionalProperties": false - }, - "TestScenario": { - "type": "object", - "allOf": [ - { - "$ref": "#/definitions/VariableScope" - } - ], - "properties": { - "scenario": { - "$ref": "#/definitions/Name", - "description": "Name of scenario" - }, - "description": { - "type": "string" - }, - "requiredVariables": { - "type": "array", - "items": { - "type": "string" - } - }, - "steps": { - "type": "array", - "items": { - "$ref": "#/definitions/TestStep" - } - }, - "dependsOn": { - "$ref": "#/definitions/Name", - "description": "Name of scenario that is depended on" - } - }, - "required": [ - "description", - "steps" - ] } } } diff --git a/documentation/samplefiles/Microsoft.YourServiceName/stable/YYYY-MM-DD/scenarios/testYourService.yaml b/documentation/samplefiles/Microsoft.YourServiceName/stable/YYYY-MM-DD/scenarios/quickStart.yaml similarity index 100% rename from documentation/samplefiles/Microsoft.YourServiceName/stable/YYYY-MM-DD/scenarios/testYourService.yaml rename to documentation/samplefiles/Microsoft.YourServiceName/stable/YYYY-MM-DD/scenarios/quickStart.yaml diff --git a/documentation/test-scenario/how-to/generateABasicTestScenario.md b/documentation/test-scenario/how-to/generateABasicTestScenario.md deleted file mode 100644 index e1302cdb59db..000000000000 --- a/documentation/test-scenario/how-to/generateABasicTestScenario.md +++ /dev/null @@ -1,57 +0,0 @@ -# Generate a basic test scenario file - -## Prerequisite - -We use `oav` tools to generate basic test scenario. `oav` analyze swagger file and use swagger example as test scenario steps. So first, you need to install the latest oav. - -## Introduction - -`oav` support rule based test scenario file generation. We use this command to generate test scenario file. - -`oav generate-static-test-scenario --readme --tag --rules ` - -- readme: swagger readme file. -- tag: which tag to generate. oav will analyze swagger file under the tag and generate test scenario. -- rules: Currently support two types. `resource-put-delete`, `operations-list`. Default: `resource-put-delete` - - `resource-put-delete`: generate resource put and delete test scenario. - - `operations-list`: generate operations list test scenario. `operations-list` is the simplest API which must be defined in swagger. - -Example: - -![](./genTestScenario.gif) - -This command will load and analyze swagger and generate a basic test scenario file (`resource-put-delete`). - -Result: the output contains two files - -- test-scenarios/signalR.yaml: The test scenario file. -- readme.test.md: The entry for SDK test generation - -The generated test scenario file: The generated test scenario file contains two steps. Create signalR and delete it. It's a basic test scenario and developer can add more step based on the basic test scenario file. - -``` -scope: ResourceGroup -testScenarios: - - description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate - steps: - - step: SignalR_CreateOrUpdate - exampleFile: ../examples/SignalR_CreateOrUpdate.json - - step: SignalR_Delete - exampleFile: ../examples/SignalR_Delete.json -``` - -If you pass rule option `operations-list`, you will get such test scenario file. - -``` -scope: ResourceGroup -testScenarios: - - description: operationsList - steps: - - step: operationsList - exampleFile: ../examples/Operations_List.json - -``` - -## Reference - -- [oav](https://github.com/Azure/oav/tree/develop) diff --git a/documentation/test-scenario/readme.md b/documentation/test-scenario/readme.md index 1103f0bcf7fa..6761413200be 100644 --- a/documentation/test-scenario/readme.md +++ b/documentation/test-scenario/readme.md @@ -1,26 +1 @@ -# Test Scenario Documentation - -Test Scenario is a YAML file defining RESTful API usage scenarios of your service with a series of API calls. Test scenario can be used for service functional test, API quality validation and SDK/CLIs test generation. - -_**Caution**: This project is in early preview phase, hence breaking changes should be expected._ -## Features -- Simple to use: Intuitive test step definition based on Swagger examples and raw REST call. -- ARM Template integration: Support creating external Azure resources with ARM Template and executing Azure Powershell or Azure CLI scripts with ARM Template deployment script. -- Implementation independent: [oav](https://github.com/Azure/oav) is the default test scenario runner, and more runners will be supported, like SDKs in different languages. - -### Demo gif - -![demo](./how-to/runApiTest.gif) - -## Quick start - -- [Example: Write and run your first test scenario file](./how-to/QuickStart.md) -- [Example: Generate a basic test scenario file](./how-to/generateABasicTestScenario.md) -- [Example: use armTemplate to generate unique resourceName](./how-to/testScenarioWithARMTemplate.md) -- [Test scenario file sample](../samplefiles/Microsoft.YourServiceName/stable/YYYY-MM-DD/scenarios/testYourService.yaml) - -## References -- [Test Scenario Definition Reference](./references/TestDefinitionReference.md) -- [Test Scenario Variable Definition Reference](./references/Variables.md) -- [Test Scenario Runner Reference](./references/Runner.md) -- [Test Scenario Schema Reference](./references/v1.0/schema.json) +Moved to [api-scenario/readme.md](../api-scenario/readme.md) diff --git a/documentation/test-scenario/references/Runner.md b/documentation/test-scenario/references/Runner.md deleted file mode 100644 index d7d553043b6c..000000000000 --- a/documentation/test-scenario/references/Runner.md +++ /dev/null @@ -1,130 +0,0 @@ -# Runner Behavior - -This document explains the expected behavior of runner. The word "runner" here references to any customer that consume the test scenario, including the runner that send out requests defined by test scenario, code generator that generate code that executes steps defined by test scenario, and any other consumer that need to understand the content of test scenario. - -## Load Test Scenario via OAV - -You could load the test scenario file via oav. It would be resolved as a simple object. - -```typescript - const readmeMd: string = - "/home/username/azure-rest-api-specs/specification/containerservice/resource-manager/readme.md"; - const argv = { - ["try-require"]: "readme.test.md", - tag: "package-2020-12", - }; - - // Get input-file config in readme.md - const autorestConfig = await getAutorestConfig(argv, readmeMd); - const swaggerFilePaths: string[] = autorestConfig["input-file"]; - const fileRoot = dirname(readmeMd); - - console.log("input-file:"); - console.log(swaggerFilePaths); - - // Create the loader from OAV - const loader = TestResourceLoader.create({ - useJsonParser: false, - checkUnderFileRoot: false, - fileRoot, - swaggerFilePaths, - }); - - // Load the test scenario file. File list could also be specified in readme.test.md - const testDef = await loader.load( - "Microsoft.ContainerService/stable/2020-12-01/test-scenarios/containerService.yaml" - ); - - console.log(testDef.testScenarios[0].steps); - - // Setup initial variable env - const env = new VariableEnv(); - env.setBatch({ - subscriptionId: "__your_subs_id_", - location: "westus", - SSH_PUBLIC_KEY: "__public_key_ssh__", - }); - - // Reference runner implementation in OAV. You need to implement your own runner. - const runner = new TestScenarioRunner({ - jsonLoader: loader.jsonLoader, - env, - client: new TestScenarioRestClient(getDefaultAzureCredential(), {}), - }); - - try { - for (const scenario of testDef.testScenarios) { - await runner.executeScenario(scenario); - } - } catch (e) { - console.log(e.message, e.stack); - } finally { - console.timeLog("TestLoad"); - await runner.cleanAllTestScope(); - } -``` - -After the test scenario is loaded, the test step will be slightly different from the file content. Every rest step will have the following resolved fields: - -- requestParameters - - Type: `object`, map of resolved parameter name and value. -- responseExpected - - Type: `any` - - The expected response body from the request. - -## Procedure of runner - -### Input - -Let's assume the following things as input of runner: - -- Test scenario definition that loaded via OAV. -- The scenario id of the test scenario. Test scenario definition file could contains multiple test scenarios, runner need to run one of them. -- Extra environment variables that required by test scenario, defined in `requiredVariables`. - -### Scope - -It's the `scope` field defined at top level of test scenario file. Now only `ResourceGroup` is supported, it means that the test scenario will: - -- Run under specified resource group. -- Runner would manage the resource group, it could create the resource group (defined by variable `subscriptionId`, `resourceGroup`, `location`) or use predefined resource group, it could also delete the resource group after the test scenario is done. Runner itself is responsible for managing the resource group, the behavior is not defined by this spec. -- Runner would run all the arm template deployment under the specified resource group. - -The scope is a convention, however it would not be enforced by test scenario. User could override the variable `resourceGroup` in any step to run that step in another resource group for example. - -### Variables - -See [Variables](./Variables.md) for variable spec. The runner must follow the variable definition in test scenario. Runner do not need to care about the variable conventions as it's already resolved by OAV. The runner must: - -- Load variables layer by layer as defined in the variable spec. -- Resolve variables like `$(variableName)` step by step in: - - requestParameter - - responseExpected (if it's used by runner) - - armTemplate payload - -### Procedure - -- Load definition via OAV, load required variables' value (runner need to specify how to load it). -- Manage the test scope, runner could create/reuse the scope as user defined in input. -- Run top level prepareSteps if it has not run. it's a list of steps defined in test scenario. -- Pass the variables from prepareSteps to the following main steps. -- For each steps defined in the test scenario array: - - If `type` field is `restCall`: - - Replace variables in `requestParameters`. - - Fill the request via parameter definition in swagger and parameter value in `requestParameters`. - - Send out the request. - - If the request is long running request, runner need to poll for the response. - - If the response's long running poll's final call is operation status, and the step itself is resource PUT/PATCH/DELETE, runner could run another GET against the resource to check. For DELETE, the final GET is expected to return 404. For PUT/PATCH, the final GET is expected to return 200, and it should represent the final response of the step. - - Check if the response status code is the same as the expected `statusCode` field defined in step. Optional. - - Check if the response body is the same as the expected `responseExpected` field defined in step. Optional. - - If `outputVariables` is defined, runner need to extract and define the variable from specified path in response body. - - If `type` field is `armTemplateDeployment`: - - Use the convention to replace arm template parameters if the parameter name matches the variable name and the parameter type is string. - - Send arm template deployment request under the resource group. - - Wait for the deployment to finish. - - Get the output variables from the deployment. Define the variables from the output variables. - - Else `type` is unsupported in runner. - -### Compare the response with expectedResponse - -It's hard for service team to make sure every field in expectedResponse is the same as the response, so here test scenario suggest to compare properties that are not `readOnly` and are not `x-ms-secret`. The detail should be defined by the runner, not this spec.