Skip to content

Commit

Permalink
[Storage] Storage ARM Template For Test Resources (Azure#6982)
Browse files Browse the repository at this point in the history
* [MQ Goals] ARM template for Storage Blob

If the ARM template works (tests-resources.json),
then it will be the same for each of the storage clients.

tests.yml should be almost the same for the other storage clients, with
some of the content changed, reflecting the appropriate name of the
client.

* feedback

* wrong path here

* all of the tests.yml for storage

* fixed the paths

* added provisionerTenantId

* using ../archetype-sdk-tests-storage.yml again

* optimization on the env variables

* optimizations on ResourceFileLocation

* lowercase string

* variables('storageApiVersion')

* tenantId resources shouldnt be necessary

* removed provisionerTenantId and uppercased the env variables

* moved the ARM specific parameters to variables

* this is closer to work

* update

* storage-blob updated tests.yml

* seems like the archetype file needed to change

* seems like the template is invalid somehow. Trying this

* tweaks, to see if things change

* this should work now

* ResourceServiceDirectory can be optimized

* add AAD secret references in the ENV vars for the resource creation - arm-template
#RequiredForCI

* run a single test

* DEBUG #1

* reduce matrix

* only keep the samples in the pipeline

* print all

* keep test job too - Linux_Node10

* last resort - move around EnvVars

* remove duplication

* update ResourceServiceDirectory as per Daniel's suggestion

* ResourceServiceDirectory: storage

* ResourceServiceDirectory: ${{ parameters.ResourceServiceDirectory }}

* Update sdk/storage/test-resources.json

* Update sdk/storage/test-resources.json

* Update sdk/storage/test-resources.json

* remove .only

* remove env from imports

* add blob contributor role to the AAD service principal

* add "apiVersion": "[variables('authorizationApiVersion')]",

* add "authorizationApiVersion": "2017-05-01",

* 2018-01-01-preview - authorizationApiVersion

* add more sas permissions

* revert the matrix back to full

* simplified CORS rules

* smaller matrix to reduce the load

* formatting changes

* nested resources

* add cors rules for file and queue as well

* queueServices unsupported with CORS

* revert matrix back to full

* revert archetype-sdk-tests-storage.yml

* update tests.yml for storage-blob and file-share to the new format

* revert unintended change

* revert changes to archetype-storage . yml

Co-authored-by: Daniel Rodríguez <[email protected]>
  • Loading branch information
HarshaNalluru and sadasant authored Jan 28, 2020
1 parent 2a39ea7 commit dfc6cbf
Show file tree
Hide file tree
Showing 5 changed files with 373 additions and 4 deletions.
64 changes: 63 additions & 1 deletion sdk/storage/storage-blob/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,68 @@ resources:
name: Azure/azure-sdk-tools
endpoint: azure
jobs:
- template: ../archetype-sdk-tests-storage.yml
- template: ../../../eng/pipelines/templates/jobs/archetype-sdk-integration.yml
parameters:
PackageName: "@azure/storage-blob"
ResourceServiceDirectory: storage
EnvVars:
AZURE_CLIENT_ID: $(aad-azure-sdk-test-client-id)
AZURE_TENANT_ID: $(aad-azure-sdk-test-tenant-id)
AZURE_CLIENT_SECRET: $(aad-azure-sdk-test-client-secret)
ACCOUNT_NAME: $(STORAGE_ACCOUNT_NAME)
ACCOUNT_KEY: $(STORAGE_ACCOUNT_KEY)
ACCOUNT_SAS: $(STORAGE_ACCOUNT_SAS)
STORAGE_CONNECTION_STRING: $(STORAGE_CONNECTION_STRING)
Matrix:
Linux_Node8:
OSVmImage: "ubuntu-16.04"
NodeVersion: "8.x"
TestType: "node"
Linux_Node10:
OSVmImage: "ubuntu-16.04"
NodeVersion: "10.x"
TestType: "node"
Linux_Node12:
OSVmImage: "ubuntu-16.04"
NodeVersion: "12.x"
TestType: "node"
Windows_Node8:
OSVmImage: "windows-2019"
NodeVersion: "8.x"
TestType: "node"
Windows_Node10:
OSVmImage: "windows-2019"
NodeVersion: "10.x"
TestType: "node"
Windows_Node12:
OSVmImage: "windows-2019"
NodeVersion: "12.x"
TestType: "node"
macOS_Node8:
OSVmImage: "macOS-10.13"
NodeVersion: "8.x"
TestType: "node"
macOS_Node10:
OSVmImage: "macOS-10.13"
NodeVersion: "10.x"
TestType: "node"
macOS_Node12:
OSVmImage: "macOS-10.13"
NodeVersion: "12.x"
TestType: "node"
Browser_Linux_Node10:
OSVmImage: "ubuntu-16.04"
NodeVersion: "10.x"
TestType: "browser"
Browser_Windows_Node10:
OSVmImage: "windows-2019"
NodeVersion: "10.x"
TestType: "browser"
Browser_macOS_Node10:
OSVmImage: "macOS-10.13"
NodeVersion: "10.x"
TestType: "browser"
Linux_Sample10:
OSVmImage: "ubuntu-16.04"
NodeVersion: "10.x"
TestType: "sample"
2 changes: 1 addition & 1 deletion sdk/storage/storage-file-datalake/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ resources:
jobs:
- template: ../archetype-sdk-tests-storage.yml
parameters:
PackageName: "@azure/storage-file-datalake"
PackageName: "@azure/storage-file-datalake"
64 changes: 63 additions & 1 deletion sdk/storage/storage-file-share/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,68 @@ resources:
name: Azure/azure-sdk-tools
endpoint: azure
jobs:
- template: ../archetype-sdk-tests-storage.yml
- template: ../../../eng/pipelines/templates/jobs/archetype-sdk-integration.yml
parameters:
PackageName: "@azure/storage-file-share"
ResourceServiceDirectory: storage
EnvVars:
AZURE_CLIENT_ID: $(aad-azure-sdk-test-client-id)
AZURE_TENANT_ID: $(aad-azure-sdk-test-tenant-id)
AZURE_CLIENT_SECRET: $(aad-azure-sdk-test-client-secret)
ACCOUNT_NAME: $(STORAGE_ACCOUNT_NAME)
ACCOUNT_KEY: $(STORAGE_ACCOUNT_KEY)
ACCOUNT_SAS: $(STORAGE_ACCOUNT_SAS)
STORAGE_CONNECTION_STRING: $(STORAGE_CONNECTION_STRING)
Matrix:
Linux_Node8:
OSVmImage: "ubuntu-16.04"
NodeVersion: "8.x"
TestType: "node"
Linux_Node10:
OSVmImage: "ubuntu-16.04"
NodeVersion: "10.x"
TestType: "node"
Linux_Node12:
OSVmImage: "ubuntu-16.04"
NodeVersion: "12.x"
TestType: "node"
Windows_Node8:
OSVmImage: "windows-2019"
NodeVersion: "8.x"
TestType: "node"
Windows_Node10:
OSVmImage: "windows-2019"
NodeVersion: "10.x"
TestType: "node"
Windows_Node12:
OSVmImage: "windows-2019"
NodeVersion: "12.x"
TestType: "node"
macOS_Node8:
OSVmImage: "macOS-10.13"
NodeVersion: "8.x"
TestType: "node"
macOS_Node10:
OSVmImage: "macOS-10.13"
NodeVersion: "10.x"
TestType: "node"
macOS_Node12:
OSVmImage: "macOS-10.13"
NodeVersion: "12.x"
TestType: "node"
Browser_Linux_Node10:
OSVmImage: "ubuntu-16.04"
NodeVersion: "10.x"
TestType: "browser"
Browser_Windows_Node10:
OSVmImage: "windows-2019"
NodeVersion: "10.x"
TestType: "browser"
Browser_macOS_Node10:
OSVmImage: "macOS-10.13"
NodeVersion: "10.x"
TestType: "browser"
Linux_Sample10:
OSVmImage: "ubuntu-16.04"
NodeVersion: "10.x"
TestType: "sample"
2 changes: 1 addition & 1 deletion sdk/storage/storage-queue/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ resources:
jobs:
- template: ../archetype-sdk-tests-storage.yml
parameters:
PackageName: "@azure/storage-queue"
PackageName: "@azure/storage-queue"
245 changes: 245 additions & 0 deletions sdk/storage/test-resources.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,245 @@
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"baseName": {
"type": "string",
"metadata": {
"description": "The base resource name."
}
},
"testApplicationOid": {
"type": "string",
"metadata": {
"description": "The client OID to grant access to test resources."
}
}
},
"variables": {
"storageApiVersion": "2019-04-01",
"location": "[resourceGroup().location]",
"accountName": "[parameters('baseName')]",
"datalakeAccountName": "[concat('d', parameters('baseName'))]",
"accountNameTidy": "[toLower(trim(variables('accountName')))]",
"blobEndPoint": "[concat('https://',variables('accountNameTidy'),'.blob.core.windows.net/')]",
"endpointSuffix": "core.windows.net",
"accountSasProperties": {
"signedServices": "bfqt",
"signedPermission": "rwdlacup",
"signedResourceTypes": "sco",
"keyToSign": "key2",
"signedExpiry": "2022-01-01T23:59:00Z"
},
"authorizationApiVersion": "2018-01-01-preview",
"blobDataContributorRoleId": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/ba92f5b4-2d11-453d-a403-e96b0029c9fe')]"
},
"resources": [
{
"type": "Microsoft.Authorization/roleAssignments",
"apiVersion": "[variables('authorizationApiVersion')]",
"name": "[guid(concat('blobDataContributorRoleId', variables('accountName')))]",
"dependsOn": [
"[variables('accountName')]"
],
"properties": {
"roleDefinitionId": "[variables('blobDataContributorRoleId')]",
"principalId": "[parameters('testApplicationOid')]"
}
},
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "[variables('storageApiVersion')]",
"name": "[variables('accountName')]",
"location": "[variables('location')]",
"sku": {
"name": "Standard_RAGRS",
"tier": "Standard"
},
"kind": "StorageV2",
"properties": {
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [
],
"ipRules": [
],
"defaultAction": "Allow"
},
"supportsHttpsTrafficOnly": true,
"encryption": {
"services": {
"file": {
"enabled": true
},
"blob": {
"enabled": true
}
},
"keySource": "Microsoft.Storage"
},
"accessTier": "Hot"
},
"resources": [
{
"name": "default",
"type": "blobServices",
"apiVersion": "[variables('storageApiVersion')]",
"dependsOn": [
"[variables('accountName')]"
],
"properties": {
"cors": {
"corsRules": [
{
"allowedOrigins": [
"*"
],
"allowedMethods": [
"DELETE",
"GET",
"HEAD",
"MERGE",
"POST",
"OPTIONS",
"PUT",
"PATCH"
],
"maxAgeInSeconds": 86400,
"exposedHeaders": [
"*"
],
"allowedHeaders": [
"*"
]
}
]
}
},
"resources": [
]
},
{
"name": "default",
"type": "fileServices",
"apiVersion": "[variables('storageApiVersion')]",
"dependsOn": [
"[variables('accountName')]"
],
"properties": {
"cors": {
"corsRules": [
{
"allowedOrigins": [
"*"
],
"allowedMethods": [
"DELETE",
"GET",
"HEAD",
"MERGE",
"POST",
"OPTIONS",
"PUT"
],
"maxAgeInSeconds": 86400,
"exposedHeaders": [
"*"
],
"allowedHeaders": [
"*"
]
}
]
}
},
"resources": [
]
}

]
},
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "[variables('storageApiVersion')]",
"name": "[variables('datalakeAccountName')]",
"location": "[variables('location')]",
"sku": {
"name": "Standard_RAGRS",
"tier": "Standard"
},
"kind": "StorageV2",
"properties": {
"isHnsEnabled": true,
"cors": {
"corsRules": [
{
"allowedOrigins": [
"*"
],
"allowedMethods": [
"DELETE",
"GET",
"HEAD",
"MERGE",
"POST",
"OPTIONS",
"PUT",
"PATCH"
],
"maxAgeInSeconds": 86400,
"exposedHeaders": [
"*"
],
"allowedHeaders": [
"*"
]
}
]
},
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [
],
"ipRules": [
],
"defaultAction": "Allow"
},
"supportsHttpsTrafficOnly": true,
"encryption": {
"services": {
"file": {
"enabled": true
},
"blob": {
"enabled": true
}
},
"keySource": "Microsoft.Storage"
},
"accessTier": "Hot"
}
}
],
"outputs": {
"STORAGE_ACCOUNT_NAME": {
"type": "string",
"value": "[variables('accountName')]"
},
"STORAGE_DATALAKE_ACCOUNT_NAME": {
"type": "string",
"value": "[variables('datalakeAccountName')]"
},
"STORAGE_ACCOUNT_KEY": {
"type": "string",
"value": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('accountName')), variables('storageApiVersion')).keys[0].value]"
},
"STORAGE_ACCOUNT_SAS": {
"type": "string",
"value": "[concat('?', listAccountSas(variables('accountNameTidy'), variables('storageApiVersion'), variables('accountSasProperties')).accountSasToken)]"
},
"STORAGE_CONNECTION_STRING": {
"type": "string",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('accountName'), ';AccountKey=', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('accountName')), variables('storageApiVersion')).keys[0].value, ';EndpointSuffix=', 'core.windows.net')]"
}
}
}

0 comments on commit dfc6cbf

Please sign in to comment.