MaxStorageAccountsCountPerSubscriptionExceeded error when deploying storage test resources

[Jinming-Hu]

Status Code:
BadRequest

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-failure-cause            : gateway
x-ms-ratelimit-remaining-subscription-writes: 1196
x-ms-request-id               : 3f56abcf-38e7-45b3-9031-373388ccfab0
x-ms-correlation-request-id   : 3f56abcf-38e7-45b3-9031-373388ccfab0
x-ms-routing-request-id       : WESTUS:20220921T092423Z:3f56abcf-38e7-45b3-9031-373388ccfab0
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
Date                          : Wed, 21 Sep 2022 09:24:23 GMT

Body:
{
  "error": {
    "code": "InvalidTemplateDeployment",
    "message": "The template deployment 'tb1defe081da1421e' is not valid according to the validation procedure. The tracking id is '3f56abcf-38e7-45b3-9031-373388ccfab0'. See inner errors for details.",
    "details": [
      {
        "code": "PreflightValidationCheckFailed",
        "message": "Preflight validation failed. Please refer to the details for the specific errors.",
        "details": [
          {
            "code": "MaxStorageAccountsCountPerSubscriptionExceeded",
            "target": "***",
            "message": "Subscription *** already contains 249 storage accounts with Standard Dns endpoints in location westus and the maximum allowed is 250."
          }
        ]
      }
    ]
  }
}


DEBUG: 9:24:23 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].
VERBOSE: Running registered exit actions
VERBOSE: Logging out of service principal '***'
VERBOSE: Performing the operation "New-TestResources.ps1" on target "***".
Retry : Exception calling "Invoke" with "0" argument(s): "The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: 9:24:23 AM - Error: Code=InvalidTemplateDeployment; Message=The template deployment 'tb1defe081da1421e' is not valid according to the validation procedure. The tracking id is '3f56abcf-38e7-45b3-9031-373388ccfab0'. See inner errors for details.
"
At D:\a\_work\1\s\eng\common\TestResources\New-TestResources.ps1:725 char:23
+         $deployment = Retry {
+                       ~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [Write-Error], MethodInvocationException
+ FullyQualifiedErrorId : ActionPreferenceStopException,Retry
##[error]PowerShell exited with code '1'.

[Jinming-Hu]

the issue is gone.

[Jinming-Hu]

"code": "PreflightValidationCheckFailed",
        "message": "Preflight validation failed. Please refer to the details for the specific errors.",
        "details": [
          {
            "code": "MaxStorageAccountsCountPerSubscriptionExceeded",
            "target": "***",
            "message": "Subscription *** already contains 249 storage accounts with Standard Dns endpoints in location westus and the maximum allowed is 250."
          }
        ]

@danieljurek We always see this error recently. Is it possible increase the limit of storage account number in the subscription?

[danieljurek]

@benbp and I have been looking into this. It's possible that there's a cluster of accounts deployed to the sub during that time... we'll need to look into it. Ben's digging in further. Thanks, Ben!

[Jinming-Hu]

@danieljurek storage accounts are created for live tests. But they are not immediately deleted after live tests finish.

We'll run into this problem if we run live tests many times a day. Is there any way to do a manual clean-up?

Or can you open a ticket to increase the limit of storage account number in West US region, to like 500 or 1000? that should be enough for sdk daily use.

[Jinming-Hu]

I only opened two new PRs today and I hit this problem again.

[danieljurek]

Thanks for clarifying, @Jinming-Hu. Indeed I'm seeing that the "Remove test resources" step on a successful test job appears to be running against our "test resources" subscription instead of our "test resources - preview" subscription and not finding the resource group it intends to delete: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1887090&view=logs&j=9c8c9cc9-6eb1-51e3-a573-73971cbc3c8b&t=fce11e25-7ef3-563a-0896-f206e1070429

It's not obvious from the run logs which subscriptions are targeted in each case, but the names suggest that the remove script is running against the wrong subscription.

@benbp, perhaps Remove-TestResources is working against the wrong sub?

@Jinming-Hu -- We do have a pipeline which cleans up "expired" resources but running that pipeline won't directly help in this specific case because the expiration date is set some hours in the future from when the resources are created... running it immediately after creating resources will not delete those resources. The pipelines should be deleting the resources they create after the tests finish running.