Webapp:Calling Set-AzWebApp with -HostNames can fail but returns an un-catchable warning that only indicates BadRequest

[shawnparslow]

Description

In production we have a script that moves a hostname/binding from an app going idle to an app going live (removes the hostname/binding from a web app, update the DNSZone and re-create the hostname and then binding) on a web app that is going live. This code has run 100's of times but recently (about a week after we migrated from AzureRm to Az) we saw the call to create a new hostname suddenly fail but return a success (or rather did not throw an exception). Also, this call in AzureRm would output info but in Az it simply shows a warning an lists 'BadRequest.

Steps to reproduce

Not an exact reproduction of our production error (since in production we set the CNAME in the DNSZone to the new app and removed the hostname/binding from the one going idle) but to test the output you can just try to add www.foobar.com to a web app. In the debug output at the bottom this command threw a warning but the call returned successful (did not catch an exception).

try {
    Set-AzWebApp -ResourceGroupName "defect-test-rg" -Name "stp-test-webapp" -HostNames "www.foobar.com"
}
catch {
    $_.Exception.Message
 }

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.17134.765
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17134.765
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Module versions

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     1.5.2      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enab...
Script     1.0.1      Az.Aks                              {Get-AzAks, New-AzAks, Remove-AzAks, Import-AzAksCredentia...
Script     1.1.0      Az.AnalysisServices                 {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServic...
Script     1.0.0      Az.ApiManagement                    {Add-AzApiManagementRegion, Get-AzApiManagementSsoToken, N...
Script     1.0.0      Az.ApplicationInsights              {Get-AzApplicationInsights, New-AzApplicationInsights, Rem...
Script     1.2.1      Az.Automation                       {Get-AzAutomationHybridWorkerGroup, Remove-AzAutomationHyb...
Script     1.1.0      Az.Batch                            {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAcc...
Script     1.0.0      Az.Billing                          {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollme...
Script     1.2.0      Az.Cdn                              {Get-AzCdnProfile, Get-AzCdnProfileSsoUrl, New-AzCdnProfil...
Script     1.1.1      Az.CognitiveServices                {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAc...
Script     2.0.0      Az.Compute                          {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAv...
Script     1.0.1      Az.ContainerInstance                {New-AzContainerGroup, Get-AzContainerGroup, Remove-AzCont...
Script     1.0.1      Az.ContainerRegistry                {New-AzContainerRegistry, Get-AzContainerRegistry, Update-...
Script     1.1.1      Az.DataFactory                      {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFa...
Script     1.0.0      Az.DataLakeAnalytics                {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalytic...
Script     1.2.0      Az.DataLakeStore                    {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeSt...
Script     1.0.0      Az.DeploymentManager                {Get-AzDeploymentManagerArtifactSource, New-AzDeploymentMa...
Script     1.0.0      Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolic...
Script     1.1.0      Az.Dns                              {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRe...
Script     1.1.1      Az.EventGrid                        {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGr...
Script     1.1.0      Az.EventHub                         {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzE...
Script     1.0.0      Az.FrontDoor                        {New-AzFrontDoor, Get-AzFrontDoor, Set-AzFrontDoor, Remove...
Script     2.0.0      Az.HDInsight                        {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wa...
Script     1.1.0      Az.IotHub                           {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-A...
Script     1.2.0      Az.KeyVault                         {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, ...
Script     1.2.1      Az.LogicApp                         {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccou...
Script     1.1.0      Az.MachineLearning                  {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssocia...
Script     1.0.0      Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms}
Script     1.1.0      Az.Media                            {Sync-AzMediaServiceStorageKey, Set-AzMediaServiceKey, Get...
Script     1.2.0      Az.Monitor                          {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile...
Script     1.8.0      Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Get-Az...
Script     1.1.0      Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorization...
Script     1.2.0      Az.OperationalInsights              {New-AzOperationalInsightsAzureActivityLogDataSource, New-...
Script     1.1.0      Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSu...
Script     1.1.0      Az.PowerBIEmbedded                  {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspa...
Script     1.4.0      Az.RecoveryServices                 {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServi...
Script     1.1.0      Az.RedisCache                       {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheSchedul...
Script     1.0.1      Az.Relay                            {New-AzRelayNamespace, Get-AzRelayNamespace, Set-AzRelayNa...
Script     1.3.1      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzR...
Script     0.7.4      Az.Security                         {Get-AzSecurityAlert, Set-AzSecurityAlert, Get-AzSecurityA...
Script     1.1.0      Az.ServiceBus                       {New-AzServiceBusNamespace, Get-AzServiceBusNamespace, Set...
Script     1.0.1      Az.ServiceFabric                    {Add-AzServiceFabricApplicationCertificate, Add-AzServiceF...
Script     1.0.2      Az.SignalR                          {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSig...
Script     1.9.0      Az.Sql                              {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlData...
Script     1.3.0      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStor...
Script     1.0.0      Az.StreamAnalytics                  {Get-AzStreamAnalyticsFunction, Get-AzStreamAnalyticsDefau...
Script     1.0.1      Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTraf...
Script     1.2.1      Az.Websites                         {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServ...

Debug output

Unfortunately I can not capture debug output as this occurred once in production and then has not happened again. In the production context the hostname had just been deleted and the binding removed, the Azure DNSZone CNAME then gets updated to the new app and the call is made to add a host name below. My best guess is that Azure had not released the hostname yet but upon seeing the error I was able to re-run the call and create the hostname fine. My concern is that there is a bug where the catch {} in my production did not actually get tripped (so the Set-AzWebApp cmdlet appeared to have not thrown a catch-able exception upon failing to create the hostname). We have wrapped retry logic around these cmdlets but the exception was not caught and the cmdlet returned successful. Also, the warning info that showed up in the logs below simply indicated BadRequest and did not have the detail level the AzureRm cmdlet returns. I tried to test with similar calls (to create a hostname of a known site) but that call does fail and is caught (though it also says BadRequest where AzureRm listed more information). That aside, the instance below from production did not throw an exception and returned successful.

Error output

WARNING: 6:33:09 PM - Could not set custom hostname 'customerportalservice.staging.elementslive.net'. Details: 
Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException: Operation returned an invalid status code 
'BadRequest'

   at 
Microsoft.Azure.Management.WebSites.WebAppsOperations.<CreateOrUpdateHostNameBindingWithHttpMessagesAsync>d__75.MoveNex
t()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at 
Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.<CreateOrUpdateHostNameBindingAsync>d__141.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.Azure.Commands.WebApps.Utilities.WebsitesClient.AddCustomHostNames(String resourceGroupName, String 
location, String webAppName, String[] hostNames)
Set-AzWebApp succeeded after 8.8377171 seconds

[shawnparslow]

There may not be enough here to move this defect forward but we did make the Set-AzWebApp with -HostNames param in a try/catch block that does retry and it seems to have failed but did not throw a catch-able exception and call succeeded. We run this code (moving hostname/binding) every day in a blue-green deployment context and just switched from AzureRm to Az (latest) last week so we have only experienced this one error that did not catch.

[markcowl]

@shawnparslow When running in the try/catch block, did you pass -ErrorAction Stop to the cmdlet? This will force it to throw an exception when writing to the Error stream, even if environment settings specify a different default error handling.

[markcowl]

@Nking92 @panchagnula This sounds like a regression in behavior from AzureRM, can you take a look?

@dcaro FYI

[shawnparslow]

@markcowl yes, the script sets the $global:ErrorActionPreference = "Stop" variable at the beginning. We noticed this behavior shortly after converting from AzureRM to Az so I thought it might be something in Az however we have not seen the error again. I am unfortunately unable to go back and enable debugging to repeat since it happened in CI/CD so I apologize if there is insufficient information here and can close if what I was able to capture is not helpful.

[panchagnula]

we will check this with the new API version changes we are making

[shawnparslow]

Just adding that we have seen this a couple more times in our CI/CD system recently. The ErrorActionPreference is set to "Stop" and the Set-AzWebApp call does NOT got through (usually setting a binding that we just removed) but returns as a warning and is not hitting catch. We catch exceptions and have retry logic (in case the binding isn't released yet or something) but this failure seems to just slip through as a warning and doesn't ever hit the catch.

WARNING: 10:48:00 PM - Could not set custom hostname 'deviceregistration.staging.elementslive.net'. Details: Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException: Operation returned an invalid status code 'BadRequest'
   at Microsoft.Azure.Management.WebSites.WebAppsOperations.CreateOrUpdateHostNameBindingWithHttpMessagesAsync(String resourceGroupName, String name, String hostName, HostNameBinding hostNameBinding, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.CreateOrUpdateHostNameBindingAsync(IWebAppsOperations operations, String resourceGroupName, String name, String hostName, HostNameBinding hostNameBinding, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.CreateOrUpdateHostNameBinding(IWebAppsOperations operations, String resourceGroupName, String name, String hostName, HostNameBinding hostNameBinding)
   at Microsoft.Azure.Commands.WebApps.Utilities.WebsitesClient.AddCustomHostNames(String resourceGroupName, String location, String webAppName, String[] hostNames)
Set-AzWebApp succeeded after 8.5922765 seconds

[panchagnula]

@Nking92 can you verify if this is fixed with this change #9496

[btardif]

@Nking92 ping?

[panchagnula]

Closing this since we believe this should now be fixed. Please let us know if this is not the case.

[cpumanaz]

This doe snot appear to be resolved. Am I missing something here?
λ Get-InstalledModule Az

Version Name Repository Description

---

2.6.0 Az PSGallery Microsoft Azure PowerShell - Cmdlets to manage resources in Azure. This module is compatible with WindowsPowerShell and PowerShell Core....

[shawnparslow]

I can say the issue went away for me since upgrading to 2.5.0 in late July. I have a CI/CD system doing 100's of Azure deployments a day and we were only seeing this once or twice a week (in June/July timeframe) and have not seen this since that release. Not definitive but just my experience.

[dcaro]

@cpumanaz please reopen if this is not solved.

[Tyriar]

This still appears to be a problem, I just ran into this.

This is the error I got:

WARNING: 12:33:12 PM - Could not set custom hostname '(snip)'. Details: Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException: Operation returned an invalid status code 'BadRequest'
   at Microsoft.Azure.Management.WebSites.WebAppsOperations.CreateOrUpdateHostNameBindingWithHttpMessagesAsync(String resourceGroupName, String name, String hostName, HostNameBinding hostNameBinding, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.CreateOrUpdateHostNameBindingAsync(IWebAppsOperations operations, String resourceGroupName, String name, String hostName, HostNameBinding hostNameBinding, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.CreateOrUpdateHostNameBinding(IWebAppsOperations operations, String resourceGroupName, String name, String hostName, HostNameBinding hostNameBinding)
   at Microsoft.Azure.Commands.WebApps.Utilities.WebsitesClient.AddCustomHostNames(String resourceGroupName, String location, String webAppName, String[] hostNames, String slotName)

Which was not helpful at all. When I tried to do the same in the portal I got this error:

Reopening because I have the power apparently 💪

[peter-bertok]

I just hit this bug. It's basically impossible to script App Service custom DNS names and SSL certificates when App Service is used indirectly (e.g.: via App Gateway, Front Door, or a third-party NVA) because you can't point directly at App Service. You need the verification TXT record up-front, but it won't give it to you.

For example, Get-AzWebSite returns "null" for CustomDomainVerificationId, which makes it impossible to script validation up-front.

If I try to add a new host name first, without the custom verification, I get this BadRequest error.

I can't do this through ARM Templates either because the DNS Zone is in a different Subscription, which makes it absurdly difficult.

What should be straight forward is the following:

$web = Get-AzWebSite ...
New-AzDnsRecordSet ... $web.CustomDomainVerificationId ...
Set-AzWebSite -HostNames @( ... )

This just doesn't work though!

[ransagy]

Just hit the very same issue as @Tyriar described with latest bits.

[planax]

Quite helpful comment from @Tyriar, which lead me to the real root cause of the error (in my case).