[Add-AzureRmServiceFabricApplicationCertificate] Potential bug during adding cert to cluster nodes

[rovinbhandari]

Hi! I have a cert in AzureKeyvault which I'd like to put in the cert stores of all the nodes in my cluster. I used Add-AzureRmServiceFabricApplicationCertificate in the following way:

Add-AzureRmServiceFabricApplicationCertificate -ResourceGroupName "MyClusterRG" -Name "MyCluster" -SecretIdentifier "https://MyKV.vault.azure.net/secrets/MyCert/LatestEnabledVersionGuid"

However, it fails with (note that it modified the secret identifier):

Add-AzureRmServiceFabricApplicationCertificate : Code: KeyVaultSecretDoesNotExist, Message: The Key Vault secret referenced with the URL https://MyKV.vault.azure.net/secrets/MyCluster20180317064812/TotallyDifferentGuidThanLatestEnabledVersionGuid does not exist.
Details:
At line:1 char:2
+  Add-AzureRmServiceFabricApplicationCertificate -ResourceGroupName "M ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Add-AzureRmServ...tionCertificate], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricApplicationCertificate

Add-AzureRmServiceFabricApplicationCertificate : One or more errors occurred.
At line:1 char:2
+  Add-AzureRmServiceFabricApplicationCertificate -ResourceGroupName "M ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Add-AzureRmServ...tionCertificate], AggregateException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricApplicationCertificate

The secret I'm trying to add does exist. I can get it in the following way:

Get-AzureKeyVaultSecret -VaultName MyKV -Name MyCert

Vault Name   : MyKV
Name         : MyCert
Version      : LatestEnabledVersionGuid
Id           : https://MyKV.vault.azure.net:443/secrets/MyCert/LatestEnabledVersionGuid
Enabled      : True
Expires      : 06.03.2020 08:32:18
Not Before   : 07.03.2018 08:32:18
Created      : 07.03.2018 08:42:22
Updated      : 07.03.2018 08:42:22
Content Type : application/x-pkcs12
Tags         :

(I'm using AzureRM 5.5.0 and AzureAD 2.0.0.131. I can provide more details if necessary.)

If I run Add-AzureRmServiceFabricApplicationCertificate in -Debug mode I can see that MyCert is actually fetched with its correct value in the Body of the response. But during applying the cert to the VMs this other secret identifier (MyCluster20180317064812) creeps in from somewhere, and is present in some of the calls as one of the "vaultCertificates".

Could it be a bug in Add-AzureRmServiceFabricApplicationCertificate?

Thank you!

[vaishnavk]

@rovinbhandari : Thank you for reporting the issue. We will investigate.

[ei-en-kei]

I have the same issue:

Add-AzureRmServiceFabricApplicationCertificate : One or more errors occurred.
At line:1 char:1
+ Add-AzureRmServiceFabricApplicationCertificate -ResourceGroupName "OA ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Add-AzureRmServ...tionCertificate], AggregateException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricApplicationCertif
   icate

DEBUG: AzureQoSEvent: CommandName - Add-AzureRmServiceFabricApplicationCertificate; IsSuccess - False; Duration -
00:00:08.2694998; Exception - System.AggregateException: One or more errors occurred. --->
System.MissingMethodException: Method not found: 'Microsoft.Azure.Commands.Common.Authentication.IAccessToken
Microsoft.Azure.Commands.Common.Authentication.Abstractions.IAuthenticationFactory.Authenticate(Microsoft.Azure.Command
s.Common.Authentication.Abstractions.IAzureAccount,
Microsoft.Azure.Commands.Common.Authentication.Abstractions.IAzureEnvironment, System.String,
System.Security.SecureString, System.String, System.String)'.
   at Microsoft.Azure.Commands.ServiceFabric.Commands.ServiceFabricCmdletBase.AuthenticationCallback(String authority,
String resource, String scope)
   at Microsoft.Azure.KeyVault.KeyVaultCredential.<PreAuthenticate>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.KeyVault.KeyVaultCredential.<ProcessHttpRequestAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.KeyVault.KeyVaultClient.<GetSecretWithHttpMessagesAsync>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.<GetSecretAsync>d__12.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at
Microsoft.Azure.Commands.ServiceFabric.Commands.ServiceFabricClusterCertificateCmdlet.GetThumbprintFromSecret(String
secretUrl)
   at
Microsoft.Azure.Commands.ServiceFabric.Commands.ServiceFabricClusterCertificateCmdlet.GetOrCreateCertificateInformation
()
   at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricApplicationCertificate.ExecuteCmdlet()
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
---> (Inner Exception #0) System.MissingMethodException: Method not found:
'Microsoft.Azure.Commands.Common.Authentication.IAccessToken
Microsoft.Azure.Commands.Common.Authentication.Abstractions.IAuthenticationFactory.Authenticate(Microsoft.Azure.Command
s.Common.Authentication.Abstractions.IAzureAccount,
Microsoft.Azure.Commands.Common.Authentication.Abstractions.IAzureEnvironment, System.String,
System.Security.SecureString, System.String, System.String)'.
   at Microsoft.Azure.Commands.ServiceFabric.Commands.ServiceFabricCmdletBase.AuthenticationCallback(String authority,
String resource, String scope)
   at Microsoft.Azure.KeyVault.KeyVaultCredential.<PreAuthenticate>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.KeyVault.KeyVaultCredential.<ProcessHttpRequestAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.KeyVault.KeyVaultClient.<GetSecretWithHttpMessagesAsync>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.<GetSecretAsync>d__12.MoveNext()<---
;
DEBUG: Finish sending metric.
DEBUG: 11:55:55 AM - AddAzureRmServiceFabricApplicationCertificate end processing.
DEBUG: 11:55:55 AM - AddAzureRmServiceFabricApplicationCertificate end processing.

[vaishnavk]

We have logged a bug on our end and we will be working on a fix.

[rajagops]

@vaishnavk Is there any workaround until the fix is available?

[rovinbhandari]

@rajagops I used the following workaround:

$vmssname = "VMSS"    # VMSS of the Service Fabric cluster
$vmssrgname = "VMSS_RG"    # Resource Group of the VMSS
$kvSecretId = "https://MyKV.vault.azure.net/secrets/MyCert/LatestEnabledVersionGuid"    # The cert to add to the VMSS
$vaultResourceId = "/subscriptions/SomeGuid/resourceGroups/RG/providers/Microsoft.KeyVault/vaults/MyKV"    # The KeyVault Resource Id where the cert is located

# Create the cert config to add to VMSS
$CertConfig = New-AzureRmVmssVaultCertificateConfig -CertificateUrl $kvSecretId -CertificateStore "My"
$newvmss = New-AzureRmVmssConfig
Add-AzureRmVmssSecret -VirtualMachineScaleSet $newvmss -SourceVaultId $vaultResourceId -VaultCertificate $CertConfig

# Get current VMSS
$vmss = Get-AzureRmVmss -VMScaleSetName $vmssname -ResourceGroupName $vmssrgname

# Add the new cert to the correct Secrets group.
$vmss.VirtualMachineProfile.OsProfile.Secrets[1].VaultCertificates.Add($newvmss.VirtualMachineProfile.OsProfile.Secrets[0].VaultCertificates[0])

# Remove the offending cert which causes the failure.
$vmss.VirtualMachineProfile.OsProfile.Secrets[1].VaultCertificates.RemoveAt(0)

# Update VMSS with the changes.
Update-AzureRmVmss -ResourceGroupName $vmssrgname -Name $vmssname -VirtualMachineScaleSet $vmss

I hope this helps more people!

[rajagops]

Thanks @rovinbhandari

[jesseik]

@vaishnavk Any news on this?

[stijnherreman]

Works for me. Was this fixed or does the bug not always occur?

[vaishnavk]

@juhacket : Can you confirm if this has been fixed?

[lukeholbertmsft]

I am having this issue currently as well. Can we get an update here?

[xaruka]

we have this issue as well. If this has been fixed, which version should we use?

[lukeholbertmsft]

I just tried to use the workaround posted above and it did the exact same thing. It failed on the Update-AzureRmVmss call with an error stating the vault and vmss were in different regions, which they were not. It also was an entirely different secret url in the error message than the one that I entered. Can we please get an update on this issue?

[lukeholbertmsft]

I think the real issue here is that the cmdlet alters the VMSS and adds the secret even when there are exceptions thrown and there is a failure. It seems every time I encounter this is when I hit a valid exception (wrong url) and then I go to fix it and I still see the same error (which is very misleading because the exception reads like the new url is the issue when in reality it is an existing secret). The second time I run it, the correct secret gets added but the wrong one is still in there. Can we get a fix for this so that validation is done before the changes are made with the cmdlet?

[ravibha]

Facing the same issue. Is the fix released? If so, which version for the Azure powershell sdk should we use?

[a-santamaria]

@ravibha this hasn't been fixed yet. As lukeholbertmsft mentioned, the issue is the cert is added to the VMSS model even if an exception is thrown. I'm following up with vmss team to see how to fix it. The workaround is to remove the offending cert in the vmss model by going to resources.azure.com or with PowerShell cmdlets as suggested by rovinbhandari.

[a-santamaria]

fix to rollback when a certificate is added to VMSS model but an exception is thrown is in AZ 1.2.0