[Az.Aks] Set-AzAksCluster returns "Operation returned an invalid status code 'Forbidden'" on Azure Automation Runbook

[jyama4ta]

Description

Set-AzAksCluster returns an error when we run the commands on Azure Automation Runbook with Az.Aks v2.1.0. The issue does not happen with Az.Aks v2.0.2.
There is no activity logs on AKS cluster when it run. So, it may caused by that it couldn't load service principal information.

Steps to reproduce

1. Create Azure Automation Account
2. Improt Az.Accounts v2.2.8 and Az.Aks 2.1.0 modules.
3. Create a Runbook with following script
4. Start the Runbook

$connectionName = "AzureRunAsConnection"
try {
    # Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection = Get-AutomationConnection -Name $connectionName
    # Wrap authentication in retry logic for transient network failures
    $logonAttempt = 0
    $connectionResult = $False
    do {
        $LogonAttempt++

        Write-Output "Logon execution count = $LogonAttempt"

        # Logging in to Azure...
        $connectionResult = Connect-AzAccount `
            -ServicePrincipal `
            -Tenant $servicePrincipalConnection.TenantId `
            -ApplicationId $servicePrincipalConnection.ApplicationId `
            -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint 
        Start-Sleep -Seconds 10
    } while (!($connectionResult) -and ($logonAttempt -le 30))
}
catch {
    if (!$servicePrincipalConnection) {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    }
    else {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}

Set-AzAksCluster -ResourceGroupName "<RG name>" -Name "<Cluster Name>" -NodeMaxCount <Any num> -NodeName "<Agent Pool Name>"

Environment data

Name                           Value                                                                                    
----                           -----                                                                                    
PSVersion                      5.1.15063.726                                                                            
PSEdition                      Desktop                                                                                  
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                  
BuildVersion                   10.0.15063.726                                                                           
CLRVersion                     4.0.30319.42000                                                                          
WSManStackVersion              3.0                                                                                      
PSRemotingProtocolVersion      2.3                                                                                      
SerializationVersion           1.1.0.1    

Module versions

    Directory: C:\Modules\User
ModuleType Version    Name                                ExportedCommands                                              
---------- -------    ----                                ----------------                                              
Script     2.2.8      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enabl...
Script     2.1.0      Az.Aks                              {Get-AzAksCluster, New-AzAksCluster, Remove-AzAksCluster, I...
Script     4.6.1      Azure.Storage                       {Get-AzureStorageTable, New-AzureStorageTableSASToken, New-...
Script     6.1.2      AzureRM.Automation                  {Get-AzureRMAutomationHybridWorkerGroup, Remove-AzureRmAuto...
Script     5.9.2      AzureRM.Compute                     {Remove-AzureRmAvailabilitySet, Get-AzureRmAvailabilitySet,...
Script     5.8.4      AzureRM.Profile                     {Disable-AzureRmDataCollection, Disable-AzureRmContextAutos...
Script     6.7.4      AzureRM.Resources                   {Get-AzureRmProviderOperation, Remove-AzureRmRoleAssignment...
Script     4.12.2     AzureRM.Sql                         {Get-AzureRmSqlDatabaseTransparentDataEncryption, Get-Azure...
Script     5.2.1      AzureRM.Storage                     {Get-AzureRmStorageAccount, Get-AzureRmStorageAccountKey, N...

    Directory: C:\Modules\Global
ModuleType Version    Name                                ExportedCommands                                              
---------- -------    ----                                ----------------                                              
Manifest   1.1.0.0    AuditPolicyDsc                                                                                    
Manifest   5.0.0.0    ComputerManagementDsc                                                                             
Script     0.2        GPRegistryPolicyParser              {Parse-PolFile, Read-RegistryPolicies, Create-RegistrySetti...
Script     0.0        Microsoft.PowerShell.Core                                                                         
Script     0.0        Microsoft.PowerShell.Diagnostics                                                                  
Script     0.0        Microsoft.PowerShell.Management                                                                   
Script     0.0        Microsoft.PowerShell.Security                                                                     
Script     0.0        Microsoft.PowerShell.Utility                                                                      
Script     0.0        Microsoft.WSMan.Management                                                                        
Binary     1.0        Orchestrator.AssetManagement.Cmd... {Get-AutomationCertificate, Get-AutomationConnection, Get-S...
Manifest   2.9.0.0    PSDscResources                                                                                    
Manifest   2.1.0.0    SecurityPolicyDsc                                                                                 
Manifest   1.0        StateConfigCompositeResources                                                                     
Manifest   1.1        xDSCDomainjoin                                                                                    
Manifest   1.1.0.0    xPowerShellExecutionPolicy          {Get-TargetResource, Test-TargetResource, Set-TargetResource} 
Manifest   1.1.0.0    xRemoteDesktopAdmin                                                                               

    Directory: C:\Program Files\WindowsPowerShell\Modules
ModuleType Version    Name                                ExportedCommands                                              
---------- -------    ----                                ----------------                                              
Script     1.0.1      Microsoft.PowerShell.Operation.V... {Get-OperationValidation, Invoke-OperationValidation}         
Binary     1.0.0.1    PackageManagement                   {Find-Package, Get-Package, Get-PackageProvider, Get-Packag...
Script     3.4.0      Pester                              {Describe, Context, It, Should...}                            
Script     1.0.0.1    PowerShellGet                       {Install-Module, Find-Module, Save-Module, Update-Module...}  
Script     1.2        PSReadline                          {Get-PSReadlineKeyHandler, Set-PSReadlineKeyHandler, Remove...

    Directory: C:\Windows\system32\WindowsPowerShell\v1.0\Modules
ModuleType Version    Name                                ExportedCommands                                              
---------- -------    ----                                ----------------                                              
Manifest   1.0.0.0    AppBackgroundTask                   {Disable-AppBackgroundTaskDiagnosticLog, Enable-AppBackgrou...
Manifest   2.0.0.0    Appx                                {Add-AppxPackage, Get-AppxPackage, Get-AppxPackageManifest,...
Manifest   2.0.0.0    BitsTransfer                        {Add-BitsFile, Complete-BitsTransfer, Get-BitsTransfer, Rem...
Manifest   1.0.0.0    CimCmdlets                          {Get-CimAssociatedInstance, Get-CimClass, Get-CimInstance, ...
Manifest   1.0        Defender                            {Get-MpPreference, Set-MpPreference, Add-MpPreference, Remo...
Manifest   1.0.0.0    DeliveryOptimization                {Get-DeliveryOptimizationStatus, Get-DeliveryOptimizationPe...
Manifest   1.0.0.0    DirectAccessClientComponents        {Disable-DAManualEntryPointSelection, Enable-DAManualEntryP...
Script     3.0        Dism                                {Add-AppxProvisionedPackage, Add-WindowsDriver, Add-Windows...
Manifest   1.0.0.0    DnsClient                           {Resolve-DnsName, Clear-DnsClientCache, Get-DnsClient, Get-...
Manifest   1.0.0.0    EventTracingManagement              {Start-EtwTraceSession, New-EtwTraceSession, Get-EtwTraceSe...
Manifest   2.0.0.0    International                       {Get-WinDefaultInputMethodOverride, Set-WinDefaultInputMeth...
Manifest   1.0.0.0    iSCSI                               {Get-IscsiTargetPortal, New-IscsiTargetPortal, Remove-Iscsi...
Script     1.0.0.0    ISE                                 {New-IseSnippet, Import-IseSnippet, Get-IseSnippet}           
Manifest   1.0.0.0    Kds                                 {Add-KdsRootKey, Get-KdsRootKey, Test-KdsRootKey, Set-KdsCo...
Manifest   1.0.1.0    Microsoft.PowerShell.Archive        {Compress-Archive, Expand-Archive}                            
Manifest   3.0.0.0    Microsoft.PowerShell.Diagnostics    {Get-WinEvent, Get-Counter, Import-Counter, Export-Counter...}
Manifest   3.0.0.0    Microsoft.PowerShell.Host           {Start-Transcript, Stop-Transcript}                           
Manifest   1.0.0.0    Microsoft.PowerShell.LocalAccounts  {Add-LocalGroupMember, Disable-LocalUser, Enable-LocalUser,...
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Content, Clear-Content, Clear-ItemProperty, Join-Path...}
Script     1.0        Microsoft.PowerShell.ODataUtils     Export-ODataEndpointProxy                                     
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {Get-Acl, Set-Acl, Get-PfxCertificate, Get-Credential...}     
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Format-List, Format-Custom, Format-Table, Format-Wide...}    
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Disable-WSManCredSSP, Enable-WSManCredSSP, Get-WSManCredSS...
Manifest   1.0        MMAgent                             {Disable-MMAgent, Enable-MMAgent, Set-MMAgent, Get-MMAgent...}
Manifest   1.0.0.0    MsDtc                               {New-DtcDiagnosticTransaction, Complete-DtcDiagnosticTransa...
Manifest   2.0.0.0    NetAdapter                          {Disable-NetAdapter, Disable-NetAdapterBinding, Disable-Net...
Manifest   1.0.0.0    NetConnection                       {Get-NetConnectionProfile, Set-NetConnectionProfile}          
Manifest   1.0.0.0    NetEventPacketCapture               {New-NetEventSession, Remove-NetEventSession, Get-NetEventS...
Manifest   2.0.0.0    NetLbfo                             {Add-NetLbfoTeamMember, Add-NetLbfoTeamNic, Get-NetLbfoTeam...
Manifest   1.0.0.0    NetNat                              {Get-NetNat, Get-NetNatExternalAddress, Get-NetNatStaticMap...
Manifest   2.0.0.0    NetQos                              {Get-NetQosPolicy, Set-NetQosPolicy, Remove-NetQosPolicy, N...
Manifest   2.0.0.0    NetSecurity                         {Get-DAPolicyChange, New-NetIPsecAuthProposal, New-NetIPsec...
Manifest   1.0.0.0    NetSwitchTeam                       {New-NetSwitchTeam, Remove-NetSwitchTeam, Get-NetSwitchTeam...
Manifest   1.0.0.0    NetTCPIP                            {Get-NetIPAddress, Get-NetIPInterface, Get-NetIPv4Protocol,...
Manifest   1.0.0.0    NetworkConnectivityStatus           {Get-DAConnectionStatus, Get-NCSIPolicyConfiguration, Reset...
Manifest   1.0.0.0    NetworkSwitchManager                {Disable-NetworkSwitchEthernetPort, Enable-NetworkSwitchEth...
Manifest   1.0.0.0    NetworkTransition                   {Add-NetIPHttpsCertBinding, Disable-NetDnsTransitionConfigu...
Manifest   1.0.0.0    PcsvDevice                          {Get-PcsvDevice, Start-PcsvDevice, Stop-PcsvDevice, Restart...
Manifest   1.0.0.0    PKI                                 {Add-CertificateEnrollmentPolicyServer, Export-Certificate,...
Manifest   1.0.0.0    PnpDevice                           {Get-PnpDevice, Get-PnpDeviceProperty, Enable-PnpDevice, Di...
Manifest   1.1        PrintManagement                     {Add-Printer, Add-PrinterDriver, Add-PrinterPort, Get-Print...
Script     3.0        Provisioning                        {Install-ProvisioningPackage, Export-ProvisioningPackage, I...
Manifest   1.1        PSDesiredStateConfiguration         {Set-DscLocalConfigurationManager, Start-DscConfiguration, ...
Script     1.0.0.0    PSDiagnostics                       {Disable-PSTrace, Disable-PSWSManCombinedTrace, Disable-WSM...
Binary     1.1.0.0    PSScheduledJob                      {New-JobTrigger, Add-JobTrigger, Remove-JobTrigger, Get-Job...
Manifest   2.0.0.0    PSWorkflow                          {New-PSWorkflowExecutionOption, New-PSWorkflowSession, nwsn}  
Manifest   1.0.0.0    PSWorkflowUtility                   Invoke-AsWorkflow                                             
Manifest   1.0.0.0    ScheduledTasks                      {Get-ScheduledTask, Set-ScheduledTask, Register-ScheduledTa...
Manifest   2.0.0.0    SecureBoot                          {Confirm-SecureBootUEFI, Set-SecureBootUEFI, Get-SecureBoot...
Manifest   2.0.0.0    SmbShare                            {Get-SmbShare, Remove-SmbShare, Set-SmbShare, Block-SmbShar...
Manifest   2.0.0.0    SmbWitness                          {Get-SmbWitnessClient, Move-SmbWitnessClient, gsmbw, msmbw...}
Manifest   1.0.0.0    StartLayout                         {Export-StartLayout, Import-StartLayout, Export-StartLayout...
Manifest   2.0.0.0    Storage                             {Add-InitiatorIdToMaskingSet, Add-PartitionAccessPath, Add-...
Manifest   2.0.0.0    TLS                                 {New-TlsSessionTicketKey, Enable-TlsSessionTicketKey, Disab...
Manifest   1.0.0.0    TroubleshootingPack                 {Get-TroubleshootingPack, Invoke-TroubleshootingPack}         
Manifest   2.0.0.0    TrustedPlatformModule               {Get-Tpm, Initialize-Tpm, Clear-Tpm, Unblock-Tpm...}          
Manifest   2.0.0.0    VpnClient                           {Add-VpnConnection, Set-VpnConnection, Remove-VpnConnection...
Manifest   1.0.0.0    Wdac                                {Get-OdbcDriver, Set-OdbcDriver, Get-OdbcDsn, Add-OdbcDsn...} 
Manifest   1.0.0.0    WindowsDeveloperLicense             {Get-WindowsDeveloperLicense, Unregister-WindowsDeveloperLi...
Script     1.0        WindowsErrorReporting               {Enable-WindowsErrorReporting, Disable-WindowsErrorReportin...
Manifest   1.0.0.0    WindowsSearch                       {Get-WindowsSearchSetting, Set-WindowsSearchSetting}          
Manifest   1.0.0.0    WindowsUpdate                       Get-WindowsUpdateLog            

Debug output

Error output

InnerException   : False
Exception        : Microsoft.Azure.Graph.RBAC.Version1_6.Models.GraphErrorException: Operation returned an invalid 
                   status code 'Forbidden'
                      at Microsoft.Azure.Graph.RBAC.Version1_6.ApplicationsOperations.<CreateWithHttpMessagesAsync>d__5.
                   MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at 
                   Microsoft.Azure.Graph.RBAC.Version1_6.ApplicationsOperationsExtensions.<CreateAsync>d__1.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.Azure.Graph.RBAC.Version1_6.ApplicationsOperationsExtensions.Create(IApplicationsOper
                   ations operations, ApplicationCreateParameters parameters)
                      at Microsoft.Azure.Commands.Aks.CreateOrUpdateKubeBase.BuildServicePrincipal(String name, String 
                   clientSecret)
                      at Microsoft.Azure.Commands.Aks.CreateOrUpdateKubeBase.EnsureServicePrincipal(String spId, String 
                   clientSecret)
                      at Microsoft.Azure.Commands.Aks.SetAzureRmAks.<>c__DisplayClass19_0.<ExecuteCmdlet>b__0()
                      at Microsoft.Azure.Commands.Aks.KubeCmdletBase.RunCmdLet(Action action)
                      at Microsoft.Azure.Commands.Aks.SetAzureRmAks.ExecuteCmdlet()
                      at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Message          : Operation returned an invalid status code 'Forbidden'
StackTrace       :    at Microsoft.Azure.Graph.RBAC.Version1_6.ApplicationsOperations.<CreateWithHttpMessagesAsync>d__5.
                   MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at 
                   Microsoft.Azure.Graph.RBAC.Version1_6.ApplicationsOperationsExtensions.<CreateAsync>d__1.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.Azure.Graph.RBAC.Version1_6.ApplicationsOperationsExtensions.Create(IApplicationsOper
                   ations operations, ApplicationCreateParameters parameters)
                      at Microsoft.Azure.Commands.Aks.CreateOrUpdateKubeBase.BuildServicePrincipal(String name, String 
                   clientSecret)
                      at Microsoft.Azure.Commands.Aks.CreateOrUpdateKubeBase.EnsureServicePrincipal(String spId, String 
                   clientSecret)
                      at Microsoft.Azure.Commands.Aks.SetAzureRmAks.<>c__DisplayClass19_0.<ExecuteCmdlet>b__0()
                      at Microsoft.Azure.Commands.Aks.KubeCmdletBase.RunCmdLet(Action action)
                      at Microsoft.Azure.Commands.Aks.SetAzureRmAks.ExecuteCmdlet()
                      at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
HelpLink         : 
ErrorDetails     : 
ErrorCategory    : CloseError: (:) [Set-AzAksCluster], GraphErrorException
InvocationInfo   : System.Management.Automation.InvocationInfo
ScriptStackTrace : at <ScriptBlock>, <No file>: line 43

InnerException   : False
Exception        : System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. 
                   Retrieve the LoaderExceptions property for more information.
                      at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
                      at System.Reflection.Assembly.GetTypes()
                      at Microsoft.PowerShell.Commands.AddTypeCommand.LoadAssemblyFromPathOrName(List`1 generatedTypes)
                      at Microsoft.PowerShell.Commands.AddTypeCommand.EndProcessing()
                      at System.Management.Automation.CommandProcessorBase.Complete()
Message          : Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more 
                   information.
StackTrace       :    at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
                      at System.Reflection.Assembly.GetTypes()
                      at Microsoft.PowerShell.Commands.AddTypeCommand.LoadAssemblyFromPathOrName(List`1 generatedTypes)
                      at Microsoft.PowerShell.Commands.AddTypeCommand.EndProcessing()
                      at System.Management.Automation.CommandProcessorBase.Complete()
HelpLink         : 
ErrorDetails     : 
ErrorCategory    : NotSpecified: (:) [Add-Type], ReflectionTypeLoadException
InvocationInfo   : System.Management.Automation.InvocationInfo
ScriptStackTrace : at <ScriptBlock>, C:\Modules\User\Az.Accounts\Az.Accounts.psm1: line 68
                   at <ScriptBlock>, C:\Modules\User\Az.Accounts\Az.Accounts.psm1: line 65
                   at <ScriptBlock>, <No file>: line 16

[wyunchi-ms]

This issue is fixed in code and will be published in the next release.