Az.Accounts v3.0.4 givimg warning in Azure Automation with PowerShell 5.1 runbooks

[Shruti-MS]

Description

Azure Automation allows customers to run PowerShell scripts as runbooks. We recently published a new Docker image for PowerShell 5.1 with Az module version 11.2.0, which includes Az.Identity v3.0.4. After switching to this image, we started encountering a warning in the Connect-AzAccount cmdlet.

Reproduction Steps:

1. Create an Azure Automation Account.
2. Upload Az.Accounts v3.0.4 as a custom module.
3. Create a PowerShell 5.1 runbook in Azure Automation.
4. Author the runbook with the following content:
   Connect-AzAccount -Identity
5. Publish the runbook.
6. Run the runbook.
7. Check the job logs, where you will see the following warning message:
   Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object."

This warning does not occur when using Az.Accounts v2.15.0.

Issue script & Debug output

Get-Module -listavailable
Disable-AzContextAutosave -Scope Process 

Connect-azaccount -identity

Environment data

Name                           Value                                                                                    
----                           -----                                                                                    
PSVersion                      5.1.17763.6189                                                                           
PSEdition                      Desktop                                                                                  
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                  
BuildVersion                   10.0.17763.6189                                                                          
CLRVersion                     4.0.30319.42000                                                                          
WSManStackVersion              3.0                                                                                      
PSRemotingProtocolVersion      2.3                                                                                      
SerializationVersion           1.1.0.1

Module versions

Directory: C:\usr\src\PSModules
ModuleType Version    Name                                ExportedCommands                                              
---------- -------    ----                                ----------------                                              
Binary     1.0        Automation.Sandbox.AssetManageme... {Get-AutomationVariable, Set-AutomationVariable, Get-Automa...

    Directory: C:\AzAt\Modules\11.2.0


ModuleType Version    Name                                ExportedCommands                                              
---------- -------    ----                                ----------------                                              
Script     3.0.4      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enabl...

Script     2.0.0      Az.Advisor                          {Disable-AzAdvisorRecommendation, Enable-AzAdvisorRecommend...

Script     6.0.0      Az.Aks                              {Get-AzAksCluster, New-AzAksCluster, Remove-AzAksCluster, I...

Script     1.1.4      Az.AnalysisServices                 {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisService...

Script     4.0.2      Az.ApiManagement                    {Add-AzApiManagementApiToGateway, Add-AzApiManagementApiToP...

Script     1.0.0      Az.App                              {Disable-AzContainerAppRevision, Enable-AzContainerAppRevis...

Script     1.3.0      Az.AppConfiguration                 {Get-AzAppConfigurationStore, Get-AzAppConfigurationStoreKe...

Script     2.2.3      Az.ApplicationInsights              {Get-AzApplicationInsights, Get-AzApplicationInsightsApiKey...

Script     1.0.0      Az.ArcResourceBridge                {Get-AzArcResourceBridge, Get-AzArcResourceBridgeApplianceC...

Script     2.0.0      Az.Attestation                      {Get-AzAttestationPolicy, Set-AzAttestationPolicy, Reset-Az...

Script     1.0.0      Az.Automanage                       {Get-AzAutomanageBestPractice, Get-AzAutomanageConfigProfil...

Script     1.10.0     Az.Automation                       {Export-AzAutomationDscConfiguration, Export-AzAutomationDs...

Script     3.5.0      Az.Batch                            {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAcco...

Script     2.0.3      Az.Billing                          {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmen...

Script     3.1.1      Az.Cdn                              {Clear-AzCdnEndpointContent, Clear-AzFrontDoorCdnEndpointCo...

Script     2.0.0      Az.CloudService                     {Get-AzCloudService, Get-AzCloudServiceInstanceView, Get-Az...

Script     1.14.1     Az.CognitiveServices                {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAcc...

Script     7.1.1      Az.Compute                          {Add-AzImageDataDisk, Add-AzVhd, Add-AzVMAdditionalUnattend...

Script     1.0.0      Az.ConfidentialLedger               {Get-AzConfidentialLedger, New-AzConfidentialLedger, New-Az...

Script     4.0.0      Az.ContainerInstance                {Add-AzContainerInstanceOutput, Get-AzContainerGroup, Get-A...

Script     4.1.3      Az.ContainerRegistry                {Connect-AzContainerRegistry, Get-AzContainerRegistryManife...

Script     1.14.0     Az.CosmosDB                         {Get-AzCosmosDBAccount, Get-AzCosmosDBAccountKey, Get-AzCos...

Script     1.1.0      Az.DataBoxEdge                      {Get-AzDataBoxEdgeJob, Get-AzDataBoxEdgeDevice, Invoke-AzDa...

Script     1.7.1      Az.Databricks                       {Get-AzDatabricksAccessConnector, Get-AzDatabricksOutboundN...

Script     1.18.0     Az.DataFactory                      {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFac...

Script     1.0.3      Az.DataLakeAnalytics                {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalytics...

Script     1.3.0      Az.DataLakeStore                    {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeSto...

Script     2.2.0      Az.DataProtection                   {Backup-AzDataProtectionBackupInstanceAdhoc, Edit-AzDataPro...

Script     1.0.1      Az.DataShare                        {New-AzDataShareAccount, Get-AzDataShareAccount, Remove-AzD...

Script     1.1.0      Az.DeploymentManager                {Get-AzDeploymentManagerArtifactSource, New-AzDeploymentMan...

Script     4.3.0      Az.DesktopVirtualization            {Disconnect-AzWvdUserSession, Expand-AzWvdMsixImage, Get-Az...

Script     1.1.0      Az.DevCenter                        {Deploy-AzDevCenterUserEnvironment, Get-AzDevCenterAdminAtt...

Script     1.0.2      Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy...

Script     1.2.0      Az.Dns                              {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRec...

Script     1.6.0      Az.EventGrid                        {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGri...

Script     4.2.0      Az.EventHub                         {New-AzEventHubAuthorizationRuleSASToken, Approve-AzEventHu...

Script     1.10.0     Az.FrontDoor                        {New-AzFrontDoor, Get-AzFrontDoor, Set-AzFrontDoor, Remove-...

Script     4.0.7      Az.Functions                        {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get...

Script     6.1.0      Az.HDInsight                        {Add-AzHDInsightClusterIdentity, Add-AzHDInsightComponentVe...

Script     2.0.0      Az.HealthcareApis                   {Get-AzHealthcareApisService, Get-AzHealthcareApisWorkspace...

Script     2.7.5      Az.IotHub                           {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-Az...

Script     5.1.0      Az.KeyVault                         {Add-AzKeyVaultCertificate, Add-AzKeyVaultCertificateContac...

Script     2.3.0      Az.Kusto                            {Add-AzKustoClusterLanguageExtension, Add-AzKustoDatabasePr...

Script     1.0.0      Az.LoadTesting                      {Get-AzLoad, New-AzLoad, Remove-AzLoad, Update-AzLoad}        

Script     1.5.0      Az.LogicApp                         {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccoun...

Script     1.1.3      Az.MachineLearning                  {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociat...

Script     1.0.0      Az.MachineLearningServices          {Get-AzMLServiceQuota, Get-AzMLServiceUsage, Get-AzMLServic...

Script     1.4.1      Az.Maintenance                      {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMa...

Script     1.2.0      Az.ManagedServiceIdentity           {Get-AzSystemAssignedIdentity, Get-AzUserAssignedIdentity, ...

Script     3.0.0      Az.ManagedServices                  {Get-AzManagedServicesAssignment, Get-AzManagedServicesDefi...

Script     2.0.0      Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Invoke-AzMarketplaceSignTerms, Set...

Script     1.1.2      Az.Media                            {Sync-AzMediaServiceStorageKey, Set-AzMediaServiceKey, Get-...

Script     2.2.0      Az.Migrate                          {Get-AzMigrateDiscoveredServer, Get-AzMigrateJob, Get-AzMig...

Script     5.0.0      Az.Monitor                          {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile,...

Script     1.1.1      Az.MySql                            {Get-AzMySqlConfiguration, Get-AzMySqlConnectionString, Get...

Script     7.3.0      Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Add-AzA...

Script     1.0.0      Az.NetworkCloud                     {Deploy-AzNetworkCloudCluster, Disable-AzNetworkCloudStorag...

Script     1.1.2      Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationR...

Script     3.2.0      Az.OperationalInsights              {New-AzOperationalInsightsAzureActivityLogDataSource, New-A...

Script     1.6.4      Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSum...

Script     1.1.0      Az.PostgreSql                       {Get-AzPostgreSqlConfiguration, Get-AzPostgreSqlConnectionS...

Script     2.0.0      Az.PowerBIEmbedded                  {Resume-AzPowerBIEmbeddedCapacity, Suspend-AzPowerBIEmbedde...

Script     1.0.4      Az.PrivateDns                       {Get-AzPrivateDnsZone, Remove-AzPrivateDnsZone, Set-AzPriva...

Script     6.7.0      Az.RecoveryServices                 {Add-AzRecoveryServicesAsrReplicationProtectedItemDisk, Bac...

Script     1.8.1      Az.RedisCache                       {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheSchedule...

Script     1.2.0      Az.RedisEnterpriseCache             {Export-AzRedisEnterpriseCache, Get-AzRedisEnterpriseCache,...

Script     2.0.0      Az.Relay                            {Get-AzRelayAuthorizationRule, Get-AzRelayHybridConnection,...

Script     1.2.0      Az.ResourceMover                    {Add-AzResourceMoverMoveResource, Get-AzResourceMoverMoveCo...

Script     6.13.0     Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyA...

Script     1.5.1      Az.Security                         {Get-AzSecuritySolution, Get-AzSecuritySolutionsReferenceDa...

Script     3.1.1      Az.SecurityInsights                 {Get-AzSentinelAlertRule, Get-AzSentinelAlertRuleAction, Ge...

Script     3.0.0      Az.ServiceBus                       {New-AzServiceBusAuthorizationRuleSASToken, Test-AzServiceB...

Script     3.3.2      Az.ServiceFabric                    {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricM...

Script     2.0.0      Az.SignalR                          {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSign...

Script     4.13.0     Az.Sql                              {Add-AzSqlDatabaseToFailoverGroup, Add-AzSqlElasticJobStep,...

Script     2.1.0      Az.SqlVirtualMachine                {Get-AzAvailabilityGroupListener, Get-AzSqlVM, Get-AzSqlVMG...

Script     2.2.3      Az.StackHCI                         {Add-AzStackHCIVMAttestation, Disable-AzStackHCIAttestation...

Script     6.1.0      Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountMan...

Script     1.3.0      Az.StorageMover                     {Get-AzStorageMover, Get-AzStorageMoverAgent, Get-AzStorage...

Script     2.1.1      Az.StorageSync                      {Get-AzStorageSyncCloudEndpoint, Get-AzStorageSyncGroup, Ge...

Script     2.0.0      Az.StreamAnalytics                  {Get-AzStreamAnalyticsCluster, Get-AzStreamAnalyticsCluster...

Script     1.0.0      Az.Support                          {Get-AzSupportService, Get-AzSupportProblemClassification, ...

Script     3.0.5      Az.Synapse                          {Add-AzSynapseDataFlowDebugSessionPackage, Add-AzSynapseTri...

Script     1.2.2      Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTraff...

Script     3.1.2      Az.Websites                         {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServi...

Error output

Warning:-Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object."

[Borgquite]

@Shruti-MS - At what point do you get this error? We also started seeing a similar error in Azure Automation starting on 13th September 2024 - but we haven't updated Az.Accounts yet (still running v2.15.0).

1. Create a new PowerShell 5.1 Runbook in Azure Automation
2. Add the following code:
   Update-AzConfig -Scope Process -CheckForUpgrade $false -EnableDataCollection $false | Out-Null
3. Check runbook logs:
   Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object."

Perhaps this is an Azure Automation issue?

[Shruti-MS]

@Borgquite , Facing error since 13th Sept for 5.1 runbooks as image changed, eventually all runbooks run on v3.0.4 of Az.Accounts.

If you are saying Az.Accounts version by seeing from modules tab it is wrong. You have to write in runbook Get-Module -listavailable, to get the Az.Accounts version currently used in runbook.

[dm9321]

@Shruti-MS Thank you for this post, I think it explains a lot of issues we are having. We are having this issue (we run PowerShell 5.1 in Azure Automation), and it shows Az. Accounts 2.15 in Modules tab but the verbose log shows it is still loading v3.0.4 as you described. Is there a way to prevent this behavior? This is concerning because there have been a lot of breaking issues lately and we depend on the automation account respecting the versions we have configured. Just a couple weeks ago we had to roll back our MSGraph and EXO modules to earlier versions due to another incompatibility issue with Az modules that broke modules in all of our environments (link below). Since Azure Automation did not offer an Az config newer than 11.2, we had to roll back the other 2 modules to get a working combination. It gave us a workaround for now, but it is concerning if the Az modules are potentially getting changed to run different versions than what is actually configured in the Modules tab without our knowledge until things start breaking. We don't run preview or beta modules in production, and it will likely be a while before there is a fix to these issues that will work in Azure Automation.

microsoftgraph/msgraph-sdk-powershell#2803

@Borgquite , as the container image used in Azure Automation changed on 13th Sept for 5.1 runbooks eventually all runbooks run on v3.0.4 of Az.Accounts.

If you are saying Az.Accounts version by seeing from modules tab it is wrong. You have to write in runbook Get-Module -listavailable, to get the Az.Accounts version currently used in runbook.

[Borgquite]

@Shruti-MS Please you let us know when and where we should have found out about this (now clearly breaking) change, if you hadn’t made this post?

Nothing here
https://learn.microsoft.com/en-us/azure/automation/whats-new

[michaelmsonne]

We see "Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object."" too in our automation!

We love the bugs like this too: microsoft/Microsoft365DSC#4720 😂

[ChawnLimited]

Same issue here with an automation acct and a powershell pipeline since updating to az.accounts 3.0.4
Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object."

Despite the exception, my pipeline jobs are still completing

I tried removing az.accounts 3.04 and install-module az.accounts -maximumversion 3.0.3
It worked, but then az.resources 7.4.0 could not load stating that it required az.accounts 3.0.4

[isra-fel]

Can confirm 3.0.4 is actually used but cannot reproduce with either Connect-AzAccount -Identity nor Update-AzConfig. Will let the runbook run periodically to see if it reproduces.

Loaded modules


ModuleType Version    Name                                ExportedCommands                                              
---------- -------    ----                                ----------------                                              
Script     3.0.4      Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-...

[isra-fel]

Our analysis showed the most likely root cause is here https://github.com/isra-fel/azure-powershell/blob/513d6c028c5f87e881b3d7d7c0f95e6c81702a44/src/Accounts/Accounts/StartupScripts/InitializePSStyle.ps1#L3 when Az.Accounts is imported it tries to initialize some utilities related to styling, but unexpectedly $host is null.
It is still under investigation why that happens.

To make sure the RCA is correct, I want to confirm with everyone if you all received warning messages, not errors?

[Borgquite]

To make sure the RCA is correct, I want to confirm with everyone if you all received warning messages, not errors?

@isra-fel Yes, I’m receiving warnings.

[Shruti-MS]

Yes, coming as warnings.

[dm9321]

Yes, warnings.

[eeode]

@isra-fel, we have the same problem, with several Automation Accounts, but only with scripts using "Disable-AzContextAutosave -Scope Process" we get warnings like the OP, e.g. the following script snippet:

Write-Output "PSVersion is : $((Get-Host).Version.Major).$((Get-Host).Version.Minor).$((Get-Host).Version.Build).$((Get-Host).Version.Revision)`n"

Write-Output "Disable AzContextAutosave..."
Disable-AzContextAutosave -Scope Process | Out-Null
Write-Output ""

try
{
    Write-Output "Logging in to Azure..."
    $azContext = (Connect-AzAccount -Identity).Context
    if ($?) { Write-Output "Login successfull." } else { Write-Error "Error Logging in??" }
}
catch {
    [...]
    exit
}

will produce the following output:

Output, PSVersion is : 5.1.17763.6189
Output, Disable AzContextAutosave...
Warning, Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object."
Output, Logging in to Azure...
Output, Login successfull.

I can not repoduce this behaviour in a normal PS or CloudShell, but my guess would be "Disable-AzContextAutosave".

[isra-fel]

We are able to find the root cause.

This issue happens when Az.Accounts module is imported. That triggers the initialization of PSStyle, the type that controls the styling of some messages. We use the PSHostUserInterface.SupportsVirtualTerminal Property to detect whether advanced styling (i.e. escape sequence) is supported, but the getter of this property throws a NullReferenceException. This is unexpected because (a) it's not documented (b) for a straightforward property like this we'd expect it to handle any exceptions internally and just return a default value.

Based on the root cause analysis above, it's certain that this warning doesn't have any impact on the execution and the result of the runbook. We are talking to Azure Automation team to see if they can do something to prevent exception from being thrown.

stack trace:

System.NullReferenceException: Object reference not set to an instance of an object.
    at System.Management.Automation.Internal.Host.InternalHostUserInterface.get_SupportsVirtualTerminal()

source code:

https://github.com/Azure/azure-powershell-common/blob/061407ff9f8ddadfe435bfd03c925f5f72dcf7ec/src/Common/ColorAndFormat/PSStyle.cs#L34C42-L34C75

[Borgquite]

@isra-fel Thanks - couldn't you suppress the warning inside Az.Accounts?

[eeode]

I see, that Az.Accounts was downgraded from 3.0.4 to 2.15.0 on two Automation Acounts in the backend. So, the warning is gone for the time beeing.

[Borgquite]

@eeode Yeah - except now if I try to use a Microsoft Graph call I get the following:

Method not found: 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])'.

This error appeared in our Automation account at the precise same time as the Az account warnings came to an end.

Come on Microsoft! This is as bad as the days of 'DLL Hell' in Windows 3.1. When on earth are you going to make your PowerShell modules play nicely together?

And for the Automation team - why aren't you respecting the modules that we've selected in the Automation Modules any more, and why is there no notification that you're doing so?

@Shruti-MS @isra-fel

[Borgquite]

@Shruti-MS If we are running PowerShell 5.1 modules on Azure Automation, you have to give us the ability to control which module versions we use. It's a well known problem that comes up again and again - if Azure Automation keeps changing the Az module version that it uses behind the scenes, it'll be unusable

microsoftgraph/msgraph-sdk-powershell#2148
https://learn.microsoft.com/en-us/powershell/scripting/dev-cross-plat/resolving-dependency-conflicts?view=powershell-7.4