[BUG] Occasional ManagedIdentity failures

[garrettlondon1]

Library name and version

Azure.Storage.Blobs" Version="12.20.0"

Describe the bug

When using Managed Identity credential and RBAC, the app running on Azure App Service has the correct credentials and network connections.

The application will succeed and get all data from blob storage, but somewhere in the middle when the application is running, Managed Identity fails and gets a bad token.

This happens intermittently, but the problem is not resolved until app is restarted

services.AddAzureClients(x =>
        {
            if (EnvironmentEx.IsOffline)
            {
                x.AddBlobServiceClient("UseDevelopmentStorage=true");
            }
            else
            {
                x.AddBlobServiceClient(new Uri(config["ApplicationConfiguration:BlobUrl"] ?? throw new Exception("BlobUrl is missing")));
                x.UseCredential(EnvironmentEx.IsDevelopment ? new AzureCliCredential() : new ManagedIdentityCredential());
            }
        });

ManagedIdentityCredential.GetToken invoked. Scopes: [ https://storage.azure.com//.default ] ParentRequestId: 7cb8a1ff-5041-46aa-a693-dcb04a13f0e1
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:08:02Z - d196e687-f152-46b0-bee9-d4bbcd4454ae] MSAL MSAL.NetCore with assembly version '4.61.3.0'. CorrelationId(d196e687-f152-46b0-bee9-d4bbcd4454ae)
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:08:02Z - d196e687-f152-46b0-bee9-d4bbcd4454ae] === AcquireTokenForClientParameters === SendX5C: False ForceRefresh: False
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:08:02Z - d196e687-f152-46b0-bee9-d4bbcd4454ae] === Request Data === Authority Provided? - True Scopes - https://storage.azure.com//.default Extra Query Params Keys (space separated) - ApiId - AcquireTokenForClient IsConfidentialClient - True SendX5C - False LoginHint ? False IsBrokerConfigured - False HomeAccountId - False CorrelationId - d196e687-f152-46b0-bee9-d4bbcd4454ae UserAssertion set: False LongRunningOboCacheKey set: False Region configured:
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:08:02Z - d196e687-f152-46b0-bee9-d4bbcd4454ae] === Token Acquisition (ClientCredentialRequest) started: Scopes: https://storage.azure.com//.default Authority Host: login.microsoftonline.com
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:08:02Z - d196e687-f152-46b0-bee9-d4bbcd4454ae] [Region discovery] Not using a regional authority.
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:08:02Z - d196e687-f152-46b0-bee9-d4bbcd4454ae] Skipping Instance discovery for Aad authority because it is not enabled.
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:08:02Z - d196e687-f152-46b0-bee9-d4bbcd4454ae] Fetched access token from host login.microsoftonline.com.

ManagedIdentityCredential.GetToken invoked. Scopes: [ https://storage.azure.com//.default ] ParentRequestId: 8755a181-7153-469f-82a8-3e8d354f56f9
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] MSAL MSAL.NetCore with assembly version '4.61.3.0'. CorrelationId(66d07183-a899-497d-8435-ea84f6ce7862)
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] === AcquireTokenForClientParameters === SendX5C: False ForceRefresh: False
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] === Request Data === Authority Provided? - True Scopes - https://storage.azure.com//.default Extra Query Params Keys (space separated) - ApiId - AcquireTokenForClient IsConfidentialClient - True SendX5C - False LoginHint ? False IsBrokerConfigured - False HomeAccountId - False CorrelationId - 66d07183-a899-497d-8435-ea84f6ce7862 UserAssertion set: False LongRunningOboCacheKey set: False Region configured:
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] === Token Acquisition (ClientCredentialRequest) started: Scopes: https://storage.azure.com//.default Authority Host: login.microsoftonline.com
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] [Region discovery] Not using a regional authority.
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] [Instance Discovery] Skipping Instance discovery because it is disabled.
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] [ClientCredentialRequest] Acquiring a token from the token provider.
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] Checking client info returned from the server..
False MSAL 4.61.3.0 MSAL.NetCore .NET 8.0.6 Microsoft Windows 10.0.20348 [2024-08-01 13:14:32Z - 66d07183-a899-497d-8435-ea84f6ce7862] Saving token response to cache..

Expected behavior

Managed Identity credential consistently works on App Service

Actual behavior

Managed Identity disconnects and fails to make calls to blob gateway

Reproduction Steps

services.AddAzureClients(x =>
        {
            if (EnvironmentEx.IsOffline)
            {
                x.AddBlobServiceClient("UseDevelopmentStorage=true");
            }
            else
            {
                x.AddBlobServiceClient(new Uri(config["ApplicationConfiguration:BlobUrl"] ?? throw new Exception("BlobUrl is missing")));
                x.UseCredential(EnvironmentEx.IsDevelopment ? new AzureCliCredential() : new ManagedIdentityCredential());
            }
        });

Environment

No response

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[christothes]

Hi @garrettlondon1

but somewhere in the middle when the application is running, Managed Identity fails and gets a bad token.

What error or logging are you seeing that indicates that the managed identity is failing and that a bad token is received?

[github-actions]

Hi @garrettlondon1. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[garrettlondon1]

Hey @christothes , I apologize, I do not have any errors or logs indicating that managed identity is failing, and a bad token is received, specifically.

All resources from blob storage cannot be loaded, and when the App Service is restarted, everything works again instantly. I've reproduced it probably 5+ times.

I know for certain that the App service has access to the storage account because the same storage account hosts the zip file which powers the app service deployment via private endpoint

This is on an "always on" app service, blazor server monolith, using the absolute default managed identity setup

[christothes]

Does your service hang or does it continue to be responsive after the problem? Is the request to storage still occurring after the problem? If so, what is the response from the service?

[github-actions]

Hi @garrettlondon1. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[garrettlondon1]

The service completely hangs, although I will say on "12.21.1" version, have not encountered in 6+ days

[christothes]

That makes sense - I think the issue won't reproduce on 12.21.1 or later. This is due to a fix in one of its dependencies in Azure.Core that was shipped in version 1.41.0. The fix is here

[github-actions]

Hi @garrettlondon1. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[github-actions]

Hi @garrettlondon1, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.