Error logs cluttered with instances CredentialUnavailableException

[rodgersmith]

Describe the bug
Please provide the description of issue you're seeing.

We recently implemented authenticating with Azure Key vault using Managed Service Identity. It seems that for the majority of the time we are authenticating as we expect to, but our error logs are overloaded with errors detailing the instances where this failed. On a typical day we see hundreds of the same typical exception. Is this expected behavior or are we doing something wrong?

Azure.Identity.CredentialUnavailableException: DefaultAzureCredential failed to retrieve a token from the included credentials.

• EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
• ManagedIdentityCredential authentication unavailable. No Managed Identity endpoint found.
• Operating system Linux 5.4.0-1034-azure [winmd servicebus] custom numeric properties are getting converted to double #35~18.04.1-Ubuntu SMP Thu Dec 10 09:13:52 UTC 2020 isn't supported.
• Stored credentials not found. Need to authenticate user in VSCode Azure Account.
• Azure CLI not installed
  ``
  Environment:
• Name and version of the Library package used: EntityFramework 5.02 Azure.Identity: 1.3
• Hosting platform or OS and .NET runtime: version Linux 5.4.0-1034-azure [winmd servicebus] custom numeric properties are getting converted to double #35~18.04.1-Ubuntu

Our Startup.cs fetches the keyVault configs we will need:

var keyVaultEndpoint = Configuration.GetValue<string>("KeyVault");
 // using Key Vault, either local dev or deployed
var configuration = new ConfigurationBuilder() .AddAzureKeyVault(new Uri(keyVaultEndpoint), new DefaultAzureCredential(new DefaultAzureCredentialOptions { ExcludeSharedTokenCacheCredential = true })).Build();
services.Configure<KeyVaultSettings>(configuration);

And the dbContext where we fetch the token to connect to the db.
(note: we have several different schemas we connect to per dbContext so we do not wire this up in the startup)

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
   {
       
       var defaultCredential = new DefaultAzureCredential(new DefaultAzureCredentialOptions
           {ExcludeManagedIdentityCredential = true});
       var token = defaultCredential.GetToken(new TokenRequestContext(new[]
           {"https://database.windows.net/.default"}));
       var connection = new SqlConnection
           {ConnectionString = _connectionString, AccessToken = token.Token};

       optionsBuilder
           .UseSqlServer(connection,
               providerOptions => providerOptions.CommandTimeout(200))
           .UseQueryTrackingBehavior(QueryTrackingBehavior.NoTracking);
          
   }

[Mohit-Chakraborty]

Thank you for your feedback. Tagging and routing to the team best able to assist.

[heaths]

This is the logging to diagnose which chained credentials failed for DefaultAzureCredential. While @schaabs owns that, I will comment that these logs can and have been very helpful in diagnosing why one credential someone thought should've worked (like MSI) didn't. If, in your environment, you know exactly which credential(s) you need, you can use just that one, or configure DefaultAzureCredentialOptions to enable just the 1 or more credentials you need, which should at least decrease the volume of errors you see.

[schaabs]

@rodgersmith Thanks for filing this issue and reaching out for more information. As @heaths mentioned the behavior of credentials in the DefaultAzureCredential throwing CredentialUnavailableException is indeed expected behavior. This exception type is used by chained credential types such as the DefaultAzureCredential to signal to continue with the next credential in the chain. If you want to avoid these internally handled exceptions you could, as @heaths suggested, configure the DefaultAzureCredential to limit the number of credentials it will try, or use the specific credential type you intend to authenticate with rather than the DefaultAzureCredential.

However, if you receive the top level error you stated above:

Azure.Identity.CredentialUnavailableException: DefaultAzureCredential failed to retrieve a token from the included credentials.

This would indicate that none of the attempted credentials were able to retrieve a token and the authentication failed. In this case the initial call to GetToken would fail with a CredentialUnavailableExcpetion. Is this the happening in your case? If so, does it only happen intermittently? Also, what environment are you running into this issue is it directly on a VM or VMSS, or is it another azure host such as an App Service or AKS cluster?

[rodgersmith]

Hi @schaabs. I have implemented the DefaultAzureCredentialOptions in order to exclude the unnecessary credential types. The relevant failure we encounter is :

---> Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. No Managed Identity endpoint found.
   at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)

This issue happens intermittently with no good way of predicting it. 9 times out of 10 it connects correctly. We are connecting to an Azure db via EntityFramework 5.02 Azure.Identity: 1.3. The environment is an AKS cluster.

[rodgersmith]

We changed the code a bit to stop using the DefaultAzureCredential entirely and the issue is gone now.
connection.AccessToken = (new Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider()).GetAccessTokenAsync("https://database.windows.net/").Result;