Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error logs cluttered with instances CredentialUnavailableException #19213

Closed
rodgersmith opened this issue Mar 4, 2021 · 5 comments
Closed
Assignees
Labels
Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@rodgersmith
Copy link

rodgersmith commented Mar 4, 2021

Describe the bug
Please provide the description of issue you're seeing.

We recently implemented authenticating with Azure Key vault using Managed Service Identity. It seems that for the majority of the time we are authenticating as we expect to, but our error logs are overloaded with errors detailing the instances where this failed. On a typical day we see hundreds of the same typical exception. Is this expected behavior or are we doing something wrong?

Azure.Identity.CredentialUnavailableException: DefaultAzureCredential failed to retrieve a token from the included credentials.

Our Startup.cs fetches the keyVault configs we will need:

var keyVaultEndpoint = Configuration.GetValue<string>("KeyVault");
 // using Key Vault, either local dev or deployed
var configuration = new ConfigurationBuilder() .AddAzureKeyVault(new Uri(keyVaultEndpoint), new DefaultAzureCredential(new DefaultAzureCredentialOptions { ExcludeSharedTokenCacheCredential = true })).Build();
services.Configure<KeyVaultSettings>(configuration);

And the dbContext where we fetch the token to connect to the db.
(note: we have several different schemas we connect to per dbContext so we do not wire this up in the startup)

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
   {
       
       var defaultCredential = new DefaultAzureCredential(new DefaultAzureCredentialOptions
           {ExcludeManagedIdentityCredential = true});
       var token = defaultCredential.GetToken(new TokenRequestContext(new[]
           {"https://database.windows.net/.default"}));
       var connection = new SqlConnection
           {ConnectionString = _connectionString, AccessToken = token.Token};

       optionsBuilder
           .UseSqlServer(connection,
               providerOptions => providerOptions.CommandTimeout(200))
           .UseQueryTrackingBehavior(QueryTrackingBehavior.NoTracking);
          
   }
@ghost ghost added needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Mar 4, 2021
@Mohit-Chakraborty Mohit-Chakraborty added Client This issue points to a problem in the data-plane of the library. KeyVault needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team labels Mar 4, 2021
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Mar 4, 2021
@Mohit-Chakraborty
Copy link
Contributor

Thank you for your feedback. Tagging and routing to the team best able to assist.

@heaths
Copy link
Member

heaths commented Mar 5, 2021

This is the logging to diagnose which chained credentials failed for DefaultAzureCredential. While @schaabs owns that, I will comment that these logs can and have been very helpful in diagnosing why one credential someone thought should've worked (like MSI) didn't. If, in your environment, you know exactly which credential(s) you need, you can use just that one, or configure DefaultAzureCredentialOptions to enable just the 1 or more credentials you need, which should at least decrease the volume of errors you see.

@schaabs
Copy link
Member

schaabs commented Mar 8, 2021

@rodgersmith Thanks for filing this issue and reaching out for more information. As @heaths mentioned the behavior of credentials in the DefaultAzureCredential throwing CredentialUnavailableException is indeed expected behavior. This exception type is used by chained credential types such as the DefaultAzureCredential to signal to continue with the next credential in the chain. If you want to avoid these internally handled exceptions you could, as @heaths suggested, configure the DefaultAzureCredential to limit the number of credentials it will try, or use the specific credential type you intend to authenticate with rather than the DefaultAzureCredential.

However, if you receive the top level error you stated above:

Azure.Identity.CredentialUnavailableException: DefaultAzureCredential failed to retrieve a token from the included credentials.

This would indicate that none of the attempted credentials were able to retrieve a token and the authentication failed. In this case the initial call to GetToken would fail with a CredentialUnavailableExcpetion. Is this the happening in your case? If so, does it only happen intermittently? Also, what environment are you running into this issue is it directly on a VM or VMSS, or is it another azure host such as an App Service or AKS cluster?

@rodgersmith
Copy link
Author

Hi @schaabs. I have implemented the DefaultAzureCredentialOptions in order to exclude the unnecessary credential types. The relevant failure we encounter is :

---> Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. No Managed Identity endpoint found.
   at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)

This issue happens intermittently with no good way of predicting it. 9 times out of 10 it connects correctly. We are connecting to an Azure db via EntityFramework 5.02 Azure.Identity: 1.3. The environment is an AKS cluster.

@rodgersmith
Copy link
Author

We changed the code a bit to stop using the DefaultAzureCredential entirely and the issue is gone now.
connection.AccessToken = (new Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider()).GetAccessTokenAsync("https://database.windows.net/").Result;

azure-sdk pushed a commit to azure-sdk/azure-sdk-for-net that referenced this issue May 25, 2022
machinelearningservice-typescript.md-update (Azure#19213)
@github-actions github-actions bot locked and limited conversation to collaborators Mar 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Development

No branches or pull requests

4 participants