Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] token retrieval intermittently stuck in BearerTokenAuthenticationPolicy in Azure.Core 1.40.0 #44817

Closed
v-mykhalchuk opened this issue Jul 1, 2024 · 9 comments · Fixed by #44882 or microsoft/azure-pipelines-agent#4921
Assignees
Labels
Azure.Core Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-author-feedback Workflow: More information is needed from author to address the issue. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@v-mykhalchuk
Copy link

v-mykhalchuk commented Jul 1, 2024

Library name and version

Azure.Core 1.40.0

Describe the bug

Hello,
After update from version 1.38.0 to 1.40.0 of Azure.Core we have started to experience intermittent issues of Requests being stuck in BearerTokenAuthenticationPolicy.
Specifically, we are trying to get a secret from key vault under user-assigned managed indentity, and we see that the SecretClient.GetSecret dependency is being stuck for a duration of our request timeout, and then fails with OperationCancelledException. Same code seem to work well after downgrade back to 1.38.0 without any other changes.

When the issue occurs, it reproduces pretty reliably in the same place for the same secret until application restart.

There is no high concurrency involved in this code path.

image

Stack Trace:

System.OperationCanceledException:
at System.Threading.CancellationToken.ThrowOperationCanceledException (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.TaskExtensions+WithCancellationTaskAwaiter`1.GetResult (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+TokenRequestState+d__19.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+d__6.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+d__12.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy+d__7.MoveNext (Azure.Security.KeyVault.Secrets, Version=4.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+d__11.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.RedirectPolicy+d__7.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.RetryPolicy+d__5.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.RetryPolicy+d__5.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.HttpPipeline+d__18.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Security.KeyVault.KeyVaultPipeline+d__29.MoveNext (Azure.Security.KeyVault.Secrets, Version=4.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Security.KeyVault.KeyVaultPipeline+d__21'1.MoveNext (Azure.Security.KeyVault.Secrets, Version=4.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Security.KeyVault.Secrets.SecretClient+d__10.MoveNext (Azure.Security.KeyVault.Secrets, Version=4.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)

Expected behavior

KeyVault call is executed

Actual behavior

The call is stuck and times out

Reproduction Steps

Unfortunately, we don't have a repro yet.

Environment

Azure App Services, Windows, .NET 8.0

@v-mykhalchuk v-mykhalchuk changed the title [BUG] token retrieval stuck in BearerTokenAuthenticationPolicy [BUG] token retrieval stuck in BearerTokenAuthenticationPolicy in Azure.Core 1.40.0 Jul 1, 2024
@github-actions github-actions bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jul 1, 2024
@jsquire jsquire self-assigned this Jul 1, 2024
@jsquire jsquire added Client This issue points to a problem in the data-plane of the library. Azure.Core labels Jul 1, 2024
@github-actions github-actions bot removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Jul 1, 2024
@jsquire
Copy link
Member

jsquire commented Jul 1, 2024

Hi @v-mykhalchuk. Thanks for reaching out and we regret that you're experiencing difficulties. Unfortunately, there's not enough information available to assist. Please collect a 5-minute slice of verbose Azure SDK logs around the behavior for analysis. Examples for gathering them can be found in this sample and further context is discussed in [Logging with the Azure SDK for .NET)(https://learn.microsoft.com/dotnet/azure/sdk/logging).

@jsquire jsquire added the needs-author-feedback Workflow: More information is needed from author to address the issue. label Jul 1, 2024
Copy link

github-actions bot commented Jul 1, 2024

Hi @v-mykhalchuk. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@PixelByProxy
Copy link

I also experienced this issue after upgrading to Azure.Identity 1.12.0 which updated the Azure.Core reference to 1.40.0. Reverting back to 1.11.2 (Azure.Core 1.38.0) resolved the issue.

Stack Trace:
[{"severityLevel":"Warning","outerId":"0","message":"The operation was canceled.","type":"System.OperationCanceledException","id":"54893588","parsedStack":[{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Threading.CancellationToken.ThrowOperationCanceledException","level":0,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+TokenRequestState+<GetCurrentHeaderValue>d__19.MoveNext","level":1,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":2,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":3,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":4,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+<GetAuthHeaderValueAsync>d__6.MoveNext","level":5,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":6,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":7,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":8,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+<AuthenticateAndAuthorizeRequestAsync>d__12.MoveNext","level":9,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":10,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":11,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":12,"line":0},{"assembly":"Azure.Storage.Blobs, Version=12.19.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Storage.StorageBearerTokenChallengeAuthorizationPolicy+<AuthorizeRequestInternal>d__7.MoveNext","level":13,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":14,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":15,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":16,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+<ProcessAsync>d__11.MoveNext","level":17,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":18,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":19,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":20,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.HttpPipelineSynchronousPolicy+<InnerProcessAsync>d__5.MoveNext","level":21,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":22,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":23,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":24,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.RedirectPolicy+<ProcessAsync>d__7.MoveNext","level":25,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":26,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":27,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":28,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__5.MoveNext","level":29,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":30,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__5.MoveNext","level":31,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":32,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":33,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":34,"line":0},{"assembly":"Azure.Storage.Blobs, Version=12.19.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Storage.Blobs.PageBlobRestClient+<UploadPagesAsync>d__11.MoveNext","level":35,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":36,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":37,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":38,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.ConfiguredTaskAwaitable``1+ConfiguredTaskAwaiter.GetResult","level":39,"line":0},{"assembly":"Azure.Storage.Blobs, Version=12.19.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Storage.Blobs.Specialized.PageBlobClient+<UploadPagesInternal>d__36.MoveNext","level":40,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":41,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":42,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":43,"line":0},{"assembly":"Azure.Storage.Blobs, Version=12.19.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Storage.Blobs.Specialized.PageBlobClient+<UploadPagesAsync>d__33.MoveNext","level":44,"line":0}

@v-mykhalchuk
Copy link
Author

@jsquire we are working on some sort of repro for this. We had to revert back the version in our main code as this was blocking our testing. I will reply back when we are able to collect any additional info on this.

@github-actions github-actions bot added needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team and removed needs-author-feedback Workflow: More information is needed from author to address the issue. labels Jul 3, 2024
@v-mykhalchuk v-mykhalchuk changed the title [BUG] token retrieval stuck in BearerTokenAuthenticationPolicy in Azure.Core 1.40.0 [BUG] token retrieval intermittently stuck in BearerTokenAuthenticationPolicy in Azure.Core 1.40.0 Jul 3, 2024
@jsquire jsquire added needs-author-feedback Workflow: More information is needed from author to address the issue. and removed needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team labels Jul 4, 2024
Copy link

github-actions bot commented Jul 4, 2024

Hi @v-mykhalchuk. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@johannesoenema
Copy link

johannesoenema commented Jul 4, 2024

We experienced the same in two of our APIs. These APIs are usually very stable, but no longer after updating to the latest Azure.Identity (probably relying on newer version of Azure.Core) package. We experienced slowness and ultimately failure. The issue is that we don't experience the whole time, but the sudden unpredictable downtimes made us decide to downgrade to the previous version of the Azure.Identity package.

image

As can be seen uploading a relatively small blob took more than 3 minutes:
image

@kevinharing
Copy link

kevinharing commented Jul 5, 2024

I'm seeing the same issue, possibly related to this new feature released in Azure.Core 1.40.0. I've have now pinned it to 1.39.0 to see if it still occurs.

I want to emphasize that it seems to hang indefinitely in this spot, unless the cancellation is triggered. One of our calls (GetIfExists on ManagedServiceIdentity) was running for 19.5 minutes before being cancelled via the supplied CancellationToken (part of a long running job).

Example stack trace:

System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.CancellationToken.ThrowIfCancellationRequested()
at Azure.Core.Pipeline.TaskExtensions.WithCancellationTaskAwaiter`1.GetResult()
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.GetCurrentHeaderValue(Boolean async, Boolean checkForCompletion, CancellationToken cancellationToken)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequestAsync(HttpMessage message, TokenRequestContext context)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.ResourceManager.ManagedServiceIdentities.FederatedIdentityCredentialsRestOperations.DeleteAsync(String subscriptionId, String resourceGroupName, String resourceName, String federatedIdentityCredentialResourceName, CancellationToken cancellationToken)
at Azure.ResourceManager.ManagedServiceIdentities.FederatedIdentityCredentialResource.DeleteAsync(WaitUntil waitUntil, CancellationToken cancellationToken)

@RemyBoyer
Copy link

Hello
Thanks for the fix!

Do you have an approximate date of when the new Azure.Core package version will be available?

@v-mykhalchuk
Copy link
Author

There was reply in similar issue #44901 that it is expected to release later this week

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Azure.Core Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-author-feedback Workflow: More information is needed from author to address the issue. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
7 participants