Server failed to authenticate the request. on docker App service with Managed identity after 24 hours

[ksstott]

• Package Name: @azure/identity & @azure/storage-queue
• Package Version: 2.0.4 & 12.9.0
• Operating system:
• nodejs
  • version: 16
• typescript
  • version: 4.8.3
• Is the bug related to documentation in
  • README.md
  • source code documentation
  • SDK API docs on https://docs.microsoft.com

Describe the bug
We are using @azure/identity & @azure/storage-queue packages within a typescript azure functions application to do custom querying on a queue that we cannot do via a regular binding. This functions app is deployed to an app service as a docker container and configured to use a managed identity which has the correct permissions on the storage account. Our typescript looks like this:

const queueServiceClient = new QueueServiceClient(process.env.AzureStorage__queueServiceUri ?? "", new DefaultAzureCredential());

const httpTrigger: AzureFunction = async function(context: Context, req: HttpRequest): Promise<void> {
    const queueClient = queueServiceClient.getQueueClient(req.params.name);

    if (await queueClient.exists() === false) {
        context.res = {
            status: 404
        };
        return;
    }

    const properties = await queueClient.getProperties();
    context.res = {
        // status: 200, /* Defaults to 200 */
        body: properties.approximateMessagesCount
    };

};

This code is running fine however after approximately 24 hours this code starts failing with the following exception

Exception while executing function: Functions.QueueLength Result: Failure
Exception: Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:fe44eaeb-d003-0025-8027-c6da7a000000
Time:2022-09-11T21:39:46.1437207Z
Stack: RestError: Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:fe44eaeb-d003-0025-8027-c6da7a000000
Time:2022-09-11T21:39:46.1437207Z
    at handleErrorResponse (/home/site/wwwroot/node_modules/@azure/core-http/src/policies/deserializationPolicy.ts:274:17)
    at /home/site/wwwroot/node_modules/@azure/core-http/src/policies/deserializationPolicy.ts:179:47
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async StorageRetryPolicy.attemptSendRequest (/home/site/wwwroot/node_modules/@azure/storage-queue/src/policies/StorageRetryPolicy.ts:168:18)
    at async StorageClientContext.sendOperationRequest (/home/site/wwwroot/node_modules/@azure/core-http/src/serviceClient.ts:520:23)
    at async QueueClient.getProperties (/home/site/wwwroot/node_modules/@azure/storage-queue/src/QueueClient.ts:821:14)
    at async QueueClient.exists (/home/site/wwwroot/node_modules/@azure/storage-queue/src/QueueClient.ts:780:7) 

I'm not sure if this is an issue with app services or an issue with one of the SDKs or an issue with the way we're creating the QueueServiceClient statically outside of the function. I could not find any information in the docs which said what sort of lifetime you should expect from the QueueServiceClient class.

[EmmaZhu]

@ksstott ,

The code seems fine. The issue could be caused by several different reasons. I'll try to run a sample for 24hours to see whether I can repro it. If the issue is reproduced in your side, could you share the exception info (as you have pasted above) for the most recent repro? I'll try to take a look in service side to find out why it returns 403 error.

Thanks
Emma

[ksstott]

@ksstott ,

The code seems fine. The issue could be caused by several different reasons. I'll try to run a sample for 24hours to see whether I can repro it. If the issue is reproduced in your side, could you share the exception info (as you have pasted above) for the most recent repro? I'll try to take a look in service side to find out why it returns 403 error.

Thanks Emma

Hi @EmmaZhu

Thanks for getting back so quickly. The exception pasted above was from yesterday afternoon within the hour before I posted this issue so it is the latest repro (it'll happen again later when it gets to 24 hours again). Is there any particular extra exception info that you need in addition to what I've pasted above?

Thanks
Kelvin

[EmmaZhu]

@ksstott ,

The exception was thrown at 2022-09-11T21:39:46.1437207Z, logs the request has been rolled out. Usually, service would keep logs for request for about 3 days.

Could you share a more recent repro?

Thanks
Emma

[ksstott]

@EmmaZhu,

Ah, that's App insights bunching together all the similar errors together and me not noticing that I'd selected one that was a few days old. Apologies, here is one that is from yesterday when it last occurred.

Exception while executing function: Functions.QueueLength Result: Failure
Exception: Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:f9710919-9003-0065-27f3-cc4a26000000
Time:2022-09-20T13:18:23.2661168Z
Stack: RestError: Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:f9710919-9003-0065-27f3-cc4a26000000
Time:2022-09-20T13:18:23.2661168Z
    at handleErrorResponse (/home/site/wwwroot/node_modules/@azure/core-http/src/policies/deserializationPolicy.ts:274:17)
    at /home/site/wwwroot/node_modules/@azure/core-http/src/policies/deserializationPolicy.ts:179:47
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async StorageRetryPolicy.attemptSendRequest (/home/site/wwwroot/node_modules/@azure/storage-queue/src/policies/StorageRetryPolicy.ts:168:18)
    at async StorageClientContext.sendOperationRequest (/home/site/wwwroot/node_modules/@azure/core-http/src/serviceClient.ts:520:23)
    at async QueueClient.getProperties (/home/site/wwwroot/node_modules/@azure/storage-queue/src/QueueClient.ts:821:14)
    at async QueueClient.exists (/home/site/wwwroot/node_modules/@azure/storage-queue/src/QueueClient.ts:780:7) 

[EmmaZhu]

The failure is because the token has been expired.

@jeremymeng , Could you help to take a look at this issue?

Seems BearerTokenAuthenticationPolicy failed to automatically refresh expired token. From the code, I think it should be able to auto refresh token?
if (cycler.mustRefresh) return refresh(tokenOptions);

[jeremymeng]

@ksstott for @azure/identity are you using the exact version 2.0.4 or with ~2.0.4 or ^2.0.4? There was an issue affecting app service managed identity credential in v2.0.5 v2.1.0. The fix is now available in the latest v3.0.0 version. Could you please give it a try?

[ksstott]

@ksstott for @azure/identity are you using the exact version 2.0.4 or with ~2.0.4 or ^2.0.4? There was an issue affecting app service managed identity credential in v2.0.5 v2.1.0. The fix is now available in the latest v3.0.0 version. Could you please give it a try?

Yes digging out the lock file from the docker image we have deployed we are indeed using v2.1.0. We're going to trial version 3 over the weekend and see if the issue disappears. Will come back to you next week with our findings
Thanks

[ksstott]

Hi

After switching to version 3 and letting it run for a while we've had no issues. Thank you for your responses