STS Client - Not authorized to perform sts:AssumeRoleWithWebIdentity #6225
Comments
meenar-se
added
bug
|
Hi @meenar-se , Thanks for reaching out. This is interesting. The error you are seeing is a permissions error and is usually unrelated to the SDK version. From the description of the problem it sounds like it started happening after upgrading to a more recent version. Does rolling back to version fastify.get('/ping', async function handler (request, reply) {
fastify.log.info("ping method started")
const client = new SSMClient({})
client.middlewareStack.add(next => async (args) => {
console.log(args.request)
const response = await next(args);
console.log(response);
return response;
}, {step: 'finalizeRequest'})
// rest of the code That would allow us to examine exactly what changed in the request to get a better idea of where the discrepancy in behavior might be coming from. Thanks again, |
RanVaknin
added
response-requested
|
Yes @RanVaknin rolling back to 3.577.0 resolves the issue. Its working perfectly fine. With the latest version 3.596.0 i have added the middleware hook and tried. Its not even printing the request and response logs. I have tried with below options
But with the version 3.577.0 middleware logs of request and response are printed |
github-actions
bot
removed
the
response-requested
|
Hi @meenar-se , I'm not sure why you are seeing this. Perhaps this is an issue with Fastify? I'm able to deploy an EKS pod using the latest version of the SDK and call SSM just fine: $ kubectl get pods
NAME READY STATUS RESTARTS AGE
repro 1/1 Running 0 45m
$ kubectl exec --stdin --tty repro -- /bin/bash
bash-5.2# cd repro/
bash-5.2# npm ls
[email protected] /repro
`-- @aws-sdk/[email protected]
bash-5.2# cat sample.mjs
import { SSMClient, GetParameterCommand } from "@aws-sdk/client-ssm";
const client = new SSMClient({})
try {
const command = new GetParameterCommand({Name: 'some-name', WithDecryption:true});
const response = await client.send(command);
console.log(response)
} catch (error) {
console.log(error)
}
bash-5.2# node sample.mjs
{
'$metadata': {
httpStatusCode: 200,
requestId: 'REDACTED',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
Parameter: {
ARN: 'arn:aws:ssm:us-east-1:REDACTED:parameter/some-name',
DataType: 'text',
LastModifiedDate: 2024-07-03T20:13:37.758Z,
Name: 'some-name',
Type: 'String',
Value: 'some-value',
Version: 1
}
}The error you are seeing is a permissions issue, where the IAM role that gets assumed using the IRSA token does not have the right permissions to call
I'm not sure why this would not get logged as this middleware just uses console.log which prints to std.out. Also if you are getting a response, you should at least see the request part of the middleware as it fires before the response returns. How are you deploying your code changes? are you using an Image from ECR? I would make sure you are deploying the correct image with the correct tag to your pod. You should also verify with doing a Thanks, |
RanVaknin
added
response-requested
|
This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing. |
github-actions
bot
added
the
closing-soon
|
@RanVaknin - Even after removing the fastify and tried with the simple example as shown your comment gives us the same error. |
github-actions
bot
removed
closing-soon
|
Hi @meenar-se , Can you please configure the logger to capture the raw request and response? You can do it like so: please add this to your client creation code, and examine and share the output from these logs for both deployments (old version, and newer version which doesnt work) This logger should capture the implicit Thanks, |
RanVaknin
added
the
response-requested
|
@RanVaknin - Here is the logs.. calling with args @aws-sdk/credential-provider-node - defaultProvider::fromSSO
@smithy/property-provider -> Skipping SSO provider in default chain (inputs do not include SSO fields).
@aws-sdk/credential-provider-node - defaultProvider::fromIni
@aws-sdk/credential-provider-ini - fromIni
default isAssumeRoleWithSourceProfile source_profile=web_token
@aws-sdk/credential-provider-ini - resolveAssumeRoleCredentials (STS)
@aws-sdk/credential-provider-ini - finding credential resolver using source_profile=[web_token]
@aws-sdk/credential-provider-web-identity - fromTokenFile
@aws-sdk/credential-provider-web-identity - fromWebToken
@aws-sdk/client-sts::resolveRegion accepting first of: undefined (provider) us-east-1 (parent client) us-east-1 (STS default)
endpoints Initial EndpointParams: {
"UseGlobalEndpoint": false,
"UseFIPS": false,
"Region": "us-east-1",
"UseDualStack": false
}
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: aws.partition($Region) = {
"dnsSuffix": "amazonaws.com",
"dualStackDnsSuffix": "api.aws",
"implicitGlobalRegion": "us-east-1",
"name": "aws",
"supportsDualStack": true,
"supportsFIPS": true,
"description": "US East (N. Virginia)"
}Note: We are using IRSA to grant kubernetes workloads access to AWS. |
|
I'm having the same issue, even downgrading the sdk version it didn't work AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n at throwDefaultError (/node_modules/.pnpm/@[email protected]/node_modules/@smithy/smithy-client/dist-cjs/index.js:839:20)\n at /node_modules/.pnpm/@[email protected]/node_modules/@smithy/smithy-client/dist-cjs/index.js:848:5\n at de_CommandError (/node_modules/.pnpm/@[email protected]/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /node_modules/.pnpm/@[email protected]/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n at async /node_modules/.pnpm/@[email protected]/node_modules/@smithy/core/dist-cjs/index.js:165:18\n at async /node_modules/.pnpm/@[email protected]/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38\n at async /node_modules/." |
|
Hi @meenar-se , Thanks. I dont see the underlying call made by the provider. We might be able to see why it is failing by overriding the provider. I'm not sure if the one that is being triggered is import { fromTokenFile } from "@aws-sdk/credential-providers";
const client = new SSMClient({
region: "us-east-1",
credentials: fromTokenFile({
clientConfig: {
logger: console
}
})
});or import { fromWebToken } from "@aws-sdk/credential-providers";
const client = new SSMClient({
region: "us-east-1",
credentials: fromWebToken({
clientConfig: {
logger: console
}
})
});Hi @RabahZeineddine this is unlikely tied to an SDK version. If the role you are assuming does not have the correct permissions, you will see an AccessDenied. You can enable logs the same way I show above, and examine the API call that the SDK is making under the hood to call Thanks, |
RanVaknin
added
response-requested
|
@RanVaknin I'm not using the SSMClient... we are using the s3 and the athena client the IRSA configuration is set using terraform so no one changed it.. |
|
Hi @RabahZeineddine , The SSMClient in my snippet was just to show that you can override the credential provider on the client you are creating to enable logging for it. In your case it will be import { fromWebToken } from "@aws-sdk/credential-providers";
const client = new S3Client({
region: "us-east-1",
credentials: fromWebToken({
clientConfig: {
logger: console
}
})
});
Which dependencies did you upgrade? The SDK version? If the SDK is the blame here, and it indeed started breaking after an upgrade, rolling back to an older version should fix the problem. In order to move forward we will need to identify either the offending version that caused the change in behavior. Or seeing the request made to Thanks, |
RanVaknin
added
response-requested
|
@RanVaknin check the logs |
|
@RanVaknin after checking and debugging I found that an internal updated changed the namespace of the IRSA.. that broke this application .. just solved and updated to the latest aws-sdk version and it is working! Thank you for your help |
|
Hey @RabahZeineddine , @meenar-se let me know if you are able to do the same sort of debugging. Thanks, |
RanVaknin
added
response-requested
|
@RanVaknin - fromWebToken is the one that being triggered. fromToken is throwing an error. Below are the detailed logs fromTokenFile:: {"message":"{"name":"CredentialsProviderError","tryNextLink":true}"}I fromWebToken: I have tried using the method fromWebToken and below is the detailed logs: endpoints Initial EndpointParams: {
"UseGlobalEndpoint": false,
"UseFIPS": false,
"Region": "us-east-1",
"UseDualStack": false
}
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: aws.partition($Region) = {
"dnsSuffix": "amazonaws.com",
"dualStackDnsSuffix": "api.aws",
"implicitGlobalRegion": "us-east-1",
"name": "aws",
"supportsDualStack": true,
"supportsFIPS": true,
"description": "US East (N. Virginia)"
}
endpoints assign: PartitionResult := {
"dnsSuffix": "amazonaws.com",
"dualStackDnsSuffix": "api.aws",
"implicitGlobalRegion": "us-east-1",
"name": "aws",
"supportsDualStack": true,
"supportsFIPS": true,
"description": "US East (N. Virginia)"
}
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints Resolving endpoint from template: {
"url": "https://sts.{Region}.{PartitionResult#dnsSuffix}",
"properties": {},
"headers": {}
}
endpoints Resolved endpoint: {
"headers": {},
"properties": {},
"url": "https://sts.us-east-1.amazonaws.com/"
}
{
clientName: 'STSClient',
commandName: 'AssumeRoleWithWebIdentityCommand',
input: {
RoleArn: 'arn:aws:iam::<account_id>:role/<role_name>',
RoleSessionName: 'session',
WebIdentityToken: '***SensitiveInformation***',
ProviderId: undefined,
PolicyArns: undefined,
Policy: undefined,
DurationSeconds: undefined
},
error: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
at throwDefaultError (/fexample/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
at /fexample/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
at de_CommandError (/fexample/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:14)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 403,
requestId: '8659fc10-ff63-4056-b411-90f01a01a098',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
Type: 'Sender',
Code: 'AccessDenied'
},
metadata: {
httpStatusCode: 403,
requestId: '8659fc10-ff63-4056-b411-90f01a01a098',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
}
}Note: Again tried with v3.577.0 its working fine and also tried with the latest version v3.624.0 getting Access Denied error. |
github-actions
bot
removed
the
response-requested
|
Hi @meenar-se , Can you please have these logs enabled in both versions and see what changes in the request logs? If something in the SDK code changed between the versions that influences the way the request is sent, you should be able to see what has changed in the outgoing request part of the logs. You redacted the
Can you tell me how you are upgrading / downgrading versions? Do you have an ECR image you are using to roll the version back and forward? Are you just execting into the pod and using You didn't address the previous suggestion:
Thanks, |
RanVaknin
added
the
response-requested
|
@RanVaknin - We have performed a detailed analysis at our end today and below are the outcome, Also we have tried out with the version v3.583.0. Its working fine. The releases in between v3.587.0 and 3.583.0 does not have any changes related to the required packages. Observations: Logs from version 3.587.0 Logs from 3.583.0 Testing & Deployment:
|
github-actions
bot
removed
the
response-requested
|
Hi @meenar-se , While we investigate this internally, can you weigh in on why are you using an INI file in the first place? In an EKS context the SDK should be able to operate without the need to configure an INI file. The EKS pod should be injected with the Role ARN env variable, and populated with the token file and the EKS provider automatically will use those to make that underlying AssumeRoleWithWebIdentity call. Can you try working around this by removing the INI altogether? Or is there a specific use case for that INI file to exist on your pod's file system? Thanks, |
RanVaknin
added
the
response-requested
|
Hi @RanVaknin Here is our ini file The With the earlier versions of sdk, we were able to chain roles and assume then recursively. In the above example, the But with recent sdk versions, it fails as it assumes |
RanVaknin
added
queued
kuhe
added
the
pending-release
|
A fix for credential assume-role chaining was released in https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.651.1. The option of having a final |
kuhe
added
closing-soon
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread. |
Checkboxes for prior research
Describe the bug
We have deployed a application which is using SSM Client to read the configuration from SSM parameter store and its deployed to EKS.
When we are using the @aws-sdk/client-ssm v3.596.0 we are getting an error Not authorized to perform sts:AssumeRoleWithWebIdentity
Detailed error stack:
But the same code is working fine with
@aws-sdk/[email protected]SDK version number
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
v20.15.0
Reproduction Steps
Deploy this test project https://github.com/meenar-se/aws-sdk-issue
Call the API
curl --location --request GET 'localhost:8080/ping'
Expected behavior:
Get the values from SSM parameter
Actual Behavior:
Getting Access Denied Exception
{"err":{"type":"STSServiceException","message":"Not authorized to perform sts:AssumeRoleWithWebIdentity","stack":"AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n at throwDefaultError (/fastify-sts-example/node_modules/@smithy/smithy-client/dist-cjs/index.js:839:20)\n at /fastify-sts-example/node_modules/@smithy/smithy-client/dist-cjs/index.js:848:5\n at de_CommandError (/fastify-sts-example/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:476:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"AccessDenied","$fault":"client","$metadata":{"httpStatusCode":403,"requestId":"1c9069d9-04ad-4e93-85b2-330b10f232c8","attempts":1,"totalRetryDelay":0},"Type":"Sender","Code":"AccessDenied"},"msg":"Error getting SSM Parameter:"}Observed Behavior
Initially we were using the @aws-sdk/client-ssm v3.577.0 and it has the peer dependency @aws-sdk/client-sts v3.577.0
there were no issues. As soon as we upgraded our project to use the latest version we are facing the issue.
And if we use the v3.577.0 there is no issue.
It seems to be an issue with @aws-sdk/client-sts library which is added as a peer dependency to all of the client libraries.
Expected Behavior
Expecting to connect to the get the credentials and make connection AWS services
Possible Solution
No response
Additional Information/Context
No response
The text was updated successfully, but these errors were encountered: