Error: Cannot find module './aws-sdk' on v3.474.0 of @aws-sdk/credential-provider-node

[affanshahab]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Our lambda function was using @aws-sdk/credential-provider-node and in package.json we had:
"@aws-sdk/credential-provider-node": "^3.95.0" with node 14.x runtime.
On Friday 15th Dec 2023 for every new build, our lambda was crashing with following error and stack trace

"Runtime.ImportModuleError: Error: Cannot find module './aws-sdk'",
        "Require stack:",
        "- /var/task/node_modules/@aws-sdk/core/dist-cjs/httpAuthSchemes/index.js",
        "- /var/task/node_modules/@aws-sdk/core/dist-cjs/index.js",
        "- /var/task/node_modules/@aws-sdk/client-sso/dist-cjs/runtimeConfig.js",
        "- /var/task/node_modules/@aws-sdk/client-sso/dist-cjs/SSOClient.js",
        "- /var/task/node_modules/@aws-sdk/client-sso/dist-cjs/index.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-sso/dist-cjs/resolveSSOCredentials.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-sso/dist-cjs/fromSSO.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-sso/dist-cjs/index.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-ini/dist-cjs/resolveSsoCredentials.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-ini/dist-cjs/resolveProfileData.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-ini/dist-cjs/fromIni.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-ini/dist-cjs/index.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js",
        "- /var/task/node_modules/@aws-sdk/credential-provider-node/dist-cjs/index.js",
        ........
        ........

There were no change in our code so we downloaded the lambda zip from past version and compared the two folders to find out in broken lambda version there were some new files added.
See comparison below and notice node_moduels > @aws_sdk > core > dist-cjs ....: stack trace showing this path

upon investigation it was discovered new version of @aws-sdk/credential-provider-node has introduced this file.
We updated our package.json from "^3.95.0" to "3.470.0" and it solved our problem.

SDK version number

3.474.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

node 14.x

Reproduction Steps

use the v3.474.0 of @aws-sdk/credential-provider-node in AWS lambda using node 14.x runtime.

Observed Behavior

Error: Cannot find module './aws-sdk' on v3.474.0 @aws-sdk/credential-provider-node of when using this library in our lambda function running node 14.x.

Expected Behavior

Expected the library to be backward compatible and to be working without crashing

Possible Solution

downgraded to v3.470.0 of @aws-sdk/credential-provider-node

Additional Information/Context

No response

[syall]

Hi @affanshahab, thanks for reporting this issue!

The main issue seems to be that the zipped lambda does not contain the complete artifact of the @aws-sdk/core dependency. The zipped dependency does not match the published NPM artifacts, which includes the aws-sdk sub-directory in @aws-sdk/[email protected].

I've taken a look at the published artifacts for @aws-sdk/credential-provider-node and the relevant @aws-sdk/core dependency, and the published artifacts all include the expected files (see section below).

I suspect there may be an issue in the specific packaging, installation, caching, or deployment process of your lambda that caused the lambda to not be packaged properly, but I am not able to debug your specific setup and unable to reproduce the issue.

Let us know if you have more questions.

---

First, investigating @aws-sdk/core published artifacts per the stacktrace.

In a previous version @aws-sdk/[email protected], the httpAuthSchemes sub-directory (and likewise aws-sdk sub-directory) are not present since the PR with the changes (#5586) had not been merged yet.

In the next version @aws-sdk/[email protected], used by the version of @aws-sdk/credential-provider-node mentioned in this issue, the PR (#5586) with httpAuthSchemes and aws-sdk sub-directories is merged and included in the NPM release.

We can confirm this by creating a new project that installs the reported @aws-sdk/[email protected] version:

// package.json
{
  "dependencies": {
    "@aws-sdk/credential-provider-node": "3.474.0"
  }
}

For this repro, I am using yarn, so after running yarn, we can inspect the node_modules/ directory. I've listed the published artifacts for the CommonJS distribution as used in your lambda application (assumed from the stacktrace, and also omitted unrelated files):

node_modules/@aws-sdk/core/dist-cjs
├── client
├── httpAuthSchemes
│   ├── aws-sdk
│   ├── index.js
│   └── utils
├── index.js
└── protocols

As seen here in the installation, the aws-sdk sub-directory exists in the CommonJS distribution, as well as the other distributions too.

Taking a look at the screenshot diff, the inclusion of the httpAuthSchemes sub-directory without the aws-sdk sub-directory seems like there is an installation or packaging issue that caused the aws-sdk sub-directory to not be included, as the published NPM artifacts are complete.

httpAuthSchemes and aws-sdk were introduced in the same PR (#5586) and published in the same NPM release: if one directory exists, the other should exist too. In the published NPM artifacts this is confirmed. In the issue description and screenshot diff, this is not the case and is likely a specific setup issue.

[jsking216]

Hi @affanshahab, thanks for reporting this issue!

The main issue seems to be that the zipped lambda does not contain the complete artifact of the @aws-sdk/core dependency. The zipped dependency does not match the published NPM artifacts, which includes the aws-sdk sub-directory in @aws-sdk/[email protected].

I've taken a look at the published artifacts for @aws-sdk/credential-provider-node and the relevant @aws-sdk/core dependency, and the published artifacts all include the expected files (see section below).

I suspect there may be an issue in the specific packaging, installation, caching, or deployment process of your lambda that caused the lambda to not be packaged properly, but I am not able to debug your specific setup and unable to reproduce the issue.

Let us know if you have more questions.

First, investigating @aws-sdk/core published artifacts per the stacktrace.

In a previous version @aws-sdk/[email protected], the httpAuthSchemes sub-directory (and likewise aws-sdk sub-directory) are not present since the PR with the changes (#5586) had not been merged yet.

In the next version @aws-sdk/[email protected], used by the version of @aws-sdk/credential-provider-node mentioned in this issue, the PR (#5586) with httpAuthSchemes and aws-sdk sub-directories is merged and included in the NPM release.

We can confirm this by creating a new project that installs the reported @aws-sdk/[email protected] version:

// package.json
{
  "dependencies": {
    "@aws-sdk/credential-provider-node": "3.474.0"
  }
}

For this repro, I am using yarn, so after running yarn, we can inspect the node_modules/ directory. I've listed the published artifacts for the CommonJS distribution as used in your lambda application (assumed from the stacktrace, and also omitted unrelated files):

node_modules/@aws-sdk/core/dist-cjs
├── client
├── httpAuthSchemes
│   ├── aws-sdk
│   ├── index.js
│   └── utils
├── index.js
└── protocols

As seen here in the installation, the aws-sdk sub-directory exists in the CommonJS distribution, as well as the other distributions too.

Taking a look at the screenshot diff, the inclusion of the httpAuthSchemes sub-directory without the aws-sdk sub-directory seems like there is an installation or packaging issue that caused the aws-sdk sub-directory to not be included, as the published artifacts are complete.

httpAuthSchemes and aws-sdk were introduced in the same PR (#5586) and published in the same NPM release: if one directory exists, the other should exist too. In the published NPM artifacts this is confirmed. In the issue description and screenshot diff, this is not the case and is likely a specific setup issue.

My team also ran into this issues with an internal wrapper around @aws-sdk/client-sqs after letting it bump from 3.468.0 to 3.474.0 and we now are having to pin specific versions. We intentionally drop the aws-sdk deps from our lambdas because they are supposed to be built-in, is it possible that the lamda runtime doesn't yet have this dependency for free?

Edit: Also seeing this with @aws-sdk/client-sfn as well, probably with anything we have touched that is on 3.474.0 so far.

We see the issue both with lambdas that have a production dependency on v3 clients in node 14 and lambdas that use node 20 runtime but don't bundle it.

[affanshahab]

Thanks @syall for looking into it.
Like @jsking216 mentioned, we intentionally drop the ‘aws-sdk’ before packaging our lambdas as it is supposed to be part of runtime by default. Reason to remove it from ‘node_modules’ is to keep the package size minimal.

Just using the v3.470.0 resolved this error for us for now but unfortunately costed me my weekend :(

[syall]

Thanks @syall for looking into it. Like @jsking216 mentioned, we intentionally drop the ‘aws-sdk’ before packaging our lambdas as it is supposed to be part of runtime by default. Reason to remove it from ‘node_modules’ is to keep the package size minimal.

Just using the v3.470.0 resolved this error for us for now but unfortunately costed me my weekend :(

@affanshahab Given this info, it sounds like the removal of the aws-sdk directory is an unintended side effect of the tool that removes dependencies due to the naming. The aws-sdk directory in the source code and dist-* artifacts is different than the top-level @aws-sdk package scope. To correctly remove AWS SDK dependencies, only packages in the @aws-sdk package scope should be removed.

[jsking216]

Thanks @syall for looking into it. Like @jsking216 mentioned, we intentionally drop the ‘aws-sdk’ before packaging our lambdas as it is supposed to be part of runtime by default. Reason to remove it from ‘node_modules’ is to keep the package size minimal.
Just using the v3.470.0 resolved this error for us for now but unfortunately costed me my weekend :(

@affanshahab Given this info, it sounds like the removal of the aws-sdk directory is an unintended side effect of the tool that removes dependencies due to the naming. The aws-sdk directory in the source code and dist-* artifacts is different than the top-level @aws-sdk package scope. To correctly remove AWS SDK dependencies, only packages in the @aws-sdk package scope should be removed.

I just tried this with one of our lambdas and it seems to be working when changing the bundler setting to ignore @aws-sdk paths. this was an old rule from aws sdk v2 that was just left there thinking it was innocuous, but apparently it is not. thanks for the update.

[syall]

My team also ran into this issues with an internal wrapper around @aws-sdk/client-sqs after letting it bump from 3.468.0 to 3.474.0 and we now are having to pin specific versions. We intentionally drop the aws-sdk deps from our lambdas because they are supposed to be built-in, is it possible that the lamda runtime doesn't yet have this dependency for free?

Edit: Also seeing this with @aws-sdk/client-sfn as well, probably with anything we have touched that is on 3.474.0 so far.

We see the issue both with lambdas that have a production dependency on v3 clients in node 14 and lambdas that use node 20 runtime but don't bundle it.

In general, "[t]he bundled SDK is provided as a convenience for developers building simpler functions or using the Lambda console for development". For more complex projects, either 1) only use dependencies included in the Lambda runtime, or 2) fully provide all @aws-sdk and @smithy node module dependencies.

SDK versions and packages availability in Lambda runtimes are managed by the Lambda team, and are updated periodically with no regular cadence. As a reference, here is a link to all SDK versions currently included in all Lambda runtimes: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.