How to include node_modules when using typescript aws-cdk?

[zoonman]

❓ How to include node_modules using typescript aws-cdk?

My lambda is supposed to rely on the whole bunch of different modules, part of which is private.
How should I tackle this problem with conformity with aws-cdk?

Environment

• CDK CLI Version: 1.6.0 (build 3a0cde0)
• OS: all
• Language: TypeScript

Other information

Could you provide example when used some external node module, axios or joi or something else?

[JoshM1994]

I have prepared a short example of how to include a set of node_modules below. The basic answer is that you need your node_modules folder to be in the same folder as specified in new lambda.AssetCode('this_folder')

1. Assume you are starting a fresh project with cdk init sample-app --language=typescript
2. Create a new top-level folder called src where you will be storing your Lambda functions
3. Run npm init inside src to generate a package.json
4. npm install your_module inside src --> you should now have a node_modules folder inside src
5. Perform all the usual steps to create your lambda and specify the code key as new lambda.AssetCode('src')
6. Run cdk synth from the root of your project
7. Look at cdk.out and you will see a folder starting asset.somethingHere --> this asset folder should contain your lambda handler and the node_modules required by it
8. Deploy and you're good to go

The final tree structure should look something like this:

├── bin
│   └── lambda.ts
├── cdk.json
├── cdk.out
│   ├── LambdaStack.template.json
│   ├── asset.lotsOfLetters
│   │   ├── basic.js
│   │   ├── node_modules # Node modules found here
│   │   ├── package-lock.json
│   │   └── package.json
│   ├── cdk.out
│   ├── manifest.json
│   └── tree.json
├── lib
│   └── lambda-stack.ts
├── package-lock.json
├── package.json
├── src
│   ├── basic.js
│   ├── node_modules # this is inside the src, (in addition to the root one required to build)
│   ├── package-lock.json
│   └── package.json

For reference, my lambda-stack.ts file looks like:

import lambda = require('@aws-cdk/aws-lambda')
import cdk = require('@aws-cdk/core');

export class LambdaStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const myLambda = new lambda.Function(this, 'iHaveNodeModules', {
      code: new lambda.AssetCode('src'),
      handler: 'basic.handler',
      runtime: lambda.Runtime.NODEJS_10_X
    })
  }
}

My basic.js example lambda (yes, the left-pad usage is meant to be ironic)

const leftPad = require('left-pad');

const handler = async () => ({
    statusCode: 200,
    body: leftPad(1, 3, '0')
});

module.exports = {
    handler
};

After deploying, see the screenshot of what the Lambda console looks like with running a test event:

[zoonman]

Josh, it looks like a complete hack & absolutely dirty solution.
As part as regular development process goes - infrastructure is an utilitary part of the project and normally has to be considered as dev dependency. Nobody wants to spend more than 5% of their project time on it and write endless boilerplate code to trivially deploy API.

Project structure supposed to look like this:

infra/cdk/index.ts
src/application-lamdbas-code.ts
src/application-lamdbas-code.spec.ts
node_modules/*
package.json
.env

I expect cdk to be able to create all the stuff inside aws by npm run deploy.
To be honest I would prefer npm run deploy to run tests, assert that code was committed and properly tagged so generated artifacts will be versioned and deployed to a correct matching stage.

I advise you to get out of cavern and look at something like serverless framework.

[JoshM1994]

I advise you to get out of cavern and look at something like serverless framework.

Yes, I agree that it is hacky and dirty but I do not believe CDK was designed to be an entire build system for every single part of AWS' enormous stack. Serverless Framework is certainly better in its ability to package node_modules for you but you lose a lot of flexibility in doing so.

It's really no extra work to have your npm run deploy do a very quick packaging step for you. Something super basic like

zip -r lambdas.zip src/* node_modules/*

and then replace the code in my example above with

const myLambda = new lambda.Function(this, 'iHaveNodeModules', {
  code: lambda.Code.fromAsset('lambdas.zip'),
  handler: 'src.basic.handler',
  runtime: lambda.Runtime.NODEJS_10_X
})

(Obviously make sure your CI only runs npm install --only=production to avoid packaging all your devDependencies as well!)

P.S. We actually switched away from Serverless Framework in favour of CDK because of CDK's much greater range of product offerings (Elastic Beanstalk) and intuitive IAM role creation (something we found to be lacking in Serverless because of its multi-vendor approach).

[NGL321]

@zoonman Sorry that no one got back to you sooner. @JoshM1994 is correct however, the CDK is not designed to be the end-all-be-all build system for everything in AWS. The AWS ecosystem is far too complex to have such specific deployment structure.

Josh's solution mirrors exactly what I would have encouraged to deploy lambdas with dependencies. It is not the most optimal solution, but it is how you would deploy a lambda without the CDK, and therefore what it is mirroring. Each step can be very easily automated within npm run deploy.

All that said, this is a fairly specific structure, and I'm not sure it makes sense to create an example for it in this repo. If you would like to request adding a specific lambda feature, I recommend you create an issue on the main CDK repo rather than the one for code samples

[SomayaB]

Closing this issue since a solution has been provided. Feel free to reopen.

[rickggaribay]

Thanks @JoshM1994 your suggestion worked great and seems like a simple solution to work around the dependency issue. I hadn't considered that you could have multiple node_modules within one "project". Simple workaround that doesn't feel like a hack at all. Thank you.

[GlennHoran]

Thanks for one of the best answers I've seen on github @JoshM1994 , it was exactly what I was looking for.

[netaisllc]

Another vector on the original ask - albeit quite limited - is to include a specific npm package as a Lambda Layer. It is trivial to do so with CDK.

[NetaNir]

Have you all tried aws-lambda-nodejs from the CDK main repo?

[simon-dk]

Have you all tried aws-lambda-nodejs from the CDK main repo?

We are using those and are really happy about it. We are experiencing small lambda deploys resulting in faster executions as well. Still experimental construct though.

[wzr1337]

Have you all tried aws-lambda-nodejs from the CDK main repo?

https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/aws-lambda-nodejs is the correct link

[xusai2014]

I have prepared a short example of how to include a set of node_modules below. The basic answer is that you need your node_modules folder to be in the same folder as specified in new lambda.AssetCode('this_folder')

1. Assume you are starting a fresh project with cdk init sample-app --language=typescript
2. Create a new top-level folder called src where you will be storing your Lambda functions
3. Run npm init inside src to generate a package.json
4. npm install your_module inside src --> you should now have a node_modules folder inside src
5. Perform all the usual steps to create your lambda and specify the code key as new lambda.AssetCode('src')
6. Run cdk synth from the root of your project
7. Look at cdk.out and you will see a folder starting asset.somethingHere --> this asset folder should contain your lambda handler and the node_modules required by it
8. Deploy and you're good to go

The final tree structure should look something like this:

├── bin
│   └── lambda.ts
├── cdk.json
├── cdk.out
│   ├── LambdaStack.template.json
│   ├── asset.lotsOfLetters
│   │   ├── basic.js
│   │   ├── node_modules # Node modules found here
│   │   ├── package-lock.json
│   │   └── package.json
│   ├── cdk.out
│   ├── manifest.json
│   └── tree.json
├── lib
│   └── lambda-stack.ts
├── package-lock.json
├── package.json
├── src
│   ├── basic.js
│   ├── node_modules # this is inside the src, (in addition to the root one required to build)
│   ├── package-lock.json
│   └── package.json

For reference, my lambda-stack.ts file looks like:

import lambda = require('@aws-cdk/aws-lambda')
import cdk = require('@aws-cdk/core');

export class LambdaStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const myLambda = new lambda.Function(this, 'iHaveNodeModules', {
      code: new lambda.AssetCode('src'),
      handler: 'basic.handler',
      runtime: lambda.Runtime.NODEJS_10_X
    })
  }
}

My basic.js example lambda (yes, the left-pad usage is meant to be ironic)

const leftPad = require('left-pad');

const handler = async () => ({
    statusCode: 200,
    body: leftPad(1, 3, '0')
});

module.exports = {
    handler
};

After deploying, see the screenshot of what the Lambda console looks like with running a test event:

Very Cool! I happened to a new issue. how to add two or more functions in Stack. I hope that trigger to deploy multi lambda function when commit the code

Could you give me help
Thank you very much!

@JoshM1994

[AntoniusGolly]

In 2022...

It was quite hard for me to figure out, whether this is still state-of-the-art or we can simplify dependencies now. Turns out it's possible now to include dependencies without hacks or complicated setup.

For me, creating the lambda with new lambda.NodeJsFunction() instead of new lambda.Function() did the trick. Yet, it was super hard for me to find a working sample. I decided to share a sample repo with you.

Sample repository

https://github.com/AntoniusGolly/cdk-lambda-typescript-boilerplate

What it does:

• it has a lot of stuff you don't need to demonstrate the purpose (e.g. it produces a API Gateway based on a config.ts, so ignore that part if you don't need it)
• it allows you to use TS and ES6
• it allows you to use a dependency: i.e. I use node-fetch here
• I just do cdk deploy and no other bundling
• according to the docu by default, all node modules are bundled except for aws-sdk.
• I don't know exactly how it works, but it does ;)

I hope this helps someone as I would have appreciated it.