SAM local invocation with CDK generated template not resolving dependencies in requirements.txt

[OFranke]

Description

Local python dependencies are installed as defined but not found using local invoke. I am using CDK and SAM together, therefore my Cloud Formation file in template.yaml contains something like:

mywidgetserviceWidgetHandlerFAB6825C:
    Type: AWS::Lambda::Function
       Properties:

It seems to work with the samples that are created using sam init. Those are using Type: AWS::Serverless::Function in their template.yaml file

my handler.py:

import pandas as pd

def main(event, context):
    # save event to logs
    print(event)
    pd.DataFrame({"hans": ["wurst"]})
    return {"statusCode": 200, "body": "test"}

my requirements.txt

pandas

Steps to reproduce

I created a sample repository to reproduce the issue: https://github.com/OFranke/sam-cdk-bug. To reproduce run:

• run yarn
• run yarn build
• run cdk bootstrap
• run sam build -> notice correct python pandas dependency in .aws-sam/build/pickletovizservicePickleToVizHandler68C6D6E2/pandas
• run sam local invoke -> notice error Unable to import module 'handler': No module named 'pandas'

Observed result

Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local invoke command is called
Collected default values for parameters: {}
13 resources found in the template
Found Lambda function with name='pickletovizservicePickleToVizHandler68C6D6E2' and CodeUri='/Users/ofranke/Projects/Customers/sam-cdk-example/src/handlers/pickleToViz'
Found one Lambda function with name 'pickletovizservicePickleToVizHandler68C6D6E2'
Invoking handler.main (python3.6)
No environment variables found for function 'pickletovizservicePickleToVizHandler68C6D6E2'
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
Resolving code path. Cwd=/Users/ofranke/Projects/Customers/sam-cdk-example/.aws-sam/build, CodeUri=/Users/ofranke/Projects/Customers/sam-cdk-example/src/handlers/pickleToViz
Resolved absolute path to code is /Users/ofranke/Projects/Customers/sam-cdk-example/src/handlers/pickleToViz
Code /Users/ofranke/Projects/Customers/sam-cdk-example/src/handlers/pickleToViz is not a zip/jar file
Skipping building an image since no layers were defined

Fetching lambci/lambda:python3.6 Docker container image......
Mounting /Users/ofranke/Projects/Customers/sam-cdk-example/src/handlers/pickleToViz as /var/task:ro,delegated inside runtime container
Starting a timer for 60 seconds for function 'pickletovizservicePickleToVizHandler68C6D6E2'
START RequestId: 6fe1f267-fa1a-11fa-76f0-0bd2c73478c8 Version: $LATEST
Unable to import module 'handler': No module named 'pandas'
END RequestId: 6fe1f267-fa1a-11fa-76f0-0bd2c73478c8
REPORT RequestId: 6fe1f267-fa1a-11fa-76f0-0bd2c73478c8  Init Duration: 214.34 ms        Duration: 7.33 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 26 MB
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 2593, 'exitReason': 'success', 'exitCode': 0, 'requestId': '2b3b1f45-3754-42ad-a68f-30be5950557f', 'installationId': '0a0f7d35-30c2-4ea0-bd16-b6fd5634ca99', 'sessionId': '2052dce3-5e4e-4ec5-a354-d7a593b25247', 'executionEnvironment': 'CLI', 'pyversion': '3.7.7', 'samcliVersion': '0.37.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

{"errorMessage":"Unable to import module 'handler'"}

Expected result

successful invocation with response

{
  "statusCode": 200, 
  "body": "test"
}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOSX
2. sam --version: tested with 0.47 and 0.37

[awood45]

Given the debug logs, what are the contents of the /Users/ofranke/Projects/Customers/sam-cdk-example/src/handlers/pickleToViz folder? Does this contain code matching the handler that is expected?

Running sam local invoke should work with an Aws::Lambda::Function resource, but local invokes look for local code source, and works a lot more smoothly with a Aws::Serverless::Function resource.

[OFranke]

I guess you mean the build?
Here's the handler code in the build directory .aws-sam/build/pickletovizservicePickleToVizHandler68C6D6E2/handler.py which exactly matches my original source at src/handlers/pickleToViz

import pandas as pd

def main(event, context):
    # save event to logs
    print(event)
    pd.DataFrame({"hans": ["wurst"]})
    return {"statusCode": 200, "body": "test"}

If youre saying that sam local invoke is looking for a local code source does that mean it would always invoke src/handlers/pickleToViz/handler.py instead of .aws-sam/build/pickletovizservicePickleToVizHandler68C6D6E2/handler.py ?

Is there a way to force sam to use its own build directory for local invocations? Haven't found anything in the CLI reference about that.

[awood45]

If the CloudFormation identifier for the function is pickletovizservicePickleToVizHandler68C6D6E2, it should check the build directory first - it's possible CDK is doing something weird here, we don't officially integrate with that tool. It is possible though that bridging sam local invoke to work with whatever CDK is doing isn't too difficult. What does your template.yaml look like? Feel free to strip out anything sensitive, I'm mainly curious about the function name in question and its properties.

The CodeUri value would be especially useful to know, here.

EDIT: I incorrectly stated we don't build Aws::Lambda::Function before editing this, which is not correct. It's just not something I see often, so it's possible we're hitting an edge case somewhere.

[c2tarun]

I looked into this more. What is happening is when you do sam build, it builds lambda and its dependencies in .aws-sam/build/pickletovizservicePickleToVizHandler68C6D6E2 folder. But when you run sam local invoke pickletovizservicePickleToVizHandler68C6D6E2 it reads template.yaml file in build folder which contains cdk-meta shows as below:

    Metadata:
      aws:cdk:path: HelloCdkStack/pickle-to-viz-service/PickleToVizHandler/Resource
      aws:asset:path: /Users/sam-cdk-bug/src/handlers/pickleToViz
      aws:asset:property: Code

Now due to this code we read asset:path and try to mount that src path in docker, which do not have installed dependencies.

I don't have enough context on why this was done this way, I'll sync with Jacob tomorrow to understand and see if this is a bug or something else. Meanwhile as a work around in your .aws-sam/build/template.yml file delete Metadata from Lambda::Function, after this sam local invoke will work.

Here is my output of local invoke:

START RequestId: 908a6325-494b-1900-1977-322be3b1e6c4 Version: $LATEST
{}
END RequestId: 908a6325-494b-1900-1977-322be3b1e6c4
REPORT RequestId: 908a6325-494b-1900-1977-322be3b1e6c4	Init Duration: 2674.53 ms	Duration: 10.44 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 68 MB

{"statusCode":200,"body":"hi du arschgeige"}

[OFranke]

@awood45 @c2tarun thank you so much for investigating! Now I understand whats happening under
the hood and would be able to work around that. Not very convenient but at least something. Thank you!

[jfuss]

@c2tarun Here is the context.

When we were looking at supporting templates written in CDK, we only added support for sam local suite of commands. This was an explicit choice to do, so effectively by design. We did discuss build with the CDK team at the time, and agreed that for CDK apps with Lambda should be built within the CDK. The CDK team looked into using aws-lambda-builders to provide this functionality within CDK directly but not sure where that ended up going.

One reason behind this was that we didn't want customers generating with CDK, then building and testing in SAM, and then going back to deploy in CDK. We wanted customers to generate and deploy with the same tool. If you do not follow this model, this would/could lead to "it working locally" but not when deployed. So the level of support we have for CDK is, If you are using CDK, use CDK for everything but local testing.

@OFranke You might still be able to accomplish what you want. You could sam build and then copy .aws-sam/build/<LogicalId> back into your Asset folder of the function. This would then allow you to deploy your functions through CDK again with a built function. I would also recommend reaching out to CDK about them supporting building Lambda Functions directly in CDK. Which I think is the solution to your issue here.

[OFranke]

@jfuss
Thank you for the brief overview!

We wanted customers to generate and deploy with the same tool.

I totally understand that, but I am also kind of surprised that it sounds like there has never been the plan to integrate both tools in the long term. My uneducated guess was that what I was trying to achieve is absolutely the way to go for building a seamless dev experience from dev to prod.

I checked what cdk is doing. It is indeed doing a build when bundling the asset, contained in cdk.out directory. So that might be an entry point.

But anyway, thank you for giving me so much insight. If that's not the intended way of sticking things together I guess I'll just fall back to docker for local dev.

I'll close that one here.

[jfuss]

@OFranke

I totally understand that, but I am also kind of surprised that it sounds like there has never been the plan to integrate both tools in the long term.

I wouldn't confuse long term with what our design choices were and why. We defiantly had the long term discussions with CDK and how we support each other.

From CDK's point of view (at least from when we discussed this in the past), they wanted to introduce building lambda from within the CDK itself. The idea being that when you cdk deploy, it can compile and deploy the template/artifacts, as well as compile the code for the functions in your CDK app. In this world, there isn't a need to write in CDK, and build within SAM or have an outside build process at all. Not having build within CDK was brought up (by me) and they had intentions to add it (here was the initial PR: aws/aws-cdk#1884). With that lens, the current design and implementation would just worked. Since CDK doesn't build, you are trying a process that we didn't design for because the original discussion was for it to be done within CDK.

I would very much encourage you to reach out to CDK and ask for building Lambda in CDK directly. I still think that is a very valid thing for them to do.

[walkafwalka]

Was this ever resolved? The PRs I looked through were closed without any closing statements. The AWS docs show integration with SAM, but not building the dependencies correctly is a non-starter.