Langchain with AWS Lambda

[ali-chaudhry8]

Has anyone deployed langchain scripts on AWS - Lambda in particular. There is some issue with the way langchain imports numpy that is causing issues. I have tried it with different version and with a docker image as well but get numpy import issues. Locally it works fine. Thanks, Ali

[ellisonbg]

@3coins from AWS is working on a deployment template for LainChain on Lambda and API Gateway. It includes a DynamoDB-based memory subclass. Should be ready in the next few days. Follow along at #1067

[ellisonbg]

We are working to get it moved over to the GitHub org of @hwchase17 but here is the repo with a working AWS deployment template for Lambda:

https://github.com/3coins/langchain-aws-template

[gitmj]

I deployed https://github.com/3coins/langchain-aws-template in my aws account.

However, API is throwing 'internal server error '. Will debug more today.

[3coins]

@gitmj
Thanks for deploying the template. Can you post the output from your logs here, I can help figure out the root cause if you do that.

[gitmj]

Here is the body used : {"prompt": "explain code: print("Hello world")", "session_id": ""} and attached logs when I tested on API gateway console

logs.txt

[gitmj]

I debugged it further and it was related to openai key. After setting up the aws secrets again, I got the following response so all is good. Thanks for the template, Now I will try to use it for my specific use case.

{"response": "Sure! This code is a simple example of printing a text message to the console. In this case, "Hello world" is the message that will be printed. The "print" function is a built-in function in Python that allows you to print messages to the console.", "session_id": "7eec9625-b034-46a6-a454-d2b87302bbf0"}

[ellisonbg]

Great, let us know if we can improve the documentation.

…

On Wed, Mar 29, 2023 at 11:52 AM gitmj ***@***.***> wrote: I debugged it further and it was related to openai key. After setting up the aws secrets again, I got the following response so all it good. Thanks for the template, Now I will try to use it for my specific use case. {"response": "Sure! This code is a simple example of printing a text message to the console. In this case, "Hello world" is the message that will be printed. The "print" function is a built-in function in Python that allows you to print messages to the console.", "session_id": "7eec9625-b034-46a6-a454-d2b87302bbf0"} — Reply to this email directly, view it on GitHub <#1364 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGXUD5SSW3AFPHAOT5KTDW6SAFPANCNFSM6AAAAAAVMFLOME> . You are receiving this because you commented.Message ID: ***@***.***>

-- Brian E. Granger Senior Principal Technologist, AWS AI/ML ***@***.***) On Leave - Professor of Physics and Data Science, Cal Poly @ellisonbg on GitHub

[AreJay-Smith]

@3coins Thank you for creating this template! I was hoping you could help me figure out an error I'm running into. I followed all the steps to deploy the "service" example, but when I get to the cdk bootstrap command, I get the following error:

(langchain-aws-service) ➜  service cdk bootstrap                            
Traceback (most recent call last):
  File "app.py", line 1, in <module>
    from aws_cdk import (
ModuleNotFoundError: No module named 'aws_cdk'

Shouldn't this module be installed when I spin up the environment using the environment.yaml file?

[3coins]

@AreJay-Smith
That’s correct, aws-cdk-lib is listed as dependency in the environment.yml file, which should add this. Can you open an issue on the template repo so I can investigate this further, also can you include info about your environment, e.g., OS, Conda version etc.

[PeterTF656]

I debugged it further and it was related to openai key. After setting up the aws secrets again, I got the following response so all is good. Thanks for the template, Now I will try to use it for my specific use case.

{"response": "Sure! This code is a simple example of printing a text message to the console. In this case, "Hello world" is the message that will be printed. The "print" function is a built-in function in Python that allows you to print messages to the console.", "session_id": "7eec9625-b034-46a6-a454-d2b87302bbf0"}

@gitmj Hi, could you please elaborate a bit more on how you set up the aws secrets and solved this issue? I'm facing the same problem here, and I tried reset my aws secrets a few times, but it didn't work...

[aarora79]

I had the same problem while using langchain in AWS Lambda, the way I resolved this by downloading the numpy files from here from the PyPi numpy files download page. Once downloaded, unzip the whl into your virtual environment's site-packages directory. Now package the dependencies for the Lambda into a zip file as usual (I was using the Python 3.9 runtime).

[francisurias]

Hi, I'm fairly new to both AWS and LangChain, and I'd like to deploy my code (similar to this found in LangChain's documentation) onto AWS Lambda. Would this be possible using JavaScript? If so, are there are examples/resources that I can refer to?

[3coins]

@francisurias
Yes, that’s possible with JavaScript. It should be very similar to the Python example in terms of CDK, but some things will differ. If you have never worked with CDK, I would recommend you start with this introductory walkthrough for CDK. https://docs.aws.amazon.com/cdk/v2/guide/hello_world.html

[francisurias]

@francisurias Yes, that’s possible with JavaScript. It should be very similar to the Python example in terms of CDK, but some things will differ. If you have never worked with CDK, I would recommend you start with this introductory walkthrough for CDK. https://docs.aws.amazon.com/cdk/v2/guide/hello_world.html

Thanks for the reply, I'll try using the CDK today.

Also, I tried simply putting ConversationalRetrievalQAChain ) into an AWS Lambda function, but I get this error:
"errorMessage": "Please install hnswlib-node as a dependency with, e.g. npm install -S hnswlib-node",
but I have already installed the dependency. How can I fix this issue?

Update: I changed the vectorstore I'm using from HNSWLib to MemoryVectorStore and it seems to be working; still not really sure why HNSWLib wasn't working though.

[samuelcardoso]

This might help:
https://github.com/yoshoku/hnswlib-node/wiki/How-to-run-hnswlib-node-on-AWS-Lambda

[reachlin]

can't install langchain on aws lambda env. python3.9. if i use aws pandas layer, the image is too large, pls help

Function code combined with layers exceeds the maximum allowed size of 262144000 bytes. The actual size is 360569987 bytes.

[aarora79]

@reachlin please see https://github.com/aws-samples/llm-apps-workshop/blob/main/blogs/rag/api/deploy.sh for a recipe to package dependencies including numpy and langchain into AWS Lambda.

[IvanCampos]

Try using Docker to create your Lambda Layer for LangChain:

AWS Lambda Layer for Python 3.10:

• docker pull public.ecr.aws/sam/build-python3.10:1.84.0-20230517004040
• docker run -it -v $(pwd):/var/task public.ecr.aws/sam/build-python3.10:1.84.0-20230517004040
• pip install langchain openai tiktoken -t ./python
• zip -r python.zip ./python

AWS Lambda Layer For Python 3.9:

• docker pull amazon/aws-sam-cli-build-image-python3.9
• docker run -it -v $(pwd):/var/task amazon/aws-sam-cli-build-image-python3.9:latest
• pip install langchain openai tiktoken -t ./python
• zip -r python.zip ./python

[hbadgery]

Try this:
https://medium.com/sopmac-labs/langchain-aws-lambda-serverless-q-a-chatbot-203470b9906f

[Duonghailee]

@reachlin please see https://github.com/aws-samples/llm-apps-workshop/blob/main/blogs/rag/api/deploy.sh for a recipe to package dependencies including numpy and langchain into AWS Lambda.

I have followed the instruction and this is what I got:
No module named 'numexpr.interpreter'",

Have you seen this? tks

[aarora79]

@Duonghailee so you need to create a Python 3.9 conda environment first and then run this script from within that environment. Yes I have seen this error and it will go away once you build from within a Python3.9 environment.

conda create -n py39 python=3.9 -y

# activate the environment
source activate py39
./deploy.sh

[ahmetsevki]

Try this: https://medium.com/sopmac-labs/langchain-aws-lambda-serverless-q-a-chatbot-203470b9906f

paywalled.

[reachlin]

up to this point, i feel it's not wise to put langchain into a lambda. even the docker image works, it will be huge. i've already put it in another micro service anyway.
good luck to those stuck with lambda!

[Deanfei]

@reachlin Hi mate. I am trying to build a new AI application using langchain from scratch. Based on the discussions above, it looks using aws lambda is not ideal.

But I want to confirm that because I have zero experience on python so want to be on the right track. I still would like to put everything on AWS, any suggestions of waht's the best approach from a high level, eg: using ECS microservices, fargate or something? I know the question is vague. I just want to have some high level ideas how to use langchain with python on production.

Another question is how big is langchain, will it impact the cold start time like lambda largely? Tks

[reachlin]

@Deanfei fyi, i put langchain part on pythonanywhere as a flask app. you can use ECS to run it. langchain start is pretty fast, but query any LLM is very slow. if you use lambda, it won't be cheap.

[Deanfei]

That's good to know. Thanks so much. 👍

[KBB99]

If someone is still looking for a Lambda layer including Langchain I just built this one and made it public: arn:aws:lambda:us-east-1:609061237212:layer:langchain:9 . It is running Langchain version 0.0.191 on ARM64 architecture and I have confirmed it's compatibility with Python3.10. To fix conflicts with boto3 I had to pin urllib<2.

[Jacob-Zinn]

I used the comments from @IvanCampos and @KBB99 to create a lambda layer that worked in my 3.10 environment:

docker pull public.ecr.aws/sam/build-python3.10:1.84.0-20230517004040
docker run -it -v $(pwd):/var/task public.ecr.aws/sam/build-python3.10:1.84.0-20230517004040
pip install langchain openai tiktoken urllib3==<compatible_version> -t ./python
zip -r python.zip ./python

Then copied the zip from the docker shell instance to my local machine and uploaded the zip to lambda as a layer.

[Deanfei]

I have migrated to aws ecs as I don't think lambda is that efficient and could be buggy for the long term.

[yoon-bob]

I have a same issue. i have trouble when using LLMChain.run().
My code is under code block

    def get_vector_memory(self) -> VectorStoreRetrieverMemory:
        loader = TextLoader('./test.txt')
        docs = loader.load()
        text_spliter = CharacterTextSplitter(chunk_size=200, chunk_overlap=0)
        texts = text_spliter.split_documents(docs)
        embeddings = OpenAIEmbeddings()
        db = Chroma.from_documents(texts, embeddings)
        retriever = db.as_retriever(search_type='similarity', search_kwargs={ "k" : 1 })
        vector_memory = VectorStoreRetrieverMemory(retriever=retriever, memory_key="vector_history" ,input_key="input")

        return vector_memory

And then my error message is under

{
  “errorType”: “Runtime.ExitError”,
  “errorMessage”: “RequestId: ... Error: Runtime exited with error: signal: illegal instruction”
}

I tried upgrade memory of lambda, but i was not working... I think memory size is appropriate

Memory Size: 10240 MB	Max Memory Used: 411 MB

plz help me....

[MarkiyanPyts]

When I have Langchain installed and try to use UnstructuredURLLoader in asks me to also install "unstructured" package which is making packages size 250MB+ this way this can not run in Lambda because of 250MB hard limit to unzipped package size

[KBB99]

@MarkiyanPyts if you want to use Lambda you should look into Lambda container images

[ellisonbg]

@3coins

[jackyyanggp]

I used the comments from @IvanCampos and @KBB99 to create a lambda layer that worked in my 3.10 environment:

docker pull public.ecr.aws/sam/build-python3.10:1.84.0-20230517004040
docker run -it -v $(pwd):/var/task public.ecr.aws/sam/build-python3.10:1.84.0-20230517004040
pip install langchain openai tiktoken urllib3==<compatible_version> -t ./python
zip -r python.zip ./python

Then copied the zip from the docker shell instance to my local machine and uploaded the zip to lambda as a layer.

I still got the error below, any tips would be appreciated:

"Unable to import module 'lambda_function': \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.10 from "/var/lang/bin/python3.10"\n * The NumPy version is: "1.25.2"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n",

cc @KBB99 @IvanCampos @3coins @ellisonbg

[Ethanmbinns]

Hey I keep getting the following error:

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'pydantic_core._pydantic_core'
Traceback (most recent call last):

Any help would be great, I'm kind of new to this to I'm totally lost on fixing it I've tried debugging it the best I can and I'm not getting anywhere with it.

[KBB99]

@Ethanmbinns try building the layer with the environment image set to amazonlinux2-x86-64-standard:5.0. Also, make sure you are using Python 3.11 runtime for your Lambda.

[KBB99]

To create a lambda layer compatible with the latest version of Langchain and Python you can follow these steps:

1. Navigate to AWS Codebuild.
2. Click create project.
3. Put a name like "langchain-layer-builder."
4. For source select "No source".
5. For artifacts pick an S3 bucket.
6. For environment pick Managed Image, Amazon Linux 2. Runtime Standard. Image aws/codebuild/amazonlinux2-x86-64-standard:5.0.
7. For buildspec, switch to editor and under build: commands: add the following command

• pip install langchain --target package/python && cd package && zip ../package.zip * -r

Under artifacts: file: add the following path

• 'package.zip'

8. Create the build project.
9. Run the build project.
10. Go to the S3 bucket you set as the artifact destination and find the 'package.zip'. Copy the S3 url.
11. Go to Lambda Layers. Select, create new layer from S3 url and paste the s3 url to the package.zip.
12. Attach the layer to a lambda that is running Python3.11.

Note: To add openai or Bedrock add them to the pip install command, i.e. - pip install langchain openai...

[pdichone]

Hi @KBB99 .

Thanks for your guidance. I did exactly what you showed here to create a lambda layer.
I was able to create the artifact, except even though I named it 'package.zip' somehow, when I go to S3, I can't find it. I do, however, find the same artifact named something different, which doesn't matter I think..,

Now the issue is, I've tried to create a layer for both openai and langchain, but every time I try to use it in the lambda function and test, I get:

{ "errorMessage": "Unable to import module 'handler': No module named 'langchain'", "errorType": "Runtime.ImportModuleError", "requestId": "46555-4027-4772-b520-f98sddddxxx", "stackTrace": [] }

So, I always get that error no matter what I do.

I am using serverless on this project.

Can you share a simple project where this works with me, please?

I appreciate any help you can provide.

[KBB99]

There is a public layer I use in the repo here. If you follow the README.md in the project it will guide you through the commands to deploy the agent to AWS using CDK.

const AIMessageProcessor = new lambda.Function(this, 'AIMessageProcessor', {
      runtime: lambda.Runtime.PYTHON_3_9,
      code: lambda.Code.fromAsset('lambda'),
      handler: 'AIMessageProcessor.lambda_handler',
      timeout: cdk.Duration.seconds(120),
      architecture: lambda.Architecture.ARM_64,
    });
    
    // This Langchain layer includes OpenAI
const langchainLayer = lambda.LayerVersion.fromLayerVersionArn(this, 'langchainLayer', 'arn:aws:lambda:us-east-1:609061237212:layer:langchain:6');
AIMessageProcessor.addLayers(langchainLayer);

[pdichone]

Thank you so much. Will give it a try. Does it have to be ARM_64?

[KBB99]

If you use my public layer, then yes. If you build your own, then it depends on what architecture you pick for the build runtime.

[howdyhyber]

Hey I keep getting the following error:

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'pydantic_core._pydantic_core' Traceback (most recent call last):

Any help would be great, I'm kind of new to this to I'm totally lost on fixing it I've tried debugging it the best I can and I'm not getting anywhere with it.

Still having this kind of error till now

[nagaaws]

If someone is still looking for a Lambda layer including Langchain I just built this one and made it public: arn:aws:lambda:us-east-1:609061237212:layer:langchain:9 . It is running Langchain version 0.0.191 on ARM64 architecture and I have confirmed it's compatibility with Python3.10. To fix conflicts with boto3 I had to pin urllib<2.

Gives numpy error though

[KBB99]

Posted a new guide for deploying a Langchain and Lambda agent here: https://github.com/aws-samples/langchain-agents . It deploys everything using CDK so it should work out of the box. Let me know if you have any questions.

[dosubot]

Hi, @ali-chaudhry8,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. It looks like you're experiencing issues deploying Langchain scripts on AWS Lambda due to problems with importing numpy. There have been various attempts to resolve the issue, including the development of a deployment template for Langchain on Lambda and API Gateway by 3coins from AWS. Additionally, users have shared their experiences and solutions, such as using Docker to create Lambda layers for Langchain and resolving conflicts with boto3. KBB99 has also provided a public Lambda layer for Langchain and shared a guide for deploying a Langchain and Lambda agent using CDK.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and contribution to LangChain!

[brunocam11]

I've followed these steps to create a Lambda layer for Python 3.12, including the langchain libraries, and it worked perfectly. Hope it helps!

# Run an Amazon Linux 2 Docker container
docker run -it --rm -v $(pwd):/lambda-layer -w /lambda-layer public.ecr.aws/sam/build-python3.12:1.118.0-20240530211142 bash

# Create a directory for your Lambda layer
mkdir -p python/lib/python3.12/site-packages

# Install dependencies in this directory
pip install langchain langchain-community langchain-core langchain-openai -t python/lib/python3.12/site-packages

# Zip the directory
zip -r layer.zip python

# Exit the container
exit

For quick access, you can also use this pre-built Lambda layer with the langchain libraries:
arn:aws:lambda:us-east-1:654654459657:layer:langchain:12

[ajithkumarkaliyappan]

hi @brunocam11 . I'm faacing this issue while trying to use your layer >
Failed to load layer version details: User: arn:aws:iam::********:user/aj is not authorized to perform: lambda:GetLayerVersion on resource: arn:aws:lambda:us-east-1:654654459657:layer:langchain:12 because no resource-based policy allows the lambda:GetLayerVersion action

[Sk4467]

arn:aws:lambda:us-east-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:51