High Memory usage - "not enough Memory" in jsii-Runtime

[jdiebold]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

ckdtf 0.0.19
python

Description

We are using terraform to provision EMR clusters on AWS from Airflow. Since we are using cdktf in python our Airflow workers are using significantly more memory up until failure with the message that the jsii-Runtime has "not enough memory".
We are generating the Terraform-Json using cdktf and then deploy using the terraform cli.

Since the same terraform commands are run with the same infrastructure, this seems to be connected to cdktf or the jsii runtime.

[skorfmann]

Thanks for opening this issue. A few questions to get a better understanding:

• You're using the hashicorp/jsii-terraform Docker image I suppose?
• Are you generating the AWS provider bindings on each run, or are they prebuilt or even using one of the published PyPi packages?
• How much memory is available to that task?
• Which Terraform version are you using?

[jsteinich]

Possibly relevant: aws/jsii#745

[jdiebold]

Thanks for your quick answer!

• We're using a basic linux base image with python, where we install Airflow, terraform and cdktf, then we run the generation of the AWS provider in the docker build
• currently we're on Terraform 0.13.5, but looking already into 0.14, we're using aws-provider version 3.23 (latest)
• our airflow workers had 8GB memory, but probably shared between tasks.

The error occurred when airflow wants to load the DAG, the stack trace shows it's when loading the aws/_jsii/__init__.py.
Would it help to use a prebuilt/published package?

[skorfmann]

Would it help to use a prebuilt/published package?

This might help, I'd assume that the cdktf get step is probably the one which takes up most resources. I think the pre-built AWS provider is still on 2.X (gonna fix that soon). You could still perform the cdktf get step as part of the Docker build.

The error occurred when airflow wants to load the DAG, the stack trace shows it's when loading the aws/_jsii/__init__.py.

Hm, not sure if that's during cdktf get or cdktf deploy, could be both I believe.

[jdiebold]

we're already performing the cdktf get during the docker build.
The weird thing about the error was, that terraform init and terraform apply both already completed and everything worked.
When the airflow worker reloads the full DAG in the step afterwards, the cdktf code was also loaded and that was when it ran out of memory.

[skorfmann]

we're already performing the cdktf get during the docker build.

👍

The weird thing about the error was, that terraform init and terraform apply both already completed and everything worked.

Hm, that sounds not too bad.

When the airflow worker reloads the full DAG in the step afterwards, the cdktf code was also loaded and that was when it ran out of memory.

Don't know much about Airflow, but does this mean that it actually reran unintentionally?

[jdiebold]

I generated the AWS provider via cdktf get and stripped down the huge python file to only the classes I needed for my stack.
The next time the scheduled jobs ran, the workers needed significantly less memory (~6gb less at the peak).
So it seems to be connected with parsing the generated __init__.py of the provider.

[skorfmann]

hm, that's interesting. Seems like this could potentially benefit from aws/jsii#2436

Related to #504

[lkdmid]

I'm also seeing cdktf (Python) using huge amounts of memory in a Dockerised AWS Lambda function, I've only managed to get it running by giving the Lambda 1GB so far, which seems unnecessary given vanilla Terraform works just fine with the defaults.

[skorfmann]

I'm also seeing cdktf (Python) using huge amounts of memory in a Dockerised AWS Lambda function, I've only managed to get it running by giving the Lambda 1GB so far, which seems unnecessary given vanilla Terraform works just fine with the defaults.

Which Docker image are you using and do you know what the default Lambda settings are?

[DanielMSchmidt]

Almost a year since last action, closing this for now 👍

[github-actions]

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.