Lambda environment paramaters can exceed 4KB limit #3594

rsavage-nozominetworks · 2023-11-03T13:43:37Z

Greetings-

Apparently the latest version (5.3.0) of terraform-aws-github-runner is breaking our lambda updates. This has never been an issue until now.

│ Error: updating Lambda Function (REDACTED-webhook) configuration: operation error Lambda: UpdateFunctionConfiguration, https response error StatusCode: 400, RequestID: 20f1f0c3-3326-4a61-895f-fccf2eb4ff64, InvalidParameterValueException: Lambda was unable to configure your environment variables because the environment variables you have provided exceeded the 4KB limit.

Were additional labels added thus causing the 4KB limit to be exceeded?

The text was updated successfully, but these errors were encountered:

rsavage-nozominetworks · 2023-11-03T13:48:14Z

Apparently changes to the runner labels have been made which seems to add additional labels to the respectful lambda environment variables. Due to this, it exceeded AWS 4kb limit and throws the error. We don't have many additional labels and they are fairly short. I can't image we are the only ones getting this error now.

I have since reverted back to 5.2.1 and the problem is gone.

npalm · 2023-11-04T08:20:39Z

Thanks for reporting. But 5.3.0 should not be breaking. It adds a feature for the multi runners to set a priority to the matcher. See here teh diff. Only terraform code philips-labs/terraform-aws-github-runner@1829721

Do you know which parameter is causing the issue? Are you using a multi runner setup?

npalm · 2023-11-04T08:25:41Z

Adding ar ref to an AWS post, total size hsould be less than 4K, https://repost.aws/knowledge-center/lambda-environment-variable-size

Which mean we have to re-thing how we configure the lambda's

npalm · 2023-11-04T08:40:09Z

Can you share your configuration, of course without the sensitive part? So we can reproduce the isse.

npalm · 2023-11-06T13:27:16Z

Quickly had a chat with Amazon, there seems no easy way to pro-active warn on the size.

npalm · 2023-11-06T19:54:59Z

This script should give some insights in the sizes of the environment variables per function.

#!/bin/bash

functions=$(aws lambda list-functions --query 'Functions[*].FunctionName' --output text)

for function in $functions
do
  env_vars=$(aws lambda get-function-configuration --function-name $function --query 'Environment.Variables' --output text)
  size=$(echo -n $env_vars | wc -c)
  printf "%-5s bytes -> %s\n" $size $function
done

rsavage-nozominetworks · 2023-11-06T21:40:31Z

Thanks for reporting. But 5.3.0 should not be breaking. It adds a feature for the multi runners to set a priority to the matcher. See here teh diff. Only terraform code 1829721

Do you know which parameter is causing the issue? Are you using a multi runner setup?

I don't know which parameter is causing the issue persay. I just know that when I reverted back to 5.2.1, it stopped erroring with the 4KB lambda update limit. Yes, I am using the multi-runner setup.

rsavage-nozominetworks · 2023-11-06T21:43:04Z

Can you share your configuration, of course without the sensitive part? So we can reproduce the isse.

Yes, I can shared it once I clean up the sensitive details. Let me ask, what specifically are you looking for in regards to the configuration, the whole multi-runner setup? Are you wanting the module output or terraform code?

rsavage-nozominetworks · 2023-11-07T13:51:10Z

@npalm please let me know specifically what you are looking for and I'll scrub it and share it. Obviously, due to the possibility of sensitive information, I want to share at least as possible. Thanks!

npalm · 2023-11-07T20:25:41Z

I have checked our example, they are all on the save side of the 4KB limit. Also our productions deployments are all on the safe side. But indeed depending on how large names you chose and how many paramaters are defined the list can differ a lot. The potential problem exists on all labmda's. In my opnion the risk is the biggest on the webhook in combinaiton with the multi-runner. In that case each matcher (runner) creates on or more rules. Which ar passed as in a varaible.

The new release introduce an extra attribute. Which seems in your case the one that is exceeding the limit. Would like just to understand how big your setup is roughtly.

In the end we should find a solution. Which will be most likely moving away from the environment varialbes for Lambda.

I ran another experiament to calculate the environment variables. Which is using the module output for the webhook as josn. Look up the environment variables and pipe it to wc -c. This results seems to give a more accurate count.

rsavage-nozominetworks · 2023-11-09T14:01:46Z

These are the env vars from one of my runner lambdas:

ENVIRONMENT     :       vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
LOG_LEVEL       :       debug
MINIMUM_RUNNING_TIME_IN_MINUTES :       5
NODE_TLS_REJECT_UNAUTHORIZED    :       1
PARAMETER_GITHUB_APP_ID_NAME    :       /github-action-runners/prod-runners/app/github_app_id
PARAMETER_GITHUB_APP_KEY_BASE64_NAME    :       /github-action-runners/prod-runners/app/github_app_key_base64
POWERTOOLS_LOGGER_LOG_EVENT     :       true
RUNNER_BOOT_TIME_IN_MINUTES     :       5
SCALE_DOWN_CONFIG       :       [{\"cron\":\"* * 9-23 * * 1-5\",\"evictionStrategy\":\"oldest_first\",\"idleCount\":10,\"timeZone\":\"Europe/Amsterdam\"}]
SERVICE_NAME    :       runners-scale-down

And I have 11 runner types.

npalm · 2023-11-09T14:48:10Z

These are the env vars from one of my runner lambdas:

ENVIRONMENT     :       vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
LOG_LEVEL       :       debug
MINIMUM_RUNNING_TIME_IN_MINUTES :       5
NODE_TLS_REJECT_UNAUTHORIZED    :       1
PARAMETER_GITHUB_APP_ID_NAME    :       /github-action-runners/prod-runners/app/github_app_id
PARAMETER_GITHUB_APP_KEY_BASE64_NAME    :       /github-action-runners/prod-runners/app/github_app_key_base64
POWERTOOLS_LOGGER_LOG_EVENT     :       true
RUNNER_BOOT_TIME_IN_MINUTES     :       5
SCALE_DOWN_CONFIG       :       [{\"cron\":\"* * 9-23 * * 1-5\",\"evictionStrategy\":\"oldest_first\",\"idleCount\":10,\"timeZone\":\"Europe/Amsterdam\"}]
SERVICE_NAME    :       runners-scale-down

And I have 11 runner types.

This would not exceed 4KB, or does it?

rsavage-nozominetworks · 2023-11-13T22:06:33Z

These are the env vars from one of my runner lambdas:

ENVIRONMENT     :       vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
LOG_LEVEL       :       debug
MINIMUM_RUNNING_TIME_IN_MINUTES :       5
NODE_TLS_REJECT_UNAUTHORIZED    :       1
PARAMETER_GITHUB_APP_ID_NAME    :       /github-action-runners/prod-runners/app/github_app_id
PARAMETER_GITHUB_APP_KEY_BASE64_NAME    :       /github-action-runners/prod-runners/app/github_app_key_base64
POWERTOOLS_LOGGER_LOG_EVENT     :       true
RUNNER_BOOT_TIME_IN_MINUTES     :       5
SCALE_DOWN_CONFIG       :       [{\"cron\":\"* * 9-23 * * 1-5\",\"evictionStrategy\":\"oldest_first\",\"idleCount\":10,\"timeZone\":\"Europe/Amsterdam\"}]
SERVICE_NAME    :       runners-scale-down

And I have 11 runner types.

This would not exceed 4KB, or does it?

Nope, however using your script, it does show that the webhook lambda to be the culprit

3646  bytes -> nnnn-runners-webhook

rsavage-nozominetworks · 2023-11-15T21:53:24Z

Hi @npalm here is the output from the trying to go from 5.2.1 to 5.4.0

│ Error: updating Lambda Function (nnnn-runners-webhook) configuration: operation error Lambda: UpdateFunctionConfiguration, https response error StatusCode: 400, RequestID: c9a4f8f7-8320-4e57-8082-3ea8aadea5c3, InvalidParameterValueException: Lambda was unable to configure your environment variables because the environment variables you have provided exceeded the 4KB limit. String measured:

{
    "POWERTOOLS_LOGGER_LOG_EVENT": "true",
    "POWERTOOLS_TRACER_CAPTURE_ERROR": "false",
    "RUNNER_CONFIG": '[{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-arm-2xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-arm-2xlarge-queued-builds.fifo","key":"linux-arm-2xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","arm64","nws-arm64-self-hosted-2xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-arm-4xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-arm-4xlarge-queued-builds.fifo","key":"linux-arm-4xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","arm64","nws-arm64-self-hosted-4xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-arm-large-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-arm-large-queued-builds.fifo","key":"linux-arm-large","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","arm64","nws-arm64-self-hosted"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-arm-xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-arm-xlarge-queued-builds.fifo","key":"linux-arm-xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","arm64","nws-arm64-self-hosted-xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-x64-2xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-x64-2xlarge-queued-builds.fifo","key":"linux-x64-2xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","x64","nws-self-hosted-2xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-x64-4xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-x64-4xlarge-queued-builds.fifo","key":"linux-x64-4xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","x64","nws-self-hosted-4xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-x64-large-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-x64-large-queued-builds.fifo","key":"linux-x64-large","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","x64","nws-self-hosted"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-x64-xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-x64-xlarge-queued-builds.fifo","key":"linux-x64-xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","x64","nws-self-hosted-xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnnnn-x64-2xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnnnn-x64-2xlarge-queued-builds.fifo","key":"windows-x64-2xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","windows","x64","nws-windows-2xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnnnn-x64-large-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnnnn-x64-large-queued-builds.fifo","key":"windows-x64-large","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","windows","x64","nws-windows-large"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnnnn-x64-xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnnnn-x64-xlarge-queued-builds.fifo","key":"windows-x64-xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","windows","x64","nws-windows-xlarge"]],"priority":999}}]',
    "ENVIRONMENT": "nnnn-runners",
    "PARAMETER_GITHUB_APP_WEBHOOK_SECRET": "/github-action-runners/nnnn-runners/app/github_app_webhook_secret",
    "POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS": "false",
    "SQS_WORKFLOW_JOB_QUEUE": "",
    "LOG_LEVEL": "debug",
    "REPOSITORY_WHITE_LIST": "[]",
    "POWERTOOLS_TRACE_ENABLED": "false",
}

│
│   with module.multi_runners["nnnn"].module.multi_runner.module.webhook.aws_lambda_function.webhook,
│   on .terraform/modules/multi_runners.multi_runner/modules/webhook/webhook.tf line 9, in resource "aws_lambda_function" "webhook":
│    9: resource "aws_lambda_function" "webhook" {
│
╵

Looks like it's sitting at 4.37 KB. Something with the update added ~ .40 KB to the payload.

npalm · 2023-11-16T08:06:29Z

I think this requires we move the RUNNER_CONFIG to SSM.

Thie PR prepare for changes to easier migrate reading config from SSM instead of environment (github-aws-runners#3594), add option to only accept messages of a defined IP list, and to introduce option to connect runners via EventBridg. - Validate input and throw validation exceptions if event cannot be accepted - Structure the code that to allow the webhook to be split in acceptiong an event and distribute to a runner (prepare for EventBridge). - Remove deprecated jest functions. - THE PR minimized changed on thest, only small structural things. This to ensure the test still validating the implemention

rsavage-nozominetworks · 2023-11-27T22:36:51Z

Hi @npalm. I saw the recent actviity, and tried with the latest 5.4.2 version, however still getting the 4KB error

╷
│ Error: updating Lambda Function (xxxx-xxxxxxx-webhook) configuration: operation error Lambda: UpdateFunctionConfiguration, https response error StatusCode: 400, RequestID: 825e0324-8a73-4915-b24c-a7d6d05b18fd, InvalidParameterValueException: Lambda was unable to configure your environment variables because the environment variables you have provided exceeded the 4KB limit. String measured:

{"POWERTOOLS_LOGGER_LOG_EVENT":"true","POWERTOOLS_TRACER_CAPTURE_ERROR":"false","RUNNER_CONFIG":"[{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-arm-2xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-arm-2xlarge-queued-builds.fifo\",\"key\":\"linux-arm-2xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"arm64\",\"xxx-xxxxx-self-hosted-2xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-arm-4xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-arm-4xlarge-queued-builds.fifo\",\"key\":\"linux-arm-4xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"arm64\",\"xxx-xxxxx-self-hosted-4xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-arm-large-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-arm-large-queued-builds.fifo\",\"key\":\"linux-arm-large\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"arm64\",\"xxx-xxxxx-self-hosted\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-arm-xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-arm-xlarge-queued-builds.fifo\",\"key\":\"linux-arm-xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"arm64\",\"xxx-xxxxx-self-hosted-xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-x64-2xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-x64-2xlarge-queued-builds.fifo\",\"key\":\"linux-x64-2xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"x64\",\"xxx-self-hosted-2xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-x64-4xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-x64-4xlarge-queued-builds.fifo\",\"key\":\"linux-x64-4xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"x64\",\"xxx-self-hosted-4xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-x64-large-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-x64-large-queued-builds.fifo\",\"key\":\"linux-x64-large\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"x64\",\"xxx-self-hosted\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-x64-xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-x64-xlarge-queued-builds.fifo\",\"key\":\"linux-x64-xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"x64\",\"xxx-self-hosted-xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-windows-x64-2xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-windows-x64-2xlarge-queued-builds.fifo\",\"key\":\"windows-x64-2xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"windows\",\"x64\",\"xxx-xxxxxxx-2xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-windows-x64-large-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-windows-x64-large-queued-builds.fifo\",\"key\":\"windows-x64-large\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"windows\",\"x64\",\"xxx-xxxxxxx-large\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-windows-x64-xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-windows-x64-xlarge-queued-builds.fifo\",\"key\":\"windows-x64-xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"windows\",\"x64\",\"xxx-xxxxxxx-xlarge\"]],\"priority\":999}}]","PARAMETER_GITHUB_APP_WEBHOOK_SECRET":"/github-action-runners/xxxx-xxxxxxx/app/github_app_webhook_secret","POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS":"false","SQS_WORKFLOW_JOB_QUEUE":"","LOG_LEVEL":"debug","REPOSITORY_WHITE_LIST":"[]","POWERTOOLS_TRACE_ENABLED":"false"}
│
│   with module.multi_runners["prod"].module.multi_runner.module.webhook.aws_lambda_function.webhook,
│   on .terraform/modules/multi_runners.multi_runner/modules/webhook/webhook.tf line 9, in resource "aws_lambda_function" "webhook":
│    9: resource "aws_lambda_function" "webhook" {
│
╵

npalm · 2023-11-28T01:21:53Z

The issue is not resolved yet. but I have refacored the lambda so the next step to move the paramters to SSM is much simpler. Help / PR's are welcome

rsavage-nozominetworks · 2023-11-30T15:00:40Z

The issue is not resolved yet. but I have refacored the lambda so the next step to move the paramters to SSM is much simpler. Help / PR's are welcome

Thanks. Unfortunately I don't have the experience or time to devote to refactoring this code base.

npalm · 2023-12-02T15:27:36Z

No problem, we keep the issue open. Hopefully our community has some time to provide a fix.

github-actions · 2024-01-02T01:48:27Z

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.

rsavage-nozominetworks · 2024-01-08T14:24:39Z

Hey guys, any way to get this prioritized? It's blocking us from updating at the moment, as we're falling way behind.

npalm · 2024-01-10T22:00:54Z

It is on our radar, but would love to get some community support here.

spcaipers-arm · 2024-01-22T11:58:51Z

Hi @npalm I have a local improvement for that. I am moving RUNNER_CONFIG to SSM as you suggested.
Question: Do you expect the ENV VAR name RUNNER_CONFIG on the webhook to change?

I changed its value to Parameters Store's name.
I am not sure what your lambda functions expect.

spcaipers-arm · 2024-01-22T12:44:02Z

I just saw there is a PR for that, so I will for that to be merged.

rsavage-nozominetworks · 2024-01-24T21:26:45Z

Thank you guys!

Upgreydd · 2024-02-23T09:44:28Z

Same issue on my end. RUNNER_CONFIG change returns

Lambda was unable to configure your environment variables because the environment variables you have provided exceeded the 4KB limit

rollback to 5.2.1 helps but it's not the solution.

This PR migrates the confugration for the webhook from environment variables to SSM to avoid the maximum size of environment variables is reached. ## Implementation The webhook will read the configuration from SSM as json string. As long the lambda is hot the configuration is cached to speed-up the lambda time. In cases of configuration changes Lambda resources will be re-created by Terraform to ensure no cached values are used. fix: #3594 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Niek Palm <[email protected]> Co-authored-by: Niek Palm <[email protected]>

npalm changed the title ~~v5.3.0 breaking lambda updates~~ Lambda environment paramaters can exceed 4KB limit Nov 7, 2023

npalm mentioned this issue Nov 19, 2023

refactor: restructure webhook lambda #3618

Merged

npalm added good first issue Good for newcomers help wanted Extra attention is needed labels Dec 2, 2023

github-actions bot added the Stale label Jan 2, 2024

npalm added stale:exempt and removed Stale labels Jan 2, 2024

GuptaNavdeep1983 mentioned this issue Jan 18, 2024

feat: migrate webhook runner configuration to SSM #3728

Merged

GuptaNavdeep1983 closed this as completed in #3728 Feb 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lambda environment paramaters can exceed 4KB limit #3594

Lambda environment paramaters can exceed 4KB limit #3594

rsavage-nozominetworks commented Nov 3, 2023

rsavage-nozominetworks commented Nov 3, 2023

npalm commented Nov 4, 2023 •

edited

Loading

npalm commented Nov 4, 2023

npalm commented Nov 4, 2023

npalm commented Nov 6, 2023

npalm commented Nov 6, 2023

rsavage-nozominetworks commented Nov 6, 2023

rsavage-nozominetworks commented Nov 6, 2023 •

edited

Loading

rsavage-nozominetworks commented Nov 7, 2023

npalm commented Nov 7, 2023

rsavage-nozominetworks commented Nov 9, 2023

npalm commented Nov 9, 2023

rsavage-nozominetworks commented Nov 13, 2023

rsavage-nozominetworks commented Nov 15, 2023 •

edited

Loading

npalm commented Nov 16, 2023

rsavage-nozominetworks commented Nov 27, 2023

npalm commented Nov 28, 2023

rsavage-nozominetworks commented Nov 30, 2023

npalm commented Dec 2, 2023

github-actions bot commented Jan 2, 2024

rsavage-nozominetworks commented Jan 8, 2024

npalm commented Jan 10, 2024

spcaipers-arm commented Jan 22, 2024

spcaipers-arm commented Jan 22, 2024

rsavage-nozominetworks commented Jan 24, 2024

Upgreydd commented Feb 23, 2024 •

edited

Loading

Lambda environment paramaters can exceed 4KB limit #3594

Lambda environment paramaters can exceed 4KB limit #3594

Comments

rsavage-nozominetworks commented Nov 3, 2023

rsavage-nozominetworks commented Nov 3, 2023

npalm commented Nov 4, 2023 • edited Loading

npalm commented Nov 4, 2023

npalm commented Nov 4, 2023

npalm commented Nov 6, 2023

npalm commented Nov 6, 2023

rsavage-nozominetworks commented Nov 6, 2023

rsavage-nozominetworks commented Nov 6, 2023 • edited Loading

rsavage-nozominetworks commented Nov 7, 2023

npalm commented Nov 7, 2023

rsavage-nozominetworks commented Nov 9, 2023

npalm commented Nov 9, 2023

rsavage-nozominetworks commented Nov 13, 2023

rsavage-nozominetworks commented Nov 15, 2023 • edited Loading

npalm commented Nov 16, 2023

rsavage-nozominetworks commented Nov 27, 2023

npalm commented Nov 28, 2023

rsavage-nozominetworks commented Nov 30, 2023

npalm commented Dec 2, 2023

github-actions bot commented Jan 2, 2024

rsavage-nozominetworks commented Jan 8, 2024

npalm commented Jan 10, 2024

spcaipers-arm commented Jan 22, 2024

spcaipers-arm commented Jan 22, 2024

rsavage-nozominetworks commented Jan 24, 2024

Upgreydd commented Feb 23, 2024 • edited Loading

npalm commented Nov 4, 2023 •

edited

Loading

rsavage-nozominetworks commented Nov 6, 2023 •

edited

Loading

rsavage-nozominetworks commented Nov 15, 2023 •

edited

Loading

Upgreydd commented Feb 23, 2024 •

edited

Loading