Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unable to find builds when first building (second time works) #204

Closed
Ranchman42 opened this issue Sep 28, 2021 · 13 comments
Closed

unable to find builds when first building (second time works) #204

Ranchman42 opened this issue Sep 28, 2021 · 13 comments

Comments

@Ranchman42
Copy link

Ranchman42 commented Sep 28, 2021

Description

On every first "build" of any of my python 3.8 lambdas i get the following error message:

Traceback (most recent call last):
│   File ".terraform/modules/xyz/package.py", line 1270, in <module>
│     main()
│   File ".terraform/modules/xyz/package.py", line 1266, in main
│     exit(args.command(args))
│   File ".terraform/modules/xyz/package.py", line 1158, in build_command
│     bpm.execute(build_plan, zs, query)
│   File ".terraform/modules/xyz/package.py", line 318, in __exit__
│     self.close()
│   File ".terraform/modules/xyz/package.py", line 308, in close
│     os.replace(self._tmp_filename, self.filename)
│ FileNotFoundError: [Errno 2] No such file or directory:
│ 'builds/fbbe17e900ebe9f3dda70cf0c686db6fae1c4487797cc3e7c5e87876b9090de4.zip.tmp'
│ ->
│ 'builds/fbbe17e900ebe9f3dda70cf0c686db6fae1c4487797cc3e7c5e87876b9090de4.zip'

each future build is successful until i delete .terraform and the builds folder.

⚠️ Note

Before you submit an issue, please perform the following first:

  1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
  2. Re-initialize the project root to pull down modules: terraform init
  3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

  • Terraform: 1.0.7, but also tried .14, and .15
Terraform v1.0.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/archive v2.2.0
+ provider registry.terraform.io/hashicorp/aws v3.45.0
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Reproduction

Steps to reproduce the behavior:

No Yes remove local cache, have a python 3.8 lambda that needs to be packaged and deployed to Lambda

Expected behavior

Successful run on the first run.

Actual behavior

Using this module to terraform apply always fails the first time after deleting .terraform and builds, but works on the second time.

This is hard because it makes building/deploying via github actions difficult since every time is the first run.

@antonbabenko
Copy link
Member

Please make sure that you are using v2.16.0 of this module or newer. See CHANGELOG.

The problem you are experiencing has been fixed by #181

Here you can see a bit more info that can help you to make it to work in CI/CD pipeline - https://github.com/terraform-aws-modules/terraform-aws-lambda#lambda-function-or-lambda-layer-with-the-deployable-artifact-maintained-separately-from-the-infrastructure

@Ranchman42
Copy link
Author

Sorry, i should have mentioned, i am using version 2.18.0 of this module and still hitting this error.

@antonbabenko
Copy link
Member

Ok, show the complete module block to see what arguments you are specifying.

@Ranchman42
Copy link
Author

Ranchman42 commented Sep 28, 2021

Here it is:


module "lambda_webhook_disabler" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "${var.region_shortcode_uc}webhook_disabler"
  handler       = "webhook_disabler.main"
  runtime       = "python3.8"
  timeout       = 300
  publish       = true
  version       = "2.18.0"
  

  source_path = "../../../lambda-src/webhook_disabler"

  attach_policies    = true
  number_of_policies = 4
  policies = [
    module.iam.aws_iam_policy_ses_full_access_arn,
    module.iam.sqs_poller_lambda_execution_role_arn,
    module.iam.webhook_basic_execution_arn,
    module.iam.aws_iam_policy_vpc_access_execution_role_arn
  ]

  vpc_subnet_ids         = [module.networking.lambda_subnet_a, module.networking.lambda_subnet_b]
  vpc_security_group_ids = [module.security_groups.allow_lambda]

}

After upgrading i also deleted .terraform and builds, and ran init again.

@antonbabenko
Copy link
Member

I can't reproduce it. Here is how I do it. I have main.tf with this content (remove VPC and policies as they are not related to the issue you described):

module "lambda_webhook_disabler" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "2.18.0"

  function_name = "webhook_disabler"
  handler       = "webhook_disabler.main"
  runtime       = "python3.8"
  timeout       = 300
  publish       = true

  # source path is relative and pointing to this directory locally:
  # https://github.com/terraform-aws-modules/terraform-aws-lambda/tree/master/examples/fixtures/python3.8-app1
  source_path = "../../terraform-aws-lambda/examples/fixtures/python3.8-app1"
}

$ terraform init

Initializing modules...
Downloading terraform-aws-modules/lambda/aws 2.18.0 for lambda_webhook_disabler...
- lambda_webhook_disabler in .terraform/modules/lambda_webhook_disabler

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/external versions matching ">= 1.0.0"...
- Finding hashicorp/local versions matching ">= 1.0.0"...
- Finding hashicorp/null versions matching ">= 2.0.0"...
- Finding hashicorp/aws versions matching ">= 3.43.0"...
- Using hashicorp/external v2.1.0 from the shared cache directory
- Using hashicorp/local v2.1.0 from the shared cache directory
- Using hashicorp/null v3.1.0 from the shared cache directory
- Using hashicorp/aws v3.60.0 from the shared cache directory

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # module.lambda_webhook_disabler.data.aws_iam_policy_document.logs[0] will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "logs"  {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "logs:CreateLogGroup",
              + "logs:CreateLogStream",
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + (known after apply),
              + (known after apply),
            ]
        }
    }

  # module.lambda_webhook_disabler.aws_cloudwatch_log_group.lambda[0] will be created
  + resource "aws_cloudwatch_log_group" "lambda" {
      + arn               = (known after apply)
      + id                = (known after apply)
      + name              = "/aws/lambda/webhook_disabler"
      + retention_in_days = 0
      + tags_all          = (known after apply)
    }

  # module.lambda_webhook_disabler.aws_iam_policy.logs[0] will be created
  + resource "aws_iam_policy" "logs" {
      + arn       = (known after apply)
      + id        = (known after apply)
      + name      = "webhook_disabler-logs"
      + path      = "/"
      + policy    = (known after apply)
      + policy_id = (known after apply)
      + tags_all  = (known after apply)
    }

  # module.lambda_webhook_disabler.aws_iam_role.lambda[0] will be created
  + resource "aws_iam_role" "lambda" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "lambda.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = true
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "webhook_disabler"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

  # module.lambda_webhook_disabler.aws_iam_role_policy_attachment.logs[0] will be created
  + resource "aws_iam_role_policy_attachment" "logs" {
      + id         = (known after apply)
      + policy_arn = (known after apply)
      + role       = "webhook_disabler"
    }

  # module.lambda_webhook_disabler.aws_lambda_function.this[0] will be created
  + resource "aws_lambda_function" "this" {
      + arn                            = (known after apply)
      + filename                       = "builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.zip"
      + function_name                  = "webhook_disabler"
      + handler                        = "webhook_disabler.main"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + memory_size                    = 128
      + package_type                   = "Zip"
      + publish                        = true
      + qualified_arn                  = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "python3.8"
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + source_code_hash               = (known after apply)
      + source_code_size               = (known after apply)
      + tags_all                       = (known after apply)
      + timeout                        = 300
      + version                        = (known after apply)

      + tracing_config {
          + mode = (known after apply)
        }
    }

  # module.lambda_webhook_disabler.local_file.archive_plan[0] will be created
  + resource "local_file" "archive_plan" {
      + content              = jsonencode(
            {
              + artifacts_dir = "builds"
              + build_plan    = [
                  + [
                      + "pip",
                      + "python3.8",
                      + "../../terraform-aws-lambda/examples/fixtures/python3.8-app1/requirements.txt",
                      + null,
                    ],
                  + [
                      + "zip",
                      + "../../terraform-aws-lambda/examples/fixtures/python3.8-app1",
                      + null,
                    ],
                ]
              + filename      = "builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.zip"
              + runtime       = "python3.8"
            }
        )
      + directory_permission = "0755"
      + file_permission      = "0644"
      + filename             = "builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.plan.json"
      + id                   = (known after apply)
    }

  # module.lambda_webhook_disabler.null_resource.archive[0] will be created
  + resource "null_resource" "archive" {
      + id       = (known after apply)
      + triggers = {
          + "filename"  = "builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.zip"
          + "timestamp" = "1632901815063644800"
        }
    }

Plan: 7 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.lambda_webhook_disabler.local_file.archive_plan[0]: Creating...
module.lambda_webhook_disabler.local_file.archive_plan[0]: Creation complete after 0s [id=e45dcb7d174c0bfac2f26ecafcaaead77edfbfc6]
module.lambda_webhook_disabler.null_resource.archive[0]: Creating...
module.lambda_webhook_disabler.null_resource.archive[0]: Provisioning with 'local-exec'...
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Executing: ["python3" ".terraform/modules/lambda_webhook_disabler/package.py" "build" "--timestamp" "1632901815063644800" "builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.plan.json"]
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: creating 'builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.zip' archive
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Installing python requirements: ../../terraform-aws-lambda/examples/fixtures/python3.8-app1/requirements.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): > mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-8eeetvks
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): > cd /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-8eeetvks
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): > python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt
module.lambda_webhook_disabler.aws_cloudwatch_log_group.lambda[0]: Creating...
module.lambda_webhook_disabler.aws_iam_role.lambda[0]: Creating...
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Collecting colorful
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec):   Using cached colorful-0.5.4-py2.py3-none-any.whl (201 kB)
module.lambda_webhook_disabler.aws_cloudwatch_log_group.lambda[0]: Creation complete after 0s [id=/aws/lambda/webhook_disabler]
module.lambda_webhook_disabler.data.aws_iam_policy_document.logs[0]: Reading...
module.lambda_webhook_disabler.data.aws_iam_policy_document.logs[0]: Read complete after 0s [id=882654373]
module.lambda_webhook_disabler.aws_iam_policy.logs[0]: Creating...
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Installing collected packages: colorful
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Successfully installed colorful-0.5.4
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding content of directory: /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-8eeetvks
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/__init__.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/ansi.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/colors.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/core.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/styles.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/terminal.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/utils.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/data/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/data/colornames.json
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/data/rgb.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/INSTALLER
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/LICENSE
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/METADATA
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/RECORD
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/REQUESTED
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/WHEEL
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/top_level.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding content of directory: ../../terraform-aws-lambda/examples/fixtures/python3.8-app1
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: ignore_please.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: index.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: requirements.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: dir1/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: dir1/dir2/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: dir1/dir2/ignore2.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: docker/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: docker/Dockerfile
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: docker/automake-1.13-to-1.16-spec.patch
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Created: builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.zip
module.lambda_webhook_disabler.null_resource.archive[0]: Creation complete after 4s [id=3200651326543575334]
module.lambda_webhook_disabler.aws_iam_role.lambda[0]: Creation complete after 2s [id=webhook_disabler]
module.lambda_webhook_disabler.aws_lambda_function.this[0]: Creating...
module.lambda_webhook_disabler.aws_iam_policy.logs[0]: Creation complete after 2s [id=arn:aws:iam::835367859851:policy/webhook_disabler-logs]
module.lambda_webhook_disabler.aws_iam_role_policy_attachment.logs[0]: Creating...
module.lambda_webhook_disabler.aws_iam_role_policy_attachment.logs[0]: Creation complete after 1s [id=webhook_disabler-20210929075024448700000001]
module.lambda_webhook_disabler.aws_lambda_function.this[0]: Still creating... [10s elapsed]
module.lambda_webhook_disabler.aws_lambda_function.this[0]: Creation complete after 15s [id=webhook_disabler]

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

All good and as expected.

Now, if I remove builds directory and run terraform apply:

terraform apply
module.lambda_webhook_disabler.local_file.archive_plan[0]: Refreshing state... [id=e45dcb7d174c0bfac2f26ecafcaaead77edfbfc6]
module.lambda_webhook_disabler.null_resource.archive[0]: Refreshing state... [id=3200651326543575334]
module.lambda_webhook_disabler.aws_cloudwatch_log_group.lambda[0]: Refreshing state... [id=/aws/lambda/webhook_disabler]
module.lambda_webhook_disabler.aws_iam_role.lambda[0]: Refreshing state... [id=webhook_disabler]
module.lambda_webhook_disabler.aws_iam_policy.logs[0]: Refreshing state... [id=arn:aws:iam::835367859851:policy/webhook_disabler-logs]
module.lambda_webhook_disabler.aws_iam_role_policy_attachment.logs[0]: Refreshing state... [id=webhook_disabler-20210929075024448700000001]
module.lambda_webhook_disabler.aws_lambda_function.this[0]: Refreshing state... [id=webhook_disabler]

Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.lambda_webhook_disabler.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
      ~ id       = "3200651326543575334" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "timestamp" = "1632901815063644800" -> "1632902479700601200"
            # (1 unchanged element hidden)
        }
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.lambda_webhook_disabler.null_resource.archive[0]: Destroying... [id=3200651326543575334]
module.lambda_webhook_disabler.null_resource.archive[0]: Destruction complete after 0s
module.lambda_webhook_disabler.null_resource.archive[0]: Creating...
module.lambda_webhook_disabler.null_resource.archive[0]: Provisioning with 'local-exec'...
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Executing: ["python3" ".terraform/modules/lambda_webhook_disabler/package.py" "build" "--timestamp" "1632902479700601200" "builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.plan.json"]
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: creating 'builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.zip' archive
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Installing python requirements: ../../terraform-aws-lambda/examples/fixtures/python3.8-app1/requirements.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): > mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-c_ae7h7e
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): > cd /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-c_ae7h7e
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): > python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Collecting colorful
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec):   Using cached colorful-0.5.4-py2.py3-none-any.whl (201 kB)
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Installing collected packages: colorful
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Successfully installed colorful-0.5.4
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding content of directory: /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-c_ae7h7e
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/__init__.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/ansi.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/colors.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/core.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/styles.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/terminal.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/utils.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/data/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/data/colornames.json
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful/data/rgb.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/INSTALLER
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/LICENSE
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/METADATA
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/RECORD
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/REQUESTED
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/WHEEL
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: colorful-0.5.4.dist-info/top_level.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding content of directory: ../../terraform-aws-lambda/examples/fixtures/python3.8-app1
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: ignore_please.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: index.py
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: requirements.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: dir1/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: dir1/dir2/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: dir1/dir2/ignore2.txt
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: docker/
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: docker/Dockerfile
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): zip: adding: docker/automake-1.13-to-1.16-spec.patch
module.lambda_webhook_disabler.null_resource.archive[0] (local-exec): Created: builds/1e6df0122a540eb2e8517370f2209510d845f67992ffb0c37b281948a0826433.zip
module.lambda_webhook_disabler.null_resource.archive[0]: Creation complete after 3s [id=7961938920746258128]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

The plan and zip archives are being recreated locally.

If I specify recreate_missing_package = false inside of the module block, zip archive won't be recreated locally after builds directory is removed.

I am not sure what else can we do here.

@hazam
Copy link

hazam commented Oct 4, 2021

I got same issue.

@maikcaru
Copy link

maikcaru commented Nov 3, 2021

I'm also seeing this behaviour when running an apply against a fresh clone of a repository.

@saurabhladhe
Copy link

using version 2.25.0 and getting the same issue. Is there any workaround at all?

@jeromevdl
Copy link

Same here with 2.28.0, I had to apply twice...

@kadrach
Copy link

kadrach commented Jan 4, 2022

This issue occurs when two Lambda functions are attempting to use the same src folder.

Specifically, the archive plan uses the (identical) source hash as a file name for the zip file. When two functions are deployed, the packager(s) race (which is why it succeeds on a subsequent attempt).

Reproducible example:

terraform {
  backend "local" {}
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

module "lambda1" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "2.28.0"

  function_name = "app1"
  handler       = "app1.lambda_handler"
  runtime       = "python3.9"

  source_path = "${path.module}/src"
}

# Same src folder, different handler
module "lambda2" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "2.28.0"

  function_name = "app2"
  handler       = "app2.lambda_handler"
  runtime       = "python3.9"

  source_path = "${path.module}/src"
}
.
├── main.tf
├── src
│   ├── app1.py
│   └── app2.py

An initial plan shows:

...

  # module.lambda1.local_file.archive_plan[0] will be created
  + resource "local_file" "archive_plan" {
      + content              = jsonencode(
            {
              + artifacts_dir = "builds"
              + build_plan    = [
                  + [
                      + "zip",
                      + "./src",
                      + null,
                    ],
                ]
              + filename      = "builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.zip"
              + runtime       = "python3.9"
            }
        )
      + directory_permission = "0755"
      + file_permission      = "0644"
      + filename             = "builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.plan.json"
      + id                   = (known after apply)
    }

  # module.lambda1.null_resource.archive[0] will be created
  + resource "null_resource" "archive" {
      + id       = (known after apply)
      + triggers = {
          + "filename"  = "builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.zip"
          + "timestamp" = "1641263342503815800"
        }
    }

...

  # module.lambda2.local_file.archive_plan[0] will be created
  + resource "local_file" "archive_plan" {
      + content              = jsonencode(
            {
              + artifacts_dir = "builds"
              + build_plan    = [
                  + [
                      + "zip",
                      + "./src",
                      + null,
                    ],
                ]
              + filename      = "builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.zip"
              + runtime       = "python3.9"
            }
        )
      + directory_permission = "0755"
      + file_permission      = "0644"
      + filename             = "builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.plan.json"
      + id                   = (known after apply)
    }

  # module.lambda2.null_resource.archive[0] will be created
  + resource "null_resource" "archive" {
      + id       = (known after apply)
      + triggers = {
          + "filename"  = "builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.zip"
          + "timestamp" = "1641263342502863000"
        }
    }

And on apply:

module.lambda2.null_resource.archive[0] (local-exec): Created: builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.zip
module.lambda1.null_resource.archive[0] (local-exec): Traceback (most recent call last):
module.lambda1.null_resource.archive[0] (local-exec):   File ".terraform/modules/lambda1/package.py", line 1271, in <module>
module.lambda1.null_resource.archive[0] (local-exec):     main()
module.lambda1.null_resource.archive[0] (local-exec):   File ".terraform/modules/lambda1/package.py", line 1267, in main
module.lambda1.null_resource.archive[0] (local-exec):     exit(args.command(args))
module.lambda1.null_resource.archive[0] (local-exec):   File ".terraform/modules/lambda1/package.py", line 1159, in build_command
module.lambda1.null_resource.archive[0] (local-exec):     bpm.execute(build_plan, zs, query)
module.lambda1.null_resource.archive[0] (local-exec):   File ".terraform/modules/lambda1/package.py", line 318, in __exit__
module.lambda1.null_resource.archive[0] (local-exec):     self.close()
module.lambda1.null_resource.archive[0] (local-exec):   File ".terraform/modules/lambda1/package.py", line 308, in close
module.lambda1.null_resource.archive[0] (local-exec):     os.replace(self._tmp_filename, self.filename)
module.lambda1.null_resource.archive[0] (local-exec): FileNotFoundError: [Errno 2] No such file or directory: 'builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.zip.tmp' -> 'builds/bbd19d35f99e7d47fb833316123b6c63187d23b81a28613f3856a2fb85c5e1dd.zip'
module.lambda2.null_resource.archive[0]: Creation complete after 4s [id=7382991278809111587]
             ^
             lambda2 wins the race.

I worked around it by explicitly re-using the packaged artefact from one function in the other.

This is already covered in #139.

@github-actions
Copy link

github-actions bot commented Feb 4, 2022

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

@github-actions github-actions bot added the stale label Feb 4, 2022
@github-actions
Copy link

This issue was automatically closed because of stale in 10 days

@github-actions
Copy link

github-actions bot commented Nov 9, 2022

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants