resource/aws_lambda_function - possible regression w/3.17.0 on us-east-2

[kyleckf]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform 0.12.25, 0.13.5
AWS Provider 3.17.0 and beyond (tested on 3.21.0 and the issue is still there)

Affected Resource(s)

• aws_lambda_function

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Any aws_lamdba_function blocks, an example can be seen here:
#16398

Debug Output

Panic Output

Expected Behavior

Terraform apply is successful

Actual Behavior

Error: error getting Lambda Function Code Signing Config: AccessDeniedException: 11:31:34 status code: 403, request id: e5539b23-7f69-478d-ae36-67bc55c8e50c

Steps to Reproduce

1. terraform apply on us-east-2

Important Factoids

This issue is very similar to #16398. People have commented that it does not work for us-east-1 also.

References

• resource/aws_lamba_function - possible regression w/3.17.0 on GovCloud #16398

[ewbankkit]

@kyleckf Thanks for raising this issue.
It could be an IAM issue - Does the role used to run Terraform have the lambda:GetCodeSigningConfig permission?

[kyleckf]

Hi @ewbankkit , we don't have the lambda:GetCodeSigningConfig permission in the IAM role. Is this a needed permission?

[kyleckf]

Just one more note @ewbankkit, we tried with the same IAM permission in us-west-2 and we don't see the error. Hence we think this is region specific.

[ewbankkit]

@kyleckf Yes, lambda:GetCodeSigningConfig is a required permission.
If I attach an IAM policy like

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": "lambda:GetCodeSigningConfig",
            "Resource": "*"
        }
    ]
}

to my test account then I get

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSLambdaFunction_codeSigningConfig'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSLambdaFunction_codeSigningConfig -timeout 120m
=== RUN   TestAccAWSLambdaFunction_codeSigningConfig
=== PAUSE TestAccAWSLambdaFunction_codeSigningConfig
=== CONT  TestAccAWSLambdaFunction_codeSigningConfig
    resource_aws_lambda_function_test.go:194: Step 1/5 error: Error running apply: 2020/12/15 11:58:49 [DEBUG] Using modified User-Agent: Terraform/0.12.26 HashiCorp-terraform-exec/0.10.0
        
        Error: error reading Lambda code signing config (arn:aws:lambda:us-west-2:123456789012:code-signing-config:csc-0db84535440db24e0): AccessDeniedException: 
        	status code: 403, request id: 19c55ac6-51b3-43a0-bc84-b291c9595eec
        
        
        
        Error: error reading Lambda code signing config (arn:aws:lambda:us-west-2:123456789012:code-signing-config:csc-00bf60596266984f7): AccessDeniedException: 
        	status code: 403, request id: d731cb3d-8fc8-4996-8892-8cb69d264a78
        
        
--- FAIL: TestAccAWSLambdaFunction_codeSigningConfig (9.72s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	9.790s
FAIL
GNUmakefile:27: recipe for target 'testacc' failed
make: *** [testacc] Error 1

See the Roadmap for a research task on how we can better describe required IAM permissions.

[bcsgh]

I found this while debugging an issue and found that I also needed to grant lambda:GetFunctionCodeSigningConfig

[jbhennin]

@ewbankkit I added the policy and still get the issue originally reported. This appears to only have a problem in the east-2 region.

Your example of getting the deny showed the error in west-2. Can you confirm that if you allow GetCodeSigningConfig in the east-2 region you get the same error.

{ "Effect": "Allow", "Action": [ "lambda:GetFunction", "lambda:GetCodeSigningConfig" ], "Resource": "*" }

[ewbankkit]

@jbhennin I have no problem in us-east-2:

$ AWS_DEFAULT_REGION=us-east-2 make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSLambdaFunction_codeSigningConfig'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSLambdaFunction_codeSigningConfig -timeout 120m
=== RUN   TestAccAWSLambdaFunction_codeSigningConfig
=== PAUSE TestAccAWSLambdaFunction_codeSigningConfig
=== CONT  TestAccAWSLambdaFunction_codeSigningConfig
--- PASS: TestAccAWSLambdaFunction_codeSigningConfig (42.09s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	42.185s

[brikis98]

I'm getting the same error in ap-northeast-3 and we are not trying to enable code signing at all. This code used to work fine, so something about introducing Lambda code signing into the AWS Provider has broken Lambda functions in general in ap-northeast-3, whether or not you are trying to use the code signing functionality.

[hostmit]

@brikis98 did you find a solution? Facing the same issue with the same region

[brikis98]

No, we haven't found any solution.

[hostmit]

@brikis98
Code Signing for AWS Lambda is available in all commercial regions except AWS China Regions, AWS GovCloud (US) Regions, and Asia Pacific (Osaka) Region.

https://aws.amazon.com/blogs/aws/new-code-signing-a-trust-and-integrity-control-for-aws-lambda/

Okey now we know why. The question is how do we solve it?

[richardgavel]

@brikis98 It really has nothing to do with whether you have code signing defined in your lambda function. To simply get the current state of the Lambda function, it will have to go get the signing config of the existing function in AWS. Think about it this way, how would you detect drift if something added code signing directly in the console but not in the TF if you couldn't find out if it was on the resource in the first place.

https://github.com/hashicorp/terraform-provider-aws/blob/main/aws/resource_aws_lambda_function.go#L842

Now, it does seem like added AWS permissions maybe should be better documented as a minor breaking change? But I guess then maybe you could say that every time they add a new resource/addition to a resource. But breaking reads is more painful to consumers, since it might impact everyone, vs impacting writes of only those who use the new capability.

[bcsgh]

Is there any way for a provider to ship values? I'm thinking for example a built in map from aws_* data and resource types to the set of IAM permissions they require. That would allow user to define their AMI policies such that any changes in what the provider needs will automatically get inserted the next time the config is applied.

[Sam-Martin]

This is a problem preventing the deployment of lambdas to ap-northeast-3 entirely.
The most trivial example fails with the issue described, even will full admin permissions because the code signing service
(AWS Signer) does not exist in ap-northeast-3. It is not a matter of permissions it is a matter of the service not existing in this region.

Try the following with the latest aws provider:

resource "aws_iam_role" "iam_for_lambda" {
  name = "role-for-ap-northeast-3"
  assume_role_policy = ""
}

resource "aws_lambda_function" "test_lambda" {
  filename      = "lambda_function_payload.zip"
  function_name = "test_function"
  role          = aws_iam_role.iam_for_lambda.arn
  handler       = "handler.lambda_handler"

  runtime = "python3.8"
}

provider "aws" {
  region ="ap-northeast-3"
}

results in:

$ terraform apply
aws_lambda_function.test_lambda: Refreshing state... [id=lambda_function_name]
╷
│ Error: error getting Lambda Function (lambda_function_name) code signing config AccessDeniedException: 
│       status code: 403, request id: 9c6cf779-8ce2-49bd-8ab3-e97e8fa5dd0f
│ 
│   with aws_lambda_function.test_lambda,
│   on main.tf line 6, in resource "aws_lambda_function" "test_lambda":
│    6: resource "aws_lambda_function" "test_lambda" {
│ 

This is not a small unusual region either anymore as Osaka ap-northeast-3 went GA on 2021-03-1 https://aws.amazon.com/blogs/aws/aws-asia-pacific-osaka-region-now-open-to-all-with-three-azs-more-services/

[github-actions]

This functionality has been released in v3.55.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.