[WIP] Wait for Hyperplane-attached ENIs for lambdas to move to 'available' state before detaching

[ewbankkit]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Closes #10044

Release note for CHANGELOG:

resources/aws_security_group: Increase default delete timeout to 30 minutes. Wait for Hyperplane-attached ENIs for lambdas to move to 'available' state before attempting to delete lingering ENIs.
resources/aws_subnet: Increase default delete timeout to 30 minutes. Wait for Hyperplane-attached ENIs for lambdas to move to 'available' state before attempting to delete lingering ENIs.

Output from acceptance testing:

$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSLambdaFunction_VPC_withInvocation'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -parallel 20 -run=TestAccAWSLambdaFunction_VPC_withInvocation -timeout 120m
=== RUN   TestAccAWSLambdaFunction_VPC_withInvocation
=== PAUSE TestAccAWSLambdaFunction_VPC_withInvocation
=== CONT  TestAccAWSLambdaFunction_VPC_withInvocation
--- PASS: TestAccAWSLambdaFunction_VPC_withInvocation (195.10s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	195.166s
$ AWS_DEFAULT_REGION=eu-west-1 make testacc TEST=./aws TESTARGS='-run=TestAccAWSLambdaFunction_VPC_withInvocation'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -parallel 20 -run=TestAccAWSLambdaFunction_VPC_withInvocation -timeout 120m
=== RUN   TestAccAWSLambdaFunction_VPC_withInvocation
=== PAUSE TestAccAWSLambdaFunction_VPC_withInvocation
=== CONT  TestAccAWSLambdaFunction_VPC_withInvocation
--- FAIL: TestAccAWSLambdaFunction_VPC_withInvocation (1288.36s)
    testing.go:630: Error destroying resource! WARNING: Dangling resources
        may exist. The full state and error is shown below.
        
        Error: errors during apply: 2 problems:
        
        - error deleting Lambda ENIs using subnet (subnet-0fae3a7fbe012c060): error waiting for ENI (eni-0606e238685023d8f) to become available: timeout while waiting for state to become 'available, deleted' (last state: 'available', timeout: 20m0s)
        - error deleting Lambda ENIs using Security Group (sg-0f12870b552344394): error waiting for ENI (eni-0606e238685023d8f) to become available: timeout while waiting for state to become 'available, deleted' (last state: 'available', timeout: 20m0s)
...
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	1288.465s
make: *** [testacc] Error 1

For some reason the new delete timeout of 30 minutes for both security groups and subnets is not being respected and even though the ENI has entered available state we are still ensuring that we get a number of availables in a row to work around any API consistency issues and hit the default 20 minute timeout.
Maybe a Terraform 0.12 issue with Timeout blocks and MigrateState functions?

[ewbankkit]

Terraform 0.11.10:

"resources": {
    "aws_security_group.example": {
        "type": "aws_security_group",
        "primary": {
            "meta": {
                "e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
                    "create": 600000000000,
                    "delete": 1800000000000
                },
                "schema_version": "1"
            },
        }
    }
}

Terraform 0.12.8:

"resources": [{
    "mode": "managed",
    "type": "aws_security_group",
    "name": "example",
    "instances": [
    {
        "schema_version": 1,
        "attributes": {
        "timeouts": null,
        }
    }
    ]
}]

The timeouts are being lost.
This doesn't seem to be related to the schema version - For resources with the default schema version timeouts is also being persisted as null.

Terraform Core issue: hashicorp/terraform#22810.

[obourdon]

The 2 funcs networkInterfaceAttachmentStateRefresh and networkInterfaceStateRefresh share over 90% of their code. Wouldn't it be nicer to have a single function which could be parametrised ?

[ewbankkit]

OK, after fixing a bug and incorporating the code changes from hashicorp/terraform#22837 I am now getting success in us-west-2, us-west-1 and us-central-1:

$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSLambdaFunction_VPC_withInvocation'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -parallel 20 -run=TestAccAWSLambdaFunction_VPC_withInvocation -timeout 120m
=== RUN   TestAccAWSLambdaFunction_VPC_withInvocation
=== PAUSE TestAccAWSLambdaFunction_VPC_withInvocation
=== CONT  TestAccAWSLambdaFunction_VPC_withInvocation
--- PASS: TestAccAWSLambdaFunction_VPC_withInvocation (66.55s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	66.576s
$  AWS_DEFAULT_REGION=eu-west-1 make testacc TEST=./aws TESTARGS='-run=TestAccAWSLambdaFunction_VPC_withInvocation'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -parallel 20 -run=TestAccAWSLambdaFunction_VPC_withInvocation -timeout 120m
=== RUN   TestAccAWSLambdaFunction_VPC_withInvocation
=== PAUSE TestAccAWSLambdaFunction_VPC_withInvocation
=== CONT  TestAccAWSLambdaFunction_VPC_withInvocation
--- PASS: TestAccAWSLambdaFunction_VPC_withInvocation (1294.06s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	1294.082s
$ AWS_DEFAULT_REGION=eu-central-1 make testacc TEST=./aws TESTARGS='-run=TestAccAWSLambdaFunction_VPC_withInvocation'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -parallel 20 -run=TestAccAWSLambdaFunction_VPC_withInvocation -timeout 120m
=== RUN   TestAccAWSLambdaFunction_VPC_withInvocation
=== PAUSE TestAccAWSLambdaFunction_VPC_withInvocation
=== CONT  TestAccAWSLambdaFunction_VPC_withInvocation
--- PASS: TestAccAWSLambdaFunction_VPC_withInvocation (1289.16s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	1289.180s

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!