aws_appautoscaling_policy resource produced new value for was present but now absent #10549
Comments
ghost
added
service/applicationautoscaling
service/dynamodb
camlow325
changed the title
github-actions
bot
added
the
needs-triage
camlow325
changed the title
|
Small update: we have since updated the Terraform AWS provider version to |
|
We were able to observe an instance of this failure with debug logging enabled during a The A subsequent In other cases where the apply succeeds, the response to the This may suggest an eventual consistency issue on the AWS side. To compensate for this, maybe it would make sense for the |
bflad
added
bug
|
Hi @camlow325 👋 Thank you for the additional details, they will be super helpful for fixing this issue. As you are suggesting and looking at your log output, this is likely an eventual consistency issue within AWS. The current flow of the The quickest fix for this could be introducing additional retry logic in the resource read function to handle the special new resource scenario here: e.g. err := resource.Retry(2*time.Minute, func() *resource.RetryError {
var err error
p, err = getAwsAppautoscalingPolicy(d, meta)
if err != nil {
if isAWSErr(err, applicationautoscaling.ErrCodeFailedResourceAccessException, "") {
return resource.RetryableError(err)
}
return resource.NonRetryableError(err)
}
if d.IsNewResource() && p == nil {
return resource.RetryableError(&resource.NotFoundError{})
}
return nil
})If you or someone else is willing to submit the above, the maintainers can review the potential fix. Thanks! |
camlow325
added a commit
to camlow325/terraform-provider-aws
that referenced
this issue
Dec 10, 2019
References: * hashicorp#10549 The AWS application-autoscaling service has eventual consistency considerations. The `aws_appautoscaling_policy` resource immediately tries to read a scaling policy after creation. If the policy is not found, the application-autoscaling service returns a 200 OK with an empty list of scaling policies. Since no scaling policy is present, the `aws_appautoscaling_policy` resource removes the created resource from state, leading to a "produced an unexpected new value for was present, but now absent" error. With the changes in this commit, the empty list of scaling policies in the response for the newly created resource will result in a NotFoundError being returned and a retry of the read request. A subsequent retry should hopefully be successful, leading to the state being preserved. Output from acceptance testing: ``` > make testacc TEST=./aws TESTARGS='-run=TestAccAWSAppautoScalingPolicy_' ... --- PASS: TestAccAWSAppautoScalingPolicy_multiplePoliciesSameName (34.27s) --- PASS: TestAccAWSAppautoScalingPolicy_dynamoDb (36.59s) --- PASS: TestAccAWSAppautoScalingPolicy_multiplePoliciesSameResource (37.73s) --- PASS: TestAccAWSAppautoScalingPolicy_disappears (68.94s) --- PASS: TestAccAWSAppautoScalingPolicy_basic (72.25s) --- PASS: TestAccAWSAppautoScalingPolicy_scaleOutAndIn (74.31s) --- PASS: TestAccAWSAppautoScalingPolicy_spotFleetRequest (77.85s) --- PASS: TestAccAWSAppautoScalingPolicy_ResourceId_ForceNew (88.97s) ```
|
@bflad Awesome - thanks much for taking a look at this. Your suggested code change makes a lot of sense to me. I went ahead and put up a PR with it. |
bflad
pushed a commit
that referenced
this issue
Dec 20, 2019
…#11222) References: * #10549 The AWS application-autoscaling service has eventual consistency considerations. The `aws_appautoscaling_policy` resource immediately tries to read a scaling policy after creation. If the policy is not found, the application-autoscaling service returns a 200 OK with an empty list of scaling policies. Since no scaling policy is present, the `aws_appautoscaling_policy` resource removes the created resource from state, leading to a "produced an unexpected new value for was present, but now absent" error. With the changes in this commit, the empty list of scaling policies in the response for the newly created resource will result in a NotFoundError being returned and a retry of the read request. A subsequent retry should hopefully be successful, leading to the state being preserved. Output from acceptance testing: ``` > make testacc TEST=./aws TESTARGS='-run=TestAccAWSAppautoScalingPolicy_' ... --- PASS: TestAccAWSAppautoScalingPolicy_multiplePoliciesSameName (34.27s) --- PASS: TestAccAWSAppautoScalingPolicy_dynamoDb (36.59s) --- PASS: TestAccAWSAppautoScalingPolicy_multiplePoliciesSameResource (37.73s) --- PASS: TestAccAWSAppautoScalingPolicy_disappears (68.94s) --- PASS: TestAccAWSAppautoScalingPolicy_basic (72.25s) --- PASS: TestAccAWSAppautoScalingPolicy_scaleOutAndIn (74.31s) --- PASS: TestAccAWSAppautoScalingPolicy_spotFleetRequest (77.85s) --- PASS: TestAccAWSAppautoScalingPolicy_ResourceId_ForceNew (88.97s) ```
|
The fix for this has been merged and will release with version 2.44.0 of the Terraform AWS Provider, early next month. Thanks to @camlow325 for the implementation. 👍 |
|
This has been released in version 2.44.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks! |
|
I'm still seeing this issue with the latest version of terraform and aws provider: module.eks_node_group.aws_iam_role_policy_attachment.ClusterAutoScaler_polattach: Creating... Error: Provider produced inconsistent result after apply When applying changes to This is a bug in the provider, which should be reported in the provider's own $ terraform version
I can reproduce it on demand and will be glad to provide additional debugging info |
|
The fix for this which landed in the It seems possible that the I'd suggest trying to reproduce the failure with TF_LOG enabled and see if the HTTP request pattern for the failure looks similar to the one I mentioned in this comment. I'd also suggest filing a separate issue for the |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Community Note
Terraform Version
Affected Resource(s)
Terraform Configuration Files
Debug Output
Unfortunately, we've only seen this problem 2 times (in automated CI runs) across hundreds of executions and didn't have debug on at the time. We also haven't been able to reproduce this on-demand yet and so don't have a way to get this output currently.
Here's a subset of the normal apply output that we saw:
Panic Output
No panic.
Expected Behavior
Both of the
aws_appautoscaling_policyresources in the config should be applied successfully.Actual Behavior
On a very infrequent basis - only 2 times across hundreds of runs for us so far - the apply results in an error for either the
aws_appautoscaling_policy.dynamodb_readoraws_appautoscaling_policy.dynamodb_writeresource. Even when the error occurs, the policy resource is actually created properly in the AWS account. If another apply is attempted after the failure, the apply is successful.Steps to Reproduce
terraform applyaws application-autoscaling describe-scaling-policies --service-namespace dynamodb --policy-names <policy_name>and confirm that the policy is present.terraform applyagain. Observe that the second apply is successful.Important Factoids
As mentioned above, the problem is highly intermittent. We're running on a bit older version of Terraform and the AWS provider at present in CI but hope to update soon and will post back here if we continue to encounter the issue.
All of the config around the DynamoDB table is listed above just in case that helps provide more context for what might be necessary to reproduce the problem - sorry for the size. In addition to updating to newer versions of Terraform and the AWS provider, we're going to attempt to make the following config changes to see if these help eliminate the problem:
aws_appautoscaling_targetresources. We only have theignore_changesblock forrole_arnargument because AWS appears to apply its own role to the resources anyway, as I believe is described in aws_appautoscaling_target role_arn is ignored or doesn't work #5023.aws_appautoscaling_policyresources - on the theory that having the resources applied in parallel might lead to the problem.The text was updated successfully, but these errors were encountered: