Missing aws_appautoscaling_policy if aws_ecs_service got re-created

[abriel]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

It is about relationship between aws_ecs_service and aws_appautoscaling_policy.

Prerequisites:
We have a template with aws_ecs_service, aws_appautoscaling_target, aws_appautoscaling_policy and aws_alb_target_group. The last one does not actually matter.

Action:
We produce any change that requires to re-create aws_ecs_service.
For example, update protocol parameter of aws_alb_target_group which causes its re-creation.
Re-creation of aws_alb_target_group causes re-creation of aws_ecs_service, if aws_ecs_service is configured with load_balancer block where it has reference to aws_alb_target_group.

Current behaviour:
aws_ecs_service got re-created and aws_appautoscaling_policy resources are lost.
Running the terraform apply once again creates missed aws_appautoscaling_policy resources.

Wanted behavior:
aws_appautoscaling_policy resources created again along with new aws_ecs_service.

Tried workarounds:
I tried to taint aws_appautoscaling_policy resources, but usually get an exception like the following:
* aws_appautoscaling_policy.cpu_down: Failed to delete scaling policy: ObjectNotFoundException: No scaling policy found for service namespace: ecs, resource ID: service/operations-dev/operations-dev-ssl, scalable dimension: ecs:service:DesiredCount, policy name: operations-dev-ssl-autoscale-cpu-down
It seems terraform tries to destroy the resource aws_appautoscaling_policy when destruction for the resource aws_ecs_service has been completed.

New or Affected Resource(s)

• aws_ecs_service
• aws_appautoscaling_policy

Potential Terraform Configuration

https://www.dropbox.com/s/ippuymh9143jvun/terraform-provider-aws-issue.zip?dl=0

During testing you may meet another issue with re-creation of aws_alb_target_group: It cannot be deleted, because is used by aws_alb_listener_rule.
I had to taint aws_alb_listener_rule.service_routing resource before apply a plan.

[AlexanderIakovlev]

@radeksimko could you please help with workaround for this problem?
I've tried:

resource "aws_appautoscaling_target" "autoscaling_target" {
  max_capacity = "${var.max_capacity}"
  min_capacity = "${var.min_capacity}"

  resource_id = "service/${data.terraform_remote_state.services.ecs_cluster}/${module.service.service_name}"

  role_arn           = "${var.ecs_service_scaling_arn}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
  depends_on         = ["module.service"]
}

So, i have explicit depends_on and output module.service.service_name which is equal to

output "service_name" {
  value = "${aws_ecs_service.main.name}"
}

aws_appautoscaling_target will be added only after second try

[bflad]

Hi folks 👋 Sorry for the unexpected behavior here. It turns out there was a very subtle bug that prevented proper recreation of Application AutoScaling Policies when only the resource_id argument changed. We also now properly ignore ObjectNotFoundException errors during Terraform resource destroy. A full writeup of these changes can be found in #7982. These fixes will be released in version 2.3.0 of the Terraform AWS Provider in the next day or two.

Even without upgrading your Terraform AWS Provider to the newer version, you may be able to workaround the original issue with a simple configuration update. Using direct references to the relevant aws_appautoscaling_target resource for the resource_id, scalable_dimension, and service_namespace arguments in the aws_appautoscaling_policy resource configuration should be enough in many cases to ensure Terraform has proper ordering information to handle recreations of the Application AutoScaling Policy resource when the underlying Application AutoScaling Target or resource associated with the target changes:

resource "aws_appautoscaling_target" "example" {
  # ... other configuration ...
}

resource "aws_appautoscaling_policy" "example" {
 # ... other configuration ...

  resource_id        = "${aws_appautoscaling_target.example.resource_id}"
  scalable_dimension = "${aws_appautoscaling_target.example.scalable_dimension}"
  service_namespace  = "${aws_appautoscaling_target.example.service_namespace}"
}

If you are still having trouble after upgrading to version 2.3.0 of the Terraform AWS Provider (when its released) and with a configuration looking similar to the above, please create a new GitHub issue with the relevant details from the issue template and we can further triage. Thanks!

[bflad]

This has been released in version 2.3.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!