Circular dependency when provisioner for destroy references array member for AWS

[artur-fijalkowski]

To resolve problem with disk attachment on AWS (disk attachement cannot be destroyed when disk is mounted on OS level) I created following code:

resource "aws_volume_attachment" "attachement_myservice" {
  count			=	"${length(var.network_myservice_subnet_ids)}"
  device_name   = "/dev/xvdg"
  volume_id     =   "${element(aws_ebs_volume.ebs_myservice.*.id, count.index)}"
  instance_id   =   "${element(aws_instance.myservice.*.id, count.index)}"

  provisioner "local-exec" {
    command = "aws ec2 stop-instances --instance-ids ${element(aws_instance.myservice.*.id, count.index)} --region ${var.region} && sleep 30"
    when = "destroy"
  }
}

When trying to rebuild instances (because AMI instances depend on) terraform reports circular dependency for this code. Experiment shows that the problem is with shell command - when reference ${element(aws_instance.myservice.*.id, count.index)} is replaced with something different terraform works like a charm (production solution for me is to use AWS CLI to first find instance IDs and then use them to stop instances).

Most probably root cause is wrong interpretation of interpolation of array member when building graph for rebuild.

Problem doesn't exist when terraform destroy is executed - only on terraform apply which is expected to re-build instances.

Terraform Version

terraform -v
Terraform v0.11.3

Your version of Terraform is out of date! The latest version
is 0.11.7. You can update by downloading from www.terraform.io/downloads.html

Terraform Configuration Files

...

Debug Output

Crash Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Additional Context

References

[jbardin]

Hi @artur-fijalkowski,

Thanks for filing this.
Is it possible for you to get the debug log output when you encounter the cycle? I haven't been able to reproduce this yet, though I may have oversimplified things a bit.

Also, have you tested this with the current release? There were a few internal changes to the evaluation of destroy provisioners recently which may have indirectly fixed this.

[artur-fijalkowski]

@jbardin

I will try to do it. The reason I haven't yet is that this is part of bigger solution and I had no time to extract this for testing. But If you can't reproduce I have to do it anyway.

[jbardin]

Hi @artur-fijalkowski,

I had some more time to experiment and expanded this out to reproduce the cycle.
Thanks!

[jbardin]

A minimal test case for investigation:

resource "null_resource" "a" {
  triggers = {
    "b" = "${null_resource.b.id}"
  }

  provisioner "local-exec" {
    command = "echo ${null_resource.b.id}"
    when = "destroy"
  }
}

resource "null_resource" "b" {
}

Tainting null_resource.b will trigger the cycle, because null_resource.a references it in its configuration as well as in the destroy provisioner.

* Cycle: null_resource.b (destroy), null_resource.b, null_resource.a (destroy)

[artur-fijalkowski]

@jbardin exactly this is simplified to maximum my case ;)

I guess that root cause is that this destroy provisioner is should not enforce dependency on resources referenced in it. Because in general this is what it is for - to remove this dependency prior to destruction ;)

[jbardin]

An interesting workaround here is adding create_before_destroy to the affected resource.

The reason for the cycle is that normally destroy dependencies flow in reverse. Node a is created before b, so b is destroyed before a. The problem arrises when you add a destroy provisioner, which is an action that depends the references existing, so the dependencies of the provisioner flow in the opposite direction of the destroy.

The dependencies showing the cycle look like:

And with create_before_destroy, the graph is is converted to:

We'll have to determine how we want to handle this after the next major release. I'm not sure if there's a good solution other than inverting the order, but at least we know it can be detected and reported fairly clearly to the user.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.