0.13 count/for_each cannot be determined while destroying even with an empty state

[zachwhaley]

Hello, we are experimenting with Terraform 0.13 and seeing an odd behavior where a destroy errors out because it reportedly cannot determine the count/for_each for a resource even though they are straight variables, and this seems to happen after the resources has been destroyed.

╭ ~/work/infra/rnd/workspaces/zacharyw (tf-v0.13>)
╰ → tf13 show

╭ ~/work/infra/rnd/workspaces/zacharyw (tf-v0.13>)
╰ → tf13 plan -destroy -out tfplan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.aws_availability_zones.available: Refreshing state...
module.db.data.aws_availability_zones.available: Refreshing state...
module.mock_ec2.data.aws_ami.amazon-linux-2: Refreshing state...
data.aws_caller_identity.current: Refreshing state...
data.aws_iam_role.ecs_task_role: Refreshing state...
module.ips_event.data.aws_iam_role.integration_role: Refreshing state...
module.vpc.data.aws_ami.ubuntu: Refreshing state...
...
...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:

Terraform will perform the following actions:

Plan: 0 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

╭ ~/work/infra/rnd/workspaces/zacharyw (tf-v0.13>)
╰ → tf13 apply tfplan

Error: Invalid count argument

  on .terraform/modules/vpc/ec2/main.tf line 14, in resource "aws_instance" "this":
  14:   count = var.create ? var.instance_count : 0

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.


Error: Invalid for_each argument

  on .terraform/modules/vpc/vpc/main.tf line 63, in resource "aws_subnet" "public":
  63:   for_each = var.public_subnets

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.


Error: Invalid for_each argument

  on .terraform/modules/vpc/vpc/main.tf line 84, in resource "aws_subnet" "private":
  84:   for_each = var.private_subnets

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

Terraform Version

$ tf13 version
Terraform v0.13.0
+ provider registry.terraform.io/hashicorp/archive v1.3.0
+ provider registry.terraform.io/hashicorp/aws v2.70.0
+ provider registry.terraform.io/hashicorp/template v2.1.2

Terraform Configuration Files

The subnet vars in the example above are declared like so

  public_subnets = {
    for i in range(local.pub_subnet_count) : "${data.aws_availability_zones.available.names[i]}" => cidrsubnet(local.vpc_cidr, 8, i)
  }
  private_nat_subnets = {
    for i in range(local.nat_subnet_count) : "${data.aws_availability_zones.available.names[i]}" => cidrsubnet(local.vpc_cidr, 8, i + local.pub_subnet_count)
  }
  private_subnets = {
    for i in range(local.pvt_subnet_count) : "${data.aws_availability_zones.available.names[i]}" => cidrsubnet(local.vpc_cidr, 8, i + local.pub_subnet_count + local.nat_subnet_count)
  }

I found the state file in question:

{
  "version": 4,
  "terraform_version": "0.13.0",
  "serial": 6,
  "lineage": "4028536c-5689-9965-d797-10827b7f5430",
  "outputs": {
    "api_dns": {
      "value": "",
      "type": "string"
    },
    "db_hostname": {
      "value": "",
      "type": "string"
    },
    "db_port": {
      "value": <REDACTED>,
      "type": "number"
    },
    "ips_event_endpoint": {
      "value": "",
      "type": "string"
    },
    "ips_lb_dns": {
      "value": "",
      "type": "string"
    }
  },
  "resources": [
    {
      "module": "module.vpc",
      "mode": "managed",
      "type": "aws_subnet",
      "name": "private_nat",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": []
    },
    {
      "module": "module.vpc",
      "mode": "managed",
      "type": "aws_vpc",
      "name": "this",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": []
    }
  ]
}

Debug Output

Crash Output

Expected Behavior

Nothing errors

Actual Behavior

Errors occur

Steps to Reproduce

1. Using a remote state, run terraform apply
2. Erase local .terraform directory
3. Run terraform init
4. Run terraform destroy

I've noticed this does not happen if I do a straight apply followed by a destroy

Additional Context

References

[ghost]

In my opinion, this bug is absolute CRITICAL

In my code base, i always run in this when a resource with count/for_each using attributes from a other resource or module output which is created with count/for_each.

This bug was alredy mentioned at #25815 but the thread opener gives up and closed it without a solution.

Because @danieldreier answered in the closed ticket: Please take attention to this bug. Terraform 0.13 is not usable anymore with this if you have a mid-complex code structure.

[danieldreier]

@jbardin if I'm understanding this correct, I think what's happening is that now that data sources are evaluated as part of the graph, functions that reference data sources return a "cannot be determined until apply" error. Am I understanding it right? If so, this does indeed seem like it would have a significant impact.

[jbardin]

I don't think this is specifically because of data sources, running destroy with an empty state is definitely and edge case that is not well tested. I don't have a full reproduction for this at the moment, but there's a good chance it will be covered by the fix for #25891.

[zachwhaley]

running destroy with an empty state is definitely and edge case

To clarify this happened when destroying a complete state, and then errorred after that state had been destroyed, subsequent commands to destroy fail for the same reason though.

[JanKoppe]

I can confirm this bug. It's even happening while not changing anything. It's completely non-deterministic behaviour in our case. If I go ahead and target just the depending resource, sometimes it works after one targeted run, sometimes only after a few, sometimes never.

This is really bad and breaking some critical code for us.

[jbardin]

While terraform should be able to run destroy with an empty state, and that is definitely a bug, the responses here indicate that this is a critical workflow for some. Does someone have an example of how running destroy with no resources is being used regularly?

[JanKoppe]

In our case it's not happening specifically in a destroy run, but in every run - plan, apply, destroy. This is code that has been working fine in 0.12.

I've read this comment #25815 (comment) which hints that maybe the code have is just "bad", but has been working until now. I'm currently trying to fix these instances by moving them over to for_each instead of count + length.

Will report back on how this is working out.

In my current case, I'm completely stuck at this error:

Error: Invalid count argument

  on ../../modules/scp-attachment/main.tf line 20, in resource "aws_organizations_policy_attachment" "policy_attachment":
  20:   count     = length(data.terraform_remote_state.master.outputs.project_policy_ids)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

For some errors, I was able to workaround this by running the suggested -target, but in this case of a datasource this is not possible, hence we probably are forced to rewrite this code.

[JanKoppe]

No luck. 👎

Error: Invalid for_each argument

  on ../../modules/scp-attachment/main.tf line 20, in resource "aws_organizations_policy_attachment" "policy_attachment":
  20:   for_each  = toset(data.terraform_remote_state.master.outputs.project_policy_ids)

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

[ghost]

@jbardin For me its the behaviour from what @JanKoppe said.
I have this not only on destroy, i have this even when i run plan or apply.

In the comment #25815 @danieldreier said the code is maybe bad, but its exactly how all the Terraform modules from Microsoft are designed. See: https://github.com/Azure/terraform-azurerm-compute (and honestly, my private modules also, because there is no other way when you run different stages with the same code base where you have different amounts of vms etc.). The network cards here are build with a count attribute and returned in the output.tf as a list: https://github.com/Azure/terraform-azurerm-compute/blob/master/outputs.tf

In the AzureRM Provider its very common that you have to connect two resources together over their id like the network_interface_application_security_group_association or the virtual_machine_data_disk_attachment. And i think its a very common usecase that you have different amounts of vms, disk, or network cards for each stage/workspace with the same code base. So if you use a assocation like in #25815 where cound depends on the length of the list, which is returnd by the compute module:

resource "azurerm_network_interface_application_security_group_association" "vms_to_asg" {
  count                         = length(module.compute_vm.network_interface_ids)
  network_interface_id          = module.compute_vm.network_interface_ids[count.index]
  application_security_group_id = azurerm_application_security_group.asg.id
}

Then on my side i run in this problem. Using for_each change nothing on that.

[jbardin]

I believe a number of different issues are being confused here. This particular issue is about being able to run destroy with an empty state. This will hopefully be remedied by the same fix as #25891, and can be closed once we confirm.

Any errors arising during plan or apply are very different issues, and not relevant here. If the count or for_each values are derived from data sources, it's likely that the root cause will be helped by #25812, but there are probably some data sources that still need updates from their respective providers to fix broken schemas as well.

[ghost]

@jbardin Maybe i understand this wrong, but with data sources you meen a data source block? Because the output from network cards from the example are just a list of objects given by the attribute of the resource? The is no data {} block in this case or?

If you think this is not the same case (even its the same exception message), i think we should reopen #25815

[zachwhaley]

To add to the discussion, I opened this issue about destroy because that was the one that I could reproduce reliably. But I am also having the same "cannot determine count" issues with other actions like apply as well.

[nicklofaso]

I believe I am having a very similar issue to @zachwhaley.

@jbardin I think @zachwhaley's issue was not that destroy on an empty state failed, but that destroy of a successfully deployed module errors (#25851 (comment)). Please correct me if I am wrong.

I have a very simple project that uses the Google GKE module https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/11.0.0.

After successfully running apply, I run destroy and all of the resources successfully delete, but the command still fails with

Error: Invalid count argument

  on .terraform/modules/gke.gcloud_delete_default_kube_dns_configmap/terraform-google-gcloud-1.4.1/main.tf line 57, in resource "random_id" "cache":
  57:   count = (! local.skip_download) ? 1 : 0

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

[kingsleyadam]

I'm also running into this exact same issue. It's completely broken one of my terraform plans. Worked without issues in 0.12.29.

I'm running a similar for_each on other resources and they don't seem to be affected.

[DarkMukke]

like @kingsleyadam said but with an example

data "aws_subnet_ids" "bastion" {
  vpc_id = data.aws_vpc.bastion.id
}

data "aws_subnet" "bastion" {
  count = length(data.aws_subnet_ids.bastion.ids)
  id = element(sort(data.aws_subnet_ids.bastion.ids), count.index)
}

or with a dirty zipmap and foreach

data "aws_subnet_ids" "bastion" {
  vpc_id = data.aws_vpc.bastion.id
}

locals {
  bastion_subnets = zipmap(data.aws_subnet_ids.bastion.ids, data.aws_subnet_ids.bastion.ids)
}

data "aws_subnet" "bastion" {
  for_each = local.bastion_subnets
  id = each.value
}

worked fine in 0.12.29, after upgrading to 0.13 it broke

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

The suggest work around has no effect however because even if i run the apply with a -target on the data blocks, it is re-read on every apply/refresh/plan

[kingsleyadam]

Here's more detailed information on my setup and where it fails.

Part of our terraform plan is to attach a server to multiple load balancers in Azure.

We're speciyfing the load balancer as a direct variable: var.load_balancers = load_balancers = ["apiserver-internal", "apiserver"]

data "azurerm_lb" "load_balancers" {
  count               = length(var.load_balancers)
  name                = "${var.lb_prefix}-${var.realm}-${var.load_balancers[count.index]}"
  resource_group_name = var.resource_group.name
}
data "azurerm_lb_backend_address_pool" "lb_backend_address_pools" {
  for_each = {
    for index, value in data.azurerm_lb.load_balancers : index => value
  }
  name            = "BackEndAddressPool"
  loadbalancer_id = each.value.id
}

Worked in 0.12.29, after upgrading to 0.13.0 we get:

Error: Invalid for_each argument

  on modules/virtual_machines/data.tf line 30, in data "azurerm_lb_backend_address_pool" "lb_backend_address_pools":
  30:   for_each = {
  31:     for index, value in data.azurerm_lb.load_balancers : index => value
  32:   }

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

Same as @DarkMukke, even if I -target=data.azurerm_lb.load_balancers it doesn't help. On targeted apply it reads in the data resource, when I re-run it throws the same error.

Hopefully a fix is found ASAP.

[zachwhaley]

This appears to have been fixed in 0.13.1

I'm no longer seeing issues during destroy.

[bharathkkb]

@nicklofaso the 0.13.1 seems to have fixed the error you were encountering for the GKE module as well.

[remche]

Fixed at first apply operation, but I still have error when reapplying.
And trying to switch to for_each way got me the error at first apply.

[jbardin]

Thanks for the confirmation @zachwhaley!

[kingsleyadam]

I don't think this issue is completely fixed. May need to open a new one that covers more than just destroy.

Multiple people have confirmed that it's happening on apply as well as destroy.

I'm not able to test since 0.13.1 broke my terraform setup completely. Once 0.13.2 is out I can confirm and open a new issue.

[kingsleyadam]

Confirming on 0.13.2 that this is still an issue for me. I can't apply my plan, I continue to get.

Error: Invalid for_each argument

  on modules/virtual_machines/data.tf line 30, in data "azurerm_lb_backend_address_pool" "lb_backend_address_pools":
  30:   for_each = {
  31:     for index, value in data.azurerm_lb.load_balancers : index => value
  32:   }

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

[nicklofaso]

@nicklofaso the 0.13.1 seems to have fixed the error you were encountering for the GKE module as well.

Thanks you are right. My issue is fixed in 0.13.1!

[kingsleyadam]

I was able to resolve my issues so I'm posting in case someone else comes across this. The error is very misleading, I was under the impression that terraform couldn't come up with the list of items to be created based on my for_each.

...Terraform cannot predict how many instances will be created....

The real issue was that I was using an unknown value for they key of the for each. I was using the id of the resource that was to be created, which it didn't know during apply.

The fix was to use my own configured values for the key, now all my issues are resolved.

This did work in 0.12.x. Hope this helps someone.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.