The "count" value depends on resource attributes that cannot be determined until apply

[eugeneromero]

Hello!

I have the following Terraform code inside of a module:

data "kubectl_path_documents" "esCustomResourcesManifests" {
  pattern = "modules/elasticsearch/all-in-one-1.3.0.yaml"
  disable_template = true
}

resource "kubectl_manifest" "esCustomResources" {
  count     = length(data.kubectl_path_documents.esCustomResourcesManifests.documents)
  yaml_body = element(data.kubectl_path_documents.esCustomResourcesManifests.documents, count.index)
}

When running terraform plan, I get:

Error: Invalid count argument

  on modules/elasticsearch/elasticSearch.tf line 24, in resource "kubectl_manifest" "esCustomResources":
  24:   count     = length(data.kubectl_path_documents.esCustomResourcesManifests.documents)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

This does not seem to be the same as issue #58, since I am not pulling any external variables. Is there something obvious I am missing?

Versions:
Terraform v0.13.5
provider registry.terraform.io/gavinbunney/kubectl v1.9.1

YAML file:
https://download.elastic.co/downloads/eck/1.3.0/all-in-one.yaml

Thanks in advance for your help!

[sathish-kumar-narayanan]

Any updates on this issue ? Am facing similar problem, this happens only if it is used inside a module

[gavinbunney]

Hi @eugeneromero my guess is the pathing isn't quite right as you need to have the ${path.module}/ as a prefix for the pattern path (obviously fixing up the relative path to that file)... so try something like this:

data "kubectl_path_documents" "esCustomResourcesManifests" {
  pattern = "${path.module}/modules/elasticsearch/all-in-one-1.3.0.yaml"
  disable_template = true
}

resource "kubectl_manifest" "esCustomResources" {
  count     = length(data.kubectl_path_documents.esCustomResourcesManifests.documents)
  yaml_body = element(data.kubectl_path_documents.esCustomResourcesManifests.documents, count.index)
}

[sunsingerus]

Hi @gavinbunney hit the same issue:

data "kubectl_path_documents" "dashboard-manifests" {
  pattern = "${path.module}/yaml/dashboard.yaml"
}

resource "kubectl_manifest" "dashboard" {
  count     = length(data.kubectl_path_documents.dashboard-manifests.documents)
  yaml_body = element(data.kubectl_path_documents.dashboard-manifests.documents, count.index)
}

Error in resource "kubectl_manifest" "dashboard":

   6:   count     = length(data.kubectl_path_documents.dashboard-manifests.documents)

The "count" value depends on resource attributes that cannot be determined

[sunsingerus]

kubectl provider is the latest one, 1.10.0

[sunsingerus]

this happens only if it is used inside a module

@gavinbunney Confirm, in my case it does not work inside a module

[rguichard]

I think we hit the same issue with a for_each inside a module. I can confirm that it works outside a module but not inside.

kubectl 1.10.0
terraform 0.14.7

locals {
  apply = [for v in data.kubectl_file_documents.apply.documents : {           
    data : yamldecode(v)                                                         
    content : v                                                                  
    }                                                                            
  ] 
}
data "kubectl_file_documents" "apply" {                                                                           
  content = data.flux_install.main.content                                    
}                                                                                
                                                                                 
# Apply manifests on the cluster                                                 
resource "kubectl_manifest" "apply" {                                            
  for_each   = { for v in local.apply : lower(join("/", compact([v.data.apiVersion, v.data.kind, lookup(v.data.metadata, "namespace", ""), v.data.metadata.name]))) => v.content }
  depends_on = [kubernetes_namespace.fluxv2]                                     
  yaml_body  = each.value                                                        
} 

Error: Invalid for_each argument

  on .terraform/modules/addons/modules/scaleway/fluxv2.tf line 72, in resource "kubectl_manifest" "apply":
  72:   for_each   = { for v in local.apply : lower(join("/", compact([v.data.apiVersion, v.data.kind, lookup(v.data.metadata, "namespace", ""), v.data.metadata.name]))) => v.content }

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

Using -target to target the Kubernetes namespace solves the problem but we'd like to run only one apply for the whole state.

[cloudoutloud]

There is a workaround is this not the same as #58

Although not ideal

[DanTulovsky]

I don't think this is the same. As mentioned in the original post, there are no variables being set here. I am hitting the same problem inside a module:

data "kubectl_path_documents" "manifests" {
  pattern = "${path.module}/yaml/k8s/*.yaml"
}

resource "kubectl_manifest" "vector-yaml" {
  count     = length(data.kubectl_path_documents.manifests.documents)
  yaml_body = element(data.kubectl_path_documents.manifests.documents, count.index)
}

gives:

Error: Invalid count argument

  on modules/vector/main.tf line 29, in resource "kubectl_manifest" "vector-yaml":
  29:   count     = length(data.kubectl_path_documents.manifests.documents)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

% terraform version                                                                                               
Terraform v0.14.7
+ provider registry.terraform.io/gavinbunney/kubectl v1.10.0
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/google v3.58.0
+ provider registry.terraform.io/hashicorp/google-beta v3.58.0
+ provider registry.terraform.io/hashicorp/helm v2.0.2
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.3
+ provider registry.terraform.io/hashicorp/kubernetes-alpha v0.2.1
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/mongey/kafka v0.2.12

[yongzhang]

I have the same issue and couldn't find any solution so far.

[bakayolo]

Having the same issue but with kubect_file_documents

data "kubectl_file_documents" "manifests" {
  content = file("${path.module}/manifests/metrics_server.yaml")
}

resource "kubectl_manifest" "metrics_server" {
  count     = length(data.kubectl_file_documents.manifests.documents)
  yaml_body = element(data.kubectl_file_documents.manifests.documents, count.index)
}

Resulting in

 6:   count     = length(data.kubectl_file_documents.manifests.documents)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

[msevastian]

same issue

  50:   count     = length(data.kubectl_path_documents.manifests.documents)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

[arodus]

I'm experiencing the same issue with the code found here: https://github.com/fluxcd/terraform-provider-flux/blob/main/examples/github/main.tf.
Everything works fine but as soon as the code is put into a module planning fails with the following error:

local.sync will be known only after apply
 
The "for_each" value depends on resource attributes that cannot be
determined until apply, so Terraform cannot predict how many instances will
be created. To work around this, use the -target argument to first apply
only the resources that the for_each depends on.

[shyblower]

The same problem manifested itself here today. Yesterday and before it was working fine.

[Arsen-Uulu]

I had the same issue with vars. You can't reference asg_name = aws_autoscaliing_group.foo.name.

[theothermike]

Having same issue, only inside a module. TF 0.15.3 and 1.10.0 provider

[theothermike]

FYI I have a hackey workaround.

Basically, I create a submodule that is its own state, that all it does it use the flux provider to generate the Manifest YAML files, and generate some terraform source code that I use in the main flux module to install. This means since I am generating terraform source, the list of manifests is well known and wont throw this error. here's some snippets (repeat same code for sync). This ought to be workable for any manifests loaded with kubectl_file_documents

# Flux
data "flux_install" "main" {
  target_path    = local.target_path
  network_policy = false
}

data "kubectl_file_documents" "install" {
  content = data.flux_install.main.content
}

# Convert documents list to include parsed yaml data
locals {
  install = [for v in data.kubectl_file_documents.install.documents : {
    data : yamldecode(v)
    content : v
    }
  ]
  install_filenames = [for v in local.install : format("%s.yaml", replace(lower(join("/", compact([v.data.apiVersion, v.data.kind, lookup(v.data.metadata, "namespace", ""), v.data.metadata.name]))), "/", "-"))]
}

# generate the install manifest files
resource "local_file" "install" {
  for_each             = { for v in local.install : lower(join("/", compact([v.data.apiVersion, v.data.kind, lookup(v.data.metadata, "namespace", ""), v.data.metadata.name]))) => v.content }
  filename             = "${path.module}/../manifests/${var.environment}/${replace(each.key, "/", "-")}.yaml"
  content              = each.value
  file_permission      = "0644"
  directory_permission = "0755"
}

// generate TF that holds the list of manifests to work around https://github.com/gavinbunney/terraform-provider-kubectl/issues/61
resource "local_file" "install-tfcode" {
  filename             = "${path.module}/../flux_install_manifests-${var.environment}.tf"
  content              = <<TF
locals {
  flux_install_manifests_${var.environment} = [
    "${join("\",\n\t\"", local.install_filenames)}"
  ]
}
TF
  file_permission      = "0644"
  directory_permission = "0755"
}

then in the main flux module i can reference these YAML and locals

resource "kubectl_manifest" "install" {
  for_each   = var.environment == "development" ? toset(local.flux_install_manifests_development) : var.environment == "staging" ? toset(local.flux_install_manifests_staging) : var.environment == "production" ? toset(local.flux_install_manifests_production) : toset(local.flux_install_manifests_development)
  depends_on = [kubernetes_namespace.flux_system]
  yaml_body  = file("${path.module}/manifests/${var.environment}/${each.value}")
}

This will also serve as a nice way to upgrade the flux manifests when there is a new provider - just re-generate these files, and run terraform apply in the actual state

[vfrans]

Exact same problem here. Was working ysterday and now it fails with:

 The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the count depends on.

My code:

data "kubectl_path_documents" "kafka-cluster-manifests" {
  pattern = "${path.module}/kafka-cluster/*.yaml"
  disable_template = true
}
resource "kubectl_manifest" "kafka-cluster" {
  count     = length(data.kubectl_path_documents.kafka-cluster-manifests.documents)
  yaml_body = element(data.kubectl_path_documents.kafka-cluster-manifests.documents, count.index)
  wait = true
}

TF version:

francois@Francoiss-Mac-mini evs-ipdvia-aws % terraform version
Terraform v0.15.5
on darwin_arm64
+ provider registry.terraform.io/gavinbunney/kubectl v1.11.1
+ provider registry.terraform.io/hashicorp/aws v3.44.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.1.2
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.4
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

[vfrans]

• exact same behaviour when I swap kubectl_path_documents to kubectl_filename_list

@gavinbunney are we doing something wrong? It seems to me like I really took the configration straight out of the documentation.

On top of that, it worked but now, with a clean state, terraform does not work because of that count...

[jaimehrubiks]

Same issue here... Only happens if inside a module

I think I will put a file called "yaml_count.txt" and read it with tonumber(file("${module.path}/.../yaml_count.txt")) and then count all the individual yaml files and put the number there :) (until this is fixed!)

[hsalluri259]

Hi, I get the same error if I do terraform apply with no existing cluster. If I have a cluster then I don't get this error.
It would be nice if terraform knows this to run with no existing cluster.

data "kubectl_path_documents" "controller_manifests" {
pattern = "${path.module}/files/controller_deploy.yaml"
vars = {
eks_cluster_name = var.eks_cluster_name
}
}
#Install the ALB controller
resource "kubectl_manifest" "install_controller_manifests" {
wait = true
count = length(data.kubectl_path_documents.controller_manifests.documents)
yaml_body = element(data.kubectl_path_documents.controller_manifests.documents, count.index)
depends_on = [kubectl_path_documents.controller_manifests]
}

when I run terraform apply first time, it throws the following error.
The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the count depends on.

[g0atcheez]

Also having the same issue as others are experiencing using the same method that's indicated in the docs.

data "kubectl_file_documents" "certmanager_crds" {
  content = file("${path.module}/kubectl/certmanager-crds.yaml")
}

resource "kubectl_manifest" "certmanager-crds" {
  count      = length(data.kubectl_file_documents.certmanager_crds.documents)
  yaml_body  = element(data.kubectl_file_documents.certmanager_crds.documents, count.index)
  depends_on = [kubectl_manifest.rancher-import]
}

Error: Invalid count argument
│ 
│   on ../../Terraform.Modules.Azure.K8s-Base/src/main.tf line 179, in resource "kubectl_manifest" "certmanager-crds":
│  179:   count      = length(data.kubectl_file_documents.certmanager_crds.documents)
│ 
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work
│ around this, use the -target argument to first apply only the resources that the count depends on.

❯ terraform version
Terraform v1.0.1
on darwin_amd64
+ provider registry.terraform.io/cloudflare/cloudflare v2.9.0
+ provider registry.terraform.io/gavinbunney/kubectl v1.11.3
+ provider registry.terraform.io/hashicorp/azuread v2.0.1
+ provider registry.terraform.io/hashicorp/google v3.81.0
+ provider registry.terraform.io/hashicorp/google-beta v3.81.0
+ provider registry.terraform.io/hashicorp/helm v2.0.3
+ provider registry.terraform.io/hashicorp/http v2.1.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.0.3
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/vault v2.23.0
+ provider registry.terraform.io/rancher/rancher2 v1.11.0

You can run terraform state show on the data resource and it returns the expected data so why would the plan say the attributes can't be determined until apply?

❯ terraform state show module.k8s-base.kubectl_manifest.certmanager-crds
# module.k8s-base.kubectl_manifest.certmanager-crds:
resource "kubectl_manifest" "certmanager-crds" {
    api_version             = "apiextensions.k8s.io/v1beta1"
    force_new               = false
    id                      = "/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/certificaterequests.cert-manager.io"
    kind                    = "CustomResourceDefinition"
...

[calexandre]

I'm having the same problem here inside a module.
I can confirm that the hack provided here does not work either inside a module - I have not tested it outside of the module.

I even tested with a slimmed down version of a single yaml document with the following configuration and it still doesn't work...

data "kubectl_path_documents" "secrets_test_hack" {
  pattern = "${path.module}/k8s/secrets-test/00-namespace.yaml"
}

resource "kubectl_manifest" "secrets_test" {
  count     = length(data.kubectl_path_documents.secrets_test_hack.documents)
  yaml_body = element(data.kubectl_path_documents.secrets_test_hack.documents, count.index)
}

╷
│ Error: Invalid count argument
│ 
│   on modules/secrets-csi-test-app/main.tf line 28, in resource "kubectl_manifest" "secrets_test":
│   28:   count     = length(data.kubectl_path_documents.secrets_test_hack.documents)
│ 
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to
│ first apply only the resources that the count depends on.
╵

[calexandre]

I managed to build a hacky workaround that might work for most of you guys...
Basically I replaced the count source using other terraform's built-in functions to fetch the document count...

Here's an example (taken from by previous comment:

data "kubectl_path_documents" "secrets_test" {
  pattern = "${path.module}/k8s/secrets-test/*.yaml"
}

resource "kubectl_manifest" "secrets_test" {
  count = length(
    flatten(
      toset([
        for f in fileset(".", data.kubectl_path_documents.secrets_test.pattern) : split("\n---\n", file(f))
        ]
      )
    )
  )
  yaml_body = element(data.kubectl_path_documents.secrets_test.documents, count.index)
}

What is happening here:

• I calculate all the possible documents in the same path that the data block is using using a fileset function
• I wrap that result in a for loop and to support the yaml's multi-document feature, I'm also splitting each file with \n---\n to ensure that I only grab the correct ones.
• In the end I must flatten the set because in case if multi-documents, there will be arrays inside arrays.

I know this is not an elegant solution and there might be other issues that I missed, but it's better than using -target to get around the issue...

I tested locally with multiple scenarios and for now this workaround works for me.

[jodem]

@calexandre I tested with terraform 1.0.5 and kubectl 1.11.3 on a fresh project and I don't reproduce this module issue.

data "kubectl_path_documents" "nginx_manifests" {
  pattern = "${path.module}/templates/deploy_nginx.yaml"
  vars = {
    domainName = var.domain_name
    namespace = var.name_space
  }
}

resource "kubectl_manifest" "deploy_nginx" {
  count     = length(data.kubectl_path_documents.nginx_manifests.documents)
  yaml_body = element(data.kubectl_path_documents.nginx_manifests.documents, count.index)
}

module "deploy_nginx_test1" {
  source = "./modules/nginx_deploy_module"
  domain_name = "test1"
  name_space = "spike-tf-test1"
}

Works in both creation and update.

[calexandre]

@jodem it doesn't happen always, that's why some folks are reporting the issue.. I've used this module for at least one year, and it never happened to me, until today...

[jodem]

Ok @calexandre thanks for the clarification, it's a bit scary. Thanks also for providing an elegant hack to mitigate the issue.

[calexandre]

Ok @calexandre thanks for the clarification, it's a bit scary. Thanks also for providing an elegant hack to mitigate the issue.

Thanks! I don't know about the "elegant" part ;) But the actual code, shouldn't be too different I guess...

[indistinctTalk]

Yep, just started hitting this with no changes 😞. How odd.

[sidh]

Hit this bug today. Very annoying. :(

[luisdavim]

This is very strange, if I put this in a file on its own:

resource "kubectl_manifest" "pixie-viziers" {
  for_each  = data.kubectl_file_documents.pixie-viziers.manifests
  yaml_body = each.value
}

resource "kubectl_manifest" "pixie-crd" {
  for_each  = data.kubectl_file_documents.pixie-crd.manifests
  yaml_body = each.value
}

data "kubectl_file_documents" "pixie-viziers" {
  content = data.http.pixie-viziers.body
}

data "kubectl_file_documents" "pixie-crd" {
  content = data.http.pixie-crd.body
}

data "http" "pixie-viziers" {
  url = "https://raw.githubusercontent.com/pixie-labs/pixie/main/k8s/operator/crd/base/px.dev_viziers.yaml"
}

data "http" "pixie-crd" {
  url = "https://raw.githubusercontent.com/pixie-labs/pixie/main/k8s/operator/helm/crds/olm_crd.yaml"
}

provider "kubectl" {
  host                   = "host"
  cluster_ca_certificate = "cert"
  token                  = "token"
  load_config_file       = false
}

terraform {
  required_version = ">= 1.0.4"

  required_providers {
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.13.1"
    }
    http = {
      source  = "hashicorp/http"
      version = "2.1.0"
    }
  }
}

it works but if I use that same code in a module, it fails with:

╷
│ Error: Invalid for_each argument
│
│   on .terraform/modules/eks/eks-init/newrelic.tf line 29, in resource "kubectl_manifest" "pixie-viziers":
│   29:   for_each  = data.kubectl_file_documents.pixie-viziers.manifests
│     ├────────────────
│     │ data.kubectl_file_documents.pixie-viziers.manifests is a map of string, known only after apply
│
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on.
╵
╷
│ Error: Invalid for_each argument
│
│   on .terraform/modules/eks/eks-init/newrelic.tf line 34, in resource "kubectl_manifest" "pixie-crd":
│   34:   for_each  = data.kubectl_file_documents.pixie-crd.manifests
│     ├────────────────
│     │ data.kubectl_file_documents.pixie-crd.manifests is a map of string, known only after apply
│
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on.
╵

[Oceanswave]

Running into this issue not inside a module with Terraform 1.2.2. The annoying part is that this worked previously and that the use case that is seemingly being protected against (a changing # of documents based on data) is probably an atypical use case...

[chap-dr]

This is a huge annoyance.. any word on a fix or even if it is being looked into? it seems to be escalating (with the latest 1.2.2 comment saying it wont even work if put outside of a module)

[justinb-shipt]

I've had clean, multiple runs from 2 different modules and then suddenly got this error on the one module. Using 1.14.0 of the kubectl provider and 1.2.9 for TF.

[bosmak]

I'm having the same issue.

I'm using it inside a module, so basically what I get is that this happens when the data needs to be updated.

If I delete everything I can apply and do whatever a want with these resources, but as soon as I leave it for some time when I come back it won't work anymore.

[nr-ey]

Oddly this just started happening to me as well, even for resources that have not been touched / modified.

[nr-ey]

I too was able to get around this by commenting out the resources and re-enabling them. Hope it lasts.

[justinb-shipt]

I'm trying out using the manifests attribute that was introduced in 1.13.0. I'm hoping to see better results. It will at least solve an issue I'm having with numbered based indexes.

[bosmak]

I wasn't able to use the HTTP provider to fetch the yaml, and ended up committing the yaml on VCS

But it seems that using this workaround, solved the issue!

[nr-ey]

It's back again. Sigh, I will just split out the files manually instead of using kubectl_path_documents.

[bosmak]

@neilrao-ey you don't need to split them, follow the workaround that I linked in my comment above!

[nr-ey]

@neilrao-ey you don't need to split them, follow the workaround that I linked in my comment above!

The workaround errors out on a windows build agent, unfortunately. I suspect it might have to do with the splitting characters used but either way I'd rather avoid the hack. Looking forward to seeing this fixed in an official capacity though.

[suvaanshkumar]

Any resolutions to this?
I faced this bug when I realized that one of my yaml file variable has a dependency on another module.

[boillodmanuel]

Hi,

I figured out that using depends_on will often produces the same cannot be determined until apply error, with data sources (not only kubectl_path_documents).

See:

• Hashicorp doc on depends_on - Processing and Planning Consequences
• Beware of depends_on for Terraform modules. It might bite you! article

This looks more like an internal terraform issue.

Similar to #215

I opened an issue on terraform directly: hashicorp/terraform#34391

[bensoer]

So i've managed to solve this by implementing what kubectl_path_documents data source offers functionally, but by using other built-in terraform functions:

If you have a single yaml file containing multiple resources, I did the following:

resource "kubectl_manifest" "cert-manager-crds" {
    for_each  = toset(split("---", file("${abspath(path.module)}/res/cert-manager-1.14.4/cert-manager.crds.yaml")))
    yaml_body = each.value
    server_side_apply = true
}

This one works by splitting the file on the --- that's between each resource within the file

And if you have a folder with multiple yaml files, I did the following:

resource "kubectl_manifest" "prometheus-crds" {
  for_each  = fileset("${abspath(path.module)}/res/prometheus-25.19.1", "*.yaml")
  yaml_body = file("${abspath(path.module)}/res/prometheus-25.19.1/${each.value}")
  server_side_apply = true
}

This one works by using fileset method, which can determine the number of files before the apply

Hope this helps someone, or you guys.

I have these running inside of a module which then I have other modules depends_on to