Helm resources are not recreated if the underlying cluster is destroyed

[n-oden]

Terraform Version and Provider Version

$ terraform -v
Terraform v0.12.28
+ provider.google v3.39.0
+ provider.helm v1.3.0

$ helm version
version.BuildInfo{Version:"v3.3.3", GitCommit:"55e3ca022e40fe200fbc855938995f40b2a68ce0", GitTreeState:"dirty", GoVersion:"go1.15.2"}

Provider Version

• 1.3.0

Affected Resource(s)

• helm_release

Terraform Configuration Files

variable "project" {
  default = "my-test-project"
}

variable "region" {
  default = "us-east1"
}

provider "google" {
  region = var.region
}

resource "google_project_service" "container" {
  project = var.project
  service = "container.googleapis.com"

  disable_dependent_services = false
  disable_on_destroy         = false
}

resource "google_container_cluster" "default" {
  name               = "helm-test-cluster"
  location           = "${var.region}"
  initial_node_count = 1
  project            = var.project
  depends_on = [
    google_project_service.container
  ]

  master_auth {
    username = ""
    password = ""
    client_certificate_config {
      issue_client_certificate = false
    }
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }

  timeouts {
    create = "30m"
    update = "40m"
  }
}

data "google_client_config" "provider" {}

provider "helm" {
  kubernetes {
    load_config_file       = false
    host                   = "https://${google_container_cluster.default.endpoint}"
    token                  = data.google_client_config.provider.access_token
    cluster_ca_certificate = base64decode(google_container_cluster.default.master_auth[0].cluster_ca_certificate)
  }
}

resource "helm_release" "redis" {
  name       = "redis"
  repository = "https://kubernetes-charts.storage.googleapis.com/"
  chart      = "redis"
  atomic     = true
  wait       = true
  namespace  = "default"
}

Debug Output

https://gist.github.com/n-oden/eb86c620c0c6c47866f2be2746491fe9

Expected Behavior

The Helm release should be re-created in the new cluster.

Actual Behavior

The old cluster is destroyed, the new cluster created in the new region, but the helm_release.redis resource is unchanged: terraform believes that it still exists, but of course it does not.

Steps to Reproduce

Create the cluster and the helm release, and then force re-creation of the cluster by changing the region variable:

1. terraform apply -var region=us-east1
2. `terraform apply -var region=us-central1

Important Factoids

I tested this running in Google Cloud Platform using GKE, but I suspect this is a more basic issue with the provider and the behavior would be the same against EKS etc.

References

• helm releases not reinstalled after cluster recreation #297 -- this basically a duplicate of that report, which was incorrectly closed because the reporter mentioned he was using Helm v2. The behavior is exactly the same when using Helm V3.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[sebglon]

For me it' the same root cause as here #593

I'd suggest setting an explicit dependency on the cluster resource in the helm_release resource, like this:

depends_on = [
  google_container_cluster.default,
]

[BunkerHarb]

For me it' the same root cause as here #593

I'd suggest setting an explicit dependency on the cluster resource in the helm_release resource, like this:

depends_on = [
  google_container_cluster.default,
]

i experience the same as n-oden (on azure) and adding a depends_on for the cluster doesn't fix it.

[n-oden]

Belatedly @sebglon using depends_on does not address this issue: depends_on hard-codes ordering of resource creation, but does not imply anything about state relationships between resources. If resource B depends on resource A, terraform will wait until A is created before creating B, but deleting A does not delete B from state.

[justinas-b]

Yep, same issue here. If cluster is recreated or helm release is deleted directly on cluster, the helm_release stays in the state and terraform does not detect that the release itself is actually missing on k8s

[michelefa1988]

same issue here on azure. Are there any workarounds for this? Currently I remove the helm_release from the statefile manually, but I really hate doing this.

[n-oden]

@michelefa1988 so far I haven't found any particularly good workarounds. You can potentially mitigate the issue by creating a random_id string, interpolating it into the name of all of your helm releases and setting its keepers field to something that will update when the cluster is rebuilt, but this invokes some pretty thorny dependency-ordering issues that you'll probalby have to manage by hand with depends_on and which will make cluster teardowns unnecessarily slow. Long story short I've built some local tooling that just manually deletes all of the resources out of tf state, and feel no better about it than you do. :)

[github-actions]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[github-actions]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[viceice]

pretty bad, still happening 😕