Kube Persistent Volumes - Create successful, refresh state fails

[jimmiebtlr]

Terraform Version

Terraform v0.11.7

• provider.google v1.14.0
• provider.kubernetes v1.1.0

Affected Resource(s)

Please list the resources as a list, for example:

• kubernetes_persistent_volume

Terraform Configuration Files

locals {
  cluster_project  = "gcloud-project"
  cluster_name     = "cluster"
  cluster_username = "admin"
  cluster_password = "passwordpassword"
}

provider "google" {
  credentials = "${file("account.json")}"
  project     = "${local.cluster_project}"
  region      = "us-central1"
}

resource "google_container_cluster" "primary" {
  name               = "${local.cluster_name}"
  zone               = "us-central1-c"
  min_master_version =  "1.10.2-gke.3"
  node_version =  "1.10.2-gke.3"

  master_auth {
    username = "${local.cluster_username}"
    password = "${local.cluster_password}"
  }

  node_pool {
    name         = "default-node-pool"

    node_config {
      image_type   = "ubuntu"

      oauth_scopes = [
        "compute-rw",
        "storage-ro",
        "logging-write",
        "monitoring",
      ]
    }
  }
}

provider "kubernetes" {
  host  = "${google_container_cluster.primary.endpoint}"

  username = "${local.cluster_username}"
  password = "${local.cluster_password}"

  client_certificate     = "${base64decode(google_container_cluster.primary.master_auth.0.client_certificate)}"
  client_key             = "${base64decode(google_container_cluster.primary.master_auth.0.client_key)}"
  cluster_ca_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"
}

resource "google_compute_disk" "pd" {
  name = "pd"
  type = "pd-ssd"
  size = "10"
  zone = "us-central1-c"
}

resource "kubernetes_persistent_volume" "persistent-volume" {
  metadata {
    name = "data-volume"

    labels {
      name = "data-volume"
    }
  }

  spec {
    capacity {
      storage = "10Gi"
    }

    access_modes                     = ["ReadWriteOnce"]

    persistent_volume_source {
      gce_persistent_disk {
        fs_type = "xfs"
        pd_name = "pd"
      }
    }
  }
}

Debug Output

The initial creation of the persistent volume is successful. (this log doesn't contain an error)
https://gist.github.com/jimmiebtlr/63daa0e4aba22189834d22468b95ad4d

The error occurs during apply after the resource has been created (I believe during the "refresh" step). (This log contains the error)
https://gist.github.com/jimmiebtlr/638610476884d98adb48b247c0088b38

Panic Output

No panic

Expected Behavior

Re-apply finds everything is created and looks good. No op.

Actual Behavior

Re-apply has trouble finding the resource and returns an error.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply
2. terraform apply

Important Factoids

Nothing unusual.

References

None that I've found.

[pdecat]

The refresh logs show it tried to reach the master at localhost:

* kubernetes_persistent_volume.persistent-volume: Get http://localhost/version: dial tcp 127.0.0.1:80: getsockopt: connection refused�[0m

Did you try to create the GKE cluster and the kubernetes resources in the same run (i.e. without specifying -target)?

If so, the kubernetes provider couldn't be initialized in the first pass.
See #144 and hashicorp/terraform#4149 for more details.

Also, I'd recommend to define load_config_file = false to avoid falling back on the current user's default .kube/config when the provider configuration is incomplete (which would explain the access to 127.0.0.1:80).

Edit: does not seem to be the issue here given the [INFO] Unable to load config file as it doesn't exist at "/root/.kube/config" messages in the logs.

[jimmiebtlr]

In one terraform apply from nothing in the gcloud project it seems to correctly

• Create kube cluster, nodes/machines
• Create the persistent volume and gcloud disks
• Create and start pods/etc (not in example)

If I'm understanding you correctly, a single terraform apply isn't expected to both create kube, and add resources to the kube cluster in a single run?

[pdecat]

In a single run, the kubernetes provider cannot be properly initialized as the GKE cluster does not exist yet.

The current workarounds are:

• to use target to create the google_container_cluster.primary resource in the first apply run, then apply again without target.
• or to separate the google resources from the kubernetes ones in two states, and use the output of the former to initialize the latter.

[pdecat]

Also related: hashicorp/terraform#12869

[dominik-lekse]

It is possible to provision both the GKE and Kubernetes resources within this cluster in a single Terraform run. See this example https://www.terraform.io/docs/providers/google/d/datasource_client_config.html#example-usage-configure-kubernetes-provider-with-oauth2-access-token on how to configure the Kubernetes provider with an IAM access token.

[pdecat]

Re-read the apply logs, it does seem to create the kubernetes PV properly (not sure in which cluster though).

In #144 (comment):

This can work in some cases, but unfortunately not every case.

No longer sure what's going on now.

[pdecat]

@dominik-lekse this example is referencing the GKE cluster from a datasource.
This corresponds to the second workaround I mentioned above in #168 (comment).

[jimmiebtlr]

What is working

• GKE creates kube cluster in the correct gcloud project
• Persistent volume is created on first run

What isn't working

• Running terraform apply a second time, fails with several

 kubernetes_persistent_volume.persistent-volume: Get http://localhost/version: dial tcp 127.0.0.1:80: getsockopt: connection refused�[0m

I'm currently running this in a docker container, so unless kube is coming up with a default (such as 127.0.0.1), it's definetly not picking up an existing .kube/config.

[jimmiebtlr]

Anyone know of an example with terraform + kube + gcloud + persistent volumes working? Maybe I can work backward.

[jimmiebtlr]

Well, seems to be working today. Removed and re-created cluster entirely (again). Afaik nothing changed in terms of versions etc though. Not sure what changed that caused it to be fixed.