Crash

[aronneagu]

Terraform Version and Provider Version

Terraform v0.12.28

• provider.azurerm v2.9.0
• provider.kubernetes v1.11.3
• provider.kubernetes-alpha (unversioned)

Used the binary from https://github.com/hashicorp/terraform-provider-kubernetes-alpha/releases/download/v0.1.0/terraform-provider-kubernetes-alpha_0.1.0_linux_amd64.zip

Kubernetes Version

kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8", GitCommit:"ec6eb119b81be488b030e849b9e64fda4caaf33c", GitTreeState:"clean", BuildDate:"2020-03-13T02:33:08Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Affected Resource(s)

Terraform Configuration Files

main.tf

provider "azurerm" {
  version = "2.9.0"
  features {}
}

provider "kubernetes" {
  version = "~> 1.11"
  # Set to true when running locally, set to false when running on CI
  # see https://www.terraform.io/docs/providers/kubernetes/index.html#in-cluster-service-account-token
  load_config_file       = false
  host                   = azurerm_kubernetes_cluster.cluster.kube_config.0.host
  username               = azurerm_kubernetes_cluster.cluster.kube_config.0.username
  password               = azurerm_kubernetes_cluster.cluster.kube_config.0.password
  client_certificate     = base64decode(azurerm_kubernetes_cluster.cluster.kube_config.0.client_certificate)
  client_key             = base64decode(azurerm_kubernetes_cluster.cluster.kube_config.0.client_key)
  cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.cluster.kube_config.0.cluster_ca_certificate)
}

provider "kubernetes-alpha" {
  server_side_planning = true
  host                 = azurerm_kubernetes_cluster.cluster.kube_config.0.host
  username               = azurerm_kubernetes_cluster.cluster.kube_config.0.username
  password               = azurerm_kubernetes_cluster.cluster.kube_config.0.password
  client_certificate     = base64decode(azurerm_kubernetes_cluster.cluster.kube_config.0.client_certificate)
  client_key             = base64decode(azurerm_kubernetes_cluster.cluster.kube_config.0.client_key)
  cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.cluster.kube_config.0.cluster_ca_certificate)
}

data "azurerm_resource_group" "rg" {
  name = "${var.rgroup}"
}

resource "azurerm_subnet" "k8s_subnet" {
  name                                           = var.cluster_name
  resource_group_name                            = data.azurerm_resource_group.rg.name
  virtual_network_name                           = var.vnet_name
  address_prefixes                               = [var.address_prefix]
  enforce_private_link_endpoint_network_policies = false
}


resource "azurerm_kubernetes_cluster" "cluster" {
  name                = var.cluster_name
  location            = var.location
  resource_group_name = data.azurerm_resource_group.rg.name
  dns_prefix          = var.cluster_name

  linux_profile {
    admin_username = "azuser"

    ssh_key {
      key_data = "ssh-rsa <replaceme>"
    }
  }

  default_node_pool {
    name                  = var.node_pool
    node_count            = var.node_count
    vm_size               = var.node_size
    vnet_subnet_id        = azurerm_subnet.k8s_subnet.id
    enable_node_public_ip = false

  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin     = "azure"
    docker_bridge_cidr = "172.17.0.1/16"
    dns_service_ip     = var.dns_service_ip
    service_cidr       = var.service_cidr
    load_balancer_sku  = "standard"
  }

  kubernetes_version = var.k8s_version

  addon_profile {
    kube_dashboard {
      enabled = false
    }
    oms_agent {
      enabled = false
    }
  }
}

resource "kubernetes_namespace" "istio_system" {
  depends_on = [azurerm_kubernetes_cluster.cluster]
  metadata {
    name = "istio-system"
  }
}

resource "kubernetes_manifest" "istiooperator_istio_controlplane" {
  depends_on = [azurerm_kubernetes_cluster.cluster]
  provider   = kubernetes-alpha
  manifest = {
    "apiVersion" = "install.istio.io/v1alpha1"
    "kind"       = "IstioOperator"
    "metadata" = {
      "name"      = "istio-controlplane"
      "namespace" = "istio-system"
    }
    "spec" = {
      "addonComponents" = {
        "egressGateways" = {
          "enabled" = true
        }
        "tracing" = {
          "enabled" = true
        }
        "prometheus" = {
          "enabled" = false
        }
      }
      "profile" = "default"
    }
  }
}

resource "kubernetes_manifest" "istio_ingressgateway" {
  depends_on = [kubernetes_manifest.istiooperator_istio_controlplane]
  provider   = kubernetes-alpha
  manifest = {
    "apiVersion" = "v1"
    "kind"       = "Service"
    "metadata" = {
      "name"      = "istio-ingressgateway"
      "namespace" = "istio-system"
      "annotations" = {
        "service.beta.kubernetes.io/azure-load-balancer-internal" = "true"
      }
    }
  }
}

variables.tf

variable "rgroup" {}
variable "location" {}
variable "cluster_name" {}
variable "dns_service_ip" {}
variable "service_cidr" {}
variable "network_plugin" {}
variable "node_count" {}
variable "node_size" {}
variable "node_pool" {}
variable "k8s_version" {}
variable "vnet_name" {}
variable "address_prefix" {}

Debug Output

Panic Output

https://gist.github.com/aronneagu/2480e338d056cd955d7a2154bd1f5a2d

Expected Behavior

What should have happened?

Actual Behavior

What actually happened?

Steps to Reproduce

Important Factoids

References

• GH-1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[aareet]

I noticed

Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8", GitCommit:"ec6eb119b81be488b030e849b9e64fda4caaf33c", GitTreeState:"clean", BuildDate:"2020-03-13T02:33:08Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Is this crash reproducible with a 1.17.x cluster @aronneagu?

[aronneagu]

Sorry, I had kubectl use a an existing context that had a 1.16 cluster inn it
I am trying to build a kubernetes cluster with version 1.17.7 via terraform, then have kubernetes-alpha provider use the variables of that terraformed cluster to deploy a manifest (for istio)

[jefflantz]

To deploy a 1.17.7 GKE cluster via terraform, consider the following:

resource "google_compute_network" "vpc" {
  name                    = "kubernetes-tf-network"
  auto_create_subnetworks = "false"
}

resource "google_compute_subnetwork" "subnet" {
  name          = "kubernetes-tf-subnet"
  region        = var.region
  network       = google_compute_network.vpc.name
  ip_cidr_range = "10.10.0.0/24"

}

data "google_container_engine_versions" "east4" {
  provider       = google-beta
  project        = var.project_id
  location       = var.region
  version_prefix = "1.17."
}

resource "google_container_cluster" "primary" {
  provider = google-beta

  name     = "kubernetes-tf-cluster"
  location = var.region

  remove_default_node_pool = true
  initial_node_count       = 1

  node_version = data.google_container_engine_versions.east4.release_channel_default_version["RAPID"]
  min_master_version = data.google_container_engine_versions.east4.release_channel_default_version["RAPID"]
  release_channel {
    channel = "RAPID"
  }

  network    = google_compute_network.vpc.name
  subnetwork = google_compute_subnetwork.subnet.name

  master_auth {
    client_certificate_config {
      issue_client_certificate = false
    }
  }
}

resource "google_container_node_pool" "primary_nodes" {
  provider = google-beta

  name     = "${google_container_cluster.primary.name}-node-pool"
  location = var.region
  cluster  = google_container_cluster.primary.name

  initial_node_count = 1
  autoscaling {
    min_node_count = 0
    max_node_count = 6
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]

    labels = {
      env = var.project_id
    }

    # preemptible  = true
    machine_type = var.machine_type
    tags         = ["gke-node", "${google_container_cluster.primary.name}"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

output "kubernetes_cluster_name" {
  value       = google_container_cluster.primary.name
  description = "GKE Cluster Name"
}
output "version" {
  value       = google_container_cluster.primary.master_version
  description = "master version"
}
output "project_id" {
  value       = var.project_id
  description = "GCP project id"
}
output "region" {
  value       = var.region
  description = "region"
}

The outputs allow you to connect to the cluster via

gcloud container clusters get-credentials $(terraform output kubernetes_cluster_name) --region $(terraform output region) --project $(terraform output project_id)

[aronneagu]

Hi @jefflantz, thanks for that snippet. But I am looking to deploy to Azure AKS

[jefflantz]

Sorry about that, I confused this issue with a different one. Perhaps a similar method exists for amazon?

[alexsomesan]

@aronneagu Your crash is caused by inconsistent credential values coming from the AKS datasources. Likely because you are creating the cluster in the same apply operation (am I right?).

This is a known limitation in Terraform that can sometimes cause problems. The situation here should improve with #65, but it's still not entirely solved.

However, you'll notice that once that's fixed, your use-case will still not work as expected because you create multiple K8s resources that depend on one-another. This currently falls under a documented limitation of the provider (see README). It will be fixed when PR #41 merges.

[aronneagu]

Hi @alexsomesan, you are right, I was trying to use the kubernetes-alpha provider in the same apply as the one that was creating the cluster. In the end, I've separated the apply in two runs, one that creates the kubernetes cluster, and the second one that uses kubernetes-alpha to deploy istio

Thanks for explaining the cause, it wasn't immediately obvious why it didn't work

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!