0.12upgrade and init have a "chicken vs egg problem"

[jhoblitt]

Terraform Version

$ $GOPATH/bin/terraform version
Terraform v0.12.0-dev
+ provider.aws v1.48.0
+ provider.google v1.19.1
+ provider.helm v0.6.2
+ provider.kubernetes v1.3.0
+ provider.template v1.0.0

Terraform Configuration Files

provider "kubernetes" {
  version = "~> 1.4.0"

  load_config_file = true

  host                   = "${module.gke.host}"
  cluster_ca_certificate = "${base64decode(module.gke.cluster_ca_certificate)}"
}

provider "grafana" {
  version = "~> 1.3"

  url  = "https://${local.grafana_fqdn}"
  auth = "${var.grafana_admin_user}:${var.grafana_admin_pass}"
}
...

Debug Output

$ TF_LOG=trace $GOPATH/bin/terraform 0.12upgrade -yes 
2018/12/20 09:31:45 [INFO] Terraform version: 0.12.0 dev 
2018/12/20 09:31:45 [INFO] Go runtime version: go1.11.2
2018/12/20 09:31:45 [INFO] CLI args: []string{"/home/jhoblitt/go/bin/terraform", "0.12upgrade", "-yes"}
2018/12/20 09:31:45 [DEBUG] Attempting to open CLI config file: /home/jhoblitt/.terraformrc
2018/12/20 09:31:45 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2018/12/20 09:31:45 [INFO] CLI command args: []string{"0.12upgrade", "-yes"}
2018/12/20 09:31:45 [DEBUG] checking for provider in "."
2018/12/20 09:31:45 [DEBUG] checking for provider in "/home/jhoblitt/go/bin"
2018/12/20 09:31:45 [DEBUG] checking for provider in ".terraform/plugins/linux_amd64"
2018/12/20 09:31:45 [DEBUG] found provider "terraform-provider-aws_v1.48.0_x4"
2018/12/20 09:31:45 [DEBUG] found provider "terraform-provider-google_v1.19.1_x4"
2018/12/20 09:31:45 [DEBUG] found provider "terraform-provider-helm_v0.6.2_x4"
2018/12/20 09:31:45 [DEBUG] found provider "terraform-provider-kubernetes_v1.3.0_x4"
2018/12/20 09:31:45 [DEBUG] found provider "terraform-provider-template_v1.0.0_x4"
2018/12/20 09:31:45 [DEBUG] found valid plugin: "helm", "0.6.2", "/home/jhoblitt/github/terraform-efd-kafka/.terraform/plugins/linux_amd64/terraform-provider-helm_v0.6.2_x4"
2018/12/20 09:31:45 [DEBUG] found valid plugin: "kubernetes", "1.3.0", "/home/jhoblitt/github/terraform-efd-kafka/.terraform/plugins/linux_amd64/terraform-provider-kubernetes_v1.3.0_x4"
2018/12/20 09:31:45 [DEBUG] found valid plugin: "template", "1.0.0", "/home/jhoblitt/github/terraform-efd-kafka/.terraform/plugins/linux_amd64/terraform-provider-template_v1.0.0_x4"
2018/12/20 09:31:45 [DEBUG] found valid plugin: "aws", "1.48.0", "/home/jhoblitt/github/terraform-efd-kafka/.terraform/plugins/linux_amd64/terraform-provider-aws_v1.48.0_x4"
2018/12/20 09:31:45 [DEBUG] found valid plugin: "google", "1.19.1", "/home/jhoblitt/github/terraform-efd-kafka/.terraform/plugins/linux_amd64/terraform-provider-google_v1.19.1_x4"
2018/12/20 09:31:45 [DEBUG] checking for provisioner in "."
2018/12/20 09:31:45 [DEBUG] checking for provisioner in "/home/jhoblitt/go/bin"
2018/12/20 09:31:45 [DEBUG] checking for provisioner in ".terraform/plugins/linux_amd64"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "grafana.tf"
2018/12/20 09:31:45 [TRACE] Resource block for kubernetes_namespace.grafana requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Resource block for helm_release.grafana requires provider "helm"
2018/12/20 09:31:45 [TRACE] Resource block for kubernetes_secret.grafana_tls requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Resource block for grafana_dashboard.confluent requires provider "grafana"
2018/12/20 09:31:45 [TRACE] Resource block for grafana_dashboard.nginx requires provider "grafana"
2018/12/20 09:31:45 [TRACE] Resource block for data.template_file.grafana_values requires provider "template"
2018/12/20 09:31:45 [TRACE] Resource block for data.template_file.confluent_grafana_dashboard requires provider "template"
2018/12/20 09:31:45 [TRACE] Resource block for data.template_file.nginx_grafana_dashboard requires provider "template"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "kafka.tf"
2018/12/20 09:31:45 [TRACE] Resource block for kubernetes_namespace.kafka requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Resource block for helm_repository.confluentinc requires provider "helm"
2018/12/20 09:31:45 [TRACE] Resource block for helm_release.confluent requires provider "helm"
2018/12/20 09:31:45 [TRACE] Resource block for data.template_file.confluent_values requires provider "template"
2018/12/20 09:31:45 [TRACE] Resource block for data.kubernetes_service.lb0 requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Resource block for data.kubernetes_service.lb1 requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Resource block for data.kubernetes_service.lb2 requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "main.tf"
2018/12/20 09:31:45 [TRACE] Provider block requires provider "template"
2018/12/20 09:31:45 [TRACE] Provider block requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Provider block requires provider "helm"
2018/12/20 09:31:45 [TRACE] Provider block requires provider "aws"
2018/12/20 09:31:45 [TRACE] Provider block requires provider "grafana"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "prometheus.tf"
2018/12/20 09:31:45 [TRACE] Resource block for kubernetes_namespace.prometheus requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Resource block for helm_release.prometheus requires provider "helm"
2018/12/20 09:31:45 [TRACE] Resource block for kubernetes_secret.prometheus_tls requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Resource block for data.template_file.prometheus_values requires provider "template"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "backend-s3.tf"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "nginx-ingress.tf"
2018/12/20 09:31:45 [TRACE] Resource block for kubernetes_namespace.nginx_ingress requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] Resource block for helm_release.nginx_ingress requires provider "helm"
2018/12/20 09:31:45 [TRACE] Resource block for data.template_file.nginx_ingress_values requires provider "template"
2018/12/20 09:31:45 [TRACE] Resource block for data.kubernetes_service.nginx_ingress requires provider "kubernetes"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "outputs.tf"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "route53.tf"
2018/12/20 09:31:45 [TRACE] Resource block for aws_route53_record.kafka_lb0 requires provider "aws"
2018/12/20 09:31:45 [TRACE] Resource block for aws_route53_record.kafka_lb1 requires provider "aws"
2018/12/20 09:31:45 [TRACE] Resource block for aws_route53_record.kafka_lb2 requires provider "aws"
2018/12/20 09:31:45 [TRACE] Resource block for aws_route53_record.grafana requires provider "aws"
2018/12/20 09:31:45 [TRACE] Resource block for aws_route53_record.prometheus requires provider "aws"
2018/12/20 09:31:45 [TRACE] configupgrade: Analyzing "variables.tf"

Error: error resolving providers: [provider.grafana: no suitable version installed
  version requirements: "~> 1.3"
  versions installed: none provider.kubernetes: no suitable version installed
  version requirements: "~> 1.4.0"
  versions installed: "1.3.0"]

$ TF_LOG=trace $GOPATH/bin/terraform init
2018/12/20 09:32:48 [INFO] Terraform version: 0.12.0 dev 
2018/12/20 09:32:48 [INFO] Go runtime version: go1.11.2
2018/12/20 09:32:48 [INFO] CLI args: []string{"/home/jhoblitt/go/bin/terraform", "init"}
2018/12/20 09:32:48 [DEBUG] Attempting to open CLI config file: /home/jhoblitt/.terraformrc
2018/12/20 09:32:48 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2018/12/20 09:32:48 [INFO] CLI command args: []string{"init"}
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Argument or block definition required

  on grafana.tf line 41, in resource "kubernetes_secret" "grafana_tls":
  41:     tls.crt = "${local.tls_crt}"

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.


Error: Argument or block definition required

  on grafana.tf line 42, in resource "kubernetes_secret" "grafana_tls":
  42:     tls.key = "${local.tls_key}"

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.


Error: Argument or block definition required

  on prometheus.tf line 48, in resource "kubernetes_secret" "prometheus_tls":
  48:     tls.crt = "${local.tls_crt}"

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.


Error: Argument or block definition required

  on prometheus.tf line 49, in resource "kubernetes_secret" "prometheus_tls":
  49:     tls.key = "${local.tls_key}"

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.

Expected Behavior

terraform 0.12upgrade --yes is able to upgrade the syntax of arbitrary tf modules (assuming that all providers are tf >= 0.12 compatible).

Actual Behavior

terraform 0.12upgrade fails if the appropriate set of providers have not already been installed by tf 0.11.x. terraform init fails if the syntax has not already been upgraded to be 0.12. compatible.

Steps to Reproduce

Additional Context

In this example, a tf >= 0.12 kubernetes provider isn't present.

[apparentlymart]

Thanks for reporting this, @jhoblitt!

There's still remaining work to do on the upgrade tool before we consider it "ready", and this is one of the things we need to figure out. We'll come up with a reasonable plan for this before we consider the upgrade tool ready to start using in the beta1 release.

[jhoblitt]

I suppose this could be a "doc fix" but I imagine that the latest tf 0.11.x code be downloaded and run for the user. Otherwise, I think there has to at least be a partial parser for for tf < 0.12

[apparentlymart]

Indeed, installing first with 0.11 is what would work without any code changes, though we'd like to do better than that for the 0.12.0 final release, even if we don't manage to get that all wired up for beta.

Since we still have all the old configuration loading codepaths in the codebase (dead code yet to be cleaned up) the most likely path will be to retain just enough of that to look up provider and module dependencies and give the init command a fallback codepath that uses it if the main loader fails. We'll investigate further before the final release and see what makes sense.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.