Plugin loading error on latest resource version

[crsimmons]

First off, thanks for the resource - it's been really helpful.

This morning when we tried running our pipeline the terraform put resulted in:

▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ Terraform Apply ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
Error: Could not load plugin
Plugin reinitialization required. Please run "terraform init".
Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.
Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".
4 problems:
- Failed to instantiate provider "registry.terraform.io/-/google" to obtain
schema: unknown provider "registry.terraform.io/-/google"
- Failed to instantiate provider "registry.terraform.io/-/kubernetes" to
obtain schema: unknown provider "registry.terraform.io/-/kubernetes"
- Failed to instantiate provider "registry.terraform.io/-/random" to obtain
schema: unknown provider "registry.terraform.io/-/random"
- Failed to instantiate provider "registry.terraform.io/-/tls" to obtain
schema: unknown provider "registry.terraform.io/-/tls"
▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ Terraform Apply ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Failed To Run Terraform Apply!
2020/09/21 09:24:10 Apply Error: Failed to run Terraform command: exit status 1

I noticed that a new version of the resource was pushed recently with terraform v0.13.3. We think the error is probably something to do with the new required_providers config.

We've fixed the issue but it's a little bit rough and might be worth documenting somewhere. We needed to:

• run terraform 0.13upgrade against each module
• configure the backend and workspace locally
• run terraform init and terraform apply locally which said nothing changed but updated the state

Our pipeline was then able to run successfully with the updated state.

Looking at the tfstate diff it changed:

"resources": [
    {
      ...
      "provider": "module.kubernetes.provider.kubernetes",

to

"resources": [
    {
      ...
      "module.kubernetes.provider[\"registry.terraform.io/hashicorp/kubernetes\"]",

for each provider in addition to bumping the terraform version.

I'm not sure why running the commands locally worked but the resource was unable to reconcile the issue. It would be great if the resource could just handle it but if not hopefully this issue can help others who hit this.

[ljfranklin]

Terraform upgrade docs are here, I'd visit that page first if you hit any upgrade issues in the future.

It would be great if the resource could just handle it

Unfortunately running terraform 0.13upgrade modifies your configuration files to add terraform.required_providers section. These changes need to be committed to your VCS to keep things consistent with terraform apply's done from outside the resource. If it was just a change to the statefile I'd consider it but config changes seem like something the resource shouldn't handle automatically.

Given that, I'm not sure if there's anything the resource can do to help but we can keep this issue open for a bit in case others hit the same issue.

[SureshGoli123]

We are also seeing similar issue.

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

4 problems:

- Failed to instantiate provider "registry.terraform.io/-/aws" to obtain
schema: unknown provider "registry.terraform.io/-/aws"
- Failed to instantiate provider "registry.terraform.io/-/random" to obtain
schema: unknown provider "registry.terraform.io/-/random"
- Failed to instantiate provider "registry.terraform.io/-/template" to obtain
schema: unknown provider "registry.terraform.io/-/template"
- Failed to instantiate provider "registry.terraform.io/-/tls" to obtain
schema: unknown provider "registry.terraform.io/-/tls"


▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ Terraform Plan ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Failed To Run Terraform Plan!
2020/09/21 12:03:18 Plan Error: Failed to run Terraform command: exit status 1

[crsimmons]

Terraform upgrade docs are here, I'd visit that page first if you hit any upgrade issues in the future.

We did go through the upgrade docs. We made the necessary changes to the terraform files with 0.13upgrade but passing those files to the resource still failed to apply.

Running a local apply with those same files succeeded (with 0 added, 0 changed, 0 deleted).

I expected the init/apply in the resource to behave the same as an init/apply on my own machine. It seems strange that the resource continues to get the same error until the state gets modified elsewhere even with the correct tf files.

[SureshGoli123]

Hi Colin,
Thanks for sharing the upgrade guide. We have updated our terraform code according to upgrade. But still, we are seeing similar issue.

I will run the local terraform init today and see if problem exists again.

We cant run terraform apply locally, the input values are read from credhub like certs, secrets, etc...

[stuart-c-moore]

We're seeing the same thing. Output from the terraform-resource is:

▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ Terraform Plan ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

2 problems:

- Failed to instantiate provider "registry.terraform.io/-/google" to obtain
schema: unknown provider "registry.terraform.io/-/google"
- Failed to instantiate provider "registry.terraform.io/-/google-beta" to
obtain schema: unknown provider "registry.terraform.io/-/google-beta"


▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ Terraform Plan ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Failed To Run Terraform Plan!
2020/09/22 09:41:50 Plan Error: Failed to run Terraform command: exit status 1

The contents of our versions.tf:

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
    }
    google-beta = {
      source = "hashicorp/google-beta"
    }
  }
  required_version = ">= 0.13"
}

Terraform init/validate works locally, and only generates this error though the terraform-resource.

[ljfranklin]

As a workaround, the docs show a new replace-provider command, e.g. terraform state replace-provider -state=some-path.tfstate registry.terraform.io/-/aws registry.terraform.io/hashicorp/aws. If you manually download your statefile from your configured backend, run that command for all providers listed in the error message, then re-upload the statefile to the backend that should unblock it.

I'm still not sure about the root cause. The Terraform docs seem to indicate that Terraform should automatically perform this rename is you run terraform init && terraform apply with no other diffs. But some posts say that this might not always happen. The commands the resource runs are:

export CHECKPOINT_DISABLE=1 TF_WARN_OUTPUT_ERRORS=1 TF_IN_AUTOMATION=1
terraform workspace select $YOUR_WORKSPACE
terraform init -input=false -get=true -backend=true -backend-config=resource_backend_config.json -get-plugins=true
terraform apply -backup='-' -input=false -auto-approve -var-file=vars.json

If someone can replicate the error locally using those commands then please open an issue on Terraform to understand why the provider paths weren't automatically migrated. If those commands work locally I'll keep looking to see if the resource is doing something else.

[mstrisoline]

Seeing something similar on a plan_only and there are no errors in the plan

...

Plan: 0 to add, 1 to change, 0 to destroy.

...

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ Terraform Plan ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Failed To Run Terraform Plan!
2020/10/01 16:06:34 Plan Error: Failed to run Terraform command: exit status 1

Ran these commands from within the container.

export CHECKPOINT_DISABLE=1 TF_WARN_OUTPUT_ERRORS=1 TF_IN_AUTOMATION=1
terraform workspace select $YOUR_WORKSPACE
terraform init -input=false -get=true -backend=true -backend-config=resource_backend_config.json -get-plugins=true
terraform apply -backup='-' -input=false -auto-approve -var-file=vars.json

The output return's exit 0

bash-5.0# terraform apply -backup='-' -input=false -auto-approve -var-file=vars.json

...

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

...

bash-5.0# echo $?
0

UPDATE: On more digging this looks to be a separate issue I am seeing with plan_only

• plan_only failing on ashald/terraform-provider-stateful provider #131

[ljfranklin]

If you see an error similar to @mstrisoline's in a plan_only run, see #131 (comment) for details and temporary workaround.