Plugin installer doesn't filter by protocol version during installation ("Incompatible API version with plugin")

[apparentlymart]

At the time of writing in master, and in the v0.12.0-alpha1 build, we have partially-implemented support for auto-installing providers via the Terraform Registry.

This is currently lacking a mechanism to filter plugins by supported protocol version, and so the alpha1 build and any other builds made before this is fixed will see terraform init seem to successfully install plugins that are not actually compatible:

$ terraform init

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "http" (1.0.1)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.http: version = "~> 1.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

...but then any follow-up command will produce a compatibility error:

$ terraform apply

Error: Failed to instantiate provider "http" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

The registry protocol changes to allow Terraform to recognize and filter by plugin protocol version are in progress but did not complete before alpha1. In the final release this will be handled during terraform init and will produce a more informative error message:

Provider "http" v1.0.0 is not compatible with Terraform v0.12.0.

Provider version v2.0.0 is the earliest compatible version.
Select it with the following version constraint:

    version = "~> 2.0"

[TheMacStack]

Im seeing this with the "AzureRM" and "Random" providers whilst testing v0.12.0-alpha2

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (2.0.0)...
- Downloading plugin for provider "azurerm" (1.18.0)...

Terraform has been successfully initialized!
$ terraform validate

Error: Failed to instantiate provider "azurerm" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

Error: Failed to instantiate provider "random" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

Its not clear if there is some workaround or different versions to get around this or i should wait it out to be fixed in the coming days?

[mildwonkey]

@TheMacStack you have a few options. The providers bundled with the alpha release will work if you drop them in the same directory as your terraform configuration files.

You can also try build any additional providers you want to test on 0.12-alpha* yourself; you will need to update them to use the latest version of terraform.

I am also aiming to release another alpha bundle early this week if you'd like to wait and use those providers, though the same caveat remains - the provider binaries bundled with the alpha need to be moved alongside your terraform configuration files are in, or in one of the well-known plugin directories.

[apparentlymart]

Hi @TheMacStack!

There are now a few different situations we've seen that lead to this symptom:

1. Using a provider that is not included in the alpha bundles at all.
2. Using the alpha against a Terraform configuration that has version constraints on the providers that effectively exclude the specific versions included with the alpha. To ensure that the alpha builds are selected, it's necessary to either have no version constraints at all or to constrain to exactly the version number (including the additional build suffix) included in the alpha.
3. Having existing plugin binaries already installed in .terraform/plugins or another search location which takes precedence over the ones in the same directory as the alpha Terraform binary.

Reasons 1 and 3 cannot apply to you here, because you are clearly using two providers that are included in the alpha2 archive and we can see terraform init actually downloading the two providers, which it would not do if it thought it already had suitable plugins available locally.

That leaves reason 2 as the remaining option for the moment. Do you have version constraints in your configuration? If so, does it help to temporarily remove them? If this is not the case then it is possible you've found a new bug we haven't seen yet.

[TheMacStack]

Thanks for the replies, I tried with and without any version constraints but it diddnt appear to help. I have a container image that i use as a build bot in ci/cd, pre built with a fork of your Kubernetes Provider (in ~/.terraform.d/plugins/) and some other utils (kubectl, helm etc), maybe the presence of the custom provider is throwing it off? (it is not used in the config from above example but is present in the image.)

I'll keep playing with it and post back if i work it out, excited for tf 12! Thanks for your efforts.

[apparentlymart]

Once terraform init has downloaded and installed providers they will "stick" in .terraform/plugins, so you may need to delete that directory and run terraform init again for Terraform to switch to the bundled alpha build plugins after removing version constraints.

[shellshock1953]

can someone describe what directories from alpha2 zip-file need to be moved into projects .terraform/ dir?

[sjwl]

This is what I did on my Mac:
download the alpha2 zip file
https://releases.hashicorp.com/terraform/0.12.0-alpha2/terraform_0.12.0-alpha2_darwin_amd64.zip

in that zip it contains

Oct 31 00:45 terraform
Oct 30 23:01 terraform-provider-aws_v1.40.0-6-gb23683732_x4
Oct 31 00:46 terraform-provider-azurerm_v1.17.0-5-ga3b48ba3_x4
Oct 30 23:59 terraform-provider-google_v1.19.1-4-gf3de5334_x4
Oct 30 23:17 terraform-provider-local_v1.1.0-5-ga2df742_x4
Oct 30 23:18 terraform-provider-null_v1.0.0-5-gf54ff98_x4
Oct 30 23:19 terraform-provider-random_v2.0.0-5-g612dff2_x4
Oct 30 23:20 terraform-provider-template_v1.0.0-5-g317c9c9_x4

I copied terraform to $(which terraform) which was /usr/local/bin for me.
Then cd to the directory where you want to run terraform on. terraform init Then delete all the plugins under .terraform/plugins/darwin_amd64/ then copy terraform-provider-* files above to .terraform/plugins/darwin_amd64/. I think you may have to run terraform init once again.

[apparentlymart]

Here's a different way to do it, which is what we've been doing for testing:

1. Unzip the entire archive into a completely new directory, such as ~/terraform-v0.12.0-alpha2
2. Add that directory to your path locally within your current shell: export PATH=$HOME/terraform-v0.12.0-alpha2:$PATH
3. In that same shell, make a fresh clone of whatever repository you want to test against, separate from any you use day-to-day.
4. Run terraform init from that directory. The executable at ~/terraform-v0.12.0-alpha2/terraform will search for plugins matching ~/terraform-v0.12.0-alpha2/terraform-provider-* and will skip trying to download plugins if they are already satisfied by executables in that directory.

The advantage of this approach is you can keep everything separate from your "real" Terraform installation and working directory and so avoid problems both with upgrading and downgrading again later.

[radekg]

I understand this is unrelated but, any plans on supporting provisioners the same way as providers. As in, making them work with terraform init?

[apparentlymart]

Installable provisioners are not in scope for Terraform v0.12, and we've not yet planned any subsequent releases. Let's keep this focused on the topic of making the plugin installer filter by provider version.

[fredrikhgrelland]

We are using the consul and vsphere provider, which are not present in the alpha bundle, and are getting the Incompatible API version with plugin. Plugin version: 4, Client versions: [5]
Is there any way of testing terraform .12 before the release version? Can we download a compatible plugin from somewhere? Desperately needs the new HCL features and would love to get a head start.

[apparentlymart]

Hi @fredrikhgrelland,

In order to keep the overhead of preparing the alpha releases low we included only a subset of the plugins since it requires doing a custom build of an existing version against an updated provider SDK. In principle you can do this yourself if you have a Go toolchain by upgrading the vendored copy of the Terraform repository to the v0.12.0-alpha2 tag and building it, after which you can copy the built binary into the same directory where you extracted everything else.

There will be 0.12-compatible provider releases available for automatic installation before we get to the beta releases though, so don't worry about the v0.12.0 final release taking you by surprise... given the scope of this release we'll have at least two beta releases before final, to maximize the opportunities to exercise as many different edges of different providers as possible.

[fredrikhgrelland]

@apparentlymart Thank you for the feedback. I'll take a stab at compiling as it is always nice to be able to give some constructive feedback.

[fii]

`$ ./terraform plan

Error: Failed to instantiate provider "kubernetes" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

$ ./terraform version
Terraform v0.12.0-alpha2

• provider.kubernetes v1.4.0

./terraform plan

Seeing the same issue - any update on this please ?

[splashx]

@apparentlymart if I wish to test the openstack module with 0.12, how to proceed? Should I ping the folks at the provider's github page? 'Cause I see some test provider bundled here at terraform core @ alpha2.. 🤔

[apparentlymart]

I can't provide instructions for individual providers since each of them has its own dependencies and thus different challenges in getting the SDK upgraded which the individual provider teams are in a better position to handle. Some providers might be straightforward for those who are already familiar with Go dependency management, but others require more care due to common transitive dependencies with Terraform Core that may need to be carefully managed during upgrade.

The easiest solution if you're not a member of a provider dev team is to wait until 0.12.0-beta1, which will come once all (or at least, almost all) providers have 0.12-compatible versions available for installation from the main releases repository. That will be the opportunity to cautiously test against real infrastructure, while the alphas are there to exercise the general language features against temporary placeholder infrastructure from the subset of providers that are included.

[sjwl]

@splashx if you feel comfortable recompiling terraform-provider-openstack with the appropriate 0.12 version of the terraform packages, then heads up that you'll run into bugs. I found one documented here: https://github.com/terraform-providers/terraform-provider-openstack/issues/493. It has a workaround PR mentioned in it as well.

Otherwise as @apparentlymart suggested, perhaps wait for a downloadable v0.12 terraform-provider-openstack

[sjwl]

This recompile method seems to work for any provider/provisioner I've run into so far.

This example is with terraform-provider-external

cd $GOPATH/src/github.com/terraform-providers
git clone https://github.com/terraform-providers/terraform-provider-external.git
cd terraform-provider-external
rm -rf vendor
govendor init
govendor fetch github.com/hashicorp/terraform/...@=v0.12.0-alpha2
govendor fetch +m
go build

you should see the 0.12 binary terraform-provider-external in the current directory

[apparentlymart]

This is now fixed in master by #19977, and will be included in the forthcoming 0.12.0 final release.

There are not any compatible providers available for installation yet, so trying master right now will lead to a different error message about that. v0.12-compatible providers (that is, providers compatible with protocol version 5) will be gradually released in the coming weeks.

[pcj]

Appreciate all your hard work. Not sure how anyone is could realistically use terraform above 0.11.x without compatible providers that can be downloaded by the init command. @apparentlymart Any more details (other than "in the coming weeks") to get providers available with protocol version 5?

[pcj]

Used the govendor recompile approach detailed by @sjwl for the azurerm provider. It compiled cleanly, but attempting to use it in a "plan" command resulted in:

2019-01-16T08:54:41.914-0700 [DEBUG] plugin: starting plugin: path=/Users/pcj/go/src/github.com/terraform-providers/terraform-provider-azurerm/terraform-provider-azurerm args=[/Users/pcj/go/src/github.com/terraform-providers/terraform-provider-azurerm/terraform-provider-azurerm]
2019-01-16T08:54:41.920-0700 [DEBUG] plugin: plugin started: path=/Users/pcj/go/src/github.com/terraform-providers/terraform-provider-azurerm/terraform-provider-azurerm pid=21256
2019-01-16T08:54:41.920-0700 [DEBUG] plugin: waiting for RPC address: path=/Users/pcj/go/src/github.com/terraform-providers/terraform-provider-azurerm/terraform-provider-azurerm
2019-01-16T08:54:41.949-0700 [INFO]  plugin.terraform-provider-azurerm: configuring server automatic mTLS: timestamp=2019-01-16T08:54:41.949-0700
2019-01-16T08:54:41.983-0700 [DEBUG] plugin.terraform-provider-azurerm: plugin address: address=/var/folders/ft/hhfr3jns16n3g89p5j86zy6h0000gn/T/plugin112259503 network=unix timestamp=2019-01-16T08:54:41.983-0700
2019-01-16T08:54:41.983-0700 [DEBUG] plugin: using plugin: version=5
2019/01/16 08:54:41 [TRACE] GRPCProvider: GetSchema
2019/01/16 08:54:42 [TRACE] GRPCProvider: Close
2019-01-16T08:54:42.053-0700 [DEBUG] plugin: plugin process exited: path=/Users/pcj/go/src/github.com/terraform-providers/terraform-provider-azurerm/terraform-provider-azurerm pid=21256
2019-01-16T08:54:42.053-0700 [DEBUG] plugin: plugin exited
[tf-err]:
Error: Failed to retrieve schema from provider "azurerm": rpc error: code = Unimplemented desc = unknown service tfplugin5.Provider

[YakDriver]

I've been able to get the latest AWS provider and latest Terraform 0.12 dev working together by using Go modules and a little trickery. I'm guessing the same method would work with other providers. It's all detailed here:
https://medium.com/@dirk.avery/terraform-0-12-development-a-step-by-step-guide-to-get-running-with-providers-cea87082ad54
(Include workaround for hashicorp/terraform-provider-aws#7661.)

[apparentlymart]

As an interim solution we've published a set of development snapshots, which are builds directly from the master branch at the date and time included in their version numbers. As I write this they are just over one week old, because there have not been significant changes since then. These are the best way to test in the mean time while we work towards the beta1 release.

There were some challenges in getting the provider upgrades done while other provider work continued, so at present only a small number of providers have 0.12.0-compatible releases available for automatic installation with terraform init. Rather than continuing to postpone the v0.12.0-beta1 release we intend to release that beta version soon along with instructions to use the dev snapshots for any providers that do not yet have v0.12-compatible stable releases, and then the individual provider development teams will make their releases as soon as they are ready, making terraform init auto-install get increasingly more functional with each release. Most of them should be updated by the time we reach v0.12.0-beta2, and of course by final we'd like to have them all working.

Look out for the v0.12.0-beta1 announcement soon. In the mean time, the snapshot releases are probably more convenient than building from source to test.

[leandro-lucarella-sociomantic]

I'm bumping into this issue now with 0.12.0-beta2. I tried to go to the providers snapshot page and downloaded the dev20190415H16 version. Since that didn't work (init works but plan still says "Error: Failed to instantiate provider "github" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]" for the github provider), I also downloaded the terraform snapshot with the same version from the same page, but still I get the same error.

$ terraform init
Initializing modules...

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.github: version = "~> 1.4"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform version
Terraform v0.12.0-dev20190415H16
+ provider.github v1.4.0-dev20190415h16-dev

$ ls .terraform/plugins/linux_amd64/
lock.json  terraform-provider-github_v1.4.0-dev20190415H16-dev_x4

$ terraform plan

Error: Failed to instantiate provider "github" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

Is this expected? Am I doing something wrong?

[apparentlymart]

It seems like some of the dev20190415H16 snapshots did not build correctly and have picked up the wrong release of the Terraform SDK. Those builds were produced by trying to force the vendored SDK onto a different commit than the provider naturally depends on, but I suspect that now that more of them are themselves using Go Modules the other dependencies in the codebase acted as constraints that prevented the upgrade from going through as expected.

You may have better luck with the dev20190216H00 snapshots... they are a little older and so may run into some SDK bugs that have already been fixed, but they should at least all consistently be using an SDK release that supports protocol 5.

The focus of the Terraform team at HashiCorp is on getting first-class releases out that are compatible with v0.12, so I expect we will not be able to make any further dev snapshot releases without delaying the release. With that said, unfortunately I think that means that for the sake of beta2 we'll need to focus testing on the providers that are either already v0.12-compatible in a real release (there are 20 of these at the time of writing, including aws, google, etc) or that have compatible dev snapshots. More real releases will show up gradually in the coming weeks as we prepare for the v0.12.0 final release.

Sorry for that oversight in the snapshot build process... it's a batch job that ran mostly unattended, building from the latest master branch of each repository, so the functionality of those releases cannot be guaranteed. (They are, essentially, "nightly builds", albeit not running nightly.)

[carrajola]

Do the following command:
terraform init -upgrade

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.