dev_overrides in .terraformrc appears to break acceptance tests (make testacc/TF_ACC=1)

[spkane]

I am working on a currently unpublished Terraform provider. The source code is here, but it is not in the registry at the moment.

The provider works, but I have dev_overrides set in my .terraformrc file this causes the acceptance tests for only the resource to fail, as the tests seem to try and run a terraform init, which does not work when dev_overrides are set.

Module version

github.com/hashicorp/terraform-plugin-framework v1.1.1

Relevant provider source code

• main.go

func main() {
	providerserver.Serve(context.Background(), inventory.New, providerserver.ServeOpts{
		Address: "registry.terraform.io/superorbital/inventory",
	})
}

• provider_test.go

package inventory

import (
	"github.com/hashicorp/terraform-plugin-framework/providerserver"
	"github.com/hashicorp/terraform-plugin-go/tfprotov6"
)

const (
	// providerConfig is a shared configuration to combine with the actual
	// test configuration so the Inventory client is properly configured.
	providerConfig = `terraform {
  required_providers {
    inventory = {
      source = "superorbital/inventory"
    }
  }
}

# Configure the connection details for the Inventory service
provider "inventory" {
}
`
)

var (
	// testAccProtoV6ProviderFactories are used to instantiate a provider during
	// acceptance testing. The factory function will be invoked for every Terraform
	// CLI command executed to create a provider server to which the CLI can
	// reattach.
	testAccProtoV6ProviderFactories = map[string]func() (tfprotov6.ProviderServer, error){
		"inventory": providerserver.NewProtocol6WithError(New()),
	}
)

• item_resource_test.go

package inventory

import (
	"testing"

	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
)

func TestAccItemResource(t *testing.T) {
	resource.Test(t, resource.TestCase{
		ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
		Steps: []resource.TestStep{
			// Create and Read testing
			{
				Config: providerConfig + `
resource "inventory_item" "test" {
    name = "bottle"
    tag  = "rare"
}
`,
				Check: resource.ComposeAggregateTestCheckFunc(
					resource.TestCheckResourceAttr("inventory_item.test", "name", "bottle"),
					resource.TestCheckResourceAttr("inventory_item.test", "tag", "rare"),
					// Verify dynamic values have any value set in the state.
					resource.TestCheckResourceAttrSet("inventory_item.test", "id"),
				),
			},
			// ImportState testing
			{
				ResourceName:      "inventory_item.test",
				ImportState:       true,
				ImportStateVerify: true,
			},
			// Update and Read testing
			{
				Config: providerConfig + `
resource "inventory_item" "test" {
    name = "jet"
    tag  = "SR-71 Blackbird"
}
`,
				Check: resource.ComposeAggregateTestCheckFunc(
					resource.TestCheckResourceAttr("item_item.test", "name", "jet"),
					resource.TestCheckResourceAttr("item_item.test", "tag", "SR-71 Blackbird"),
					// Verify dynamic values have any value set in the state.
					resource.TestCheckResourceAttrSet("item_item.test", "id"),
				),
			},
			// Delete testing automatically occurs in TestCase
		},
	})
}

Terraform Configuration Files

• ~/.terraformrc

provider_installation {
  dev_overrides {
    "superorbital/inventory" = "/Users/spkane/dev/go/path/bin/"
  }
  direct {}
}

• Makefile

default: install

generate:
	$(info ************  Generating Provider Documentation  ************)
	go generate ./...

install:
	$(info ************  Building and Installing Binary  ************)
	go install .

test:
	$(info ************  Running Unit Tests  ************)
	go test -v -count=1 -parallel=4 ./...

testacc:
	$(info ************  Running Acceptance Tests  ************)
	TF_ACC=1 go test -count=1 -parallel=4 -timeout 10m -v ./...

Debug Output

N/A

Expected Behavior

I expect the acceptance tests to work just fine with the local provider code, and to not complain about the provider not existing in the registry.

Actual Behavior

The tests appear to try to run terraform init which appears to cause issues in either case.

• With dev_overerides set. The data source test works, but the resource test fails (despite using the same terraform and provider block).

$ make testacc

************  Running Acceptance Tests  ************
TF_ACC=1 go test -count=1 -parallel=4 -timeout 10m -v ./...
?   	github.com/superorbital/terraform-provider-inventory	[no test files]
=== RUN   TestAccItemDataSource
--- PASS: TestAccItemDataSource (1.44s)
=== RUN   TestAccItemResource
    item_resource_test.go:10: Error running init: exit status 1

        Error: Failed to query available provider packages

        Could not retrieve the list of available versions for provider
        superorbital/inventory: provider registry registry.terraform.io does not have
        a provider named registry.terraform.io/superorbital/inventory

        All modules should specify their required_providers so that external
        consumers will get the correct providers when using a module. To see which
        modules are currently depending on superorbital/inventory, run the following
        command:
            terraform providers

--- FAIL: TestAccItemResource (2.13s)
FAIL
FAIL	github.com/superorbital/terraform-provider-inventory/inventory	3.579s
FAIL
make: *** [testacc] Error 1

• With dev_overerides disabled. Neither test works.

$ make testacc

************  Running Acceptance Tests  ************
TF_ACC=1 go test -count=1 -parallel=4 -timeout 10m -v ./...
?   	github.com/superorbital/terraform-provider-inventory	[no test files]
=== RUN   TestAccItemDataSource
    item_data_source_test.go:11: Step 1/1 error: Error running pre-apply refresh:
        Error: Inconsistent dependency lock file

        The following dependency selections recorded in the lock file are
        inconsistent with the current configuration:
          - provider registry.terraform.io/superorbital/inventory: required by this configuration but no version is selected

        To make the initial dependency selections that will initialize the dependency
        lock file, run:
          terraform init
--- FAIL: TestAccItemDataSource (0.23s)
=== RUN   TestAccItemResource
    item_resource_test.go:10: Step 1/3 error: Error running pre-apply refresh:
        Error: Inconsistent dependency lock file

        The following dependency selections recorded in the lock file are
        inconsistent with the current configuration:
          - provider registry.terraform.io/superorbital/inventory: required by this configuration but no version is selected

        To make the initial dependency selections that will initialize the dependency
        lock file, run:
          terraform init
--- FAIL: TestAccItemResource (0.21s)
FAIL
FAIL	github.com/superorbital/terraform-provider-inventory/inventory	0.451s
FAIL
make: *** [testacc] Error 1

Steps to Reproduce

1. TF_ACC=1 go test -count=1 -parallel=4 -timeout 10m -v ./...

References

N/A

[spkane]

I have published the provider and found a bug in the code, which made things work for now, but I still think there is an underlying problem here.

The code had a search-and-replace error in the update test for the resource, which became very apparent once I published the provider and got around the terraform init registry issue.

  diff --git a/inventory/item_resource_test.go b/inventory/item_resource_test.go
  index 43d4b41..3ccc54a 100644
  --- a/inventory/item_resource_test.go
  +++ b/inventory/item_resource_test.go
  @@ -40,10 +40,10 @@ resource "inventory_item" "test" {
   }
   `,
              Check: resource.ComposeAggregateTestCheckFunc(
  -              resource.TestCheckResourceAttr("item_item.test", "name", "jet"),
  -              resource.TestCheckResourceAttr("item_item.test", "tag", "SR-71 Blackbird"),
  +              resource.TestCheckResourceAttr("inventory_item.test", "name", "jet"),
  +              resource.TestCheckResourceAttr("inventory_item.test", "tag", "SR-71 Blackbird"),
                 // Verify dynamic values have any value set in the state.
  -              resource.TestCheckResourceAttrSet("item_item.test", "id"),
  +              resource.TestCheckResourceAttrSet("inventory_item.test", "id"),
              ),
           },
           // Delete testing automatically occurs in TestCase

However, I am pretty sure that this was just a secondary issue and is not related to the failure that was occurring when the provider was unpublished. Now that it is published, I can not reproduce the error when dev_overrides is set. Without the dev_overrides set it still fails as before.

[spkane]

Also, changing the dev_overrides line between these two settings makes no difference (at least now that it is published).

• "registry.terraform.io/superorbital/inventory" = "/Users/spkane/dev/go/path/bin/"
  versus
• "superorbital/inventory" = "/Users/spkane/dev/go/path/bin/"

[muratcelep]

I'm running into the same issue like OP. For a use such as 'Create and Read testing' or 'Update and Read testing' one can get away setting resource.TestStep.ImportState to false. However, for use case 'ImportState testing' , resource.TestStep.ImportState needs to be set true and then my test fails with the same error mentioned by OP.

[bflad]

Hi @spkane 👋 Thank you for raising this issue and sorry you ran into trouble here.

From reading through this report, it appears that this issue is limited to functionality for acceptance testing providers, which in your case is being managed by the github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource package. I do not believe there is anything to be done in the terraform-plugin-framework Go module for this report, so I'm going to use GitHub's issue transfer functionality to move this issue to the https://github.com/hashicorp/terraform-plugin-sdk repository.

[bflad]

Now that this was transferred, some additional context --

The Terraform CLI dev_overrides functionality directly interferes with how Terraform Providers are typically orchestrated by the testing code in helper/resource. Declaring a dev_overrides configuration is considered an escape hatch for provider developers wishing to manually test their provider code with real Terraform commands. The testing code meanwhile expects to fully manage the lifecycle of Terraform Provider binaries by starting and stopping the provider server via the TestCase/TestStep type ProtoV5ProviderFactories/ProtoV6ProviderFactories field definitions (similarly the Providers and ProviderFactories fields represent older ways to declare providers without explicitly writing the terraform-plugin-sdk provider server code; it was just implicitly handled). When managing the provider server lifecycle, Terraform CLI is executed with a special TF_REATTACH_PROVIDERS environment variable that enables the running provider server to be discovered within Terraform's internals without its own provider server management. This enables the provider code-under-test to be instrumented with tooling such as debuggers, etc.

This leaves the testing code in an awkward position here. Detecting whether a provider is being managed via dev_overrides is difficult as Terraform CLI does not implement commands or an interface to retrieve this type of configuration. Reimplementing some or all of that internal Terraform logic to detect the expected configuration file (based off Terraform's loading preferences), parse it, then see if this may be an issue for the test seems problematic.

The best recommendation for now I think would be to disable the usage of dev_overrides when performing automated testing, either by commenting out the entry, moving the file, or pointing the TF_CLI_CONFIG_FILE environment variable elsewhere. Installing the provider server binary and using dev_overrides is not necessary for successfully testing unreleased provider code.

[spkane]

So, this is good to know.

However, I am not sure that this completely resolves the issue. In the description, I also mention that With dev_overerides disabled. Neither test works.

Re-working my workflow to avoid using dev_overrides, except when actually testing using real terraform commands is probably fine, but I was working on a provider that had yet to be published on the Hashicorp plugin registry and it looked like something in the test was trying to reach out to the registry during the init phase.

$ make testacc

************  Running Acceptance Tests  ************
TF_ACC=1 go test -count=1 -parallel=4 -timeout 10m -v ./...
?   	github.com/superorbital/terraform-provider-inventory	[no test files]
=== RUN   TestAccItemDataSource
    item_data_source_test.go:11: Step 1/1 error: Error running pre-apply refresh:
        Error: Inconsistent dependency lock file

        The following dependency selections recorded in the lock file are
        inconsistent with the current configuration:
          - provider registry.terraform.io/superorbital/inventory: required by this configuration but no version is selected

        To make the initial dependency selections that will initialize the dependency
        lock file, run:
          terraform init
--- FAIL: TestAccItemDataSource (0.23s)
=== RUN   TestAccItemResource
    item_resource_test.go:10: Step 1/3 error: Error running pre-apply refresh:
        Error: Inconsistent dependency lock file

        The following dependency selections recorded in the lock file are
        inconsistent with the current configuration:
          - provider registry.terraform.io/superorbital/inventory: required by this configuration but no version is selected

        To make the initial dependency selections that will initialize the dependency
        lock file, run:
          terraform init
--- FAIL: TestAccItemResource (0.21s)
FAIL
FAIL	github.com/superorbital/terraform-provider-inventory/inventory	0.451s
FAIL
make: *** [testacc] Error 1

Maybe this is caused by something else in the environment?

[bflad]

Apologies if I misunderstood something about your report. One potential quirk with having an explicit:

terraform {
  required_providers {
    inventory = {
      source = "superorbital/inventory"
    }
  }
}

In the test configuration, is that the testing code may require an additional TF_ACC_PROVIDER_NAMESPACE=superorbital environment variable set. This configuration is not normally necessary for acceptance testing (as it will get implicitly added), but the explicit required_providers configuration may be causing it to be required.

I believe you can verify whether this is the case by enabling TF_LOG=TRACE in your test execution and looking for a TRACE entry with Setting Terraform CLI reattach configuration -- if the reattach configuration has registry.terraform.io/hashicorp/inventory then Terraform would raise this error since there would be a mismatch of provider addresses between the reattach configuration and the one defined explicitly in the configuration. Setting TF_ACC_PROVIDER_NAMESPACE should swap out hashicorp in that reattach configuration.