Make binary testing compatible with Terraform 0.15

[kmoe]

Terraform 0.15 deprecates a number of CLI arguments, including the [DIR] positional argument used in terraform init and terraform apply. The binary test framework made use of these arguments as a way of working around plugin source limitations in v1. This workaround is no longer needed.

The main difference here is that previously we would instantiate one tfexec.Terraform per test config file, but wd.workDir set as the working directory. Since we no longer have to symlink the plugin binary itself in wd.workDir, we can now just use wd.configDir as the Terraform working dir instead.

Provider source files (testdata in directories alongside test source files) must still be symlinked into the Terraform working directory. Since this is now wd.configDir, not wd.workDir, and each wd.workDir has several wd.configDirs, this symlinking must unfortunately be carried out more often. Performance should not be impacted as we should still be symlinking only a small number of directories.

Example test temp directory structure before this change:

/tmp/plugintest594266265
└── work819712804
    ├── config403653211
    │   └── terraform_plugin_test.tf
    ├── config502955059
    │   └── terraform_plugin_test.tf
    ├── terraform.tfstate
    ├── testdata -> /home/katy/dev/go/src/github.com/hashicorp/terraform-provider-random/internal/provider/testdata
    └── tfplan

Example test temp directory structure after this change:

/tmp/plugintest099393048
└── work460918423
    ├── config243863818
    │   ├── terraform_plugin_test.tf
    │   └── testdata -> /home/katy/dev/go/src/github.com/hashicorp/terraform-provider-random/internal/provider/testdata
    └── config848766255
        ├── terraform_plugin_test.tf
        ├── terraform.tfstate
        ├── testdata -> /home/katy/dev/go/src/github.com/hashicorp/terraform-provider-random/internal/provider/testdata
        └── tfplan

Update following 515c542

In order to reinstate the automatic persistence of terraform.tfstate between test steps, 515c542 simplifies the test temp directory structure by removing config dirs altogether. The structure is now:

/tmp/plugintest594266265
└── work819712804
    ├── terraform_plugin_test.tf
    ├── terraform.tfstate
    ├── testdata -> /home/katy/dev/go/src/github.com/hashicorp/terraform-provider-random/internal/provider/testdata
    └── tfplan

where terraform_plugin_test.tf gets overwritten each time a new test config is set.

Testing

Tested this change locally with the following providers:

• terraform-provider-random
• terraform-provider-archive
• terraform-provider-dns

and the following Terraform versions:

• 0.14.5
• master as of 2nd February 2021 (c8f83e184b67a5f79255b2fffb86524c6c8ef811)

fixes #697

[paultyng]

This LGTM, not sure if you want @paddycarver to take a look too.

[kmoe]

Actually @paddycarver hold on, there seems to be a failure in the DNS provider testing. That provider may have been a bad choice. Sorting it out now.

[kmoe]

In fact I believe this might break certain test flows. Back to draft for now. State needs to be passed between steps more carefully.

[kmoe]

I believe we can simplify this even further by removing config dirs altogether. This allows for Terraform state to persist automatically between test steps, as it does currently on master. Please see "Update following 515c542" in the PR description and compare the three directory structures.

Either way, this PR is a substantial enough change in the binary test framework that I am running tests on at least one major cloud provider to catch regressions.

[kmoe]

AWS TC run ongoing.

[bflad]

515c542 looks good to me with AWS Provider acceptance testing with Terraform CLI 0.12.30, 0.13.5, 0.14.6, 0.15.0alpha20210127, and HEAD. 🚀

[paddycarver]

This makes sense to me and the test runs make me more confident.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.