Not Helpful Err When Running Tests For Provider

[cicoyle]

SDK version

$ cat go.mod | grep terraform-plugin-sdk
github.com/hashicorp/terraform-plugin-sdk/v2 v2.26.1

$ terraform version
Terraform v1.4.6
on darwin_arm64

$ terraform version -json
{
  "terraform_version": "1.4.6",
  "platform": "darwin_arm64",
  "provider_selections": {},
  "terraform_outdated": false
}

$ /Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform version     
zsh: killed    

$ /Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform version -json
zsh: killed  

Relevant provider source code

Generic test code, since the repo I am working with is not yet public...

func TestAccMyResource(t *testing.T) {
	resource.Test(t, resource.TestCase{
		ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
		Steps: []resource.TestStep{
			{
				Config: providerConfig + `resource "my_resource" "example" {}`,
...
			},
		},
	})
}

Terraform Configuration Files

resource "my_resource" "example" {
  my_name           = "example"
}

Debug Output

TLDR: I have my terraform binary in my /usr/local/bin, but apparently I have another terraform binary within this directory too: /Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform. When running my tests with the below command, you can see the main problem is: test_terraform_path=/Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform where the sdk testing code is looking for any terraform binary on my machine and using the one I didn't expect.

$ TF_LOG=TRACE TF_ACC=1 go test -count=1 -run='TestAccMyResource' -v ./internal/provider
=== RUN   TestAccMyResource
2023-05-26T14:01:13.909-0500 [TRACE] sdk.helper_resource: Validating TestCase: test_name=TestAccMyResource
2023-05-26T14:01:13.909-0500 [TRACE] sdk.helper_resource: Validating TestStep: test_step_number=1 test_name=TestAccMyResource
2023-05-26T14:01:13.909-0500 [DEBUG] sdk.helper_resource: Starting TestCase: test_name=TestAccMyResource
2023-05-26T14:01:13.909-0500 [TRACE] sdk.helper_resource: Adding potential Terraform CLI source of local filesystem PATH lookup: test_name=TestAccMyResource
2023-05-26T14:01:13.909-0500 [TRACE] sdk.helper_resource: Adding potential Terraform CLI source of checkpoint.hashicorp.com latest version for installation in: /var/folders/wp/bzngx46j2r7f9bpr56w_gg_h0000gq/T/plugintest-terraform2323115692: test_name=TestAccMyResource
2023-05-26T14:01:13.909-0500 [DEBUG] sdk.helper_resource: Found Terraform CLI: test_name=TestAccMyResource test_terraform_path=/Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform
2023-05-26T14:01:13.909-0500 [TRACE] sdk.helper_resource: Symlinking source directories to work directory: test_name=TestAccMyResource test_working_directory=/var/folders/wp/bzngx46j2r7f9bpr56w_gg_h0000gq/T/plugintest3579151791/work1719932836
    my_resource_test.go:11: failed to create new working directory: unable to disable terraform-exec provider verification: signal: killed
        
--- FAIL: TestAccMyResource (0.01s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-<my-provider>/internal/provider	0.098s
FAIL

After switching my command to include the TF_ACC_TERRAFORM_PATH env var: $ TF_ACC_TERRAFORM_PATH=/usr/local/bin TF_LOG=TRACE TF_ACC=1 go test -count=1 -run='TestAccMyResource' -v ./internal/provider I actually see a permission denied (probably since my binary is in /usr/local/bin?):

$ TF_ACC_TERRAFORM_PATH=/usr/local/bin TF_LOG=TRACE TF_ACC=1 go test -count=1 -run='TestAccMyResource' -v ./internal/provider
=== RUN   TestAccMyResource
2023-05-26T14:00:47.702-0500 [TRACE] sdk.helper_resource: Validating TestCase: test_name=TestAccMyResource
2023-05-26T14:00:47.703-0500 [TRACE] sdk.helper_resource: Validating TestStep: test_name=TestAccMyResource test_step_number=1
2023-05-26T14:00:47.703-0500 [DEBUG] sdk.helper_resource: Starting TestCase: test_name=TestAccMyResource
2023-05-26T14:00:47.703-0500 [TRACE] sdk.helper_resource: Adding potential Terraform CLI source of exact path: /usr/local/bin: test_name=TestAccMyResource
cannot run Terraform provider tests: failed to find or install Terraform CLI from [0x140000bb080]: unable to find, install, or build from 1 sources: 1 error occurred:
	* permission denied


FAIL	github.com/hashicorp/terraform-provider-<my-provider>/internal/provider	0.407s
FAIL

Upon looking at my terraform binary file permissions and trying to update it, I'm not sure why I see permission denied in terminal:

$ ls -la /usr/local/bin | grep terraform
-rwxr-xr-x@   1 cassiecoyle  staff   71044112 Apr 26 13:10 terraform

$ chmod 777 /usr/local/bin/terraform  
$ cassiecoyle@cassiecoyle-J9Y2X4P6L4 terraform-provider-waypoint % ls -la /usr/local/bin | grep terraform
-rwxrwxrwx@   1 cassiecoyle  staff   71044112 Apr 26 13:10 terraform

I still see the same permission denied.

But, when I run the tests in my Goland IDE, I somehow get past the permission denied and it runs just fine. This is my Goland IDE config:

Expected Behavior

Ideally, I should get notified of any errs upfront before giving an error that doesn't help me out as a user. Seeing the err: failed to create new working directory: unable to disable terraform-exec provider verification: signal: killed when running my tests doesn't give me anything helpful to work with. I have no clue how to debug this based on that err output. Like, why is it trying to create a new working directory at the point when I'm running my provider tests, and why does it need to disable terraform-exec? And why is the signal killed? I would like a better err before getting to this point.

Secondly, regarding the signal: killed, that is also equally unhelpful as output from the terraform version. Why is the signal killed when running a simple command like terraform version?

I would like to see better errs outputted upfront rather then at the point where I'm running tests, and something from terraform version that says, oh, looks like you you have multiple terraform binaries. I can't use this terraform binary version for what you are doing. try upgrading or specifying the terraform binary with this env var...

Last, I would expect the sdk test code to look for a terraform binary in the typical /usr/local/bin first, then if not there look elsewhere, or if there are more than 1 binary, then /usr/local/bin should be default or override the other locations. I expect most binaries to be in /usr/local/bin.

Actual Behavior

I can't run my provider tests from terminal due to the above errs.

Steps to Reproduce

Build out a simple provider. Have more than 1 terraform binary, both of diff versions. Allow the sdk to pick the binary you don't expect and when running the go tests see this output: failed to create new working directory: unable to disable terraform-exec provider verification: signal: killed.

References

Internal references in Slack

[bendbennett]

Thank you for the detailed bug report @cicoyle, appreciate you taking the time to provide all of the relevant details.

As discussed out of band, this will be addressed both within the terraform-plugin-sdk and terraform-plugin-testing repos.

[cicoyle]

Also worth noting, I see the same err when using the terraform-plugin-docs so the same fix will likely need to be applied there. I was running make docs and see the signal killed:

$ make docs                         
go run github.com/hashicorp/terraform-plugin-docs/cmd/tfplugindocs
rendering website for provider "terraform-provider-myprovider" (as "terraform-provider-myprovider")
exporting schema from Terraform
compiling provider "myprovider"
using Terraform CLI binary from PATH if available, otherwise downloading latest Terraform CLI binary
running terraform init
Error executing command: unable to generate website: signal: killed


exit status 1
make: *** [docs] Error 1

The moment I mv my second binary to a backup it works just fine. so:

$ mv /Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform /Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform-backup 

$ make docs
go run github.com/hashicorp/terraform-plugin-docs/cmd/tfplugindocs
rendering website for provider "terraform-provider-myprovider" (as "terraform-provider-myprovider")
exporting schema from Terraform
compiling provider "myprovider"
using Terraform CLI binary from PATH if available, otherwise downloading latest Terraform CLI binary
running terraform init
getting provider schema
rendering missing docs
generating missing resource content
generating template for ...
generating missing provider content
generating template for "terraform-provider-myprovider"
rendering static website
cleaning rendered website dir
rendering templated website to static markdown
rendering "data-sources/item.md.tmpl"
...

With the tfplugindocs, I am unable to set which path to my binary to choose. The randomly choosing a binary in my path doesn't work for me :/

[bflad]

Hi @cicoyle 👋

To ensure we're all on the same page, what is in your PATH environment variable when executing these tools? If this is within editor-specific functionality (you show a particular GoLand dialog), sometimes those will have their own PATH value that differs from a shell started in another manner, such as a separate terminal program or even a terminal within the same editor. Sometimes editors also support more global/project settings which could adjust the PATH as well. Unfortunately I'm not familiar with GoLand and couldn't immediately find documentation on how it creates/determines the PATH when running via that manner, but maybe someone else out there is familiar and can help in that regard. Another option may be to try making a quick test that prints os.Getenv("PATH") and running that test in a similar manner to those selecting the "wrong" executable.

Similar to other tooling which calls other executables, the provider testing and documentation generation tools should be using the PATH environment variable value to find a terraform (hopefully executable) file and return the first match in the precedence declared by the ordering of the PATH. I would anticipate the "fix" in this particular case is not actually changing the executable lookup logic, unless there is a specific bug in that logic where it is not respecting PATH ordering, but rather providing an improved error message when the selected file does not exist, does not have the correct executable permissions, or does not work as "expected" (in our context, likely double checking the output of a sub-command like terraform version before attempting other executable calls and returning a more confusing error).

Maybe to state it differently or more explicitly, if the given PATH includes an invalid file of higher precedence, the lookup logic should error rather than do something non-standard like skip the selected file to a further available executable or try to detect that multiple potential executables exist across PATH. The tooling logic does not (and likely should not) treat any directories as special or of higher precedence outside of the given PATH as this should match the expected behavior of other industry tooling dependent on PATH and should help prevent any unexpected behaviors introduced by doing something different within these projects.

---

After switching my command to include the TF_ACC_TERRAFORM_PATH env var: $ TF_ACC_TERRAFORM_PATH=/usr/local/bin TF_LOG=TRACE TF_ACC=1 go test -count=1 -run='TestAccMyResource' -v ./internal/provider I actually see a permission denied (probably since my binary is in /usr/local/bin?)

The TF_ACC_TERRAFORM_PATH environment variable is specific to this particular project and should specify the full path (including the filename) to the expected Terraform executable. The GoLand dialog correctly has terraform at the end, which is likely why it doesn't get a permission error like the other output. This is case where we could potentially raise a better error message including the path to the expected executable and potentially provide hints to the developer based on return the Go error type.

[cicoyle]

Hello!

From terminal, my path is:

$ echo $PATH                                                                                  
/usr/local/bin/google-cloud-sdk/bin:/Users/cassiecoyle/go/src/github.com/cloud-makefiles/scripts/tools/../../bin:/Users/cassiecoyle/go/bin:/Users/cassiecoyle/go/src/github.com/cloud-sre/bin:/Library/PostgreSQL/14/bin:/opt/homebrew/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin

From in Goland, I added to my acceptance test: fmt.Printf("HERE is my PATH: %s", os.Getenv("PATH")) and see:

HERE is my PATH: /usr/local/bin/google-cloud-sdk/bin:/Users/cassiecoyle/go/src/github.com/cloud-makefiles/scripts/tools/../../bin:/Users/cassiecoyle/go/bin:/Users/cassiecoyle/go/src/github.com/cloud-sre/bin:/Library/PostgreSQL/14/bin:/opt/homebrew/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/usr/local/go/bin:/Users/cassiecoyle/go/bin--- PASS: TestAccMyResource (0.68s)

Im not entirely sure why my Goland vs terminal commands return diff results when the path is relatively the same order wise, with cloud-makefiles coming up as one of the first ones.

Fair enough, I've not looked at the ordering of things in my PATH before bc it's never been a problem before, I guess I can update that accordingly. Better err messaging upfront would be sufficient. I had no clue I had multiple binaries, let alone the fact that I could specify a certain one with only 1 terraform tool. Maybe that could help get around this too if there were env vars or flags that could be provided to the tfplugindocs? Although maybe that is just a bandaid to the problem, and I should update my PATH ordering regardless, but it is interesting you can specify the binary location for 1 terraform tool and not the others