Not Helpful Err When Running Tests For Provider about terraform-plugin-sdk

[bendbennett]

Thank you for the detailed bug report @cicoyle, appreciate you taking the time to provide all of the relevant details.

As discussed out of band, this will be addressed both within the terraform-plugin-sdk and terraform-plugin-testing repos.

[bflad]

Hi @cicoyle 👋

To ensure we're all on the same page, what is in your PATH environment variable when executing these tools? If this is within editor-specific functionality (you show a particular GoLand dialog), sometimes those will have their own PATH value that differs from a shell started in another manner, such as a separate terminal program or even a terminal within the same editor. Sometimes editors also support more global/project settings which could adjust the PATH as well. Unfortunately I'm not familiar with GoLand and couldn't immediately find documentation on how it creates/determines the PATH when running via that manner, but maybe someone else out there is familiar and can help in that regard. Another option may be to try making a quick test that prints os.Getenv("PATH") and running that test in a similar manner to those selecting the "wrong" executable.

Similar to other tooling which calls other executables, the provider testing and documentation generation tools should be using the PATH environment variable value to find a terraform (hopefully executable) file and return the first match in the precedence declared by the ordering of the PATH. I would anticipate the "fix" in this particular case is not actually changing the executable lookup logic, unless there is a specific bug in that logic where it is not respecting PATH ordering, but rather providing an improved error message when the selected file does not exist, does not have the correct executable permissions, or does not work as "expected" (in our context, likely double checking the output of a sub-command like terraform version before attempting other executable calls and returning a more confusing error).

Maybe to state it differently or more explicitly, if the given PATH includes an invalid file of higher precedence, the lookup logic should error rather than do something non-standard like skip the selected file to a further available executable or try to detect that multiple potential executables exist across PATH. The tooling logic does not (and likely should not) treat any directories as special or of higher precedence outside of the given PATH as this should match the expected behavior of other industry tooling dependent on PATH and should help prevent any unexpected behaviors introduced by doing something different within these projects.

---

After switching my command to include the TF_ACC_TERRAFORM_PATH env var: $ TF_ACC_TERRAFORM_PATH=/usr/local/bin TF_LOG=TRACE TF_ACC=1 go test -count=1 -run='TestAccMyResource' -v ./internal/provider I actually see a permission denied (probably since my binary is in /usr/local/bin?)

The TF_ACC_TERRAFORM_PATH environment variable is specific to this particular project and should specify the full path (including the filename) to the expected Terraform executable. The GoLand dialog correctly has terraform at the end, which is likely why it doesn't get a permission error like the other output. This is case where we could potentially raise a better error message including the path to the expected executable and potentially provide hints to the developer based on return the Go error type.

[cicoyle]

Also worth noting, I see the same err when using the terraform-plugin-docs so the same fix will likely need to be applied there. I was running make docs and see the signal killed:

$ make docs                         
go run github.com/hashicorp/terraform-plugin-docs/cmd/tfplugindocs
rendering website for provider "terraform-provider-myprovider" (as "terraform-provider-myprovider")
exporting schema from Terraform
compiling provider "myprovider"
using Terraform CLI binary from PATH if available, otherwise downloading latest Terraform CLI binary
running terraform init
Error executing command: unable to generate website: signal: killed


exit status 1
make: *** [docs] Error 1

The moment I mv my second binary to a backup it works just fine. so:

$ mv /Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform /Users/cassiecoyle/go/src/github.com/cloud-makefiles/bin/terraform-backup 

$ make docs
go run github.com/hashicorp/terraform-plugin-docs/cmd/tfplugindocs
rendering website for provider "terraform-provider-myprovider" (as "terraform-provider-myprovider")
exporting schema from Terraform
compiling provider "myprovider"
using Terraform CLI binary from PATH if available, otherwise downloading latest Terraform CLI binary
running terraform init
getting provider schema
rendering missing docs
generating missing resource content
generating template for ...
generating missing provider content
generating template for "terraform-provider-myprovider"
rendering static website
cleaning rendered website dir
rendering templated website to static markdown
rendering "data-sources/item.md.tmpl"
...

With the tfplugindocs, I am unable to set which path to my binary to choose. The randomly choosing a binary in my path doesn't work for me :/

[cicoyle]

Hello!

From terminal, my path is:

$ echo $PATH                                                                                  
/usr/local/bin/google-cloud-sdk/bin:/Users/cassiecoyle/go/src/github.com/cloud-makefiles/scripts/tools/../../bin:/Users/cassiecoyle/go/bin:/Users/cassiecoyle/go/src/github.com/cloud-sre/bin:/Library/PostgreSQL/14/bin:/opt/homebrew/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin

From in Goland, I added to my acceptance test: fmt.Printf("HERE is my PATH: %s", os.Getenv("PATH")) and see:

HERE is my PATH: /usr/local/bin/google-cloud-sdk/bin:/Users/cassiecoyle/go/src/github.com/cloud-makefiles/scripts/tools/../../bin:/Users/cassiecoyle/go/bin:/Users/cassiecoyle/go/src/github.com/cloud-sre/bin:/Library/PostgreSQL/14/bin:/opt/homebrew/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/usr/local/go/bin:/Users/cassiecoyle/go/bin--- PASS: TestAccMyResource (0.68s)

Im not entirely sure why my Goland vs terminal commands return diff results when the path is relatively the same order wise, with cloud-makefiles coming up as one of the first ones.

Fair enough, I've not looked at the ordering of things in my PATH before bc it's never been a problem before, I guess I can update that accordingly. Better err messaging upfront would be sufficient. I had no clue I had multiple binaries, let alone the fact that I could specify a certain one with only 1 terraform tool. Maybe that could help get around this too if there were env vars or flags that could be provided to the tfplugindocs? Although maybe that is just a bandaid to the problem, and I should update my PATH ordering regardless, but it is interesting you can specify the binary location for 1 terraform tool and not the others