Make detection for test-binary more universal

[thaJeztah]

This has been reverted by #2235 in release v1.9.1

• relates to adds check to see if command name and os.Args[0] is the same before... #155
• relates to test spring-cleaning docker/cli#5224

When running tests in verbose mode (or other options), tests involving Cobra may fail if the test does not explicitly set Command.args to an empty slice; in this case, Cobra defaults to using os.Args, which will contain arguments passed to the test (such as -v (verbose)).

Commits e576205 and 1ef0913 implemented a workaround for this when running (unit) tests for Cobra itself, but this check is specifig to Cobra (checking for cobra.test), and don't work on Windows (which will have a .exe extension),

This patch implements a more universal check, so that users of Cobra as a module also benefit from this workaround.

go1.21 and up provides a testing.Testing() utility (1); as the Cobra module still supports Go1.16 and up, an alternative implementation was added for older versions, based on golang.org/x/mod/lazyregexp 2.

[marckhouzam]

@thaJeztah Thanks for this. I had never dug into the history of this cobra.test check, and I'm glad you've explained it so well.

I'm trying to reproduce the issue by removing this cobra.test check but haven't been able to.
I added this test to cobra:

func Test1(t *testing.T) {
	root := &Command{Use: "root", Run: func(_ *Command, args []string) {}}

	if _, err := root.ExecuteC(); err != nil {
		t.Errorf("error: %v", err)
	}
}

Then removed the cobra.test check, and added a printout of the args that are read by Cobra. Then ran go test:

$ go test -v -run ^Test1$ github.com/spf13/cobra
=== RUN   Test1
args:  [-test.testlogfile=/var/folders/k0/lwnx9y_n7393cfx7mj7dct580000gq/T/go-build3346663322/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true -test.run=^Test1$]
--- PASS: Test1 (0.00s)
PASS
ok  	github.com/spf13/cobra	(cached)

Notice that the test flags are processed by cobra, but the test still passes.
This led me to this piece of code in spf13/pflag, which explicitly ignores test.* flags.
https://github.com/spf13/pflag/blob/d5e0c0615acee7028e1e2740a11102313be88de1/flag.go#L1017

I wonder why things fail in your case?
Could you point me to a way to reproduce this issue so that I can see your PR at work?

[thaJeztah]

Ah, yeah, I probably should've provided a good reproducer when I was working on this 😂 (It's been a while). Let me try come with one 👍

IIRC this happened either when the test was pre-compiled and run, but also in some cases when running tests from my IDE (I recall I ran into these when working on docker/cli, and re-generating the .golden files generated by gotest.tools.

[thaJeztah]

I think I have an example (I saw I wrote one up in docker/cli#5224), and it was indeed when pre-compiling the tests;

Before this patch:

go test -c -o foo.test

./foo.test -test.run TestNoArgs
--- FAIL: TestNoArgs (0.00s)
    args_test.go:37: Unexpected output: Error: unknown command "TestNoArgs" for "c"
        Usage:
          c [flags]

        Flags:
          -h, --help   help for c

    args_test.go:40: Unexpected error: unknown command "TestNoArgs" for "c"
FAIL

After this patch:

go test -c -o foo.test

./foo.test -test.run TestNoArgs
PASS

I can update the commit message with that information 👍

[thaJeztah]

Updated the commit message for future reference 😄

[marckhouzam]

I think I have an example (I saw I wrote one up in docker/cli#5224), and it was indeed when pre-compiling the tests;

./foo.test -test.run TestNoArgs
--- FAIL: TestNoArgs (0.00s)
   args_test.go:37: Unexpected output: Error: unknown command "TestNoArgs" for "c"

[...]

Aha, I see. So the -test.* flags get ignored properly, but the flag values remain and become arguments to the cobra command.

I see that using the = form works, as I would expect: ./foo.test -test.run=TestNoArgs.
Let me review the proposed change and see.

[marckhouzam]

I pushed a commit addressing the comments and adding a unit test.
Thanks @thaJeztah for this.

[marckhouzam]

@thaJeztah If you have comments on the latest commit I pushed, please let me know and we can fix it in a follow up.

[thaJeztah]

Ah! Sorry, I was away from my keyboard, so didn't push updates; thanks @marckhouzam !

[marckhouzam]

This is causing some unit tests to fail from some projects using Cobra.
It turns out that the better detection of a test-binary is preventing those projects from setting os.Args in their test, since cobra no longer reads os.Args in unit tests.

For example, in the tanzu CLI:
https://github.com/vmware-tanzu/tanzu-cli/blob/624420e9f6db5ae55731800700075ae2d4bc8c72/pkg/command/root_test.go#L2218-L2222

And in helm (pretty funky test if you ask me):
https://github.com/helm/helm/blob/3a94215585b91d5ac41ebb258e376aa11980b564/cmd/helm/helm_test.go#L169-L173

I am able to fix the tests for the tanzu CLI, but it is more difficult for helm.
I still think this PR is correct, but I think that we need a way for programs be able to set the equivalent of os.Args in their tests. I'm still thinking of options.

cc @thaJeztah @jpmcb

[thaJeztah]

Hm.. that's unfortunate! I'd not be horribly upset if it was reverted; it was really a small quality-of-life improvement so that consumers wouldn't have to adjust their tests (I think I got most of them in our tests, but once in a while I run into another one 😂)

[marckhouzam]

I feel like the case that got broken is much less common than the case this PR fixed.

I was thinking Cobra could provide a global array of strings that a test can set when it needs to set the os.Args. Something like:

var ArgsFromTest []string

if c.args == nil {
  if !isTesting() {
    args = os.Args[1:]
  } else {
    args = ArgsFromTest
  }
}

However, I’m thinking that for developers to find this special variable when they need it, will be almost impossible, so I’m leaning toward reverting this PR.

@thaJeztah @jpmcb what do you think?

[thaJeztah]

Reverting for now sounds good to me; we can open a new PR to un-revert to work on an approach that would work and/or check with affected projects to see what a reasonable approach would be.

[jpmcb]

Reverting sounds fine although, I agree, I think there should be something besides developers having to set os.Args to get around a test binaries arguments: we probably need a more robust way to approach test scaffolding - I'd be curious to understand how other libraries, like Rust's clap, handle this case or enable testing.

[marckhouzam]

I've opened #2235 to blindly revert for the moment, so that we can do a 1.9.1 release today.