Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: TestMain on other packages are run when executing Run File (neotest) if the test package has _test suffix #222

Closed
2 tasks done
rbroggi opened this issue Nov 28, 2024 · 23 comments · Fixed by #225
Closed
2 tasks done
Labels
bug Something isn't working

Comments

@rbroggi
Copy link

rbroggi commented Nov 28, 2024

Did you check docs and existing issues?

Neovim version (nvim -v)

NVIM v0.10.2

Operating system/version

arch

Output from :checkhealth neotest-golang

neotest-golang: require("neotest-golang.health").check()

Requirements ~
- OK Binary 'go' found on PATH: /home/rb/.local/bin/go
- OK Found go.mod file for /home/rb/repo/efg/watch-catalogue/test/setup_test.go in /home/rb/repo/efg/watch-catalogue/go.mod
- OK Treesitter parser for go is installed
- OK neotest is available
- OK nvim-treesitter is available
- OK nio is available
- OK plenary is available

DAP (optional) ~
- OK Binary 'dlv' found on PATH: /home/rb/.local/share/nvim/mason/bin/dlv
- OK dap is available
- OK dapui is available
- OK dap-go is available

Gotestsum (optional) ~
- OK Binary 'gotestsum' not found on PATH

Describe the bug

neotest-golang: /home/rb/.local/share/nvim/lazy/nvim-nio/lua/nio/init.lua:119: The coroutine failed with this message: 
Vim:E974: Expected a Number or a String, Blob found
stack traceback:
	[C]: in function 'writefile'
	.../nvim/lazy/neotest-golang/lua/neotest-golang/process.lua:132: in function 'test_results'
	...are/nvim/lazy/neotest-golang/lua/neotest-golang/init.lua:149: in function 'results'
	...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:132: in function '_run_spec'
	...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:89: in function <...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:88>
stack traceback:
	...ocal/share/nvim/lazy/neotest/lua/neotest/client/init.lua:89: in function <...ocal/share/nvim/lazy/neotest/lua/neotest/client/init.lua:88>
	[C]: in function 'error'
	/home/rb/.local/share/nvim/lazy/nvim-nio/lua/nio/init.lua:119: in function 'gather'
	...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:94: in function '_run_tree'
	...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:65: in function <...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:22>
	[C]: in function 'xpcall'
	...ocal/share/nvim/lazy/neotest/lua/neotest/client/init.lua:84: in function 'run_tree'
	...al/share/nvim/lazy/neotest/lua/neotest/consumers/run.lua:85: in function 'func'
	/home/rb/.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:173: in function </home/rb/.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:172>

Steps To Reproduce

lazyvim config taken from the very repo and removed the argument "-count=1":

return {
  {
    "nvim-neotest/neotest",
    config = function()
      require("neotest").setup({
        adapters = {
          require("neotest-golang")({
            go_test_args = { "-count=1", "-tags=integration" },
            go_list_args = { "-tags=integration" },
            dap_go_opts = {
              delve = {
                build_flags = { "-tags=integration" },
              },
            },
          }),
        },
      })
    end,
  },
}

Expected Behavior

I would expect to see the test pass normally.

Your Lua setup

return {
  {
    "nvim-neotest/neotest",
    config = function()
      require("neotest").setup({
        adapters = {
          require("neotest-golang")({
            go_test_args = { "-count=1", "-tags=integration" },
            go_list_args = { "-tags=integration" },
            dap_go_opts = {
              delve = {
                build_flags = { "-tags=integration" },
              },
            },
          }),
        },
      })
    end,
  },
}
@rbroggi rbroggi added the bug Something isn't working label Nov 28, 2024
@rbroggi
Copy link
Author

rbroggi commented Nov 28, 2024

I would also add that by restoring the -count=1 the situation is maybe even worse:

the command to run test on file does not report any outcome.

@fredrikaverpil
Copy link
Owner

fredrikaverpil commented Nov 30, 2024

Hi @rbroggi and thanks for the report!

I can't reproduce this. I've replaced my config with the one you supplied and I can run tests just fine when "count=1" is present as well as when I remove it.

All the -count=1 option does is telling go test not to cache test results, so that the test will always run. This leads me to think it could be the integration build tag that is causing some sort of issue on your particular code base. Removing -count=1 or leaving it in likely just toggles caching of the test result.

  • Are you using an //go:build integration build tag in your Go code?
  • Have you tried using this neotest-golang config in a different project which does not use the integration build tag and had similar results? (for ruling out whether your config is universally not working)

Judging from the traceback, something unexpected is passed into async.fn.writefile, which is meant to write the test command output to file:

async.fn.writefile(cmd_output, cmd_output_path)

Incidentally, a rather recent addition to the codebase here was colorization of output. You can try disabling the colorize_test_output option, so we can rule out whether that is part of causing this problem.

But to really understand what the test command output looks like on your end, would it be possible for you to supply a debug-level log?
The README contains instructions on how to achieve this.
Or you can enter process.lua locally and add print(vim.inspect(cmd_output)) just before the async.fn.writefile operation.
Alternatively, you can attach to a running test (hit a when in the test summary on a running test) and observe any errors returned by go list or go test and report back here.

@rbroggi
Copy link
Author

rbroggi commented Nov 30, 2024

Thank you for the explanation. I will prolly not be able to do that today but will definitely follow up. Cheers

@fredrikaverpil
Copy link
Owner

fredrikaverpil commented Nov 30, 2024

No stress!
But in case this is a persistent problem for you, I would like to learn more about what might be happening - as you are unlikely to be alone in experiencing something like this.

By the way, I merged an unrelated change just now, which improves on showing build errors in the output. It's possible you had build errors but that it wasn't obvious that this was the case.

You can try it out by making sure you set neotest-golang to use what's on the main branch (e.g. by setting setting branch = "main" if you're using the lazy.nvim package manager). This will be made into a proper relase in the next couple of days: #221

@rbroggi
Copy link
Author

rbroggi commented Nov 30, 2024

Ok I've collected some more info:

  1. The program indeed fails on projects where I'm actively using the integration build tag.

Have you tried using this neotest-golang config in a different project which does not use the integration build tag and had similar results? (for ruling out whether your config is universally not working)

I've tried in more than one project but most of the projects I use actually have an integration build tags.

The issue happens also in tests not having the build tag.

I have done the branch = "main" thing but still facing the error:

neotest-golang: /home/rb/.local/share/nvim/lazy/nvim-nio/lua/nio/init.lua:119: The coroutine failed with this message: 
Vim:E974: Expected a Number or a String, Blob found
stack traceback:
	[C]: in function 'writefile'
	.../nvim/lazy/neotest-golang/lua/neotest-golang/process.lua:155: in function 'test_results'
	...are/nvim/lazy/neotest-golang/lua/neotest-golang/init.lua:149: in function 'results'
	...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:132: in function '_run_spec'
	...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:89: in function <...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:88>
stack traceback:
	...ocal/share/nvim/lazy/neotest/lua/neotest/client/init.lua:89: in function <...ocal/share/nvim/lazy/neotest/lua/neotest/client/init.lua:88>
	[C]: in function 'error'
	/home/rb/.local/share/nvim/lazy/nvim-nio/lua/nio/init.lua:119: in function 'gather'
	...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:94: in function '_run_tree'
	...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:65: in function <...al/share/nvim/lazy/neotest/lua/neotest/client/runner.lua:22>
	[C]: in function 'xpcall'
	...ocal/share/nvim/lazy/neotest/lua/neotest/client/init.lua:84: in function 'run_tree'
	...al/share/nvim/lazy/neotest/lua/neotest/consumers/run.lua:85: in function 'func'
	/home/rb/.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:173: in function </home/rb/.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:172>

I've managed to activate the debug log but the logs contain too much data that I think it's not a good idea to post in the open internet (private paths/folders and organization code).

The same thing unfortunately applies to the print statement - I couldn't see anything weird about it TBH. I will think a bit longer and try to come out with more evidences.

p.s.: I noticed that running the tests from nvim (I use lazyvim) takes WAY longer than running go test. Is that expected?

Also maybe another interesting evidence is that the issue happens only when running the Run File (neotest) target, whenever running the Run Nearest (neotest) the issue does not happen - and the times of execution are as expected (fast)

@fredrikaverpil
Copy link
Owner

For me, the absolutely most helpful next step (for figuring out what is wrong) is if I can reproduce this on my end.

Would it be possible for you to create a minimal main.go and main_test.go file that exhibits this problem?

@fredrikaverpil
Copy link
Owner

p.s.: I noticed that running the tests from nvim (I use lazyvim) takes WAY longer than running go test. Is that expected?

I don't think it taking "WAY longer" is expected, no. This depends on your project though.

Go does not run tests of individual files. Instead, it runs tests of Go packages. However, Neotest talks files, so when you say you want to run all tests in a file neotest-golang has to figure out which Go packages are actually related to the file(s) and which test functions reside in a certain file.

I'm leveraging go list for this file/package mapping. My guess is this is either go list is taking a long time or the build itself, done by go test.

You can see from the debug logs what the actual go list and go test commands are, and you can run them in the terminal and maybe easily spot what is going on.

Btw, the README explains how you can filter the (very large) debug log so it filters out only logs from neotest-golang.

@rbroggi
Copy link
Author

rbroggi commented Dec 1, 2024

Thank you for the advises, I now have a better visibility over what is going on...

I have sanitized the debug log for the Run File (neotest):

DEBUG | 2024-12-01T18:08:32Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Searching for go.mod in /my/project/path/internal/core/dto 
DEBUG | 2024-12-01T18:08:32Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Searching for go.mod in /my/project/path/internal/core
DEBUG | 2024-12-01T18:08:32Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Searching for go.mod in /my/project/path/internal
DEBUG | 2024-12-01T18:08:32Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Searching for go.mod in /my/project/path
DEBUG | 2024-12-01T18:08:32Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Found go.mod at /my/project/path/go.mod
DEBUG | 2024-12-01T18:08:32Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Running Go list: go list -json -tags=integration ./... in /my/project/path
DEBUG | 2024-12-01T18:08:33Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] JSON-decoded 'go list' output:  { {
INFO | 2024-12-01T18:08:33Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:59 | [neotest-golang] Test command: go test -json -count=1 -tags=integration ./... -run ^(^TestName1$|^TestName2$)$

It seems that it correctly goes up and find my go.mod

but instead of launching the process with the specific file package /my/project/path/internal/core/dto it runs from project root.

go test -json -count=1 -tags=integration ./... -run ^(^TestName1$|^TestName2$)$

instead of:

go test -json -count=1 -tags=integration ./internal/core/dto -run ^(^TestName1$|^TestName2$)$

seems to also ignore the run command altogether and I'm trying to understand why...

The Run Nearest (neovim) target instead works as I would expect:

DEBUG | 2024-12-01T18:32:34Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Running Go list: go list -json -tags=integration ./... in /my/project/path/internal/core/dto
DEBUG | 2024-12-01T18:32:34Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] JSON-decoded 'go list' output:  { {
INFO | 2024-12-01T18:32:34Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:59 | [neotest-golang] Test command: go test -json -count=1 -tags=integration /my/project/path/internal/core/dto -run ^TestName1$
DEBUG | 2024-12-01T18:32:34Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] RunSpec: {
INFO | 2024-12-01T18:32:34Z+0100 | ...nvim/lazy/neotest/lua/neotest/client/strategies/init.lua:36 | Starting process neotest-golang:/my/project/path-/my/project/path/internal/core/dto/query_test.go::TestNewName1 with strategy integrated
INFO | 2024-12-01T18:32:34Z+0100 | ...nvim/lazy/neotest/lua/neotest/client/strategies/init.lua:56 | Process for position neotest-golang:/my/project/path-/my/project/path/internal/core/dto/query_test.go::TestNewGetMatchQuery exited with code 0
DEBUG | 2024-12-01T18:32:34Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Runner 'go', raw output:  
DEBUG | 2024-12-01T18:32:34Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Final internal test result data {
DEBUG | 2024-12-01T18:32:34Z+0100 | .../nvim/lazy/neotest-golang/lua/neotest-golang/logging.lua:47 | [neotest-golang] Final Neotest result data {
DEBUG | 2024-12-01T18:32:34Z+0100 | ...hare/nvim/lazy/neotest/lua/neotest/client/state/init.lua:70 | New results for adapter neotest-golang:/my/project/path
DEBUG | 2024-12-01T18:32:34Z+0100 | ...hare/nvim/lazy/neotest/lua/neotest/client/state/init.lua:70 | New results for adapter neotest-golang:/my/project/path

Notice how here the target path was successfully used:

go test -json -count=1 -tags=integration /my/project/path/internal/core/dto -run ^TestName1$

@rbroggi
Copy link
Author

rbroggi commented Dec 1, 2024

FYI:

❯ go version
go version go1.23.3 linux/amd64

@rbroggi
Copy link
Author

rbroggi commented Dec 1, 2024

It seems like the go list -json ./... executes from different directories whether the target is to run Run file (executes from repo dir where go.mod is found), or Run Nearest (executes from the directory of the test file)

@fredrikaverpil
Copy link
Owner

fredrikaverpil commented Dec 7, 2024

Ok, interesting. Is it correct to say that the test command is only wrong for you in the case of running all tests of a file then?
Meaning, running all tests in a folder (in the neotest summary) or individual tests works ok?

@rbroggi
Copy link
Author

rbroggi commented Dec 7, 2024

Ok, interesting. Is it correct to say that the test command is only wrong for you in the case of running all tests of a file then?
Meaning, running all tests in a folder (in the neotest summary) or individual tests works ok?

Yes, that is right

@fredrikaverpil
Copy link
Owner

fredrikaverpil commented Dec 7, 2024

I'm having trouble reproducing this on my end.
I've set up the following: https://github.com/fredrikaverpil/go-playground/tree/main/bugs/neotest-golang

Could you please have a look and see if you can spot something which differs from the setup on your end?

This is what I do:

  1. cd bugs/neotest-golang.
  2. Launch nvim.
  3. I then open up internal/core/dto/dto_test.go in a neovim buffer.
  4. I execute the test using the test, file and dir strategies (from the neotest summary pane) and I can't see what you are seeing.

This is the output I get (I am using the gotestsum runner):

# run nearest test
Info  13:03:06 notify.info { "go", "list", "-json", "-tags=wireinject,tools,integration", "./..." }
Info  13:03:06 notify.info { "gotestsum", "--jsonfile=/var/folders/85/kqpgt4g50kg91tm0j34bmtc80000gp/T/nvim.fredrik/L3nhYX/18", "--format=standard-verbose", "--", "-v", "-count=1", "-race", "-coverprofile=/Users/fredrik/code/public/go-playground/bugs/neotest-golang/coverage.out", "-parallel=1", "-tags=wireinject,tools,integration", "/Users/fredrik/code/public/go-playground/bugs/neotest-golang/internal/core/dto", "-run", "^TestSomething$" }

# run file (dto_test.go)
Info  13:04:10 notify.info { "go", "list", "-json", "-tags=wireinject,tools,integration", "./..." }
Info  13:04:10 notify.info { "gotestsum", "--jsonfile=/var/folders/85/kqpgt4g50kg91tm0j34bmtc80000gp/T/nvim.fredrik/L3nhYX/22", "--format=standard-verbose", "--", "-v", "-count=1", "-race", "-coverprofile=/Users/fredrik/code/public/go-playground/bugs/neotest-golang/coverage.out", "-parallel=1", "-tags=wireinject,tools,integration", "github.com/fredrikaverpil/go-playground/bugs/neotest-golang/internal/core/dto", "-run", "^(^TestSomething$)$" }

# run dir (dto/)
Info  13:04:46 notify.info { "go", "list", "-json", "-tags=wireinject,tools,integration", "./..." }
Info  13:04:46 notify.info { "gotestsum", "--jsonfile=/var/folders/85/kqpgt4g50kg91tm0j34bmtc80000gp/T/nvim.fredrik/L3nhYX/26", "--format=standard-verbose", "--", "-v", "-count=1", "-race", "-coverprofile=/Users/fredrik/code/public/go-playground/bugs/neotest-golang/coverage.out", "-parallel=1", "-tags=wireinject,tools,integration", "github.com/fredrikaverpil/go-playground/bugs/neotest-golang/internal/core/dto" }

And here's the same sequence (nearest test, file, dir) with the go test runner:

# run nearest test
Info  13:05:42 notify.info { "go", "list", "-json", "-tags=wireinject,tools,integration", "./..." }
Info  13:05:42 notify.info { "go", "test", "-json", "-v", "-count=1", "-race", "-coverprofile=/Users/fredrik/code/public/go-playground/bugs/neotest-golang/coverage.out", "-parallel=1", "-tags=wireinject,tools,integration", "/Users/fredrik/code/public/go-playground/bugs/neotest-golang/internal/core/dto", "-run", "^TestSomething$" }

# run file
Info  13:05:51 notify.info { "go", "list", "-json", "-tags=wireinject,tools,integration", "./..." }
Info  13:05:51 notify.info { "go", "test", "-json", "-v", "-count=1", "-race", "-coverprofile=/Users/fredrik/code/public/go-playground/bugs/neotest-golang/coverage.out", "-parallel=1", "-tags=wireinject,tools,integration", "github.com/fredrikaverpil/go-playground/bugs/neotest-golang/internal/core/dto", "-run", "^(^TestSomething$)$" }

# run dir
Info  13:05:59 notify.info { "go", "list", "-json", "-tags=wireinject,tools,integration", "./..." }
Info  13:05:59 notify.info { "go", "test", "-json", "-v", "-count=1", "-race", "-coverprofile=/Users/fredrik/code/public/go-playground/bugs/neotest-golang/coverage.out", "-parallel=1", "-tags=wireinject,tools,integration", "github.com/fredrikaverpil/go-playground/bugs/neotest-golang/internal/core/dto" }

@fredrikaverpil
Copy link
Owner

fredrikaverpil commented Dec 7, 2024

Btw, since you are having this issue when executing all tests in a file, I'm pretty sure that for some reason, this part of the code is failing to figure out what your package name is.

-- find the go package that corresponds to the pos.path
local package_name = "./..."
local pos_path_filename = vim.fn.fnamemodify(pos.path, ":t")
local pos_path_foldername = vim.fn.fnamemodify(pos.path, ":h")
for _, golist_item in ipairs(golist_data) do
if golist_item.TestGoFiles ~= nil then
if
pos_path_foldername == golist_item.Dir
and vim.tbl_contains(golist_item.TestGoFiles, pos_path_filename)
then
package_name = golist_item.ImportPath
break
end
end
end

As you can see, it's parsing the output from go list -tags=integration ./... and from that attempt to infer which package name corresponds to the given file.

@rbroggi
Copy link
Author

rbroggi commented Dec 7, 2024

in your bug example it works well indeed. Lemme see if I can spot what differences are there.

@rbroggi
Copy link
Author

rbroggi commented Dec 7, 2024

I noticed something interesting:

I have a package that contains 2 *_test.go files, in one of the files the package is dto and in the other the package is dto_test.

You know, in go there is this good practice stated by some to append _test to package names in your test files so that you concentrate on testing your public methods (you only have access to the package public interface).

It seems that all works well for the package dto kind of test file and the issue happens for the package dto_test kind of test file.

@fredrikaverpil
Copy link
Owner

I'm trying to reproduce your use case, but I still can't see any errors:
https://github.com/fredrikaverpil/go-playground/tree/main/bugs/neotest-golang/internal/core/dto

Unless you can provide a minimal reproducible case, I don't think we will get anywhere here, unfortunately.

@rbroggi
Copy link
Author

rbroggi commented Dec 8, 2024

I will try to put a reproducer together. I just need to find the time. Thx for the support so far

@rbroggi
Copy link
Author

rbroggi commented Dec 8, 2024

Ok I think I found the error...

Here is the reproducer

It seems that when running Run File (neotest) it will still run TestMain files that are located in other directories...

Something quite interesting and suspect is that if we rename the package from dto_test to only dto the error does not occur:

image

image

So, it seems that when the package has the _test prefix - for some reason the execution will find and run the TestMain on other packages and when the package does not have the _test prefix it will not.

@rbroggi rbroggi changed the title bug: removal of count=1 from test arguments breaks execution of tests bug: TestMain on other packages are run when executing Run File (neotest) if the test package has _test suffix Dec 8, 2024
@rbroggi
Copy link
Author

rbroggi commented Dec 8, 2024

maybe some ideas could come from how Goland implements this (here I might be out of my way :D ):

go test -c -o <test-executable> <package-path> #gosetup
go tool test2json -t <test-executable> -test.v=test2json -test.paniconexit0 -test.run <regex>

@fredrikaverpil
Copy link
Owner

Allright, I found the bug now and I believe I know how to fix this. Many thanks for being persistent and providing all these details 🙏

@fredrikaverpil
Copy link
Owner

fredrikaverpil commented Dec 8, 2024

@rbroggi if you can, please give this a whirl:

return {
  {
    "nvim-neotest/neotest",
    dependencies = {
      "nvim-neotest/nvim-nio",
      "nvim-lua/plenary.nvim",
      "antoinemadec/FixCursorHold.nvim",
      "nvim-treesitter/nvim-treesitter",
      { 
        "fredrikaverpil/neotest-golang",
-       version = "*"
+       branch = "xtestgofiles",
	  },
    },
    config = function()
      require("neotest").setup({
        adapters = {
          require("neotest-golang"),
        },
      })
    end,
  },
}

fredrikaverpil added a commit that referenced this issue Dec 8, 2024
fredrikaverpil added a commit that referenced this issue Dec 8, 2024
fredrikaverpil added a commit that referenced this issue Dec 8, 2024
@rbroggi
Copy link
Author

rbroggi commented Dec 8, 2024

seems to have worked 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants