CTest integration not picking up test names on first compile

[reverendbedford]

Brief Issue Summary

I have a project where I am managing tests through Catch2. The CTest integration can successfully find, run, and display the results from my tests, but I'm facing an odd issue where it only works on the second compile of the project.

The first compile and run of Ctest finds no tests because instead of running the tests with a command of the form:

ctest -jxx -C Debug -T test --output-on-failure -R ^my_test_name-buildid

it instead runs

ctest -jxx -C Debug -T test --output-on-failure -R ^my_test_name_NOT_BUILT-buildid

The build itself actually looks fine, running ctest with no arguments in the build directory will run all the tests.

What's odd is that if I recompile (say by selecting a different build variant) and recompile/retest, changing nothing else, it will correctly pick up and run all the tests.

I can't figure out where the extension is picking up the test names, for example grepping for NOT_BUILT in the build directory returns nothing.

CMake Tools Diagnostics

No response

Debug Log

No response

Additional Information

No response

[benmcmorran]

Thanks for the report! I suspect the NOT_BUILT suffix is coming from Catch2, although I'm not familiar enough with Catch2 to understand why it does this. We will attempt to reproduce the issue to see if there's a way the CMake Tools extension can better handle this case and let you know if we need any more information.

[reverendbedford]

Thanks. Things do seem to work fine with the C++ TestMate extension, which is probably better integrated with Catch. It would be nice to be able to run the test properly from cmake-tools directly, but it's not a high priority if it doesn't happen.

[jeremicmilan]

I'm hitting the same issue but with GTest, but for me rebuilding is not helping. The fix for me is to refresh the tests (either through UI or with command cmake.refreshTests).

@benmcmorran, I have a similar file (<test_name>[1]_include.cmake) being generated in the build folder. Note that with GTest the suffix is slightly different.

if(EXISTS "<path>/<test_name>[1]_tests.cmake")
  include("<path>/<test_name>[1]_tests.cmake")
else()
  add_test(<test_name>_NOT_BUILT <test_name>_NOT_BUILT)
endif()

The file <path>/<test_name>[1]_tests.cmake does exist, and refresh is making the vscode integration pick it up.

A solid repro for me is to delete the build folder (so starting fresh) and clicking run all tests in the UI.
So, I guess the fix is to run test refresh after the build finishes and prior to running the tests (but automatically).

[dcoles]

I'm hitting the same issue but with GTest, but for me rebuilding is not helping. The fix for me is to refresh the tests (either through UI or with command cmake.refreshTests).

I spent quite a bit of time debugging this today. From what I can tell it looks like VSCode is querying and caching the tests before the initial build is complete. As such the corresponding _tests.cmake file is not in the build tree and the test gets stubbed out with a _NOT_BUILT test.

So I agree that it would make sense to refresh the tests any time that build is run that touches a test target.

[v-frankwang]

@dcoles Thank you for the follow up. Our developers will be working on this issue, thank you for your patience.

[dabrahams]

We are having the _NOT_BUILT problem with GTest (on first compile or recompile) and have discovered that disabling "test explorer integration" in the CMake plugin eliminates the problem. When test explorer integration is enabled, it changes the way ctest is invoked: instead of invoking ctest once for each test executable and letting it discover the tests, the plugin invokes ctest for each test the test explorer integration feature has (incorrectly?) discovered, with a regex filter designed to select just that test. But something is wrong with those invocations; I'm not sure what. You can reproduce this problem with this project

[v-frankwang]

@dabrahams Thank you very much for being able to ask this question, when I use this project you provided to reproduce the problem it will fail to configure, I can't be sure if this is something wrong with my operation, can you provide a simple reproduction project or a recording of the problem reproduced?

[dabrahams]

You need to add a minimal CMakePresets.json in the root, IIRC. I let the plugin generate one for me; it allowed me to reproduce:

Sorry, scratch that. I'll fix the project so you can reproduce it.

[sean-parent]

I just pushed an update to make it easier to reproduce.

Run CTest using the test-debug preset to see the error:

The result will contain failures like this on some of the tests:

[proc] Executing command: /usr/local/bin/ctest -C Debug --output-on-failure -R ^verbose_configuration_tests_NOT_BUILT$
[ctest] Test project /Users/sean-parent/repos/github.com/stlab/adobe-contract-checks/out/build/test-debug
[ctest] No tests were found!!!

The test explorer gets wedged -

My current list of extensions (not all are enabled, but there is no filter to list only enabled ones).

code --install-extension andruhovski.html-md-converter
code --install-extension antfu.slidev
code --install-extension bierner.github-markdown-preview
code --install-extension bierner.markdown-checkbox
code --install-extension bierner.markdown-emoji
code --install-extension bierner.markdown-footnotes
code --install-extension bierner.markdown-mermaid
code --install-extension bierner.markdown-preview-github-styles
code --install-extension bierner.markdown-yaml-preamble
code --install-extension cs128.cs128-clang-tidy
code --install-extension cschlosser.doxdocgen
code --install-extension dafny-lang.ide-vscode
code --install-extension davidanson.vscode-markdownlint
code --install-extension dbaeumer.vscode-eslint
code --install-extension eamodio.gitlens
code --install-extension editorconfig.editorconfig
code --install-extension esbenp.prettier-vscode
code --install-extension fredericbonnet.cmake-test-adapter
code --install-extension github.copilot
code --install-extension github.copilot-chat
code --install-extension github.vscode-github-actions
code --install-extension github.vscode-pull-request-github
code --install-extension gitworktrees.git-worktrees
code --install-extension hbenl.vscode-test-explorer
code --install-extension ivanhofer.git-assistant
code --install-extension jeff-hykin.better-cpp-syntax
code --install-extension kruemelkatze.vscode-dashboard
code --install-extension moshfeu.compare-folders
code --install-extension ms-azuretools.vscode-docker
code --install-extension ms-python.debugpy
code --install-extension ms-python.isort
code --install-extension ms-python.python
code --install-extension ms-python.vscode-pylance
code --install-extension ms-toolsai.jupyter
code --install-extension ms-toolsai.jupyter-keymap
code --install-extension ms-toolsai.jupyter-renderers
code --install-extension ms-toolsai.vscode-jupyter-cell-tags
code --install-extension ms-toolsai.vscode-jupyter-slideshow
code --install-extension ms-vscode-remote.remote-containers
code --install-extension ms-vscode-remote.remote-ssh
code --install-extension ms-vscode-remote.remote-ssh-edit
code --install-extension ms-vscode-remote.remote-wsl
code --install-extension ms-vscode-remote.vscode-remote-extensionpack
code --install-extension ms-vscode.cmake-tools
code --install-extension ms-vscode.cpptools
code --install-extension ms-vscode.cpptools-extension-pack
code --install-extension ms-vscode.cpptools-themes
code --install-extension ms-vscode.powershell
code --install-extension ms-vscode.remote-explorer
code --install-extension ms-vscode.remote-server
code --install-extension ms-vscode.test-adapter-converter
code --install-extension ms-vscode.vscode-speech
code --install-extension ms-vsliveshare.vsliveshare
code --install-extension ms-vsliveshare.vsliveshare-pack
code --install-extension mushan.vscode-paste-image
code --install-extension parallelsdesktop.parallels-desktop
code --install-extension pdconsec.vscode-print
code --install-extension raymondcamden.languagetool
code --install-extension redhat.java
code --install-extension rust-lang.rust-analyzer
code --install-extension ryu1kn.edit-with-shell
code --install-extension stkb.rewrap
code --install-extension streetsidesoftware.code-spell-checker
code --install-extension tomoki1207.pdf
code --install-extension twxs.cmake
code --install-extension tyriar.vscode-terminal-here
code --install-extension vadimcn.vscode-lldb
code --install-extension visualstudioexptteam.intellicode-api-usage-examples
code --install-extension visualstudioexptteam.vscodeintellicode
code --install-extension vscjava.vscode-gradle
code --install-extension vscjava.vscode-java-debug
code --install-extension vscjava.vscode-java-dependency
code --install-extension vscjava.vscode-java-pack
code --install-extension vscjava.vscode-java-test
code --install-extension vscjava.vscode-maven
code --install-extension xaver.clang-format
code --install-extension znck.grammarly

[dabrahams]

I have reproduced it with a much shorter list of extensions attached

[v-frankwang]

@dabrahams Thank you very much for your reply as well as the update, we are having the following problem with the project you provided, on the first Run tests all the tests give an error, on every subsequent click a different test gives an error. Does this reproduce the problem you mentioned?

Actual result:
First click:

Second click:

Third click:

[dabrahams]

I don't understand what this means:

Does this reproduce the problem you mentioned?

You're talking about a different set of steps. Whether it's the same problem is unclear. Are you asking whether I see the same symptoms if I follow your steps?

Please clarify, thanks.
@sean-parent maybe you can answer?

[v-frankwang]

@dabrahams I'm very sorry that I didn't understand your question, I thought your question was the same as the one mentioned in this issue.

[v-frankwang]

@gcampbell-msft I'm not really sure what the call logic is for this question: #3358 (comment), so I hope you can provide some help. :)

[dabrahams]

@dabrahams I'm very sorry that I didn't understand your question, I thought your question was the same as the one mentioned in this issue.

Now I'm even more confused. What question? Before this reply saying I didn't understand your question to me, I had no question. What I had was a reproducible example that shows some of the same symptoms as the OP, a workaround, and a partial explanation for why the workaround is effective. I assumed that information might be helpful to you in fully diagnosing and then fixing the problem.

Please let me know if there's anything further I can do to make this information more useful. If you think it's unrelated maybe it would help for me to file a separate issue?

[gcampbell-msft]

@dabrahams To be clear, it sounds this like this problem is well understood now.

It sounds like, when the test explorer integration is enabled, it works as expected, but when it isn't enabled, it doesn't behave as expected in this scenario.

@v-frankwang I don't believe this needs any further triaging, @dabrahams please confirm and I will label this accordingly and put it in our backlog.

[dabrahams]

It sounds like, when the test explorer integration is enabled, it works as expected, but when it isn't enabled, it doesn't behave as expected in this scenario.

No, it's exactly the opposite.

[gcampbell-msft]

@dabrahams Ah, thanks for the clarification, my apologies. I'll update the context.