Test dicovery doesn't provide feedback on failure

[david-waterworth]

I find quite often my tests "disappear" from the test window, or discovery appears to do nothing. This is almost always because I've introduced a syntax error in the package I'm testing which results in an import failure. The reason is clear in the Python Test Log, i.e. below I forgot to install a dependency culminating in discovery failing with exit code 2. If this should happen at the very least the test sidebar should provide some indication that test discovery exited abnormally - perhaps with a highlight on the "Show Test Output" icon since clicking on that is the first step in resolving the issue.

Test Discovery failed: 
Error: ============================= test session starts ==============================

<snip>

    import ahocorasick # pip install pyahocorasick
E   ModuleNotFoundError: No module named 'ahocorasick'
=========================== short test summary info ============================
ERROR tests/test_label_model.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== 5 tests collected, 1 error in 2.00s ======================

Traceback (most recent call last):
  File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

[karthiknadig]

We have major improvements coming to testing UI. See here #16769 this is addressed there.

[karthiknadig]

Sorry. I misunderstood. The changes in #16769 provides a way for us to report test discovery errors in a better way. We need to hook this information we get from python to the node where the error occurs in the UI.

[aholmes]

Sorry. I misunderstood. The changes in #16769 provides a way for us to report test discovery errors in a better way. We need to hook this information we get from python to the node where the error occurs in the UI.

To confirm, does this mean the Test UI will not clear out when test discovery fails from, e.g., a syntax error?

[karthiknadig]

You will see something like this. We add a node for that workspace with the error details in the test-UI itself.

[aholmes]

Does that mean previously discovered tests will not still be displayed?

[karthiknadig]

Depends on the test framework, and type of error. On unittest, the test in that particular file with syntax error will be skipped. In pytest, currently syntax errors means it won't be able to see any of the other tests. We have plans on making that part better when we update our integration with pytest.

[aholmes]

In pytest, currently syntax errors means it won't be able to see any of the other tests.

Yeah this is a huge bummer w/ pytest.

We have plans on making that part better when we update our integration with pytest.

Great! Looking forward to it.

[eleanorjboyd]

Hello! We have just finished our testing rewrite and are beginning the roll out to users. I have tested this issue with the re-write and I am no longer reproducing the bug! If you would like to try it yourself, you need to add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. Let me know if it doesn’t work for you and we can re-open this issue. Thanks!

[aholmes]

Hi @eleanorjboyd, is there something else we need to do for this setting to take effect?

[aholmes]

Actually, it looks like it kind of works after putting the setting in my WSL config.

[eleanorjboyd]

You can confirm you have the rewrite enabled by setting "python.analysis.logLevel": "Trace", in your user settings then check for Experiment 'pythonTestAdapter' is active in your python logs. Do you see that in your logs?

[aholmes]

Looks like it is.

Here are my observations so far:

• The test extension takes a few seconds to appear on the left. It's the last one to "load."
• When I click on it, I first see this:
  
• A couple seconds later I see what's in my prior screenshot.

[aholmes]

By the way, I selected to install the pre-release version of the Python extension. I'm not sure whether that's needed to test the pythonTestAdapter change. I was running the release version prior to this.

[eleanorjboyd]

Hi! Could you send me your logs? I am wondering if there is an error there about UUIDs and if so would be connected to this: #21599