Test execution randomly ends prematurely #17282

pewscorner · 2021-09-04T17:31:45Z

Environment data

VS Code version: 1.60.0
Extension version (available under the Extensions sidebar): v2021.9.1191016588
OS and version: Windows 10
Python version (& distribution if applicable, e.g. Anaconda): 3.9.7
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
Relevant/affected Python packages and their versions: N/A
Relevant/affected Python-related VS Code extensions and their versions: Pylance v2021.9.0
Value of the python.languageServer setting: Pylance

Expected behaviour

When clicking on the "Run Tests" button, all discovered tests should run.

Actual behaviour

A random number of tests are run (sometimes all of them). There is nothing in the UI indicating why the testing stopped (but there's an error in the Python output log, see below). This behavior started after upgrading VS Code to 1.60.0.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

I simply run my 64-test test suite.

Logs

Test output terminal:

Running tests (unittest): d:\Documents\PythonProjects\PythonQuickRefTest\unittests
Running tests: d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py::TestClasses::test_about_classes_types_and_objects
./classes_unittest.py::TestClasses::test_about_classes_types_and_objects Passed
Running tests: d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py::TestClasses::test_class_creation_and_instantiation
./classes_unittest.py::TestClasses::test_class_creation_and_instantiation Passed
Running tests: d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py::TestClasses::test_class_inheritance
./classes_unittest.py::TestClasses::test_class_inheritance Passed
Running tests: d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py::TestClasses::test_descriptors
./classes_unittest.py::TestClasses::test_descriptors Passed
Running tests: d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py::TestClasses::test_property_attributes
Total number of tests expected to run: 64
Total number of tests run: 64
Total number of tests passed: 4
Total number of tests failed: 0
Total number of tests failed with errors: 0
Total number of tests skipped: 0
Finished running tests!

Python output

User belongs to experiment group 'pythonaa'
User belongs to experiment group 'pythonJediLSP'
User belongs to experiment group 'pythonSendEntireLineToREPL'
User belongs to experiment group 'pythonNotDisplayLinterPrompt'
User belongs to experiment group 'pythonDiscoveryModuleWithoutWatcher'
User belongs to experiment group 'pythonTensorboardExperiment'
User belongs to experiment group 'PythonPyTorchProfiler'
Python interpreter path: D:\Python\Python39\python.exe
> pyenv root
> conda --version
> python3.7 -c "import sys;print(sys.executable)"
> python3.6 -c "import sys;print(sys.executable)"
> python3 -c "import sys;print(sys.executable)"
> python2 -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> py -3.7 -c "import sys;print(sys.executable)"
> py -3.6 -c "import sys;print(sys.executable)"
> py -3 -c "import sys;print(sys.executable)"
> py -2 -c "import sys;print(sys.executable)"
> D:\Python\Python39\python.exe -c "import sys;print(sys.executable)"
> conda info --json
Error 2021-09-04 19:21:02: Detection of Python Interpreter for Command python3 and args  failed as file  does not exist
Error 2021-09-04 19:21:03: Failed to get interpreter information for 'D:\Python\Python25\python.exe' [Error: Command failed: "D:\\Python\Python25\python.exe" "c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\interpreterInfo.py"
Traceback (most recent call last):
  File "c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\interpreterInfo.py", line 4, in <module>
    import json
ImportError: No module named json

	at ChildProcess.exithandler (child_process.js:317:12)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)] {
  killed: false,
  code: 1,
  signal: null,
  cmd: '"D:\\\\Python\\Python25\\python.exe" "c:\\Users\\User\\.vscode\\extensions\\ms-python.python-2021.9.1191016588\\pythonFiles\\interpreterInfo.py"'
}
Error 2021-09-04 19:21:03: Failed to get interpreter information for 'D:\Python\Python24\python.exe' [Error: Command failed: "D:\\Python\Python24\python.exe" "c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\interpreterInfo.py"
Traceback (most recent call last):
  File "c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\interpreterInfo.py", line 4, in ?
    import json
ImportError: No module named json

	at ChildProcess.exithandler (child_process.js:317:12)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)] {
  killed: false,
  code: 1,
  signal: null,
  cmd: '"D:\\\\Python\\Python24\\python.exe" "c:\\Users\\User\\.vscode\\extensions\\ms-python.python-2021.9.1191016588\\pythonFiles\\interpreterInfo.py"'
}
Starting Pylance language server.
Error 2021-09-04 19:21:04: Failed to get interpreter information for 'C:\Users\User\AppData\Local\Microsoft\WindowsApps\python.exe' [Error: Command failed: "C:\\Users\User\AppData\Local\Microsoft\WindowsApps\python.exe" "c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\interpreterInfo.py"
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

	at ChildProcess.exithandler (child_process.js:317:12)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Socket.<anonymous> (internal/child_process.js:439:11)
	at Socket.emit (events.js:315:20)
	at Pipe.<anonymous> (net.js:673:12)] {
  killed: false,
  code: 9009,
  signal: null,
  cmd: '"C:\\\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\python.exe" "c:\\Users\\User\\.vscode\\extensions\\ms-python.python-2021.9.1191016588\\pythonFiles\\interpreterInfo.py"'
}
Error 2021-09-04 19:21:04: Failed to get interpreter information for 'C:\Users\User\AppData\Local\Microsoft\WindowsApps\python3.exe' [Error: Command failed: "C:\\Users\User\AppData\Local\Microsoft\WindowsApps\python3.exe" "c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\interpreterInfo.py"
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

	at ChildProcess.exithandler (child_process.js:317:12)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Socket.<anonymous> (internal/child_process.js:439:11)
	at Socket.emit (events.js:315:20)
	at Pipe.<anonymous> (net.js:673:12)] {
  killed: false,
  code: 9009,
  signal: null,
  cmd: '"C:\\\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\python3.exe" "c:\\Users\\User\\.vscode\\extensions\\ms-python.python-2021.9.1191016588\\pythonFiles\\interpreterInfo.py"'
}
Error 2021-09-04 19:21:04: Failed to get interpreter information for 'C:\Users\User\AppData\Local\Microsoft\WindowsApps\python3.exe' [Error: Command failed: "C:\\Users\User\AppData\Local\Microsoft\WindowsApps\python3.exe" "c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\interpreterInfo.py"
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

	at ChildProcess.exithandler (child_process.js:317:12)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Socket.<anonymous> (internal/child_process.js:439:11)
	at Socket.emit (events.js:315:20)
	at Pipe.<anonymous> (net.js:673:12)] {
  killed: false,
  code: 9009,
  signal: null,
  cmd: '"C:\\\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\python3.exe" "c:\\Users\\User\\.vscode\\extensions\\ms-python.python-2021.9.1191016588\\pythonFiles\\interpreterInfo.py"'
}
Error 2021-09-04 19:21:04: Failed to get interpreter information for 'C:\Users\User\AppData\Local\Microsoft\WindowsApps\python.exe' [Error: Command failed: "C:\\Users\User\AppData\Local\Microsoft\WindowsApps\python.exe" "c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\interpreterInfo.py"
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

	at ChildProcess.exithandler (child_process.js:317:12)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Socket.<anonymous> (internal/child_process.js:439:11)
	at Socket.emit (events.js:315:20)
	at Pipe.<anonymous> (net.js:673:12)] {
  killed: false,
  code: 9009,
  signal: null,
  cmd: '"C:\\\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\python.exe" "c:\\Users\\User\\.vscode\\extensions\\ms-python.python-2021.9.1191016588\\pythonFiles\\interpreterInfo.py"'
}
> conda --version
> D:\Python\Python39\python.exe -c "
import unittest
import inspect

def get_sourceline(obj):
    s, n = inspect.getsourcelines(obj)
    for i, v in enumerate(s):
        if v.strip().startswith('def'):
            return str(n+i)
    return '*'

def generate_test_cases(suite):
    for test in suite:
        if isinstance(test, unittest.TestCase):
            yield test
        else:
            for test_case in generate_test_cases(test):
                yield test_case

loader = unittest.TestLoader()
suite = loader.discover("./unittests", pattern="*unittest.py")

print("start")  # Don't remove this line
loader_errors = []
for s in generate_test_cases(suite):
    tm = getattr(s, s._testMethodName)
    testId = s.id()
    if testId.startswith("unittest.loader._FailedTest"):
        loader_errors.append(s._exception)
    else:
        print(testId.replace(".", ":") + ":" + get_sourceline(tm))

for error in loader_errors:
    try:
        print("=== exception start ===")
        print(error.msg)
        print("=== exception end ===")
    except:
        pass
"
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
> D:\Python\Python39\python.exe -c "
import unittest
import inspect

def get_sourceline(obj):
    s, n = inspect.getsourcelines(obj)
    for i, v in enumerate(s):
        if v.strip().startswith('def'):
            return str(n+i)
    return '*'

def generate_test_cases(suite):
    for test in suite:
        if isinstance(test, unittest.TestCase):
            yield test
        else:
            for test_case in generate_test_cases(test):
                yield test_case

loader = unittest.TestLoader()
suite = loader.discover("./unittests", pattern="*unittest.py")

print("start")  # Don't remove this line
loader_errors = []
for s in generate_test_cases(suite):
    tm = getattr(s, s._testMethodName)
    testId = s.id()
    if testId.startswith("unittest.loader._FailedTest"):
        loader_errors.append(s._exception)
    else:
        print(testId.replace(".", ":") + ":" + get_sourceline(tm))

for error in loader_errors:
    try:
        print("=== exception start ===")
        print(error.msg)
        print("=== exception end ===")
    except:
        pass
"
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_about_classes_types_and_objects --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_about_classes_types_and_objects --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
Error 2021-09-04 19:21:21: Error: read ECONNRESET
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_class_creation_and_instantiation --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_class_creation_and_instantiation --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
Error 2021-09-04 19:21:21: Error: read ECONNRESET
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_class_inheritance --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_class_inheritance --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
Error 2021-09-04 19:21:21: Error: read ECONNRESET
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_descriptors --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_descriptors --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
Error 2021-09-04 19:21:22: Error: read ECONNRESET
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_property_attributes --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
> D:\Python\Python39\python.exe c:\Users\User\.vscode\extensions\ms-python.python-2021.9.1191016588\pythonFiles\visualstudio_py_testlauncher.py --us=./unittests --up=*unittest.py --uvInt=2 --result-port=49811 -tclasses_unittest.TestClasses.test_property_attributes --testFile=d:\Documents\PythonProjects\PythonQuickRefTest\unittests\classes_unittest.py
cwd: d:\Documents\PythonProjects\PythonQuickRefTest
Error 2021-09-04 19:21:22: Error: read ECONNRESET
Error 2021-09-04 19:21:22: undefined

The text was updated successfully, but these errors were encountered:

karthiknadig · 2021-09-04T19:07:53Z

@pewscorner Are you using pytest or unittest?

pewscorner · 2021-09-04T19:42:39Z

@karthiknadig I'm using unittest.

Josephkready · 2021-09-05T01:34:25Z

I am also experiencing this after updating to 1.60.0

pvenderbosch · 2021-09-07T06:21:35Z

Using unittest and have the same issue since the update to 1.60.0

mrucker · 2021-09-07T20:52:25Z

I'm using untitest and have the same issue.

I have over 300 unit tests in my project. I haven't had it finish all 300+ test in a single pass one time since the update.

mrucker · 2021-09-08T01:28:01Z

One more small thing, I know this probably isn't the place for it, but I do really really like the look of the new test interface. Thanks for all your hard work!!!

karthiknadig · 2021-09-08T04:41:21Z

@mrucker Thanks for the feedback. Can you try this build and see if it improves the situation? https://github.com/microsoft/vscode-python/suites/3707450658/artifacts/90551474

mrucker · 2021-09-08T13:26:10Z

@karthiknadig This build is better but still wasn't 100% for me. I reran my entire test suite a few times and it completed all the tests once. The other five or so times it missed a handful of random tests (like 4/350 tests). Here's a screenshot to give you an idea.

pewscorner · 2021-09-11T10:27:32Z

I'm also seeing an improvement but not a 100% fix with the new Python extension v2021.9.1218897484. Out of 10 runs, my 64-test suite ran to completion 8 times, and skipped 1 test 2 times (not the same test).

@karthiknadig, shouldn't this be reopened?

pewscorner added bug Issue identified by VS Code Team member as probable bug triage-needed Needs assignment to the proper sub-team labels Sep 4, 2021

pvenderbosch mentioned this issue Sep 7, 2021

Unittest run all tests is extremely slow #17295

Closed

karthiknadig self-assigned this Sep 7, 2021

karthiknadig added area-testing needs PR and removed triage-needed Needs assignment to the proper sub-team labels Sep 7, 2021

karthiknadig mentioned this issue Sep 7, 2021

Fix issue with in complete unittest runs. #17313

Merged

karthiknadig closed this as completed in #17313 Sep 8, 2021

karthiknadig added this to the August Recovery milestone Sep 8, 2021

github-actions bot locked as resolved and limited conversation to collaborators Dec 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test execution randomly ends prematurely #17282

Test execution randomly ends prematurely #17282

pewscorner commented Sep 4, 2021

karthiknadig commented Sep 4, 2021

pewscorner commented Sep 4, 2021

Josephkready commented Sep 5, 2021

pvenderbosch commented Sep 7, 2021

mrucker commented Sep 7, 2021

mrucker commented Sep 8, 2021

karthiknadig commented Sep 8, 2021

mrucker commented Sep 8, 2021

pewscorner commented Sep 11, 2021

Test execution randomly ends prematurely #17282

Test execution randomly ends prematurely #17282

Comments

pewscorner commented Sep 4, 2021

Environment data

Expected behaviour

Actual behaviour

Steps to reproduce:

Logs

karthiknadig commented Sep 4, 2021

pewscorner commented Sep 4, 2021

Josephkready commented Sep 5, 2021

pvenderbosch commented Sep 7, 2021

mrucker commented Sep 7, 2021

mrucker commented Sep 8, 2021

karthiknadig commented Sep 8, 2021

mrucker commented Sep 8, 2021

pewscorner commented Sep 11, 2021