Tests do not respect unittest.subTest

[jakelishman]

Environment

• Qiskit Terra version: 0.19.0dev0 @ f6ab3a9
• Python version: any
• Operating system: any

What is happening?

Tests do not run subsequent unittest.subTest blocks if one fails, and do not report the parameterisations in use at the time of the failure.

How can we reproduce the issue?

An example test file, called test_subtests.py:

import unittest
import qiskit.test.base

class Direct(unittest.TestCase):
    def test_failure(self):
        with self.subTest("failure1"):
            self.assertTrue(False)
        with self.subTest("failure2"):
            self.assertTrue(False)

class Base(qiskit.test.base.BaseQiskitTestCase):
    def test_failure(self):
        with self.subTest("failure1"):
            self.assertTrue(False)
        with self.subTest("failure2"):
            self.assertTrue(False)

class Full(qiskit.test.base.FullQiskitTestCase):
    def test_failure(self):
        with self.subTest("failure1"):
            self.assertTrue(False)
        with self.subTest("failure2"):
            self.assertTrue(False)

and run with either python -munittest or stestr run gives an example output:

F
======================================================================
FAIL: test_failure (test_subtests.Base) [failure1]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jake/code/qiskit/terra/test_subtests.py", line 14, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

======================================================================
FAIL: test_failure (test_subtests.Base) [failure2]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jake/code/qiskit/terra/test_subtests.py", line 16, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

======================================================================
FAIL: test_failure (test_subtests.Direct) [failure1]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jake/code/qiskit/terra/test_subtests.py", line 7, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

======================================================================
FAIL: test_failure (test_subtests.Direct) [failure2]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jake/code/qiskit/terra/test_subtests.py", line 9, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

======================================================================
FAIL: test_failure (test_subtests.Full)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/Users/jake/code/qiskit/terra/test_subtests.py", line 21, in test_failure
    self.assertTrue(False)
  File "/Users/jake/.miniconda3/envs/qiskit/lib/python3.9/unittest/case.py", line 682, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true


----------------------------------------------------------------------
Ran 3 tests in 0.005s

FAILED (failures=5)

What should happen?

The test_subtests.Full case should also report two failures, and display the message "failure1" or "failure2" on a fail.

Any suggestions?

Likely a problem with testtools or the code we've imported from there into FullQiskitTestCase, similar to #7307, but I didn't want to put them in the same issue in case one part is much easier to solve than the other.

[mtreinish]

I believe this is a bug in testtools result handler (especially after #7311) that we use primarily so we can get attachments of stdout, stderr, and pylogging when running in parallel with stestr (without this they just print all at the same time making it hard to debug). We'll have to investigate and report upstream I think to try and fix this.

[jakelishman]

It's more than just the results handler - the tests actually fail out and stop running on the first failure. I made a simple test that leaked its run state out by touching some files:

import unittest
import testtools
import subprocess

class Unittest(unittest.TestCase):
    def test_failure(self):
        with self.subTest("failure1"):
            with open("u1.txt", "w"):
                pass
            self.assertTrue(False)
        with self.subTest("failure2"):
            with open("u2.txt", "w"):
                pass
            self.assertTrue(False)

class Testtools(testtools.TestCase):
    def test_failure(self):
        with self.subTest("failure1"):
            with open("t1.txt", "w"):
                pass
            self.assertTrue(False)
        with self.subTest("failure2"):
            with open("t2.txt", "w"):
                pass
            self.assertTrue(False)


if __name__ == "__main__":
    subprocess.run(["rm", "-f", "u1.txt", "u2.txt", "t1.txt", "t2.txt"])
    unittest.main()

and the result is that t2.txt isn't created. I've opened testing-cabal/testtools#317 to track.