Tests do not respect unittest.expectedFailure

[jakelishman]

Environment

• Qiskit Terra version: 0.19.0.dev0 @ f6ab3a9
• Python version: any
• Operating system: any

What is happening?

If you use the unittest.expectedFailure decorator on a test in a QiskitTestCase class, the test just reports a regular failure rather than a correct xfail.

How can we reproduce the issue?

This is a sample test file test_failures.py:

import unittest
import qiskit.test.base

class Direct(unittest.TestCase):
    @unittest.expectedFailure
    def test_failure(self):
        self.assertTrue(False)

class Base(qiskit.test.base.BaseQiskitTestCase):
    @unittest.expectedFailure
    def test_failure(self):
        self.assertTrue(False)

class Full(qiskit.test.base.FullQiskitTestCase):
    @unittest.expectedFailure
    def test_failure(self):
        self.assertTrue(False)

The output of unittest is something like:

xxF
======================================================================
FAIL: test_failure (test_failures.Full)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/Users/jake/code/qiskit/terra/test_failures.py", line 20, in test_failure
    self.assertTrue(False)
  File "/Users/jake/.miniconda3/envs/qiskit/lib/python3.9/unittest/case.py", line 682, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true


----------------------------------------------------------------------
Ran 3 tests in 0.005s

FAILED (failures=1, expected failures=2)

What should happen?

The test in the FullQiskitTestCase should also report an expected failure.

Any suggestions?

Likely related to some of the testtools code we've imported into FullQiskitTestCase, or maybe even testtools itself.