Add support for attachments to test execution (#3982)

* Add support for attachments to test execution

By default in CI and with tox locally we run all the tests in parallel.
However, this makes things like printing to stdout, warnings on stderr,
or logging during tests difficult to deal with. This is because we'll
have multiple writers to the output streams simultaneously and no way
to figure out which text goes to which test. Luckily stestr and the
subunit protocl it uses internally has a method for adding attachments
to test results to handle this kind of case. Using this each output
stream is associated with the test which generated it and will get
displayed as such. The only issue with this is that the stdlib unittest
doesn't know how to write attachments to it's result stream. There is a
unittest extension library testtools[1] that provides this functionality
in a strictly compatible manner with unittest, however the current state
of the library needs some work before it can work well with more modern
features of stdlib unittest from python>3.5 so we either have to change
the tests to not use these features or come up with an alternative
solution for test attachments.

This commit ports the necessary result streaming functionality from
testtools into the QiskitTestCase base test class and then sets up the
appropriate fixtures to attach output streams from each test. This works
around the current limitation with testtools (I'm slowly working on
fixing this in testtools) but still provides the functionality. When
testtools is at the point it can be a drop in replacement for unittest
as terra is using it we can drop this code.

As part of this change all the test classes that define a setUp have a
missing super() call added. This framework does not work without setUp
in the base class being defined since that's what is used to setup the
stream capture. Moving forward the base test class actually enforces
that super() is called, otherwise the tests will fail.

[1] https://github.com/testing-cabal/testtools

* Fix lint

* Update contributing doc

* Fix copy paste error

* Add missing super calls from new tests

* Add missing super() calls and fix skip message

* Fix lint

Co-authored-by: Luciano Bello <[email protected]>

.travis.yml

@@ -46,6 +46,7 @@ jobs:
       python: 3.6
       env:
         - PYTHON="coverage run --source qiskit --parallel-mode"
+        - QISKIT_TEST_CAPTURE_STREAMS=1
       after_success:
         - coverage combine || true
         - coveralls || true

CONTRIBUTING.md

@@ -280,6 +280,16 @@ C:\..\> set LOG_LEVEL="INFO"
 C:\..\> python -m unittest test/python/circuit/test_circuit_operations.py
 ```
 
+##### STDOUT/STDERR and logging capture
+
+When running tests in parallel either via tox, the makefile, or in CI we set
+the env variable `QISKIT_TEST_CAPTURE_STREAMS` which will capture any text
+written to stdout, stderr, and log messages and add them as attachments to
+the tests run so output can be associated with the test case it originated
+from. However, if you run tests outside of these mechanisms by default the
+streams are not captured. To enable stream capture just set the
+`QISKIT_TEST_CAPTURE_STREAMS` env variable to `1`.
+
 ##### Test Skip Options
 
 How and which tests are executed is controlled by an environment

Makefile

@@ -67,7 +67,7 @@ pytest_randomized:
 
 test_ci:
 	echo "Detected $(NPROCS) CPUs running with $(CONCURRENCY) workers"
-	stestr run --concurrency $(CONCURRENCY)
+	QISKIT_TEST_CAPTURE_STREAMS=1 stestr run --concurrency $(CONCURRENCY)
 
 test_randomized:
 	python3 -m unittest discover -s test/randomized -t . -v

azure-pipelines.yml

@@ -132,6 +132,7 @@ stages:
       variables:
         QISKIT_SUPPRESS_PACKAGING_WARNINGS: Y
         PIP_CACHE_DIR: $(Pipeline.Workspace)/.pip
+        QISKIT_TEST_CAPTURE_STREAMS: 1
       steps:
         - task: UsePythonVersion@0
           inputs:
@@ -285,6 +286,7 @@ stages:
       variables:
         QISKIT_SUPPRESS_PACKAGING_WARNINGS: Y
         PIP_CACHE_DIR: $(Pipeline.Workspace)/.pip
+        QISKIT_TEST_CAPTURE_STREAMS: 1
       steps:
         - task: UsePythonVersion@0
           inputs:
@@ -358,6 +360,7 @@ stages:
       variables:
         QISKIT_SUPPRESS_PACKAGING_WARNINGS: Y
         PIP_CACHE_DIR: $(Pipeline.Workspace)/.pip
+        QISKIT_TEST_CAPTURE_STREAMS: 1
       steps:
         - task: UsePythonVersion@0
           inputs:
@@ -426,6 +429,7 @@ stages:
       variables:
         QISKIT_SUPPRESS_PACKAGING_WARNINGS: Y
         PIP_CACHE_DIR: $(Pipeline.Workspace)/.pip
+        QISKIT_TEST_CAPTURE_STREAMS: 1
       steps:
         - task: UsePythonVersion@0
           inputs:
@@ -495,6 +499,7 @@ stages:
       variables:
         QISKIT_SUPPRESS_PACKAGING_WARNINGS: Y
         PIP_CACHE_DIR: $(Pipeline.Workspace)/.pip
+        QISKIT_TEST_CAPTURE_STREAMS: 1
       steps:
         - task: UsePythonVersion@0
           inputs: