Pytest related changes

.github/workflows/health_check.yml

@@ -18,13 +18,12 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Installing dependencies
         run: |
-          pip install coverage tensorflow
+          pip install tensorflow pytest pytest-cov
           pip install -e ./
       - name: Code instrumentation
         run: |
-          coverage run -m unittest discover -v
-          coverage run -a datum.py -h
-          coverage xml
+          pytest -v --cov --cov-report xml:coverage.xml
+          datum.py -h
       - name: Sending coverage results
         if: matrix.python-version == '3.6'
         run: |

.github/workflows/pr_checks.yml

@@ -21,9 +21,9 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Installing dependencies
         run: |
-          pip install tensorflow
+          pip install tensorflow pytest
           pip install -e ./
       - name: Unit testing
         run: |
-          python -m unittest discover -v
+          pytest -v
           datum -h

.gitignore

@@ -54,4 +54,7 @@ coverage.xml
 cover/
 
 # Sphinx documentation
-docs/_build/
+docs/_build/
+
+#Pycharm config files
+.idea/

CHANGELOG.md

@@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Allowed arbitrary subset count and names in classification and detection splitters (<https://github.com/openvinotoolkit/datumaro/pull/207>)
 - Annotation-less dataset elements are now participate in subset splitting (<https://github.com/openvinotoolkit/datumaro/pull/211>)
 - Classification task in LFW dataset format (<https://github.com/openvinotoolkit/datumaro/pull/222>)
+- Testing is now performed with pytest instead of unittest (<https://github.com/openvinotoolkit/datumaro/pull/248>)
 
 ### Deprecated
 -

CONTRIBUTING.md

@@ -1,9 +1,12 @@
 ## Table of Contents
 
+- [Design document](docs/design.md)
+- [Developer guide](docs/developer_guide.md)
 - [Installation](#installation)
 - [Usage](#usage)
+- [Code style](#code-style)
+- [Development environment](#environment)
 - [Testing](#testing)
-- [Design](#design-and-code-structure)
 
 ## Installation
 
@@ -64,15 +67,73 @@ python datum.py --help
 import datumaro
 ```
 
-## Testing
+## Code style
+
+Try to be readable and consistent with the existing codebase.
+
+The project mostly follows PEP8 with little differences.
+Continuation lines have a standard indentation step by default,
+or any other, if it improves readability. For long conditionals use 2 steps.
+No trailing whitespaces, 80 characters per line.
+
+Example:
+
+```python
+def do_important_work(parameter1, parameter2, parameter3,
+        option1=None, option2=None, option3=None) -> str:
+    """
+    Optional description. Mandatory for API.
+    Use comments for implementation specific information, use docstrings
+    to give information to user / developer.
+
+    Returns: status (str) - Possible values: 'done', 'failed'
+    """
+
+    ... do stuff ...
+
+    # Use +1 level of indentation for continuation lines
+    variable_with_a_long_but_meaningful_name = \
+        function_with_a_long_but_meaningful_name(arg1, arg2, arg3,
+            kwarg1=value_with_a_long_name, kwarg2=value_with_a_long_name)
+
+    # long conditions, loops, with etc. also use +1 level of indentation
+    if condition1 and long_condition2 or \
+            not condition3 and condition4 and condition5 or \
+            condition6 and condition7:
+
+        ... do other stuff ...
+
+    elif other_conditions:
+
+        ... some other things ...
+
+    # in some cases special formatting can improve code readability
+    specific_case_formatting = np.array([
+        [0, 1, 1, 0],
+        [1, 1, 0, 0],
+        [1, 1, 0, 1],
+    ], dtype=np.int32)
+
+    return status
+```
+
+## Environment
+
+The recommended editor is VS Code with the Python language plugin.
+
+## Testing <a id="testing"></a>
 
 It is expected that all Datumaro functionality is covered and checked by
 unit tests. Tests are placed in `tests/` directory.
+Currently, we use [`pytest`](https://docs.pytest.org/) for testing, but we
+also compatible with `unittest`.
 
 To run tests use:
 
 ``` bash
-python -m unittest discover -s tests
+pytest -v
+# or
+python -m pytest -v
 ```
 
 If you're working inside of a CVAT environment, you can also use:
@@ -81,19 +142,121 @@ If you're working inside of a CVAT environment, you can also use:
 python manage.py test datumaro/
 ```
 
-## Design and code structure
 
-- [Design document](docs/design.md)
-- [Developer guide](docs/developer_guide.md)
+### Test cases <a id="Test_case_description"></a>
 
-## Code style
+### Test marking <a id="Test_marking"></a>
 
-Try to be readable and consistent with the existing codebase.
-The project mostly follows PEP8 with little differences.
-Continuation lines have a standard indentation step by default,
-or any other, if it improves readability. For long conditionals use 2 steps.
-No trailing whitespaces, 80 characters per line.
+For better integration with CI and requirements tracking,
+we use special annotations for tests.
 
-## Environment
+A test needs to marked with a requirement it is related to. To mark a test, use:
+
+```python
+from unittest import TestCase
+from .requirements import Requirements, mark_requirement
+
+class MyTests(TestCase):
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
+    def test_my_requirement(self):
+        ... do stuff ...
+```
+
+Such marking will apply markings from the requirement specified.
+They can be overriden for a specific test:
+
+```python
+import pytest
+
+    @pytest.mark.proirity_low
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
+    def test_my_requirement(self):
+        ... do stuff ...
+```
+
+#### Requirements <a id="Requirements"></a>
+
+Requirements and other links need to be added to [`tests/requirements.py`](tests/requirements.py):
+
+```python
+DATUM_244 = "Add Snyk integration"
+DATUM_BUG_219 = "Return format is not uniform"
+```
+
+```python
+# Fully defined in GitHub issues:
+@pytest.mark.reqids(Requirements.DATUM_244, Requirements.DATUM_333)
+
+# And defined ony other way:
+@pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+```
+
+
+##### Available annotations for tests and requirements
+
+Markings are defined in [`tests/conftest.py`](tests/conftest.py).
+
+**A list of requirements and bugs**
+```python
+@pytest.mark.requids(Requirements.DATUM_123)
+@pytest.mark.bugs(Requirements.DATUM_BUG_456)
+```
+
+**A priority**
+```python
+@pytest.mark.priority_low
+@pytest.mark.priority_medium
+@pytest.mark.priority_high
+```
+
+**Component**
+The marking used for indication of different system components
+
+```python
+@pytest.mark.components(DatumaroComponent.Datumaro)
+```
+
+**Skipping tests**
+
+```python
+@pytest.mark.skip(SkipMessages.NOT_IMPLEMENTED)
+```
+
+**Parametrized runs**
+
+Parameters are used for running the same test with different parameters e.g.
+
+```python
+@pytest.mark.parametrize("numpy_array, batch_size", [
+    (np.zeros([2]), 0),
+    (np.zeros([2]), 1),
+    (np.zeros([2]), 2),
+    (np.zeros([2]), 5),
+    (np.zeros([5]), 2),
+])
+```
+
+### Test documentation <a id="TestDoc"></a>
+
+Tests are documented with docstrings. Test descriptions must contain
+the following: sections: `Description`, `Expected results` and `Steps`.
+
+```python
+def test_can_convert_polygons_to_mask(self):
+    """
+    <b>Description:</b>
+    Ensure that the dataset polygon annotation can be properly converted
+    into dataset segmentation mask.
+
+    <b>Expected results:</b>
+    Dataset segmentation mask converted from dataset polygon annotation
+    is equal to an expected mask.
 
-The recommended editor is VS Code with the Python plugin.
+    <b>Steps:</b>
+    1. Prepare dataset with polygon annotation
+    2. Prepare dataset with expected mask segmentation mode
+    3. Convert source dataset to target, with conversion of annotation
+      from polygon to mask.
+    4. Verify that resulting segmentation mask is equal to the expected mask.
+    """
+```

pytest.ini

@@ -0,0 +1,3 @@
+[pytest]
+python_classes =
+python_functions =

requirements.txt

@@ -11,3 +11,4 @@ PyYAML>=5.3.1
 scikit-image>=0.15.0
 tensorboardX>=1.8
 pandas>=1.1.5
+pytest>=5.3.5

tests/cli/test_diff.py

@@ -15,9 +15,11 @@
 )
 from datumaro.util.image import Image
 from datumaro.util.test_utils import TestDir
+from ..requirements import Requirements, mark_requirement
 
 
 class DiffTest(TestCase):
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
     def test_can_compare_projects(self): # just a smoke test
         label_categories1 = LabelCategories.from_iterable(['x', 'a', 'b', 'y'])
         mask_categories1 = MaskCategories.make_default(len(label_categories1))

tests/cli/test_voc_format.py

@@ -9,13 +9,15 @@
 from datumaro.components.extractor import Bbox, Mask, Image, Label
 from datumaro.cli.__main__ import main
 from datumaro.util.test_utils import TestDir, compare_datasets
+from ..requirements import Requirements, mark_requirement
 
 DUMMY_DATASETS_DIR = osp.join(__file__[:__file__.rfind(osp.join('tests', ''))],
             'tests', 'assets', 'voc_dataset')
 
 def run(test, *args, expected_code=0):
     test.assertEqual(expected_code, main(args), str(args))
 
+
 class VocIntegrationScenarios(TestCase):
     def _test_can_save_and_load(self, project_path, source_path, source_dataset,
             dataset_format, result_path=None, label_map=None):
@@ -30,6 +32,7 @@ def _test_can_save_and_load(self, project_path, source_path, source_dataset,
         target_dataset = Dataset.import_from(result_path, dataset_format)
         compare_datasets(self, source_dataset, target_dataset)
 
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
     def test_preparing_dataset_for_train_model(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='c', subset='train',
@@ -80,6 +83,7 @@ def test_preparing_dataset_for_train_model(self):
             parsed_dataset = Dataset.import_from(export_path, format='voc')
             compare_datasets(self, source_dataset, parsed_dataset)
 
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
     def test_convert_to_voc_format(self):
         label_map = OrderedDict(('label_' + str(i), [None, [], []]) for i in range(10))
         label_map['background'] = [None, [], []]
@@ -122,6 +126,7 @@ def test_convert_to_voc_format(self):
             parsed_dataset = Dataset.import_from(voc_export, format='voc')
             compare_datasets(self, source_dataset, parsed_dataset)
 
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
     def test_can_save_and_load_voc_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',
@@ -164,6 +169,7 @@ def test_can_save_and_load_voc_dataset(self):
             self._test_can_save_and_load(test_dir, voc_dir, source_dataset,
                 'voc', label_map='voc')
 
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
     def test_can_save_and_load_voc_layout_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',
@@ -196,6 +202,7 @@ def test_can_save_and_load_voc_layout_dataset(self):
             self._test_can_save_and_load(test_dir, voc_layout_path, source_dataset,
                 'voc_layout', result_path=result_voc_path, label_map='voc')
 
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
     def test_can_save_and_load_voc_detect_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',
@@ -234,6 +241,7 @@ def test_can_save_and_load_voc_detect_dataset(self):
             self._test_can_save_and_load(test_dir, voc_detection_path, source_dataset,
                 'voc_detection', result_path=result_voc_path, label_map='voc')
 
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
     def test_can_save_and_load_voc_segmentation_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',
@@ -252,6 +260,7 @@ def test_can_save_and_load_voc_segmentation_dataset(self):
             self._test_can_save_and_load(test_dir, voc_segm_path, source_dataset,
                 'voc_segmentation', result_path=result_voc_path, label_map='voc')
 
+    @mark_requirement(Requirements.DATUM_GENERAL_REQ)
     def test_can_save_and_load_voc_action_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',