Pytest related changes

.gitignore

@@ -54,4 +54,7 @@ coverage.xml
 cover/
 
 # Sphinx documentation
-docs/_build/
+docs/_build/
+
+#Pycharm config files
+.idea/

.travis.yml

@@ -15,12 +15,11 @@ matrix:
     - dist: bionic
       python: '3.6'
       before_install:
-        - pip install coverage
+        - pip install pytest-cov
       script:
-        - coverage run -m unittest discover -v
-        - coverage run -a datum.py -h
+        - pytest -v --cov --cov-report xml:coverage.xml
+        - datum -h
       after_success:
-        - coverage xml
         - bash <(curl -Ls https://coverage.codacy.com/get.sh) report -r coverage.xml
 
     - dist: bionic
@@ -34,5 +33,5 @@ install:
   - pip install pandas
 
 script:
-  - python -m unittest discover -v
+  - pytest -v
   - datum -h

datumaro/components/extractor.py

@@ -563,6 +563,8 @@ def get(self, id, subset=None) -> Optional[DatasetItem]:
         raise NotImplementedError()
 
 class Extractor(IExtractor):
+    __test__ = False
+
     def __init__(self, length=None, subsets=None):
         self._length = length
         self._subsets = subsets

datumaro/util/test_utils.py

@@ -52,6 +52,8 @@ class TestDir(FileRemover):
         ...
     """
 
+    __test__ = False
+
     def __init__(self, path=None):
         if path is None:
             path = osp.abspath('temp_%s-' % current_function_name(2))
@@ -143,7 +145,7 @@ def compare_datasets_strict(test, expected, actual):
                 '%s:\n%s\nvs.\n%s\n' % \
                 (idx, item_a, item_b))
 
-def test_save_and_load(test, source_dataset, converter, test_dir, importer,
+def check_save_and_load(test, source_dataset, converter, test_dir, importer,
         target_dataset=None, importer_args=None, compare=None, **kwargs):
     converter(source_dataset, test_dir)
 

docs/test_marking.md

@@ -0,0 +1,182 @@
+# Source code elements related to pytest
+## Contents
+- [Test case description](#Test_case_description)
+  - [Test marking](#Test_marking)
+    - [Requirements](#Requirements)
+    - [Type](#Type)
+    - [Priority](#Priority)
+    - [Component](#Component)
+    - [Skip](#Skip)
+    - [Bug(s)](#Bug)
+    - [Parameters](#Parameters)
+    - [Fixtures](#Fixtures)
+  - [Test naming convention](#TestNaming)
+    - [Test name](#TestName)
+  - [Test documentation](#TestDoc)
+    - [Docstring](#Docstring)
+    - [Steps in test body](#Step)
+
+<a id="Test_case_description"></a>
+## Test case description
+<a id="Test_marking"></a>
+### Test marking
+<a id="Requirements"></a>
+#### Requirements 
+Fully defined in GitHub issues, e.g.
+```
+    @pytest.mark.reqids(Requirements.DATUM_244, Requirements.DATUM_333)
+```
+Requirements constants need to be added to datumaro/tests/constants/requirements.py for example:
+```
+# [DATUMARO] Add SNYK scan integration
+DATUM_244 = "DATUM-244 Add Snyk integration"
+```  
+Recommended notation for requirements 
+
+```python
+@pytest.mark.requids(Requirements.DATUM_123)
+``` 
+where DATUM is a keyword for datumaro indication, and number is a datumaro github issue number.
+
+<a id="Type"></a>
+#### Type
+Test types: gui_smoke, gui_regression, manual, gui_other, gui_long, api, component, unit e.g.
+```python
+    @pytest.mark.component
+    @pytest.mark.unit
+```
+
+<a id="Priority"></a>
+#### Priority 
+Test priorities: low, medium, high, e.g.,
+```python
+     @pytest.mark.priority_low
+     @pytest.mark.priority_medium
+     @pytest.mark.priority_high
+```
+<a id="Component"></a>
+#### Component
+
+Component marking used for indication of different system components e.g.
+```python
+    @pytest.mark.components(DatumaroComponent.Datumaro)
+```
+<a id="Skip"></a>
+#### Skip 
+
+For marking tests, which should be skipped for example for not yet tests, e.g.,
+```python
+   @pytest.mark.skip(reason=SkipMessages.NOT_IMPLEMENTED)
+```
+<a id="Bug"></a>
+#### Bug(s):
+
+In case of test failure, bug should be entered into github issues, and test can be marked e.g.
+```python
+    @pytest.mark.bugs("DATUM-219 - Return format is not uniform")
+```
+<a id="Parameters"></a>
+#### Parameters: 
+
+Parameters are used for running the same test with different parameters e.g. 
+```python
+@pytest.mark.parametrize("numpy_array, batch_size", [  
+        (np.zeros([2]), 0),  
+        (np.zeros([2]), 1),
+        (np.zeros([2]), 2),
+        (np.zeros([2]), 5),
+        (np.zeros([5]), 2),
+    ])
+```
+
+<a id="Fixtures"></a>
+#### Fixtures 
+
+If needed pytest fixtures can be used - for more details see pytest documentation <br>
+https://docs.pytest.org/en/6.2.x/contents.html <br>
+(be aware that fixtures are supported in pytest, and they are not supported in unittests) 
+
+<a id="TestNaming"></a>
+### Test naming convention
+
+<a id="TestName"></a>
+#### Test name:
+
+Test method should be prefixed with "test_" prefix e.g.   
+```python
+    test_*()
+```
+"test_" prefix is an indication for pytest to treat method as a test for execution. 
+It's important not to start other methods with the "test_" prefix.
+
+<a id="DestDoc"></a>
+### Test documentation
+
+<a id="Docstring"></a>
+#### Docstring 
+
+Tests are documented with Docstring. Every test method documentation string should contain: Description, Expected results 
+and Steps. These fields are required but, not limited e.g.
+```python
+        def test_can_convert_polygons_to_mask(self):
+        """
+        <b>Description:</b>
+        Ensure that the dataset polygon annotation can be properly converted into dataset segmentation mask.
+
+        <b>Expected results:</b>
+        Dataset segmentation mask converted from dataset polygon annotation is equal to expected mask.
+
+        <b>Steps:</b>
+        1. Prepare dataset with polygon annotation (source dataset)
+        2. Prepare dataset with expected mask segmentation mode (target dataset)
+        3. Convert source dataset to target, with conversion of annotation from polygon to mask. Verify that result
+        segmentation mask is equal to expected mask.
+
+        """
+
+```
+<a id="Steps"></a>
+#### Steps in test body: 
+
+Steps description in test body are placed as a code comment lines e.g.
+```python
+      # 1. Prepare dataset with polygon annotation (source dataset)
+        source_dataset = Dataset.from_iterable([
+            DatasetItem(id=1, image=np.zeros((6, 10, 3)),
+                annotations=[
+                    Polygon([0, 0, 4, 0, 4, 4],
+                        label=3, id=4, group=4),
+                    Polygon([5, 0, 9, 0, 5, 5],
+                        label=3, id=4, group=4),
+                ]
+            ),
+        ], categories=[str(i) for i in range(10)])
+
+        # 2. Prepare dataset with expected mask segmentation mode (target dataset)
+        target_dataset = Dataset.from_iterable([
+            DatasetItem(id=1, image=np.zeros((6, 10, 3)),
+                annotations=[
+                    Mask(np.array([
+                            [0, 1, 1, 1, 0, 1, 1, 1, 1, 0],
+                            [0, 0, 1, 1, 0, 1, 1, 1, 0, 0],
+                            [0, 0, 0, 1, 0, 1, 1, 0, 0, 0],
+                            [0, 0, 0, 0, 0, 1, 0, 0, 0, 0],
+                            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+                            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
+                            # only internal fragment (without the border),
+                            # but not everywhere...
+                        ),
+                        attributes={ 'is_crowd': True },
+                        label=3, id=4, group=4),
+                ], attributes={'id': 1}
+            ),
+        ], categories=[str(i) for i in range(10)])
+
+        # 3. Convert source dataset to target, with conversion of annotation from polygon to mask. Verify that result
+        # segmentation mask is equal to expected mask.
+        with TestDir() as test_dir:
+            self._test_save_and_load(source_dataset,
+                partial(CocoInstancesConverter.convert, segmentation_mode='mask'),
+                test_dir, target_dataset=target_dataset)
+
+```

requirements.txt

@@ -6,8 +6,9 @@ lxml>=4.4.1
 matplotlib>=3.3.1
 opencv-python-headless>=4.1.0.25
 Pillow>=6.1.0
-pycocotools>=2.0.0
+pycocotools==2.0.0
 PyYAML>=5.3.1
 scikit-image>=0.15.0
 tensorboardX>=1.8
 pandas>=1.1.5
+pytest>=5.3.5

tests/cli/test_diff.py

@@ -16,8 +16,16 @@
 from datumaro.util.image import Image
 from datumaro.util.test_utils import TestDir
 
+import pytest
+from tests.requirements import Requirements
+from tests.requirements import DatumaroComponent
 
+
+@pytest.mark.components(DatumaroComponent.Datumaro)
 class DiffTest(TestCase):
+    @pytest.mark.priority_medium
+    @pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+    @pytest.mark.component
     def test_can_compare_projects(self): # just a smoke test
         label_categories1 = LabelCategories.from_iterable(['x', 'a', 'b', 'y'])
         mask_categories1 = MaskCategories.make_default(len(label_categories1))

tests/cli/test_voc_format.py

@@ -10,12 +10,18 @@
 from datumaro.cli.__main__ import main
 from datumaro.util.test_utils import TestDir, compare_datasets
 
+import pytest
+from tests.requirements import Requirements
+from tests.requirements import DatumaroComponent
+
 DUMMY_DATASETS_DIR = osp.join(__file__[:__file__.rfind(osp.join('tests', ''))],
             'tests', 'assets', 'voc_dataset')
 
 def run(test, *args, expected_code=0):
     test.assertEqual(expected_code, main(args), str(args))
 
+
+@pytest.mark.components(DatumaroComponent.Datumaro)
 class VocIntegrationScenarios(TestCase):
     def _test_can_save_and_load(self, project_path, source_path, source_dataset,
             dataset_format, result_path=None, label_map=None):
@@ -30,6 +36,9 @@ def _test_can_save_and_load(self, project_path, source_path, source_dataset,
         target_dataset = Dataset.import_from(result_path, dataset_format)
         compare_datasets(self, source_dataset, target_dataset)
 
+    @pytest.mark.priority_medium
+    @pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+    @pytest.mark.component
     def test_preparing_dataset_for_train_model(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='c', subset='train',
@@ -80,6 +89,9 @@ def test_preparing_dataset_for_train_model(self):
             parsed_dataset = Dataset.import_from(export_path, format='voc')
             compare_datasets(self, source_dataset, parsed_dataset)
 
+    @pytest.mark.priority_medium
+    @pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+    @pytest.mark.component
     def test_convert_to_voc_format(self):
         label_map = OrderedDict(('label_' + str(i), [None, [], []]) for i in range(10))
         label_map['background'] = [None, [], []]
@@ -122,6 +134,9 @@ def test_convert_to_voc_format(self):
             parsed_dataset = Dataset.import_from(voc_export, format='voc')
             compare_datasets(self, source_dataset, parsed_dataset)
 
+    @pytest.mark.priority_medium
+    @pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+    @pytest.mark.component
     def test_can_save_and_load_voc_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',
@@ -164,6 +179,9 @@ def test_can_save_and_load_voc_dataset(self):
             self._test_can_save_and_load(test_dir, voc_dir, source_dataset,
                 'voc', label_map='voc')
 
+    @pytest.mark.priority_medium
+    @pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+    @pytest.mark.component
     def test_can_save_and_load_voc_layout_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',
@@ -196,6 +214,9 @@ def test_can_save_and_load_voc_layout_dataset(self):
             self._test_can_save_and_load(test_dir, voc_layout_path, source_dataset,
                 'voc_layout', result_path=result_voc_path, label_map='voc')
 
+    @pytest.mark.priority_medium
+    @pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+    @pytest.mark.component
     def test_can_save_and_load_voc_detect_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',
@@ -234,6 +255,9 @@ def test_can_save_and_load_voc_detect_dataset(self):
             self._test_can_save_and_load(test_dir, voc_detection_path, source_dataset,
                 'voc_detection', result_path=result_voc_path, label_map='voc')
 
+    @pytest.mark.priority_medium
+    @pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+    @pytest.mark.component
     def test_can_save_and_load_voc_segmentation_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',
@@ -252,6 +276,9 @@ def test_can_save_and_load_voc_segmentation_dataset(self):
             self._test_can_save_and_load(test_dir, voc_segm_path, source_dataset,
                 'voc_segmentation', result_path=result_voc_path, label_map='voc')
 
+    @pytest.mark.priority_medium
+    @pytest.mark.reqids(Requirements.DATUM_GENERAL_REQ)
+    @pytest.mark.component
     def test_can_save_and_load_voc_action_dataset(self):
         source_dataset = Dataset.from_iterable([
             DatasetItem(id='2007_000001', subset='train',