Pytest related changes #248

sstrehlk · 2021-05-21T12:56:47Z

Summary

Tests moved to pytest. Updated CI. Updated requirements.
Updated contribution guide
Added annotations for tests
Updated tests
Added code style guide

This PR introduces this capability to use pytest for testing purposes. Pytest gives some advantages like:

running tests parallelly by multiple processes
run the same test a few times with different parameters
marking tests with predefined marks, and use these marks to run only selected group of tests
generating html reports if needed with installed pytest-html library

How to test

run tests with command:
pytest -v -n 6 tests/

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG
I have updated the documentation accordingly
I have added tests to cover my changes ( it's not applicable)
I have linked related issues)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT

zhiltsov-max

Currently, it looks heavy and hard to maintain. In many places, it also looks extraneous and introduces code duplication.
Please, don't change the formatting in the code.
I expect that you change how tests are executed on CI and adjust requirements accordingly. Because currently it does not work.
If possible, consider marking requirements instead of tests. It is easier to maintain and makes more sense.

datumaro/util/test_utils.py

docs/test_marking.md

zhiltsov-max · 2021-05-21T13:46:03Z

docs/test_marking.md

+        1. Prepare dataset with polygon segmentation mode (source dataset)
+        2. Prepare dataset with expected mask segmentation mode (target dataset)
+        3. Convert source dataset to target, with segmentation mode changed from polygon to mask and verify that result
+        segmentation mask is equal to expected mask.


I feel it can quickly become a story similar to the story with comments in code. If a test is so complex it is impossible to understand quickly, it must be split and simplified. For many unit tests such description will be equal, or even bigger that the test code itself. It might be ok for complex or long integration/e2e tests, which are supposed to be like this.

This high level description is dedicated to reports viewers. Viewer don't need to know particular programming language to have understanding what was tested and how.

docs/test_marking.md

tests/constants/requirements.py

tests/constants/datumaro_components.py

tests/test_voc_format.py

zhiltsov-max

Let's collect all the definitions from pytest_makring_constants in tests/requirements.py.
pip install pytest doesn't allow to use -n 6 flag
Could you also update our Travis config and launch tests in parallel?

requirements.txt

Fixed pycocotools version, to eliminate conflicts with numpy during tests.

Modifications related to test marking and documentation.

TestDir and Extractor classes marked with __test__ variable not to collect by pytest. Moved pytest related constants to requirements.py. Added pytest and pytest-xdist to requirements.txt. Corrected DUMMY_DATASET_DIR in test_voc_format.py.

Updated test_cityscapes_format.py file.

sstrehlk · 2021-05-26T08:57:00Z

pytest-xdist needs to be installed to run
pytest -n 6

requirements.txt

TestDir and Extractor classes marked with __test__ variable not to collect by pytest. Moved pytest related constants to requirements.py. Added pytest and pytest-xdist to requirements.txt. Corrected DUMMY_DATASET_DIR in test_voc_format.py.

Updated test_cityscapes_format.py file.

Added pytest.ini file. Reverted name of check_save_and_load function to test_save_and_load.

#253

…oolkit/datumaro into pytest_related_changes

CONTRIBUTING.md

tests/conftest.py

zhiltsov-max · 2021-06-01T13:04:05Z

tests/requirements.py

+
+    # GitHub issues (bugs)
+    # https://github.com/openvinotoolkit/datumaro/issues
+    DATUM_BUG_219 = "Return format is not uniform"


Please comment on this layout. I suggest to list bugs as other reqids.

As we agreed, bugs and component marking should be added.

zhiltsov-max · 2021-06-02T12:12:45Z

Can we merge this PR?

sstrehlk · 2021-06-02T12:35:16Z

I restored "component" run test type marking in decorator in requirements.py. In my opinion this PR can be merged.

zhiltsov-max · 2021-06-02T13:17:47Z

Thanks for the PR!

* Rename 'openvino' plugin to 'openvino_plugin' (#205) Co-authored-by: Jihyeon Yi <[email protected]> * Make remap labels more accurate, allow explicit label deletion, add docs, update tests (#203) * Kate/handling multiple attributes and speed up detection split (#207) * better handling multi-attributes for classification_split * handling multi-attributes better for detection * bugfix in calculating required number of images for splitting 2 correct side effect of the changes for re-id split * allow multiple subsets with arbitrary names * rename _is_number to _is_float and improve it * Fix voc to coco example (#209) * Fix export filtering * update example in readme * Fix export filename for LabelMe format (#200) * change export filename for LabelMe format * Allow simple merge for datasets with no labels * Add a more complex test on relative paths * Support escaping in attributes * update changelog Co-authored-by: Maxim Zhiltsov <[email protected]> * split unlabeled data into subsets for task-specific splitters (#211) * split unlabeled data into subsets for classification, detection. for re-id, 'not-supported' subsets for this data * Fix image ext on saving in cvat format (#214) * fix image saving in cvat format * update changelog * Label "face" for bounding boxes in Wider Face (#215) * add face label * update changelog * Adding "difficult", "truncated", "occluded" attributes when converting to Pascal VOC if they are not present (#216) * remove check for 'difficult' attribute * remove check for 'truncated' and 'occluded' attributes * update changelog * Ignore empty lines in YOLO annotations (#221) * Ignore empty lines in yolo annotations * Add type hints for image class, catch image opening errors in image.size * update changelog * Classification task in LFW dataset format (#222) * add classification * update changelog * update documentation * Add splitter for segmentation task (#223) * added segmentation_split * updated changelog * rename reidentification to reid * Support for CIFAR-10/100 format (#225) * add CIFAR dataset format * add CIFAR to documentation * update Changelog * add validation item for instance segmentation (#227) * add validation item for instance segmentation * Add panoptic and stuff COCO format (#210) * add coco stuff and panoptic formats * update CHANGELOG Co-authored-by: Maxim Zhiltsov <[email protected]> * update detection splitter algorithm from # of samples to # of instances (#235) * add documentation for validator (#233) * add documentation for validator * add validation item description (#237) * Fix converter for Pascal VOC format (#239) * User documentation for Pascal VOC format (#228) * add user documentation for Pascal VOC format * add integration tests * update changelog * Support for MNIST dataset format (#234) * add mnist format * add mnist csv format * add mnist to documentation * make formats docs folder, create COCO format documentation (#241) * Make formats docs folder, move format docs * Create COCO format documentation * Fixes in CIFAR dataset format (#243) * Add folder creation * Update changelog * Add user documentation file and integration tests for YOLO format (#246) * add user documentation file for yolo * add integraion tests * update user manual * update changelog * Add Cityscapes format (#249) * add cityscapes format * add format docs * update changelog * Fix saving attribute in WiderFace extractor (#251) * add fixes * update changelog * Fix spelling errors (#252) * Configurable Threshold CLI support (#250) * add validator cli * add configurable validator threshold * update changelog * CI. Move to GitHub actions. (#263) * Moving to GitHub Actions * Sending a coverage report if python3.6 (#264) * Rename workflows (#265) * Rename workflows * Update repo config and badge (#266) * Update PR template * Update build status badge * Fix deprecation warnings (#270) * Update RISE docs (#255) * Update rise docs * Update cli help * Pytest related changes (#248) * Tests moved to pytest. Updated CI. Updated requirements. * Updated contribution guide * Added annotations for tests * Updated tests * Added code style guide * Fix CI (#272) * Fix script call * change script call to binary call * Fix help program name, add mark_bug (#275) * Fix prog name * Add mark_bug test annotation * Fix labelmap parameter in CamVid (#262) * Fix labelmap parameter in camvid * Release 0.1.9 (dev) (#276) * Update version * Update changelog * Fix numpy conflict (#278) Co-authored-by: Emily Chun <[email protected]> Co-authored-by: Jihyeon Yi <[email protected]> Co-authored-by: Kirill Sizov <[email protected]> Co-authored-by: Anastasia Yasakova <[email protected]> Co-authored-by: Harim Kang <[email protected]> Co-authored-by: Zoya Maslova <[email protected]> Co-authored-by: Roman Donchenko <[email protected]> Co-authored-by: Seungyoon Woo <[email protected]> Co-authored-by: Dmitry Kruchinin <[email protected]> Co-authored-by: Slawomir Strehlke <[email protected]>

* Rename 'openvino' plugin to 'openvino_plugin' (#205) Co-authored-by: Jihyeon Yi <[email protected]> * Make remap labels more accurate, allow explicit label deletion, add docs, update tests (#203) * Kate/handling multiple attributes and speed up detection split (#207) * better handling multi-attributes for classification_split * handling multi-attributes better for detection * bugfix in calculating required number of images for splitting 2 correct side effect of the changes for re-id split * allow multiple subsets with arbitrary names * rename _is_number to _is_float and improve it * Fix voc to coco example (#209) * Fix export filtering * update example in readme * Fix export filename for LabelMe format (#200) * change export filename for LabelMe format * Allow simple merge for datasets with no labels * Add a more complex test on relative paths * Support escaping in attributes * update changelog Co-authored-by: Maxim Zhiltsov <[email protected]> * split unlabeled data into subsets for task-specific splitters (#211) * split unlabeled data into subsets for classification, detection. for re-id, 'not-supported' subsets for this data * Fix image ext on saving in cvat format (#214) * fix image saving in cvat format * update changelog * Label "face" for bounding boxes in Wider Face (#215) * add face label * update changelog * Adding "difficult", "truncated", "occluded" attributes when converting to Pascal VOC if they are not present (#216) * remove check for 'difficult' attribute * remove check for 'truncated' and 'occluded' attributes * update changelog * Ignore empty lines in YOLO annotations (#221) * Ignore empty lines in yolo annotations * Add type hints for image class, catch image opening errors in image.size * update changelog * Classification task in LFW dataset format (#222) * add classification * update changelog * update documentation * Add splitter for segmentation task (#223) * added segmentation_split * updated changelog * rename reidentification to reid * Support for CIFAR-10/100 format (#225) * add CIFAR dataset format * add CIFAR to documentation * update Changelog * add validation item for instance segmentation (#227) * add validation item for instance segmentation * Add panoptic and stuff COCO format (#210) * add coco stuff and panoptic formats * update CHANGELOG Co-authored-by: Maxim Zhiltsov <[email protected]> * update detection splitter algorithm from # of samples to # of instances (#235) * add documentation for validator (#233) * add documentation for validator * add validation item description (#237) * Fix converter for Pascal VOC format (#239) * User documentation for Pascal VOC format (#228) * add user documentation for Pascal VOC format * add integration tests * update changelog * Support for MNIST dataset format (#234) * add mnist format * add mnist csv format * add mnist to documentation * make formats docs folder, create COCO format documentation (#241) * Make formats docs folder, move format docs * Create COCO format documentation * Fixes in CIFAR dataset format (#243) * Add folder creation * Update changelog * Add user documentation file and integration tests for YOLO format (#246) * add user documentation file for yolo * add integraion tests * update user manual * update changelog * Add Cityscapes format (#249) * add cityscapes format * add format docs * update changelog * Fix saving attribute in WiderFace extractor (#251) * add fixes * update changelog * Fix spelling errors (#252) * Configurable Threshold CLI support (#250) * add validator cli * add configurable validator threshold * update changelog * CI. Move to GitHub actions. (#263) * Moving to GitHub Actions * Sending a coverage report if python3.6 (#264) * Rename workflows (#265) * Rename workflows * Update repo config and badge (#266) * Update PR template * Update build status badge * Fix deprecation warnings (#270) * Update RISE docs (#255) * Update rise docs * Update cli help * Pytest related changes (#248) * Tests moved to pytest. Updated CI. Updated requirements. * Updated contribution guide * Added annotations for tests * Updated tests * Added code style guide * Fix CI (#272) * Fix script call * change script call to binary call * Fix help program name, add mark_bug (#275) * Fix prog name * Add mark_bug test annotation * Fix labelmap parameter in CamVid (#262) * Fix labelmap parameter in camvid * Release 0.1.9 (dev) (#276) * Update version * Update changelog * Fix numpy conflict (#278) * Add changelog stub (#279) * tests/requirements.py: remove the test_wrapper functions (#285) * Subformat importers for VOC and COCO (#281) * Document find_sources * Add VOC subformat importers * Add coco subformat importers * Fix LFW * Reduce voc detect dataset cases * Reorganize coco tests, add subformat tests * Fix default subset handling in Dataset * Fix getting subset * Fix coco tests * Fix voc tests * Update changelog * Add image zip format (#273) * add tests * add image_zip format * update changelog Co-authored-by: Maxim Zhiltsov <[email protected]> * Add KITTI detection and segmentation formats (#282) * Add KITTI detection and segmentation formats * Remove unused import * Add KITTI user manual Co-authored-by: Maxim Zhiltsov <[email protected]> * Fix loading file and image processing in CIFAR (#284) * Fix image layout and encoding problems * Update Changelog Co-authored-by: Maxim Zhiltsov <[email protected]> * CLI tests for convert command for VOC dataset (#286) * Add tests for convert command * Convert most enum definitions from the functional style to the class style (#290) * yolo format documentation update (#295) * add info about coordinates in yolo format doc * Fix merged dataset item filtering (#258) * Add tests * Fix xpathfilter transform * Update changelog * Sms/pytest marking cityscapes and zip (#298) * Updated pytest marking for cityscapes and imagezip. * Introduce Validator plugin type (#299) * Introduce Validator plugin type * Fix validator definitions (#303) * update changelog * Fixes in validator definitions * Update validator cli * Make TF availability check optional (#305) * Make tf availability check optional * update changelog * Update pylint (#304) * Add import order check in pylint * Fix some linter problems * Remove warning suppression comments * Add lazy loading for builtin plugins (#306) * Refactor env code * Load builtin plugins lazily * update changelog * Update transforms handling in Dataset (#297) * Update builtin transforms * Optimize dataset length computation when no source * Add filter test * Fix transforms affecting categories * Optimize categories transforms * Update filters * fix imports * Avoid using default docstrings in plugins * Fix patch saving in VOC, add keep_empty export parameter * Fix flush_changes * Fix removed images and subsets in dataset patch * Update changelog * Update voc doc * Skip item transform base class in plugins * Readable COCO and datumaro format for CJK (#307) * Do not force ASCII in COCO and Datumaro JSONs for readable CJK * Add tests * Use utf-8 encoding for writing Co-authored-by: Maxim Zhiltsov <[email protected]> * Force utf-8 everywhere (#309) * Fix in ImageNet_txt (#302) * Add extensions for images to annotation file * Remove image search in extractor * Update changelog Co-authored-by: Maxim Zhiltsov <[email protected]> * Reduce duplication of dependency information (#308) * Move requirements from setup.py to requirements-base.txt * Add whitespace error checking to GitHub Actions (#311) * Fix whitespace errors As detected with `git diff --check`. * Add a job to check for whitespace errors I called it "lint" so that other checks could be added to it later. * Bump copyright years in changed files * Add initial support for the Open Images dataset (#291) * Support reading or Labels in Open Images (v4, v5, v6) * Add tests for the Open Images extractor/importer * Add Open Images documentation * Update changelog * Fix tensorboardX dependency (#318) * Fixing remark-lint issues. Adding remark-linter check. (#321) * Fix remark-lint issues. * Align continuation lines with the first line. Apply comments * Added remark check * Add an upper bound on the Pillow dependency to work around a regression in 8.3 (#323) * open_images_user_manual.md: fix image description file URLs I accidentally swapped the URLs for test and validation sets. * Fix COCO Panoptic (#319) * add test * Fix integer overflow in bgr2index * Fix pylint issues. Added pylint checking. (#322) * Added pylint job for CI * Rework pip install * Fixed remaining pylint warnings Co-authored-by: Andrey Zhavoronkov <[email protected]> * Open Images: add writing support (#315) * open_images_user_manual.md: fix image description file URLs * open_images_format: add conversion support * open_images_format: add support for images in subdirectories * open_images_format: add tests for writing support * open_images_format: add documentation for the writing support * Update the changelog entry for the Open Images support * Add python bandit checks. (#316) * Add bandit dependency * Add bandit checks on CI * Disable some warnings Co-authored-by: Andrey Zhavoronkov <[email protected]> Co-authored-by: Maxim Zhiltsov <[email protected]> * Remove Pylint unused-import warning suppressions (#326) * Remove Pylint unused-import warning suppressions * Add a job to check import formatting using isort (#333) * Reformat all imports using isort * Implement a workflow for checking import formatting based on isort * Reformat the enabled checker list in .pylintrc (#335) Put each code on its own line and add a comment with its symbolic name. That makes the list more understandable and easier to edit. * Merge all linting jobs into one workflow file (#331) Doing it this way means that on GitHub's Checks page, all jobs are displayed under one "Linter" category, instead of multiple indistinguishable "Linter" categories with one job each. Move the whitespace checking job into the Linter workflow as well, since that's where it logically belongs. I also took the opportunity to slightly rename the jobs in order to spell the linter names correctly. * Fix cuboids / 3d / M6 (#320) * CVAT-3D Milestone-6: Added Supervisely Point Cloud and KITTI Raw 3D formats * Added Cuboid3d annotations * Added docs for new formats Co-authored-by: cdp <cdp123> Co-authored-by: Jayraj <[email protected]> Co-authored-by: Roman Donchenko <[email protected]> * Clean up .pylintrc (#340) * Clean up the list of messages in .pylintrc * Remove obsolete Pylint options * .pylintrc: move the disable setting and its documentation together * Remove the commented-out setting. * Revert "Add an upper bound on the Pillow dependency to work around a regression in 8.3 (#323)" (#341) The regression was fixed in 8.3.1. This reverts commit 9a85616. * Enable pylint checkers that find invalid escape sequences (#344) Fix the issues that they found. * Factor out the images.meta loading code from YoloExtractor (#343) * Factor out the images.meta loading code from YoloExtractor It looks like the same thing will be needed for Open Images, so I'm moving it to a common module. * Rework image.meta parsing code to use shell syntax This allows comments and improves extensibility. * Support for CIFAR-100 (#301) * Add support for CIFAR-100 * Update Changelog * Update user_manual.md * Add notes about differences in formats * Fix importing for VGG Face 2 (#345) * correct asset according the original vgg_face2 dataset * fix importing of the original dataset Co-authored-by: Maxim Zhiltsov <[email protected]> * Dataset caching fixes (#351) * Fix importing arbitrary file names in COCO subformats * Optimize subset iteration in a simple scenario * Fix subset iteration in dataset with transforms * Cuboid 3D for Datumaro format (#349) * Support cuboid_3d and point cloud in datumaro format * Add cuboid_3d and point cloud tests in datumaro format * Add image size type conversions Co-authored-by: Maxim Zhiltsov <[email protected]> * Add e2e tests for cuboids (#353) * Add attr name check in kitti raw * Add sly pcd e2e test * Rename "object" attribute to "track_id" in sly point cloud * Add kitti raw e2e test * Update kitti raw example * update changelog * Release 0.1.10 (dev) (#354) * Update changelog * Add cifar security notice * Update version Co-authored-by: Emily Chun <[email protected]> Co-authored-by: Jihyeon Yi <[email protected]> Co-authored-by: Kirill Sizov <[email protected]> Co-authored-by: Anastasia Yasakova <[email protected]> Co-authored-by: Harim Kang <[email protected]> Co-authored-by: Zoya Maslova <[email protected]> Co-authored-by: Roman Donchenko <[email protected]> Co-authored-by: Seungyoon Woo <[email protected]> Co-authored-by: Dmitry Kruchinin <[email protected]> Co-authored-by: Slawomir Strehlke <[email protected]> Co-authored-by: Jaesun Park <[email protected]> Co-authored-by: Andrey Zhavoronkov <[email protected]> Co-authored-by: Jayraj <[email protected]>

sstrehlk requested a review from zhiltsov-max May 21, 2021 12:58

zhiltsov-max suggested changes May 21, 2021

View reviewed changes

zhiltsov-max reviewed May 21, 2021

View reviewed changes

tests/test_voc_format.py Outdated Show resolved Hide resolved

zhiltsov-max suggested changes May 25, 2021

View reviewed changes

requirements.txt Show resolved Hide resolved

sstrehlk added 20 commits May 26, 2021 09:38

Adapt project to run tests with pytest.

4f244d3

Update requirements.txt

4094609

Fixed pycocotools version, to eliminate conflicts with numpy during tests.

updated tests marking

cfd9e93

Update conftest.py

fbe66bd

Update conftest.py

daca7b8

Added test_marking manual.

2f13f6e

Modifications related to test marking and documentation.

Cosmetic corrections in test_marking.md

f80eea5

adapting cli/test_yolo_format.py for pytest

aab6bd7

Updated test_yolo_format.py

45e9470

Corrected datumaro_components.py, updated license in added files.

46e1fe7

Updated test_marking.md file

bd8a335

Updated test_marking.md file

f0f343b

Updates after review.

7df661c

Removed unnecessary marking.

ea804b1

removed unneeded comments in requirements.py

7cc26a6

Corrected requirement constant.

e7d845a

experiment related to test_can_import test.

299cc26

Corrected accidentally changed indentation.

60ace83

TestDir and Extractor classes marked with __test__ variable not to collect by pytest. Moved pytest related constants to requirements.py. Added pytest and pytest-xdist to requirements.txt. Corrected DUMMY_DATASET_DIR in test_voc_format.py.

Updated .travis.yml to work with pytest

cfeb7d9

Updated .travis.yml to work with pytest.

e98f5e3

Updated test_cityscapes_format.py file.

sstrehlk force-pushed the pytest_related_changes branch from 17b01c8 to e98f5e3 Compare May 26, 2021 08:08

Updated .gitignore, .travis.yml

fe8e510

sstrehlk added 2 commits May 26, 2021 13:11

Updated .travis.yml to check pytest performance in Travis.

f0fe579

Updated .travis.yml and requirements.txt.

9a164ce

zhiltsov-max reviewed May 26, 2021

View reviewed changes

requirements.txt Outdated Show resolved Hide resolved

sstrehlk and others added 19 commits June 1, 2021 11:43

removed unneeded comments in requirements.py

96c4c4a

Corrected accidentally changed indentation.

1e40654

TestDir and Extractor classes marked with __test__ variable not to collect by pytest. Moved pytest related constants to requirements.py. Added pytest and pytest-xdist to requirements.txt. Corrected DUMMY_DATASET_DIR in test_voc_format.py.

Updated .travis.yml to work with pytest.

2772c04

Updated test_cityscapes_format.py file.

Updated .gitignore, .travis.yml

91b58bb

Updated .travis.yml and requirements.txt.

3cce32d

Merged two imports into one line in test files.

0b4758a

Added pytest.ini file. Reverted name of check_save_and_load function to test_save_and_load.

Removed unneded __test__ variable.

9c06cc2

Reverted change related to pycocotools version. It's discussed in Issue

a5feca0

#253

Updated CHANGELOG.md

90cb339

Merge branch 'pytest_related_changes' of https://github.com/openvinot…

ced967c

…oolkit/datumaro into pytest_related_changes

github testing scripts adapted to pytest

83204ce

Corrected rebase.

f856e47

Merge branch 'pytest_related_changes' of https://github.com/openvinot…

6f8328f

…oolkit/datumaro into pytest_related_changes

Merge branch 'develop' into pytest_related_changes

83cd7c2

update changelog

84eb6f7

Remove extra comments

3fa8112

Move test descriptions to contributing guide

2158ebe

Update imports

b95013a

Update test annotations

5ad70cf

zhiltsov-max reviewed Jun 1, 2021

View reviewed changes

Update markings

a0faa20

Added component test run type marking for datumaro tests.

e32153a

zhiltsov-max merged commit eb572d9 into develop Jun 2, 2021

zhiltsov-max deleted the pytest_related_changes branch June 3, 2021 17:53

zhiltsov-max mentioned this pull request Jun 8, 2021

CVAT-3D Milestone-6 PCD and Kitti formats #245

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pytest related changes #248

Pytest related changes #248

sstrehlk commented May 21, 2021 •

edited by zhiltsov-max

Loading

zhiltsov-max left a comment •

edited

Loading

zhiltsov-max May 21, 2021

sstrehlk May 24, 2021

zhiltsov-max left a comment

sstrehlk commented May 26, 2021

zhiltsov-max Jun 1, 2021

sstrehlk Jun 2, 2021

zhiltsov-max commented Jun 2, 2021

sstrehlk commented Jun 2, 2021

zhiltsov-max commented Jun 2, 2021

Pytest related changes #248

Pytest related changes #248

Conversation

sstrehlk commented May 21, 2021 • edited by zhiltsov-max Loading

Summary

How to test

Checklist

License

zhiltsov-max left a comment • edited Loading

Choose a reason for hiding this comment

zhiltsov-max May 21, 2021

Choose a reason for hiding this comment

sstrehlk May 24, 2021

Choose a reason for hiding this comment

zhiltsov-max left a comment

Choose a reason for hiding this comment

sstrehlk commented May 26, 2021

zhiltsov-max Jun 1, 2021

Choose a reason for hiding this comment

sstrehlk Jun 2, 2021

Choose a reason for hiding this comment

zhiltsov-max commented Jun 2, 2021

sstrehlk commented Jun 2, 2021

zhiltsov-max commented Jun 2, 2021

sstrehlk commented May 21, 2021 •

edited by zhiltsov-max

Loading

zhiltsov-max left a comment •

edited

Loading