Exporting Datumaro Project - no occluded and z_order shape attributes in the output

[AhmadM-DL]

I have annotated a video using interpolation mode in CVAT.
I then exported the task as a datumaro dataset.
I tried to output annotated frames using the following command:

datum project export -e "/item/annotation" --filter-mode "i+a" -f yolo

But I am having the following error:

2020-03-14 16:09:39,387 INFO: Loading the project...
2020-03-14 16:09:39,388 ERROR: 'cvat_rest_api_task_images'
Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\*\*\.venv\Scripts\datum.exe\__main__.py", line 7, in <module>
  File "c:\users\*\*\.venv\lib\site-packages\datumaro\cli\__main__.py", line 156, in main
    return args.command(args)
  File "c:\users\*\*\.venv\lib\site-packages\datumaro\cli\contexts\project\__init__.py", line 347, in export_command
    dataset = project.make_dataset()
  File "c:\users\*\*\.venv\lib\site-packages\datumaro\components\project.py", line 762, in make_dataset
    return ProjectDataset(self)
  File "c:\users\*\*\.venv\lib\site-packages\datumaro\components\project.py", line 473, in __init__
    url, **options)
  File "c:\users\*\*\.venv\lib\site-packages\datumaro\components\project.py", line 274, in make_extractor
    return self.extractors.get(name)(*args, **kwargs)
  File "c:\users\*\*\.venv\lib\site-packages\datumaro\components\project.py", line 64, in get
    return self.items[key] # returns a class / ctor
KeyError: 'cvat_rest_api_task_images'

I tried to figure out the problem with no results.

I am working on windows 7.
I installed datamaru as following:

pip install 'git+https://github.com/opencv/cvat#egg=datumaro&subdirectory=datumaro'

[zhiltsov-max]

• Ensure you have the corresponding file in <project_dir>/.datumaro/extractors or <project_dir>/.datumaro/plugins (depending on the CVAT version, the latter is for actual). If there is no such file for some reason, you can find it here.
• Ensure you have <project_dir> in your PYTHONPATH env. variable
• Install cvat/util/cli/requirements.txt if you haven't done this

[AhmadM-DL]

@zhiltsov-max

<project_dir>/.datumaro/plugins does exist

with the following structure:

plugins
    cvat_rest_api_task_images.py
    __pycache__
        cvat_rest_api_task_images.cpython-37

I also have a config yaml file in the project dir:

format_version: 1
models: {}
project_name: undefined
sources:
  task_11_images:
    format: cvat_rest_api_task_images
    options: {}
    url: ''
subsets: []

During datumaro requirements installation I had an error when installing pycocotools:

cl : Command line error D8021 : invalid numeric argument '/Wno-cpp'

I resolved by checking the following cocoapi issue: cocodataset/cocoapi#51
and downloading a cocoapi version:

pip install git+https://github.com/philferriere/cocoapi.git#subdirectory=PythonAPI

Any thoughts?

Thanks for your time and consideration.

[zhiltsov-max]

Try running the command with --loglevel=debug flag set like datum --loglevel=debug project ...

[AhmadM-DL]

Before going further I want to mention that initially my exported datumaro project didn't contain
a config yaml file inside <project_dir>/.datamaro. And when I run without the file I get the following error:

2020-03-16 10:37:21,692 ERROR: [Errno 2] No such file or directory: 'C:\\Users\\PC\\Downloads\\t\\.datumaro\\config.yaml'
Traceback (most recent call last):
  File "C:\Program Files\Python37\Scripts\datum-script.py", line 11, in <module>
    load_entry_point('datumaro==0.1.0', 'console_scripts', 'datum')()
  File "c:\program files\python37\lib\site-packages\datumaro\cli\__main__.py", line 156, in main
    return args.command(args)
  File "c:\program files\python37\lib\site-packages\datumaro\cli\contexts\project\__init__.py", line 322, in export_command
    project = load_project(args.project_dir)
  File "c:\program files\python37\lib\site-packages\datumaro\cli\util\project.py", line 12, in load_project
    return Project.load(project_dir)
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 711, in load
    config = Config.parse(config_path)
  File "c:\program files\python37\lib\site-packages\datumaro\components\config.py", line 209, in parse
    with open(path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\PC\\Downloads\\t\\.datumaro\\config.yaml'

To resolve the problem I am copying the config yaml file in the <project_dir> into <project_dir>/.datumaro. Is that normal?

Now, debug log level did reveal some problems:

2020-03-16 10:33:42,839 DEBUG: Popen(['git', 'init'], cwd=C:\Users\PC\Downloads\t, universal_newlines=False, shell=None, istream=None)
2020-03-16 10:33:42,927 DEBUG: Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'The system cannot find the file specified', None, 2, None)
2020-03-16 10:33:42,930 DEBUG: Failed to import module 'cvat_rest_api_task_images.py': No module named 'requests'
2020-03-16 10:33:42,931 INFO: Loading the project...
2020-03-16 10:33:42,931 ERROR: 'cvat_rest_api_task_images'
Traceback (most recent call last):
  File "C:\Program Files\Python37\Scripts\datum-script.py", line 11, in <module>
    load_entry_point('datumaro==0.1.0', 'console_scripts', 'datum')()
  File "c:\program files\python37\lib\site-packages\datumaro\cli\__main__.py", line 156, in main
    return args.command(args)
  File "c:\program files\python37\lib\site-packages\datumaro\cli\contexts\project\__init__.py", line 347, in export_command
    dataset = project.make_dataset()
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 762, in make_dataset
    return ProjectDataset(self)
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 473, in __init__
    url, **options)
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 274, in make_extractor
    return self.extractors.get(name)(*args, **kwargs)
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 64, in get
    return self.items[key] # returns a class / ctor
KeyError: 'cvat_rest_api_task_images'

It is saying that importing "cvat_rest_api_task_imaegs" failed due to missing module "requests"
pip install requests resolved the problem but now I am getting another missing
module "cvat"

2020-03-16 10:43:59,343 DEBUG: Popen(['git', 'init'], cwd=C:\Users\PC\Downloads\t, universal_newlines=False, shell=None, istream=None)
2020-03-16 10:43:59,435 DEBUG: Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'The system cannot find the file specified', None, 2, None)
2020-03-16 10:43:59,440 DEBUG: Failed to import module 'cvat_rest_api_task_images.py': No module named 'cvat'
2020-03-16 10:43:59,442 INFO: Loading the project...
2020-03-16 10:43:59,442 ERROR: 'cvat_rest_api_task_images'
Traceback (most recent call last):
  File "C:\Program Files\Python37\Scripts\datum-script.py", line 11, in <module>
    load_entry_point('datumaro==0.1.0', 'console_scripts', 'datum')()
  File "c:\program files\python37\lib\site-packages\datumaro\cli\__main__.py", line 156, in main
    return args.command(args)
  File "c:\program files\python37\lib\site-packages\datumaro\cli\contexts\project\__init__.py", line 347, in export_command
    dataset = project.make_dataset()
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 762, in make_dataset
    return ProjectDataset(self)
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 473, in __init__
    url, **options)
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 274, in make_extractor
    return self.extractors.get(name)(*args, **kwargs)
  File "c:\program files\python37\lib\site-packages\datumaro\components\project.py", line 64, in get
    return self.items[key] # returns a class / ctor
KeyError: 'cvat_rest_api_task_images'

It seems that I have ran into some problems during datum installation process!
Though I followed all the steps exactly except for downloading cocoapi from an altered github repo as aforementioned.

Where should I get "cvat" module from?

[zhiltsov-max]

If you haven't found config.yml in .datumaro, I believe you exported the task from an outdated version of CVAT, and installed the actual version from develop, where project layout has been changed a bit. The corresponding to CVAT version of Datumaro is available at <project_dir>. It is installed as described in the readme there.

Regarding requests package - it should have been installed from <project_dir>/cvat/utils/cli/requirements.txt. This step is described in the README. The cvat package required should be taken from the <project_dir> by providing PYTHONPATH=<project_dir>, which is also mentioned in the readme.

I've just rechecked if these steps work (on Ubuntu, though) - they did. However, I've found that some attributes were not being passed this way, specifically occluded and z_order. This can be fixed by this patch (#1271). Meanwhile, exporting as CVAT xml and importing CVAT project in Datumaro should work without an issue.