OpenImage dataset json files are mismatched with OpenImagesChallengeEval ("OIC-EVAL") class

[herok97]

Hello,

I'm testing my model with compressai-vision. I have successfully completed experiments for SFU (with some name matching modifications). However, I'm facing difficulties with the mpeg-oiv6 dataset.

While there appear to be no issues with the codec or pipeline, I'm having trouble properly loading the annotation file for the mpeg-oiv6 dataset, as shown below:

[2023-10-30 16:02:57,485][fvcore.common.checkpoint][INFO] - [Checkpointer] Loading from /data/herok97/CompressAI-Vision/compressai_vision/model_wrappers/../../weights/detectron2/COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x/139173657/model_final_68b088.pkl ...
[2023-10-30 16:02:57,624][fvcore.common.checkpoint][INFO] - Reading a file from 'Detectron2 Model Zoo'
[2023-10-30 16:02:57,812][detectron2.data.datasets.coco][INFO] - Loaded 5000 images in COCO format from /data/dataset/MPEG-FCVCM/vcm_testdata/mpeg-oiv6/annotations/mpeg-oiv6-detection-coco.json
[2023-10-30 16:02:57,836][Detectron2Dataset][INFO] - "mpeg-oiv6-detection" successfully registred.
[2023-10-30 16:02:57,836][detectron2.data.common][INFO] - Serializing 5000 elements to byte tensors and concatenating them all ...
[2023-10-30 16:02:57,855][detectron2.data.common][INFO] - Serialized dataset takes 1.93 MiB
[2023-10-30 16:02:57,856][detectron2.data.dataset_mapper][INFO] - [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333, sample_style='choice')]
Error executing job with overrides: ['++pipeline.type=image', '++codec.codec_paths.encoder_path=/data/herok97/CompressAI-Vision/weights/lmsfc_v2/lmsfc_v2_detection_encoder.pth', '++codec.codec_paths.decoder_path=/data/herok97/CompressAI-Vision/weights/lmsfc_v2/lmsfc_v2_detection_decoder.pth', '++codec.encoder_config.qp=1', '++codec.eval_encode=bpp', '++codec.experiment=', '++codec.device=cuda', '++pipeline.output_dir_root=/data/herok97/CompressAI-Vision/results', '++pipeline.type=image', '++pipeline.conformance.save_conformance_files=True', '++pipeline.conformance.subsample_ratio=9', '++vision_model.arch=faster_rcnn_X_101_32x8d_FPN_3x', '++dataset.type=Detectron2Dataset', '++dataset.datacatalog=MPEGOIV6', '++dataset.config.root=/data/dataset/MPEG-FCVCM/vcm_testdata/mpeg-oiv6', '++dataset.config.annotation_file=annotations/mpeg-oiv6-detection-coco.json', '++dataset.config.dataset_name=mpeg-oiv6-detection', '++evaluator.type=OIC-EVAL', '++misc.device=cuda']
Traceback (most recent call last):
  File "/data/herok97/CompressAI-Vision/compressai_vision/run/eval_split_inference.py", line 125, in main
    pipeline, modules = setup(conf)
  File "/data/herok97/CompressAI-Vision/compressai_vision/run/eval_split_inference.py", line 69, in setup
    evaluator = create_evaluator(
  File "/data/herok97/CompressAI-Vision/compressai_vision/config/config.py", line 130, in create_evaluator
    return EVALUATORS[conf.type](catalog, datasetname, dataset, conf.output)
  File "/data/herok97/CompressAI-Vision/compressai_vision/evaluators/evaluators.py", line 123, in __init__
    assert "oic_labelmap" in json_dict
AssertionError

My dataset directory structure is as follows:

.vcm_testdata/mpeg-oiv6/
├── annotations
│   ├── mpeg-oiv6-detection-coco.json
│   └── mpeg-oiv6-segmentation-coco.json
└── images
     ├── 0a1bd356f90aaab6.jpg
     └── ...

The JSON files in the annotations folder were originally in the scripts folder and were copied over.

Upon checking the error, it seems that in compressai_vision/evaluators/evaluators.py, it's trying to find keys from the JSON file, which seem similar to those in mpeg-oiv6-detection-coco.json, but with different hierarchy and names (e.g., 'group_of' in evaluator.py and 'isgroupof' in the JSON file).

Is there any solution to this issue?

@register_evaluator("OIC-EVAL")
class OpenImagesChallengeEval(BaseEvaluator):
    def __init__(
        self, datacatalog_name, dataset_name, dataset, output_dir="./vision_output/"
    ):
        super().__init__(datacatalog_name, dataset_name, dataset, output_dir)

        with open(dataset.annotation_path) as f:
            json_dict = json.load(f)

        def _search_category(name):
            for e, item in enumerate(self.thing_classes):
                if name == self._normalize_labelname(item):
                    return e

            self._logger.error(f"Not found Item {name} in 'thing_classes'")
            raise ValueError

        assert len(json_dict["annotations"]) > 0
        has_segmentation = (
            True if "segmentation" in json_dict["annotations"][0] else False
        )

        self._valid_contiguous_id = []
        self._oic_labelmap_dict = {}
        self._oic_categories = []

        assert "oic_labelmap" in json_dict
        assert "oic_annotations" in json_dict

        valid_categories = json_dict["**oic_labelmap**"]
        gt_annotations = json_dict["**oic_annotations**"]

        for item in valid_categories:
            e = _search_category(self._normalize_labelname(item["name"]))

            if e is not None:
                self._valid_contiguous_id.append(e)

            self._oic_labelmap_dict[self._normalize_labelname(item["name"])] = item[
                "id"
            ]
            self._oic_categories.append({"id": item["id"], "name": item["name"]})

        self._oic_evaluator = OpenImagesChallengeEvaluator(
            self._oic_categories, evaluate_masks=has_segmentation
        )

        self._logger.info(
            f"Loading annotations for {len(gt_annotations)} images and reformatting them for OpenImageChallenge Evaluator."
        )
        for gt in tqdm(gt_annotations):
            img_id = gt["image_id"]
            annotations = gt["annotations"]

            anno_dict = {}
            anno_dict[InputDataFields.groundtruth_boxes] = np.array(annotations["bbox"])
            anno_dict[InputDataFields.groundtruth_classes] = np.array(
                annotations["**cls**"]
            )
            anno_dict[InputDataFields.groundtruth_group_of] = np.array(
                annotations["**group_of**"]
            )

            img_cls = []
            for name in annotations["img_cls"]:
                img_cls.append(self._oic_labelmap_dict[self._normalize_labelname(name)])
            anno_dict[InputDataFields.groundtruth_image_classes] = np.array(img_cls)

            if "**mask**" in annotations:
                segments, _ = decode_gt_raw_data_into_masks_and_boxes(
                    annotations["**mask**"], annotations["**img_size**"]
                )
                anno_dict[InputDataFields.groundtruth_instance_masks] = segments

            self._oic_evaluator.add_single_ground_truth_image_info(img_id, anno_dict)

        self._logger.info(
            f"All groundtruth annotations for {len(gt_annotations)} images are successfully registred to the evaluator."
        )

        self.reset()

Below is the shell script I executed.

#!/usr/bin/env bash
#
set -eu
ROOT_DIR="/data/herok97/CompressAI-Vision"
OUTPUT_DIR="${ROOT_DIR}/results"
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
ENTRY_CMD="compressai-vision-eval"
VCM_TESTDATA="/data/dataset/MPEG-FCVCM/vcm_testdata"    # YOUR DATASET DIR, ex) '../../vcm_testdata'


MPEG_OIV6_SRC="${VCM_TESTDATA}/mpeg-oiv6"
CODEC="lmsfc_v2"
DATASET="MPEGOIV6"
QP=$1   # 1, ..., 6

ENCODER_PATH="/data/herok97/CompressAI-Vision/weights/lmsfc_v2/lmsfc_v2_detection_encoder.pth"
DECODER_PATH="/data/herok97/CompressAI-Vision/weights/lmsfc_v2/lmsfc_v2_detection_decoder.pth"

# MPEGOIV6 - Detection with Faster RCNN
${ENTRY_CMD} --config-name=eval_lmsfc_v2.yaml \
             ++pipeline.type=image \
             ++codec.codec_paths.encoder_path=${ENCODER_PATH} \
             ++codec.codec_paths.decoder_path=${DECODER_PATH} \
             ++codec.encoder_config.qp=${QP} \
             ++codec.eval_encode='bpp' \
             ++codec.experiment="" \
             ++codec.device=${DEVICE} \
             ++pipeline.output_dir_root=${OUTPUT_DIR} \
             ++pipeline.type=image \
             ++pipeline.conformance.save_conformance_files=True \
             ++pipeline.conformance.subsample_ratio=9 \
             ++vision_model.arch=faster_rcnn_X_101_32x8d_FPN_3x \
             ++dataset.type=Detectron2Dataset \
             ++dataset.datacatalog=MPEGOIV6 \
             ++dataset.config.root=${MPEG_OIV6_SRC} \
             ++dataset.config.annotation_file=annotations/mpeg-oiv6-detection-coco.json \
             ++dataset.config.dataset_name=mpeg-oiv6-detection \
             ++evaluator.type=OIC-EVAL \
             ++misc.device="${DEVICE}"

[bblgbr]

Hello,
I'm using compressAi-Vision right now. I install from docker and it seems work successfully when running
compressai-vision-eval --help

but I run compressai-vision info, it has the problem. I really don't know why I have this problem.If you can help me, I would really appreciate it!

bash: compressai-vision: command not found