Standalone evaluation code

[alcinos]

Hello,

Thank you for providing the code, and congratulation on the paper acceptance, this is a very interesting task.

It is very helpful to provide the baseline model, however, I’d argue that it would facilitate adoption if a standalone reference evaluator for the task was provided. Something with minimal dependencies (eg no assumption on the DeepLearning framework used, although numpy/sklearn are probably ok), that takes in the path to the annotations, as well as the model predictions in a documented format (json or dict for eg), and spits out the official metrics. Some basic checks could also be carried out, eg that all predictions are present, that there is no duplicate, that the masks have the correct resolution,...

As far as I can tell, the current evaluation code is too tightly integrated with the model to be used independently, for example taking care of dealing with distributed aspect, relying on internal configuration classes, and more importantly, integrating the forward pass directly inside the evaluation function.

For examples of such standalone evaluators, I refer to the panoptic evaluation toolkit for coco: https://github.com/cocodataset/panopticapi or for a more closely related task, the referring expression segmentation evaluator on PhraseCut: https://github.com/ChenyunWu/PhraseCutDataset

Looking forward to working on this task,

Best regards,
Nicolas Carion