Model does not generate masks. (segm.json is empty)

[akashpalrecha]

Questions and Help

Hello everyone!
First of all, I'm a beginner in AI, please bear with me as this is the first thing I've ever written on Github. I'll try making my issue as clear as possible.

OUTLINE - Model trains normally - loss_mask goes down - Evaluation shows zero results for masks - corresponding segm.json file in output directory is empty.

Here's what I did:
I tried using this repo to generate masks for an entirely different dataset (Massachusetts Roads Dataset - Contains satellite imagery of urban areas and corresponding masks of roads present in them). I converted the dataset to COCO format using pycococreatortools and placed everything in correct default folders.
I then started training like so- (a short schedule for the purposes of this question, longer schedules produce similar results)
python tools/train_net.py --config-file "configs/e2e_mask_rcnn_R_50_FPN_1x.yaml" SOLVER.IMS_PER_BATCH 1 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 180 SOLVER.STEPS "(120, 160)" TEST.IMS_PER_BATCH 2 OUTPUT_DIR './output'

In the config file, I added NUM_CLASSES: 2 under ROI_BOX_HEAD as suggested in #273 (one for roads and one for background.
This is what the results looks like -
Evaluate annotation type *bbox*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.01s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.008
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.168
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.005
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.055
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.055
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.167
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.035
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.00s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

In the output folder under inference, the segm.json file looks like this for every image:
{"image_id": 112, "category_id": 1, "segmentation": {"size": [500, 500], "counts": "Td7"}, "score": 0.06615357100963593}For all images, the"counts":"Td7"` part is exactly the same.

How do I get the masks to be generated ? What am I doing wrong ?

Surprisingly, the first time I ran the exact same command, the masks were generated. But I coudln't reproduce the results again. I haven't changed anythin since then and yet it is behaving differently. (except the fact that I previously trained on 300x300 images and now they are 500x500)

I'm attaching the output of the training here.
output3.txt
Also, these are the annotations that I have used in case there is something wrong with them (changed the extension to upload on Github) -

instances_train2014.txt

[fmassa]

Hi,

Your training schedule seems to be too short (only 180 SGD steps).
This means that the model didn't have the time to start learning anything I believe.