Cannot reproduce the results. far from the paper.

[happycoding1996]

Dear author,

Thank you for sharing your work.

Howevr, I cannot reproduce the performance on VOC with the inductive setting. Here are my results obtained by training vpt_seg_zero_vit-b_512x512_20k_12_10.py with 4GPUs.

I have run the experiments twice and found that the results are highly fluctuating and not reproducible, still far away from the results reported in your paper.

I strictly followed the data preparation of MMseg, and testing with your pre-trained model can get the results same to that in your paper. Could you help me address this issue?

+++++++++++ Total classes +++++++++++++
per class results:
+-------------+-------+-------+
| Class | IoU | Acc |
+-------------+-------+-------+
| aeroplane | 96.77 | 97.32 |
| bicycle | 87.05 | 95.63 |
| bird | 98.13 | 99.09 |
| boat | 93.51 | 99.11 |
| bottle | 92.13 | 92.82 |
| bus | 97.2 | 98.1 |
| car | 93.62 | 96.36 |
| cat | 97.36 | 98.1 |
| chair | 43.48 | 48.82 |
| cow | 95.41 | 96.28 |
| diningtable | 77.64 | 87.08 |
| dog | 94.73 | 97.51 |
| horse | 96.44 | 97.52 |
| motorbike | 93.35 | 96.78 |
| person | 95.4 | 97.42 |
| pottedplant | 31.27 | 31.47 |
| sheep | 92.1 | 99.11 |
| sofa | 51.73 | 96.23 |
| train | 98.03 | 99.9 |
| tvmonitor | 33.86 | 35.61 |
+-------------+-------+-------+
Summary:
+-------+-------+-------+
| aAcc | mIoU | mAcc |
+-------+-------+-------+
| 92.51 | 82.96 | 88.01 |
+-------+-------+-------+

+++++++++++ Seen classes +++++++++++++
seen per class results:
+-------------+-------+-------+
| Class | IoU | Acc |
+-------------+-------+-------+
| aeroplane | 96.77 | 97.32 |
| bicycle | 87.05 | 95.63 |
| bird | 98.13 | 99.09 |
| boat | 93.51 | 99.11 |
| bottle | 92.13 | 92.82 |
| bus | 97.2 | 98.1 |
| car | 93.62 | 96.36 |
| cat | 97.36 | 98.1 |
| chair | 43.48 | 48.82 |
| cow | 95.41 | 96.28 |
| diningtable | 77.64 | 87.08 |
| dog | 94.73 | 97.51 |
| horse | 96.44 | 97.52 |
| motorbike | 93.35 | 96.78 |
| person | 95.4 | 97.42 |
+-------------+-------+-------+
Seen Summary:
+-------+-------+-------+
| aAcc | mIoU | mAcc |
+-------+-------+-------+
| 92.51 | 90.15 | 93.19 |
+-------+-------+-------+

+++++++++++ Unseen classes +++++++++++++
unseen per class results:
+-------------+-------+-------+
| Class | IoU | Acc |
+-------------+-------+-------+
| pottedplant | 31.27 | 31.47 |
| sheep | 92.1 | 99.11 |
| sofa | 51.73 | 96.23 |
| train | 98.03 | 99.9 |
| tvmonitor | 33.86 | 35.61 |
+-------------+-------+-------+
Unseen Summary:
+-------+------+-------+
| aAcc | mIoU | mAcc |
+-------+------+-------+
| 92.51 | 61.4 | 72.46 |
+-------+------+-------+

================================================
When I rerun again, I got the results as:
+++++++++++ Total classes +++++++++++++
per class results:
+-------------+-------+-------+
| Class | IoU | Acc |
+-------------+-------+-------+
| aeroplane | 99.22 | 99.69 |
| bicycle | 88.86 | 96.46 |
| bird | 98.13 | 99.12 |
| boat | 96.76 | 99.1 |
| bottle | 92.91 | 94.62 |
| bus | 97.59 | 98.34 |
| car | 93.77 | 96.84 |
| cat | 96.35 | 97.38 |
| chair | 53.86 | 74.51 |
| cow | 94.98 | 95.66 |
| diningtable | 84.4 | 88.17 |
| dog | 94.04 | 97.52 |
| horse | 97.11 | 98.16 |
| motorbike | 93.14 | 97.56 |
| person | 95.74 | 97.57 |
| pottedplant | 56.7 | 58.07 |
| sheep | 93.85 | 97.81 |
| sofa | 52.77 | 81.53 |
| train | 97.64 | 99.92 |
| tvmonitor | 52.98 | 54.49 |
+-------------+-------+-------+
Summary:
+-------+-------+-------+
| aAcc | mIoU | mAcc |
+-------+-------+-------+
| 93.76 | 86.54 | 91.13 |
+-------+-------+-------+

+++++++++++ Seen classes +++++++++++++
seen per class results:
+-------------+-------+-------+
| Class | IoU | Acc |
+-------------+-------+-------+
| aeroplane | 99.22 | 99.69 |
| bicycle | 88.86 | 96.46 |
| bird | 98.13 | 99.12 |
| boat | 96.76 | 99.1 |
| bottle | 92.91 | 94.62 |
| bus | 97.59 | 98.34 |
| car | 93.77 | 96.84 |
| cat | 96.35 | 97.38 |
| chair | 53.86 | 74.51 |
| cow | 94.98 | 95.66 |
| diningtable | 84.4 | 88.17 |
| dog | 94.04 | 97.52 |
| horse | 97.11 | 98.16 |
| motorbike | 93.14 | 97.56 |
| person | 95.74 | 97.57 |
+-------------+-------+-------+
Seen Summary:
+-------+-------+-------+
| aAcc | mIoU | mAcc |
+-------+-------+-------+
| 93.76 | 91.79 | 95.38 |
+-------+-------+-------+

+++++++++++ Unseen classes +++++++++++++
unseen per class results:
+-------------+-------+-------+
| Class | IoU | Acc |
+-------------+-------+-------+
| pottedplant | 56.7 | 58.07 |
| sheep | 93.85 | 97.81 |
| sofa | 52.77 | 81.53 |
| train | 97.64 | 99.92 |
| tvmonitor | 52.98 | 54.49 |
+-------------+-------+-------+
Unseen Summary:
+-------+-------+-------+
| aAcc | mIoU | mAcc |
+-------+-------+-------+
| 93.76 | 70.79 | 78.36 |
+-------+-------+-------+

[ZiqinZhou66]

Sorry for the late response. Did it fix now? Actually, I did not occur such extremely unstable experimental results.

Another similar issue was solved by correcting the batch size:
#7 (comment)

Have you ever tested on other datasets?

[aliman80]

@happycoding1996 : I have tried to run it for the coco stuff dataset and i got resuts for first 11 classes and rest all are zero. Did you change anything in the repository before validating or testing ? I just update the dataset path.

[DeserveLars]

@happycoding1996：我尝试对 coco stuff 数据集运行它，我得到了前 11 类结果，其余的都是零。在验证或测试之前，您是否更改了存储库中的任何内容？我只是更新了数据集路径。

Hello,did you solve this issue?