How to evaluate DSTC7?

[Zhouziyi828]

Could you please tell me how to use the github link to evaluate DSTC7?

[Chenrj233]

Hello, the github link contains the data and introduction of the DSTC7-AVSD task. Firstly, you need to replace UNDISCLOSED in the test data set with the answer predicted by the model, and then use the evaluation script to get the metrics.

[Zhouziyi828]

Thanks for your reply.
But when I ran the ./dstc7avsd_eval.sh sample/baseline_i3d_rgb-i3d_flowh.json I found an error:

sentences = '\n'.join([c['caption'].replace('\n', ' ') for k, v in list(captions_for_image.items()) for c in v])

AttributeError: 'NoneType' object has no attribute 'replace'

It seems that it is because that the 'caption' annotations are not included in the result file: sample/baseline_i3d_rgb-i3d_flowh.json.
Did you have the same problem?

[Chenrj233]

We did not encounter this issue during the evaluation. It's possible that your environment may differ from the requirements specified in dstc7avs_eval README. We recommend creating a new virtual environment to run the evaluation script.