A significantly large number of queries of the NLQ task have a response window of 0 seconds.

[md-mohaiminul]

Thanks for this wonderful work!

A significantly large number of queries of the NLQ task have a response window of 0 seconds according to the annotations provided. Particularly, 1304 (8.59%) windows of the train and validation sets have 0s duration. According to the standard evaluation method (https://github.com/EGO4D/episodic-memory/blob/main/NLQ/VSLNet/utils/evaluate_ego4d_nlq.py), these windows will always produce an IoU of 0 irrespective of the predictions. Is this a problem with the dataset?

Moreover, a large number of windows have very small durations. Particularly, 2641 (17.41%) windows of the train and validation sets have a duration of less than 1 second.

I was wondering if these are errors with annotations? How should we handle these cases?

[ebyrne]

@md-mohaiminul We can confirm that this is in fact an issue with the annotation. We won't adjust the annotations while the current challenge round is running (till June 1), but we will update the annotations in the near future. We apologize for the additional noise here!

[ebyrne]

We'll post more updates here as well as they're available: https://discuss.ego4d-data.org/t/nlq-annotation-zero-temporal-windows/36