About the effects of sliding_window_stride

[xiangyh9988]

Hi Bain,
I see you mentioned that setting sliding_window_stride=12 when evaluating retrieval on MSR-VTT (finetuned) helps improve the performance in other issues. I tryied this but didn't get the improvement.

After finetuned with msrvtt_4f_i21k.json, the model is tested as the command presented in the README. Results are:

[t2v_metrics] epoch 0, R@1: 28.9, R@5: 55.6, R@10 66.2, R@50 86.8MedR: 4, MeanR: 29.9                                                                                                                                                                  
[v2t_metrics] epoch 0, R@1: 28.4, R@5: 56.5, R@10 66.2, R@50 88.1MedR: 4, MeanR: 25.6  

After setting --sliding_window_stride=12 for test.py, the results are:

[t2v_metrics] epoch 0, R@1: 28.8, R@5: 57.7, R@10 68.5, R@50 88.0MedR: 4, MeanR: 27.3
[v2t_metrics] epoch 0, R@1: 30.0, R@5: 58.8, R@10 68.8, R@50 89.7MedR: 4, MeanR: 22.5

It shows no obvious improvement in my test.

In #41, sliding_window_stride indeed helps improve the evaluation performance. I don't know why it doesn't work here. I keep codes in test.py unchanged and only modify some codes in base_dataset.py to fit my environment (i.e., lower version of PyTorch ans TorchVision due to the limitation on the computing cluster). Besides, the version of ffmpeg on my cluster is low and is hard to update. Is the difference of enviroments the reason leading to the poor results?

I just want to use one trained model with sound performance for some experiments in test phase (e.g., adversarial attacks), so I would like to see the finetuned Frozen-in-Time with R@1 over 30% as results in your paper shows. However, I failed to get such a model. :(

The phenomenon seems weird and I will check further to try to reproduce higher results. Besides, if possible, would you mind share a finetuned model?