Distillation is broken

[eu9ene]

The hypothesis that we also discussed in #161 is that distillation might not work as expected with the on-the-fly augmentation since the student is supposed to be train on the exact outputs of the teacher. The gap between the teacher and student models is way too large. See also #231. I'll try to disable augmentation on the fly for the student model first to see how much it helps.

If it's the case, the proper fix would be to do augmentation of the corpus before decoding by the teachers.

[eu9ene]

I removed the augmentation but the new student model is still training https://firefox-ci-tc.services.mozilla.com/tasks/eyHs_KRMRbONpKLXIgj2fA/runs/1

[eu9ene]

Retraining with disabled augmentation didn't help, the results are almost the same. There's likely a bug somewhere else.

[eu9ene]

It seems the issue is in the merged training corpus. Source and target sentences don't match in the output of merge-translated step: https://firefox-ci-tc.services.mozilla.com/tasks/QlmjiyNmTJ-fzGGM520NMQ/runs/0

[eu9ene]

It's supposed to be fixed now. Retesting here https://firefox-ci-tc.services.mozilla.com/tasks/groups/V-OmRM1yS_GwESTVDPl0VQ