OpusTrainer should disable early stopping until the final stage

[gregtatum]

While training one of my teachers in the ensemble started out very poorly, and was stopped due to early stopping before it could get to the later stages. Here this is the yellow graph of bad behavior.

Loss:

chrF:

This is the in task group PCOkERaaRtu6s6I7-xE5aA.

[gregtatum]

@eu9ene I'm thinking the work here is to have two different training steps. The first step would be to apply N-1 of the training schedule with early-stopping set to 0. Once that completes, we then would run the final stage with early-stopping being taken from the config.

[eu9ene]

@eu9ene I'm thinking the work here is to have two different training steps. The first step would be to apply N-1 of the training schedule with early-stopping set to 0. Once that completes, we then would run the final stage with early-stopping being taken from the config.

Yes, to do that we should implement support of training parameters for each stage on the OpusTrainer side. However we used to train on the mixed dataset for 2 epochs with default early stopping (20) and it worked fine. So, we should investigate what changed here. Maybe in fact it did early stop sometimes but since we used a different task for finetuning it didn't affect it. I think the main issue might be just the proportion of the back-translated data + pre-training on the original. We might be able to fix it even without using different parameters for now. See #314.

[eu9ene]

Created an issue for OpusTrainer: hplt-project/OpusTrainer#44

[eu9ene]

Closing. See discussion in hplt-project/OpusTrainer#44. The issue was #352 and not early stopping. It should train fine with the same parameters now.