Release 2.1.16: metrics file loses a line every time training is stopped and then continued

[tuglat]

Let's say you train for 10 checkpoints and stop. The metrics file will have info for 10 checkpoints, and the latest params file will be params.00010.

Now, suppose you continue training and stop after an additional 10 checkpoints. The metrics file will have info for 19 checkpoints, and the latest params file will be params.00020. The line that was lost was the line corresponding to params.00010. All of the lines from the 10'th on are now incorrect. Line (9+n) of the metrics file actually contains information about checkpoint (9 + n + 1).

[fhieber]

Hi @tuglat thanks for reporting this issue! I was able to reproduce and it turns out we incorrectly save the training state before adding metrics to it. #860 should fix it.