bug of resume model:

[luckycaicai]

Original code:

mmcv/mmcv/runner/iter_based_runner.py

Line 203 in 54ece10

meta.update(self.meta)

Fix to:
self.meta.update(meta)

As well as,

mmcv/mmcv/runner/epoch_based_runner.py

Line 159 in 54ece10

meta.update(self.meta)

[zhouzaida]

meta is the information to be saved in the checkpoint rather than self.meta

[luckycaicai]

I know that, but the trouble is, you have initialized self.meta in base_runner.resume, like this:

mmcv/mmcv/runner/base_runner.py

Line 362 in 59ed0dd

self.meta = checkpoint['meta']

but not do any update it after epoch/iter.

For example, when I resume model from epoch=4, iter=200, and save model after 1 epoch, after the line,

mmcv/mmcv/runner/iter_based_runner.py

Line 203 in 54ece10

meta.update(self.meta)

meta is updated by self.meta, the saved information is always epoch=4, iter=200.
It will be always the meta you initialized.

so, the following line maybe the simplest solution to fix it.
self.meta.update(meta)

[ChiefGodMan]

I think you

I know that, but the trouble is, you have initialized self.meta in base_runner.resume, like this:

mmcv/mmcv/runner/base_runner.py

Line 362 in 59ed0dd

self.meta = checkpoint['meta']

but not do any update it after epoch/iter.
For example, when I resume model from epoch=4, iter=200, and save model after 1 epoch, after the line,

mmcv/mmcv/runner/iter_based_runner.py

Line 203 in 54ece10

meta.update(self.meta)

meta is updated by self.meta, the saved information is always epoch=4, iter=200.
It will be always the meta you initialized.
so, the following line maybe the simplest solution to fix it.
self.meta.update(meta)

I think you are right. Actually I had trained my model to 7 epoches, but after I resume from ckpt it returned to previous 4 epoches.

[zhouzaida]

@luckycaicai @ailias , thanks for your feedback, the issue will be resolved by the PR #1108