[Hotfix] Synchronization upon Checkpointing

[justusschock]

This adds a barrier before checkpointing.

Fixes #5604

[ananthsub]

could you describe why the barrier here fixes it?

[SeanNaren]

Nice, can you just add a comment to explain why this is done?

Suggested change

	trainer.accelerator.barrier()
	# Synchronize all processes if using distributed
	trainer.accelerator.barrier()

[SeanNaren]

Just FYI @justusschock I'm not sure if you found the root cause, but I noticed that if I omit this broadcast, the reproducible code runs for me:

https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/callbacks/model_checkpoint.py#L599

[codecov]

Codecov Report

Merging #6185 (bd50df7) into master (ee35907) will decrease coverage by 2%.
The diff coverage is 96%.

@@           Coverage Diff            @@
##           master   #6185     +/-   ##
========================================
- Coverage      93%     91%     -2%     
========================================
  Files         116     159     +43     
  Lines        8833   11368   +2535     
========================================
+ Hits         8235   10350   +2115     
- Misses        598    1018    +420     

[carmocca]

The branch this PR is based off is very old (November). Can you rebase master?

[marrrcin]

Just FYI @justusschock I'm not sure if you found the root cause, but I noticed that if I omit this broadcast, the reproducible code runs for me:

https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/callbacks/model_checkpoint.py#L599

I can confirm that removing this line fixes the issue my case (with more advanced setup than in the reproduction steps: complex model, optimizer with custom scheduler and ddp). Would also like to know more what is the root cause.

[justusschock]

Will investigate further