Should Lightning provide a default worker_init_fn?

[awaelchli]

When your dataset generates random numbers (e.g. for data augmentation), the seed in the workers is only correct for torch but not other libraries like numpy will set an identical seed in all workers [1]. The result is that data augmentations are the same for all workers which can obviously affect performance.

Lightning could provide a default worker init fn which takes care of this issue. Even if we don't add a default, the question is how would the user do it properly today in Lightning? The tricky part is that it is required to reset the seed every epoch (when the dataloader is consumed completely).

Note: An easy way out of this is to not use numpy at all to generate random numbers in the dataloader. When using torch, you're fine.

[1] https://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/

[ethanwharris]

A default worker_init_fn might be the way forward, particularly if using seed_everything where the user has directly asked us to take control of seeding things properly.

[Borda]

I like the worker_init_fn :]

[awaelchli]

Made a quick PR #6960
One thing to consider, this will be a breaking change even if it helps users who have this "bug". By breaking change I mean parity is not guaranteed, performance will differ after releasing this change (for those who use numpy.random).

If we value parity between lightning versions more, we should not add this change.

[ethanwharris]

That's a very good point. It's definitely nice to have the option, so maybe a seed_workers flag in seed_everything is the way to go? We could even default this flag to True and document that it should be switched to False if parity is important to the user.