Enable Deepspeed on Windows

[guillochon]

🚀 Feature

Currently, deepspeed is explicitly disabled on Windows systems:

_DEEPSPEED_AVAILABLE = not _IS_WINDOWS and _module_available('deepspeed')

I checked by commenting out the not _IS_WINDOWS part that deepspeed indeed does not currently work on Windows, the following error is emitted:

    595         elif backend == Backend.NCCL:
    596             if not is_nccl_available():
--> 597                 raise RuntimeError("Distributed package doesn't have NCCL "
    598                                    "built in")
    599             pg = ProcessGroupNCCL(
RuntimeError: Distributed package doesn't have NCCL built in

Motivation

It would be awesome to get the speed and memory benefits of deepspeed on Windows machines too!

Pitch

My understanding is that enabling this requires using our own implementation of parallelism via MPI rather than Pytorch distributed, which doesn't support Windows. Not sure how involved this is.

Alternatives

N/A

Additional context

Discussion with @SeanNaren about it here: https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1616436343124400

[aribornstein]

@SeanNaren is this something the deep speed team can support?

[SeanNaren]

DeepSpeed already supports MPI which is windows compatible! In order to integrate this, we'll need to refactor our code such that we use DeepSpeed's initialization function: https://deepspeed.readthedocs.io/en/latest/initialize.html#deepspeed.init_distributed

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[SeanNaren]

Support has been tentatively added in #8488, without end to end testing. So if anyone gets the chance to try this out on Windows end to end and report back it would be awesome!