document deepspeed.initialize()

[stas00]

As a follow up to the very helpful suggestion I received at #633 I propose this doc update.

Thank you!

Fixes: #633

@jeffra

[jeffra]

We currently briefly reference this call related to MPI/AzureML environments (https://www.deepspeed.ai/getting-started/#mpi-and-azureml-compatibility) but thanks for adding it higher up since it seems others will also find this useful.

[stas00]

Honestly I'd have never looked there unless I had MPI or Azure ;)

Should I tweak the doc to add that it uses the NCCL backend?

[jeffra]

Haha totally understandable, it was kind of hidden away it seems :) It also needs to be added to our RTD documentation as well. I'll do that right now actually. You can choose the backend as a parameter, but it does default to NCCL and that's primarily what we've tested everything with across the project. Feel free to add some text about NCCL though.

[stas00]

Thank you, @jeffra

I added the note about the default, but I couldn't find init_distributed's documentation so I couldn't link to it for further details.

[jeffra]

Just added it to RTD docs with this PR #645

[jeffra]

If you want to add a link to it, it's live now: https://deepspeed.readthedocs.io/en/latest/initialize.html#distributed-initialization

[stas00]

That's excellent. Added the link.