Add installation instructions through conda

[h-vetinari]

Since merging conda-forge/fairscale-feedstock#2, conda-forge now has fairscale-packages also with GPU support, which I think is good enough to let people know also "officially" through the installation instructions here. :)

While editing the docs, I saw that the formatting of https://github.com/facebookresearch/fairscale/blob/main/docs/source/installation_instructions.rst didn't really work (rst vs. md), so I added some cosmetic touch-ups as I went, as well as the key insight I needed to arrive at for building the GPU packages from source (things fail without setting TORCH_CUDA_ARCH_LIST).

[anj-s]

@h-vetinari thank you for the PR as well as adding support for the conda install! Is there anything we need to do to ensure that we maintain conda installs for FairScale?

[h-vetinari]

Glad to hear it!

Is there anything we need to do to ensure that we maintain conda installs for FairScale?

You don't need to do anything, but you're welcome to join as a maintainer of the "feedstock" that contains the recipe for packaging, as well as the CI that executes it (joining as a maintainer would mean you get the rights on that github repo; it's not strictly necessary, normal pull requests work as well). In particular, I'd welcome some help on debugging various issues - currently there are some errors in the test suite on OSX, see conda-forge/fairscale-feedstock#5 (but since 0.4.3 at least, the final package passes the test suite on linux within conda-forge CI).

[min-xu-ai]

Thanks for the PR. I merged it in.

We actually don't have any test coverage on OSX. Do you know how big is a user base on OSX that runs GPU based training? I thought virtually nobody does that since I have never encountered a user like that. I do like to have that OS supported but we don't have a test env for it.

In fact, we only test linux. We don't test windows or OSX. Do conda-forge test both windows and osx? From the error you have seen in the other PR, I think pytorch on OSX is also somewhat spotty in terms of coverage. Pytorch on windows seems to be experimental. Any insight from you would be appreciated.

[blefaudeux]

oh wow, this is great, thanks @h-vetinari for all the work on the conda side !

[h-vetinari]

We actually don't have any test coverage on OSX. Do you know how big is a user base on OSX that runs GPU based training? I thought virtually nobody does that since I have never encountered a user like that. I do like to have that OS supported but we don't have a test env for it.

Conda-forge does not build GPU-enabled packages for OSX, currently things are CPU only (apple makes it quite hard with their lack of open source support in the graphics stack). Might change in the future, but it's gonna take a while I believe.

In fact, we only test linux. We don't test windows or OSX. Do conda-forge test both windows and osx? From the error you have seen in the other PR, I think pytorch on OSX is also somewhat spotty in terms of coverage. Pytorch on windows seems to be experimental. Any insight from you would be appreciated.

So in principle, conda-forge can support all of: linux-x64, linux-aarch64, linux-ppc64le, osx-x64, osx-arm64, win-x64. The non-x64 linux architectures are obviously still more niche but generally not too hard to support (I haven't gone through this for fairscale yet though; and neither for osx-arm64, where there are no native CI agents available).

Windows and osx on x64 are well-supported in general, the only problem for pytorch is that the builds run into the 6h timeout of the free-for-open-source azure agents, and so the development/iteration on the builds for this is slowed down quite a lot because it takes a conda-forge core member to manually build & upload them.

The long story short is that there are no windows builds for pytorch in conda-forge yet. This is something we'll hopefully fix soon(-ish) though. Once that dependency is there, there's no reason in principle why we couldn't also build (& test) fairscale on windows.

[h-vetinari]

oh wow, this is great, thanks @h-vetinari for all the work on the conda side !

Happy to hear it! As I mentioned, you're all welcome to take a look at the feedstock and participate, if you want to have a say in how things are packaged there.