CUDA 11/A100 Support

[afiaka87]

I saw in another thread that there are plans to target the A100 next. This sounds very useful to me as I'm trying to use sparse attention in another project and I've had luck getting access to A100's recently.

CUDA 11 is quickly becoming the de facto standard on a lot of cloud servers. I'm aware that I could roll back to 10.2, but I'd love to get the improvements of both.

Anyway, keep me posted on this feature if you can.

Thanks

[awan-10]

Hi @afiaka87, can you please elaborate the problem a bit more?

Did you try to build from source on a CUDA11 machine and it failed?

We do support CUDA11 and A100.

[afiaka87]

We're having a lot of trouble getting Sparse attention specifically to work. The issue remains even after the ZeRO Infinity update. I'll follow up tomorrow with an error message

@awan-10

[helena-balabin]

Do you have any advice on how to configure deepspeed for CUDA11 and an A100? I'm currently trying to set it up, but I'm constantly running into incompatibility issues.

[loadams]

Hi @afiaka87 and @helena-balabin - I'm closing this issue as stale given the age of it. Cuda/Torch for A100s should be more well supported now there and in DeepSpeed. If you are still having any issues, please open a new issue and link this one and I'd be happy to take a look at any issues.

Thanks!