Include cublas error details when getting cublas handle fails

[jli]

I've been getting hard-to-debug errors in some DeepSpeed runs. During initialization, one of the worker processes raises RuntimeError: Fail to create cublas handle. with no further details, which feels pretty mysterious.

This change includes details of the failure status by using https://docs.nvidia.com/cuda/cublas/#cublasgetstatusname and https://docs.nvidia.com/cuda/cublas/#cublasgetstatusstring

---

original error message (using deepspeed 0.9.2): RuntimeError: Fail to create cublas handle.

new error message with this change: RuntimeError: Failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED the library was not initialized

This is still not a great error message, but it has better search results (most results suggest that it's due to running out of GPU memory; bizarrely some people also report removing ~/.nv fixes it...).

[jli]

@microsoft-github-policy-service agree

[jli]

CI checks are failing because the CI environment is using a version of CUDA/cuBLAS that predates these functions. cublasGetStatusName and cublasGetStatusString were added in CUDA 11.4.2 (released in late 2021 I believe).

@loadams It seems like DeepSpeed doesn't specify any minimum version of CUDA, so I'm guessing you'd rather not include this change. If not, maybe we could instead just include the raw enum number, which the user could look up manually?

details

Example error:

/tmp/actions-runner/_work/DeepSpeed/DeepSpeed/unit-test-venv/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/context.h(56): error: identifier "cublasGetStatusName" is undefined

According to the ds_report output, the CI checks use these versions:

• nv-accelerate-v100: torch cuda version 11.7, nvcc 11.1
• nv-torch19-p40: torch cuda version 11.1, nvcc 11.1
• nv-torch19-v100: torch cuda version 11.1, nvcc 11.1

I'm not sure why nv-accelerate-v100 still failed with cuda 11.7, but maybe it's because it's using nvcc 11.1.

(the nv-inference failure seems unrelated? TypeError: can't assign a NoneType to a torch.cuda.HalfTensor)

[loadams]

Thanks for the info @jli.

cc: @mrwyattii and @jeffra for FYI. At least until we got a min CUDA version, I think it would make sense to at least print the raw enum number to add more debug info for users.