[REVIEW] Add GPU and CUDA validations

[galipremsagar]

Closes #4620, #870, #499

• Add GPU driver and runtime validations
• Test on a GPU machine
• Test on non-GPU machine with libcuda & toolkit installed.

[galipremsagar]

Since we are actually using cupy for the driver & GPU check. If cupy doesn't see a GPU it actually errors out like this:

>>> import cudf
Traceback (most recent call last):
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/__init__.py", line 21, in <module>
    from cupy import core  # NOQA
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/core/__init__.py", line 1, in <module>
    from cupy.core import core  # NOQA
  File "cupy/core/core.pyx", line 1, in init cupy.core.core
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/cuda/__init__.py", line 5, in <module>
    from cupy.cuda import compiler  # NOQA
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/cuda/compiler.py", line 14, in <module>
    from cupy.cuda import function
  File "cupy/cuda/texture.pxd", line 6, in init cupy.cuda.function
  File "cupy/cuda/texture.pyx", line 1, in init cupy.cuda.texture
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/conda/envs/cudf/lib/python3.7/site-packages/cudf/__init__.py", line 6, in <module>
    import cupy
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/__init__.py", line 42, in <module>
    six.reraise(ImportError, ImportError(msg), exc_info[2])
  File "/conda/envs/cudf/lib/python3.7/site-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/__init__.py", line 21, in <module>
    from cupy import core  # NOQA
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/core/__init__.py", line 1, in <module>
    from cupy.core import core  # NOQA
  File "cupy/core/core.pyx", line 1, in init cupy.core.core
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/cuda/__init__.py", line 5, in <module>
    from cupy.cuda import compiler  # NOQA
  File "/conda/envs/cudf/lib/python3.7/site-packages/cupy/cuda/compiler.py", line 14, in <module>
    from cupy.cuda import function
  File "cupy/cuda/texture.pxd", line 6, in init cupy.cuda.function
  File "cupy/cuda/texture.pyx", line 1, in init cupy.cuda.texture
ImportError: CuPy is not correctly installed.

If you are using wheel distribution (cupy-cudaXX), make sure that the version of CuPy you installed matches with the version of CUDA on your host.
Also, confirm that only one CuPy package is installed:
  $ pip freeze

If you are building CuPy from source, please check your environment, uninstall CuPy and reinstall it with:
  $ pip install cupy --no-cache-dir -vvvv

Check the Installation Guide for details:
  https://docs-cupy.chainer.org/en/latest/install.html

original error: libcuda.so.1: cannot open shared object file: No such file or directory

@kkraus14 @jrhemstad
Will this be okay for us?

[codecov]

Codecov Report

Merging #4692 into branch-0.14 will decrease coverage by 0.14%.
The diff coverage is 47.05%.

@@               Coverage Diff               @@
##           branch-0.14    #4692      +/-   ##
===============================================
- Coverage         88.3%   88.16%   -0.15%     
===============================================
  Files               49       51       +2     
  Lines             9707     9741      +34     
===============================================
+ Hits              8572     8588      +16     
- Misses            1135     1153      +18

Impacted Files	Coverage Δ
python/cudf/cudf/errors.py	0% <0%> (ø)
python/cudf/cudf/utils/gpu_utils.py	53.33% <53.33%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e9d9d79...2e23210. Read the comment docs.

[jrhemstad]

That error occurs if libcuda isn't available, probably if the toolkit isn't installed.

[kkraus14]

No that isn't okay because libcuda and the toolkit could exist on a machine without GPUs for example.

We should also much more gracefully error if we can't find the proper libraries.

[kkraus14]

@jrhemstad would you mind reviewing again?

[jrhemstad]

Sure would be nice if someone were to make a standard set of Python wrappers for the CUDA runtime APIs!

[kkraus14]

There's a reason this is in cudf/_cuda as opposed to cudf/_libxx ;)

[kkraus14]

@galipremsagar as a follow up to this, we likely need to modify setup.py to ship the .pxd files of _cuda and subdirectories in our package. Can you raise an issue and tackle that in a follow up PR?

[galipremsagar]

@galipremsagar as a follow up to this, we likely need to modify setup.py to ship the .pxd files of _cuda and subdirectories in our package. Can you raise an issue and tackle that in a follow up PR?

Sure, here is a link to the FEA: #4785