unclear error message with older driver

[rongou]

Feature request

Reporting a bug

• I am using the latest version of cuDF from conda, built from master, or
  built from the latest tagged release.
• I have included the version or commit hash I am using for reference.
• I have included the following environment details:
  Linux Distro, Linux Kernel, GPU Model
• I have included the following version information for:
  Arrow, cmake, CUDA, gcc/g++, Numpy, Pandas, Python
• I have included below a minimal working reproducer (if you are unsure how
  to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

I followed the setup instructions for cuDF and cuML and tried to run the kmeans sample code, but got an error that turned out be caused by the nvidia driver being too old.

Steps to reproduce:

• On Ubuntu 18.04, the default nvidia driver is 390.87.
• Install CUDA 9.2.
• Install cuDF:

conda install -c numba -c conda-forge -c rapidsai -c defaults cudf=0.2.0

• Install cuML:

conda install -c pytorch faiss-gpu cuda92
conda install -c rapidsai cuml

Then run this code:

import cudf
import numpy as np
import pandas as pd

def np2cudf(df):
    # convert numpy array to cuDF dataframe
    df = pd.DataFrame({'fea%d'%i:df[:,i] for i in range(df.shape[1])})
    pdf = cudf.DataFrame()
    for c,column in enumerate(df):
        pdf[str(c)] = df[column]
    return pdf


a = np.asarray([[1.0, 1.0], [1.0, 2.0], [3.0, 2.0], [4.0, 3.0]],dtype=np.float32)
b = np2cudf(a)
print("input:")
print(b)

This produces the error:

Traceback (most recent call last):
  File "foo.py", line 16, in <module>
    b = np2cudf(a)
  File "foo.py", line 11, in np2cudf
    pdf[str(c)] = df[column]
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/cudf/dataframe.py", line 213, in __setitem__
    self.add_column(name, col)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/cudf/dataframe.py", line 454, in add_column
    series = self._prepare_series_for_add(data, forceindex=forceindex)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/cudf/dataframe.py", line 427, in _prepare_series_for_add
    self._sanitize_columns(col)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/cudf/dataframe.py", line 386, in _sanitize_columns
    series = Series(col)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/cudf/series.py", line 66, in __init__
    index = GenericIndex(data.index)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/cudf/index.py", line 209, in __new__
    values = NumericalColumn(data=Buffer(values), dtype=values.dtype)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/cudf/buffer.py", line 35, in __init__
    self.mem = cudautils.to_device(mem)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/cudf/cudautils.py", line 20, in to_device
    dary, _ = rmm.auto_device(ary)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/librmm_cffi/wrapper.py", line 199, in auto_device
    devobj.copy_to_device(obj, stream=stream)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/numba/cuda/cudadrv/devices.py", line 212, in _require_cuda_context
    return fn(*args, **kws)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/numba/cuda/cudadrv/devicearray.py", line 205, in copy_to_device
    _driver.host_to_device(self, ary, sz, stream=stream)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/numba/cuda/cudadrv/driver.py", line 1807, in host_to_device
    fn(device_pointer(dst), host_pointer(src), size, *varargs)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/numba/cuda/cudadrv/driver.py", line 290, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/rou/anaconda3/envs/cuml/lib/python3.5/site-packages/numba/cuda/cudadrv/driver.py", line 325, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to cuMemcpyHtoD results in CUDA_ERROR_INVALID_VALUE

Finally, upgrading the nvidia driver fixes this:

$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo apt update
$ ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0 ==
modalias : pci:v000010DEd00001D81sv000010DEsd00001218bc03sc00i00
vendor   : NVIDIA Corporation
model    : GV100 [TITAN V]
driver   : nvidia-driver-396 - third-party free
driver   : nvidia-driver-410 - third-party free recommended
driver   : nvidia-driver-390 - third-party free
driver   : xserver-xorg-video-nouveau - distro free builtin
$ sudo ubuntu-drivers autoinstall

[harrism]

I think the action to take for this issue is to add a section on testing, recommending that users run numba -s on the command line in their conda environment, showing what correct output is.

I had a similar issue today where I had a new driver but not properly installed toolkit (or something) and got a different error in tests, and numba -s indeed reported a CUDA misconfiguration even though nvidia-smi reported copasetic.

[kkraus14]

Fixed by #4692