Warn if CUDA context is created on incorrect device with LocalCUDACluster

[pentschev]

Warns if for some reason the creation of CUDA context has already happened or occurs on the incorrect device in LocalCUDACluster. This can be a problem if something initializes the CUDA runtime library too early.

Because these things are related to the global Python context, it's difficult to test it with pytest. But below is a reproducer for both warnings:

gpu_assignment.py

import create_context  # `create_context.py` file on running directory -- Triggers CUDA context has already been created

from dask.distributed import Client
from dask_cuda import LocalCUDACluster
from numba import cuda

cuda_ver = cuda.runtime.get_version()  # Doesn't create a context, but causes all workers to create context on device 0 when `numba.cuda.current_context()` is called


if __name__ == '__main__':
    cluster = LocalCUDACluster()
    client = Client(cluster)

create_context.py

from numba import cuda
cuda.current_context()

Fixes #384

[pentschev]

rerun tests

[pentschev]

cc @cjnolet @rlratzel

[codecov-commenter]

Codecov Report

Merging #719 (bd6a63f) into branch-21.10 (8e6ab70) will increase coverage by 1.46%.
The diff coverage is 88.01%.

❗ Current head bd6a63f differs from pull request most recent head 9aaefc9. Consider uploading reports for the commit 9aaefc9 to get more accurate results

@@               Coverage Diff                @@
##           branch-21.10     #719      +/-   ##
================================================
+ Coverage         87.63%   89.10%   +1.46%     
================================================
  Files                15       15              
  Lines              1658     1734      +76     
================================================
+ Hits               1453     1545      +92     
+ Misses              205      189      -16     

Impacted Files	Coverage Δ
dask_cuda/cuda_worker.py	77.64% <ø> (ø)
dask_cuda/get_device_memory_objects.py	90.00% <0.00%> (+21.94%)	⬆️
dask_cuda/local_cuda_cluster.py	77.88% <50.00%> (ø)
dask_cuda/utils.py	84.79% <65.95%> (-2.49%)	⬇️
dask_cuda/proxify_device_objects.py	95.45% <80.00%> (+6.56%)	⬆️
dask_cuda/initialize.py	94.28% <88.88%> (+5.39%)	⬆️
dask_cuda/proxify_host_file.py	93.46% <92.15%> (-5.94%)	⬇️
dask_cuda/proxy_object.py	90.59% <98.11%> (+0.94%)	⬆️
dask_cuda/device_host_file.py	71.66% <100.00%> (+1.50%)	⬆️
dask_cuda/explicit_comms/dataframe/shuffle.py	98.69% <100.00%> (+0.65%)	⬆️
... and 10 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 33e5d3e...9aaefc9. Read the comment docs.

[pentschev]

rerun tests

[madsbk]

LGTM (only a minor typo), thanks @pentschev

[jakirkham]

@gpucibot merge

[jakirkham]

Thanks Peter for the PR and Mads for the review! 😄