Assigning Deterministic rank to Dask Workers Based on CUDA_VISIBLE_DEVICES

[VibhuJawa]

Previously, dask-raft non-deterministically maps a process to a GPU.

In this PR, we assign a deterministic order to each worker based on the CUDA_VISIBLE_DEVICES environment variable.
as NCCL>1.11 expects a process with rank r to be mapped to r % num_gpus_per_node .

This fixes rapidsai/cugraph#3478
and this raft-test in MNMG setting

raft/python/raft-dask/raft_dask/test/test_comms.py

Lines 82 to 84 in c1a7b7c

	def func_test_device_multicast_sendrecv(sessionId, n_trials):
	handle = local_handle(sessionId, dask_worker=get_worker())
	return perform_test_comms_device_multicast_sendrecv(handle, n_trials)

CC: @jnke2016

[jnke2016]

@VibhuJawa thanks a lot for this PR. It looks good. However, it seems to uncover an issue with batch edge betweenness centrality as it is the only one failing. @ChuckHastings we were thinking about merging this PR as it fixes the nccl failure that we have been debugging and later, find a fix for batch edge betweenness centrality unless you have something else in mind. In the mean time, I have been working on bindings for edge betweenness centrality(with the hope that this failure doesn't occur in the new implementation) as you are implementing it but we have to make a call for this PR as raft is entering code freeze soon.

[ChuckHastings]

@VibhuJawa thanks a lot for this PR. It looks good. However, it seems to uncover an issue with batch edge betweenness centrality as it is the only one failing. @ChuckHastings we were thinking about merging this PR as it fixes the nccl failure that we have been debugging and later, find a fix for batch edge betweenness centrality unless you have something else in mind. In the mean time, I have been working on bindings for edge betweenness centrality(with the hope that this failure doesn't occur in the new implementation) as you are implementing it but we have to make a call for this PR as raft is entering code freeze soon.

Definitely merge this to fix the nccl failure. The batch edge betweenness centrality failure is clearly a separate issue. We would need to resolve that separately anyway.

[cjnolet]

@VibhuJawa can you open a quick cuml PR that pins your branch in get_raft.cmake just so we can make sure this doesn't somehow break cuml?

[VibhuJawa]

@VibhuJawa can you open a quick cuml PR that pins your branch in get_raft.cmake just so we can make sure this doesn't somehow break cuml?

Started PR : https://github.com/rapidsai/cuml/pull/5462/files

[VibhuJawa]

Closing as covered by #1587