Passing rmm_pool_size larger than memory will not create a pool

[pentschev]

During some conversations offline, I noticed passing rmm_pool_size that's larger than the total GPU memory will succeed, although no pool gets created or warnings get thrown. A simple reproducer is below:

from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster(rmm_pool_size="100GB", silence_logs=False)

By checking nvidia-smi, it's clear that the cluster gets created but without a memory pool.

[pentschev]

This should be fixed by dask/distributed#4297 .

[github-actions]

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

[quasiben]

This is still an issue

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[pentschev]

With dask/distributed#4297 merged in, this is now fixed:

$ dask-cuda-worker tcp://localhost:8786 --rmm-pool-size 100GB
...
distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:~/miniconda3/envs/test_dev/include/rmm/mr/device/pool_memory_resource.hpp:183: Maximum pool size exceeded
Traceback (most recent call last):
  File "~/miniconda3/envs/test_dev/lib/python3.8/site-packages/distributed/utils.py", line 681, in log_errors
    yield
  File "~/miniconda3/envs/test_dev/lib/python3.8/site-packages/distributed/worker.py", line 3283, in plugin_add
    result = plugin.setup(worker=self)
  File "~/miniconda3/envs/test_dev/lib/python3.8/site-packages/dask_cuda/utils.py", line 85, in setup
    rmm.reinitialize(
  File "~/miniconda3/envs/test_dev/lib/python3.8/site-packages/rmm/rmm.py", line 84, in reinitialize
    rmm.mr._initialize(
  File "rmm/_lib/memory_resource.pyx", line 672, in rmm._lib.memory_resource._initialize
  File "rmm/_lib/memory_resource.pyx", line 732, in rmm._lib.memory_resource._initialize
  File "rmm/_lib/memory_resource.pyx", line 270, in rmm._lib.memory_resource.PoolMemoryResource.__cinit__
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:~/miniconda3/envs/test_dev/include/rmm/mr/device/pool_memory_resource.hpp:183: Maximum pool size exceeded