Address CI failures caused by upstream distributed and cupy changes

[rjzamora]

After dask/distributed#7028, the "distributed.comm.ucx"-specific config options being passed down to the LocalCluser super class are no longer merged with the global options in dask.config.config on the worker. This means that the workers only inherit the "distributed.comm.ucx"-specific options.

This PR explicitly merges the "distributed.comm.ucx" options with dask.config.config within LocalCUDACluster. Without this change, CI will fail.

The original purpose of this PR has been resolved upstream (see: dask/distributed#7069). This PR now only modifies the tests that are still failing in CI.

[pentschev]

This looks good to me. There are some new failures which don't seem related. I'm still unable to reproduce them locally though, I'll see if I can reproduce it on Monday but please keep me posted if you happen to find something in the meantime.

[rjzamora]

This looks good to me. There are some new failures which don't seem related. I'm still unable to reproduce them locally though, I'll see if I can reproduce it on Monday but please keep me posted if you happen to find something in the meantime.

Unfortunately, the failure is indeed related. It turns out that, since we are merging in the global config dictionary in the LocalCUDACluster initializer, we do not pickl up the proper option for the "DASK_DISTRIBUTED__COMM__COMPRESSION" environment variable. It is not yet clear to me if there is a way to fix this.

The good news is that d893881 demonstrates that using dask.config now works fine for disabling compression.

[rjzamora]

The good news is that the config approach seems to work fine. The bad news is that this env approach does not :/

Not yet sure if there is a clean way to support both.

[wence-]

My reading of dask/distributed#7028 is that what we were doing with config was correct, but that there is a bug in the way the localcluster merges config options (it should do dask.config["distributed"] | config), so that most of this should not be needed once that is fixed?

[rjzamora]

My reading of dask/distributed#7028 is that what we were doing with config was correct, but that there is a bug in the way the localcluster merges config options (it should do dask.config["distributed"] | config), so that most of this should not be needed once that is fixed?

Yes, exactly - Leaving this as "draft" with the hope that the fix can be upstream (and then we can close this).

[rjzamora]

It looks like dask/distributed#7069 resolved the issue of global config options not being passed through to the worker. However, we will still see failures in dask_cuda/tests/test_spill.py::test_cudf_cluster_device_spill, because the dask.config defaults will already be set before the environment variable is patched. I submitted dask/distributed#7070 to document this behavior, but I'm not sure it should be considered a real "bug".

[pentschev]

@rjzamora thanks for pushing on this. However, the errors are different this time, they're in dask_cuda.tests.test_proxy.test_cupy_matmul (and imatmul). I briefly discussed this with @madsbk earlier, and this happens on CuPy >= 11, probably because the array access has changed. Since JIT-Unspill and CuPy don't currently work without compatibility mode, Mads suggested we disable this test. Could mark it to xfail for now?

[rjzamora]

Could mark it to xfail for now?

Thanks for explaining that @pentschev - I marked those tests as xfail for now (for cupy>=11.0)

[pentschev]

Thanks @rjzamora for fixing this!

[pentschev]

@gpucibot merge