DEBUG log messages imply UCX config options are invalid

[rlratzel]

When debugging is enabled, distributed outputs the following messages:

distributed.comm.ucx - DEBUG - Key: cuda_copy with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: tcp with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: nvlink with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: infiniband with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: rdmacm with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: net-devices with value: None not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: reuse-endpoints with value: None not a valid UCX configuration option.

When the corresponding env vars as set, such as export DASK_UCX__CUDA_COPY=True, the code in ucx.py here seems to behave correctly, but the check here still fails and prints the debug messages.

This can be seen simply by setting:

export DASK_LOGGING__DISTRIBUTED="DEBUG"

and running the scheduler like so:

python -m distributed.cli.dask_scheduler

Since the default values even cause the debug messages to be printed, there's no need to set the UCX config options to see the debug messages above.

@quasiben @pentschev

[pentschev]

Thanks @rlratzel , I think this was just a bug in how we were checking variables but wasn't harmful, that should now be addressed in #4850 .