Update test_spill to use cuDF's memory_usage()

[pentschev]

With the cuDF move from using __sizeof__() to memory_usage() we need to update spilling tests to check against the new method. Requires rapidsai/cudf#9544 .

Together with this change we fix #79 too.

[pentschev]

rerun tests

[madsbk]

LGTM, thanks @pentschev

[pentschev]

rerun tests

[pentschev]

rerun tests

[pentschev]

rerun tests

[pentschev]

Tests are still failing because gpuCI isn't picking latest dask-cudf packages, waiting for that to be resolved.

[pentschev]

rerun tests

[quasiben]

rerun tests

[pentschev]

I wasted far too much time figuring this out. To summarize, the problem is gpuCI installs python-snappy which isn't installed by other means, such as when following official RAPIDS Conda Install Instructions. When python-snappy is installed, it gets automatically used by Dask for compressing files to disk, as distributed.comm.compression="auto" by default, which then breaks checking for spilled size. We should be able to disable it via dask.config.set or dask.config.update, but for whatever reason neither is respected and compression happens even when disabling it. The only way I found that actually is respected is disabling via environment variables, which is what I've done now.

If anybody knows why this happens or has ideas to fix, please let me know, otherwise I'll merge it as is in 24 hours provided tests pass.

[pentschev]

Actually, this is supposed to be merged before the release, so it has to be merged by EOD today.

[madsbk]

I wasted far too much time figuring this out. To summarize, the problem is gpuCI installs python-snappy which isn't installed by other means, such as when following official RAPIDS Conda Install Instructions. When python-snappy is installed, it gets automatically used by Dask for compressing files to disk, as distributed.comm.compression="auto" by default, which then breaks checking for spilled size. We should be able to disable it via dask.config.set or dask.config.update, but for whatever reason neither is respected and compression happens even when disabling it. The only way I found that actually is respected is disabling via environment variables, which is what I've done now.

If anybody knows why this happens or has ideas to fix, please let me know, otherwise I'll merge it as is in 24 hours provided tests pass.

Great debugging work @pentschev, thanks!