Use TrackingResourceAdaptor to get better debug info

[madsbk]

For better out of memory message, JIT-unspill now check the current RMM resource stack for resources such as StatisticsResourceAdaptor and TrackingResourceAdaptor that can report the current allocated bytes.

Enable by running dask-cuda-worker with --rmm-track-allocations=True or calling dask_cuda.LocalCUDACluster with rmm_track_allocations=True.

This is very useful for debugging RMM fragmentation.

[codecov-commenter]

Codecov Report

Base: 87.17% // Head: 63.31% // Decreases project coverage by -23.85% ⚠️

Coverage data is based on head (e3e3a4e) compared to base (bdb7b56).
Patch coverage: 84.21% of modified lines in pull request are covered.

Additional details and impacted files

@@                Coverage Diff                @@
##           branch-23.02    #1079       +/-   ##
=================================================
- Coverage         87.17%   63.31%   -23.86%     
=================================================
  Files                18       26        +8     
  Lines              2253     3127      +874     
=================================================
+ Hits               1964     1980       +16     
- Misses              289     1147      +858     

Impacted Files	Coverage Δ
dask_cuda/local_cuda_cluster.py	89.01% <ø> (ø)
dask_cuda/proxify_host_file.py	93.15% <84.21%> (-1.58%)	⬇️
dask_cuda/benchmarks/local_cudf_merge.py	0.00% <0.00%> (ø)
dask_cuda/benchmarks/local_cudf_shuffle.py	0.00% <0.00%> (ø)
dask_cuda/benchmarks/local_cupy_map_overlap.py	0.00% <0.00%> (ø)
dask_cuda/benchmarks/common.py	0.00% <0.00%> (ø)
dask_cuda/benchmarks/local_cupy.py	0.00% <0.00%> (ø)
dask_cuda/benchmarks/local_cudf_groupby.py	0.00% <0.00%> (ø)
dask_cuda/benchmarks/utils.py	0.00% <0.00%> (ø)
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[pentschev]

This looks really nice @madsbk , I can see this being very useful in the long-term. I've left a few minor change requests.

[pentschev]

Suggested change

	StatisticsResourceAdaptor and TrackingResourceAdaptor that
	can report the current allocated bytes. Returns None, if
	``StatisticsResourceAdaptor`` and ``TrackingResourceAdaptor`` that
	can report the current allocated bytes. Returns ``None``, if

[madsbk]

I think we should only use a single ` for function, variables etc.
https://numpydoc.readthedocs.io/en/latest/format.html#common-rest-concepts

[pentschev]

TBH, I'm not 100% confident I know what's the difference from a single ` to double ``, IIRC the change to use double was started by @charlesbluca when he was working on RTD, could you remind us why that change was made?

In any case, I won't block this PR for this right now, if needed be we can address this on a follow-up PR.

[pentschev]

LGTM now, thanks @madsbk !

[pentschev]

/merge