Add a dashboard component that displays RMM memory #5740
Conversation
|
Thanks @shwina this will be very useful when trying to understand memory consumption with RAPIDS workloads. cc @charlesbluca & @rjzamora |
|
CC: @ayushdg , This will be useful when deciding on the choice of 32GB vs 40 GB vs 80 GB for gpu-bdb workflows . |
Unit Test Results 12 files ± 0 12 suites ±0 5h 57m 7s ⏱️ + 2m 32s For more details on these failures, see this check. Results for commit 34a3cc7. ± Comparison against base commit 1b4d13f. ♻️ This comment has been updated with latest results. |
Adds the `rmm_track_allocations` option that enables workers to query the amount of RMM memory allocated at any time via `mr.get_allocated_bytes()`. This is used in dask/distributed#5740. Authors: - Ashwin Srinath (https://github.com/shwina) - Charles Blackmon-Luca (https://github.com/charlesbluca) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #842
There was a problem hiding this comment.
This looks great!
I noticed that RMM doesn't get capitalised correctly in the UI. We special case CPU and GPU so we should update that list.
I'm also not super confident on the colours. Purple for RMM makes sense, but it vibrates a bit with the green. I wonder if we would either reduce the opacity of the green or switch to a different purple. In the worker memory plot we use three opacities of the same blue. It would be good to try and stay consistent.
|
Just wanted to bump this @shwina . What do you think about my comment on the colours?
|
|
Hi @jacobtomlinson -- thanks for the ping and sorry to let this PR languish. I'll bring it back to speed and go with "varying opacities of green", if that sounds alright to you. |
Tracking RMM allocation will be useful together with dask/distributed#5740 , and will help with the analysis of memory fragmentation when comparing regular pool and the async memory allocator. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Benjamin Zaitlen (https://github.com/quasiben) URL: #1145
pre-commit run --all-filesThis PR adds a visualization that provides a bit more information than the current GPU memory utilization plot. It also shows the amount of GPU memory being managed by RMM, and in the case of a RMM pool, the amount of pool memory actually used:
The purple region of the bar represents the total amount of memory managed by RMM. Dark purple shows the amount of memory actively being utilized (this is relevant when a memory pool is used). The green region represents GPU memory used outside of RMM.