Implement DeviceHostFile.evict()
#2
Conversation
| try: | ||
| _, _, weight = self.host_buffer.fast.evict() | ||
| return weight | ||
| except Exception: # We catch all `Exception`s, just like zict.LRU |
There was a problem hiding this comment.
I would strongly advise to log it. Otherwise your users will have no clue why they're running out of memory.
There was a problem hiding this comment.
Currently we can't log appropriately, we may consider this in the future. Could you point me to what Dask currently logs? I'm not sure we even intended to log it this way but could you point to where Dask is logging this currently?
There was a problem hiding this comment.
https://github.com/dask/distributed/blob/main/distributed/spill.py
In SpillBuffer.evict()
https://github.com/dask/distributed/blob/7a69b5e24c115245005cb83129b4b4b46f05f151/distributed/spill.py#L168-L180
and its context manager SpillBuffer.handle_errors()
https://github.com/dask/distributed/blob/7a69b5e24c115245005cb83129b4b4b46f05f151/distributed/spill.py#L81-L134
There was a problem hiding this comment.
Thanks @crusaderky for the pointers, for now I opened rapidsai#873 so we can consider doing the same, we'll also have to think how to add proper logging in Dask-CUDA without just hacking into Distributed's.
|
Based on an offline chat, I'm going to go ahead and merge this here and we'll address the error logging elsewhere. Thank you, @pentschev and @crusaderky! |
No description provided.