[BUG] Spilling logic can spill data that cannot be freed

[jlowe]

The spill code in RapidsBufferStore will spill data from a buffer that is currently not spillable. This can be very wasteful when the thread that has locked the buffer is busy blocked on an allocate call that is causing spilling to occur. Such buffers cannot be freed to satisfy the allocation, and spilling them in the hopes the buffer can be relinquished later can be a complete waste of time and I/O bandwidth if the buffer is never unlocked.

Rather than spilling all buffers, the code should track spillable (i.e.: not currently acquired) buffers and only spill those. The waiting logic for pending free buffers should be changed to waiting logic for buffers to be released (and thus spilled only at that point).

[abellina]

Related issue: #6561, since in that one we want to make sure the spill framework is told when the underlying buffer is actually freed. For consideration when working on this specific issue.

[abellina]

This is the cuDF side of this change: rapidsai/cudf#12125. We have decided that a callback is all we need from the memory buffer, where we would make something spillable once the memory buffer reaches refCount=1 (e.g. it is held only by the spillable framework).