[FEA] Add synchronize and retry before OOM with async allocator

[abellina]

In a dev build, I added a Cuda.deviceSynchronize when we are unable to handle a failed allocation (we can't spill anymore) in DeviceMemoryEventHandler.onAllocFailure and I had a customer query pass consistently after doing this.

This is one such example that shows 15GB allocated, and us unable to allocate 2.6GB in a 40GB pool:

Device store exhausted, unable to allocate 2570310640 bytes. Total RMM allocated is 15015998208 bytes

After the synchronize, a retry succeeded. This could be related to internals of how the ASYNC allocator works or it could also be partly due to how the limiting memory resource and the async memory resource (which is wrapped by the limiting memory resource) are not in sync. It is worth noting that I tried a similar with with ARENA, I still run OOM, but a synchronize doesn't seem to be helping in that case.

So, I believe there is value in making a change either in the plugin or in RmmJni.cpp.

Essentially what I think we want is:

1. Try to handle the failure via spilling the regular way (this could be also tweaked, see below).
2. When we can't spill anymore (https://github.com/NVIDIA/spark-rapids/blob/branch-22.12/sql-plugin/src/main/scala/com/nvidia/spark/rapids/DeviceMemoryEventHandler.scala#L52):
   a. If we haven't retried this allocation more than N times (some sort of limit, or time based limit), Cuda.deviceSynchronize and then allow the calling code to retry.
   b. If we have gone above our retry limits, we can let the executor fail as before.

We'll need to reset the limit if the job starts succeeding again without help. That said, there could be many threads failing and spilling and several concurrent retries. So I think one approach may be a time based retry. If the last time we failed for any thread is more than say 1 or 2 seconds, we can retry again. Otherwise, we know we retried and we are still retrying, it is probably best to be done. Perhaps we can keep a retry counter per task thread, that seems like an option as well. I'd like to hear input on preferences to the above.

I believe we should experiment with synchronize before we try to spill to see if (1) above could be prevented in some cases. If we are seeing fragmentation that can be handled via a sync, it's probably cheaper to sync than to spill in some scenarios.