Crash in rmm async allocator in test_java_url_decode

[nvdbaranec]

Build failure in CICD. Likely upstream of the async allocator itself.

https://prod.blsm.nvidia.com/sw-gpu-spark-jenkins/job/examples-udf-examples-native/121/

src/main/python/rapids_udf_test.py::test_java_url_decode

[2024-01-22T09:44:17.989Z] 24/01/22 09:44:17 ERROR RapidsExecutorPlugin: Stopping the Executor based on exception being a fatal
 CUDA error: ai.rapids.cudf.CudaFatalException: std::bad_alloc: CUDA error at: /home/jenkins/agent/workspace/jenkins-spark-rapids-
jni_nightly-dev-652-cuda11/thirdparty/cudf/cpp/build/_deps/rmm-src/include/rmm/mr/device
/cuda_async_view_memory_resource.hpp:128: cudaErrorIllegalAddress an illegal memory access was encountered

[jlowe]

@GaryShen2008 at first I thought this was related to #347, but then upon digging I discovered that's not in the build that failed. #347 accidentally was committed to main instead of branch-24.02. 🤦

I'll put up reverts to main and re-PR to branch-24.02 to fix that mistake, but the failure noted in this issue may be related to the ccache change in #345, as that's the only change listed between the good run and the failing run.

[GaryShen2008]

Not reproduced in local. Trigger CI again.

[GaryShen2008]

@NvTimLiu CI can reproduced the issue.
Please help to figure it out.

[NvTimLiu]

checking

[NvTimLiu]

branch-24.04 UDF native examples app with native CUDF is still building to branch-23.12,

https://github.com/NVIDIA/spark-rapids-examples/blob/branch-24.04/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/cpp/CMakeLists.txt#L87-L90

This caused the failure due to latest branch-24.04 rapidsai cuDF/rmm changes not built into UDF native examples jar file, e.g. : rapidsai/rmm#1437

[NvTimLiu]

Close as fix #358 #359 got merged