[BUG] Build is broken, possibly with CCCL 2.2 changes upstream

[mythrocks]

The spark-rapids-jni build seems to be borked currently. A fresh checkout seems to produce the following error:

[INFO]      [exec] -- Could NOT find libcudacxx (missing: libcudacxx_DIR)
[INFO]      [exec] -- Configuring incomplete, errors occurred!
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24.617s
[INFO] Finished at: Tue Dec 19 23:05:26 UTC 2023
[INFO] Final Memory: 30M/1924M
[INFO] ------------------------------------------------------------------------

It appears that the error is related to upstream changes (in libcudf and rmm) to support CCCL 2.2. From the build log:

[INFO]      [exec] -- CPM: rmm: Adding package [email protected] (v2.2.0)
[INFO]      [exec] CMake Error at build/_deps/cccl-src/thrust/thrust/cmake/thrust-config.cmake:708 (find_package):
[INFO]      [exec]   Could not find a configuration file for package "libcudacxx" that is
[INFO]      [exec]   compatible with requested version "2.2.0.0".
[INFO]      [exec] 
[INFO]      [exec]   The following configuration files were considered but not accepted:
[INFO]      [exec] 
[INFO]      [exec]     /usr/local/cuda/lib64/cmake/libcudacxx/libcudacxx-config.cmake, version: 1.8.1.0
[INFO]      [exec] 
[INFO]      [exec] Call Stack (most recent call first):
[INFO]      [exec]   build/_deps/cccl-src/lib/cmake/cccl/cccl-config.cmake:65 (find_package)
[INFO]      [exec]   build/_deps/cccl-src/cmake/CCCLAddSubdir.cmake:1 (find_package)
[INFO]      [exec]   build/_deps/cccl-src/CMakeLists.txt:21 (include)
[INFO]      [exec] 
[INFO]      [exec] 
[INFO]      [exec] -- Could NOT find libcudacxx (missing: libcudacxx_DIR)
[INFO]      [exec] -- Configuring incomplete, errors occurred!

It appears that rmm is pulling in [email protected]. libcudacxx is then unable to be found.

[mythrocks]

Note: There have been changes underway in libcudf to support CCCL 2.2. While rapidsai/cudf#14576 has been merged in libcudf, the cudf submodule in spark-rapids-jni remains at a prior commit (dadc7a0).

But the rmm dependency does appear to include the [email protected] change.

[mythrocks]

Note, also, that the spark-rapids-jni changes to support CCCL 2.2 are being worked on here: #1662.

We have a couple of options:

1. Change the rmm version that spark-rapids-jni depends on.
2. Bite the bullet, and complete Use cuda::proclaim_return_type on device lambdas #1662 to switch over to CCCL 2.2.

[bdice]

Thanks @mythrocks. I agree there are two choices - pin both cudf and rmm (and potentially rapids-cmake/cuCollections), or complete the work to adopt CCCL 2.2.0.

Notes on pinning:

• The last RMM commit before CCCL 2.2.0 is rapidsai/rmm@77b5500
• You may also need to pin rapids-cmake and/or cuCollections, so that the versions.json points to a build of cuCollections before CCCL 2.2.0.
  • The last commit of rapids-cmake without the updated cuCollections is rapidsai/rapids-cmake@ef3388f.
  • The pinning of cuCollections RAPIDS used before updating to CCCL 2.2.0 is NVIDIA/cuCollections@7c76a12.

[ttnghia]

Close this as it is resolved by #1662.