[BUG] submodule sync failed with rapids_logger json entry error

[jlowe]

cudf submodule sync job failed with this error:

[2024-12-10T20:00:53.181Z] [INFO] --- maven-antrun-plugin:3.0.0:run (build-libcudf) @ spark-rapids-jni ---
[2024-12-10T20:00:53.181Z] [INFO] Executing tasks
[2024-12-10T20:00:53.181Z] [INFO]     [mkdir] Created dir: /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4070/target/libcudf-install
[2024-12-10T20:00:53.181Z] [INFO]     [mkdir] Created dir: /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4070/target/libcudf/cmake-build
[2024-12-10T20:00:53.181Z] [INFO]      [exec] loading initial cache file /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4070/thirdparty/cudf-pins//setup.cmake
[2024-12-10T20:00:53.181Z] [INFO]      [exec] -- Pinning rapids-cmake SHA1 [7e3122c579b19ccaa3fe670fa6ad8efb8d3f738c]
[2024-12-10T20:00:53.181Z] [INFO]      [exec] -- Found Git: /usr/bin/git (found version "2.43.5") 
[2024-12-10T20:00:53.181Z] [INFO]      [exec] -- Configuring incomplete, errors occurred!
[2024-12-10T20:00:53.181Z] [INFO]      [exec] CMake Error at /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4070/target/libcudf/cmake-build/_deps/rapids-cmake-src/rapids-cmake/cpm/detail/package_details.cmake:72 (message):
[2024-12-10T20:00:53.181Z] [INFO]      [exec]   rapids_cmake can't parse 'rapids_logger' json entry, it is missing a
[2024-12-10T20:00:53.181Z] [INFO]      [exec]   `version` entry
[2024-12-10T20:00:53.181Z] [INFO]      [exec] Call Stack (most recent call first):
[2024-12-10T20:00:53.181Z] [INFO]      [exec]   /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4070/target/libcudf/cmake-build/_deps/rapids-cmake-src/rapids-cmake/cpm/package_override.cmake:119 (rapids_cpm_package_details)
[2024-12-10T20:00:53.181Z] [INFO]      [exec]   /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4070/thirdparty/cudf-pins/add_dependency_pins.cmake:27 (rapids_cpm_package_override)
[2024-12-10T20:00:53.181Z] [INFO]      [exec]   /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4070/thirdparty/cudf-pins/add_dependency_pins.cmake:37 (add_override_if_requested)
[2024-12-10T20:00:53.181Z] [INFO]      [exec]   CMakeLists.txt:27 (project)
[2024-12-10T20:00:53.181Z] [INFO]      [exec] 
[2024-12-10T20:00:53.181Z] [INFO]      [exec] 
[2024-12-10T20:00:53.181Z] [INFO]      [exec] CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
[2024-12-10T20:00:53.181Z] [INFO]      [exec] CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
[2024-12-10T20:00:53.181Z] [INFO]      [exec] CMake Error: CMAKE_CUDA_COMPILER not set, after EnableLanguage

[jlowe]

This appears to be an issue either in rapids-cmake because it doesn't fallback to the git tag when no version is specified or a bug in cudf because it manually adds the rapids_logger package without a specified VERSION. Manually overriding the version in the thirdparty/cudf/versions.json file to match the git tag seems to fix the issue.

[abellina]

Manually overriding the version in the thirdparty/cudf/versions.json file to match the git tag seems to fix the issue.

Where did you find the git tag to use here?

[ttnghia]

@jlowe Can you try reverting rapidsai/cudf#17307 to see if that PR is the source of issue?

Maybe we need to adopt the changes in https://github.com/rapidsai/cudf/pull/17307/files#diff-1bba462ab050e89360fd88110a689e85ee037749cea091a1848ab574381d3795 to fix this issue.

[jlowe]

Where did you find the git tag to use here?

From the new rapids_logger dependency entry in thirdparty/cudf-pins/versions.json file after building with latest dependencies. Building libcudf with latest dependencies works, it's building with pinned dependencies that fails.

Can you try reverting rapidsai/cudf#17307 to see if that PR is the source of issue?

It should be that issue, since that's the PR that added the rapids_logger package dependency. I can try reverting to verify.

Maybe we need to adopt the changes in https://github.com/rapidsai/cudf/pull/17307/files#diff-1bba462ab050e89360fd88110a689e85ee037749cea091a1848ab574381d3795 to fix this issue.

This is a build failure in libcudf, not our spark-rapids-jni native build. The fix needs to be either in rapids-cmake or libcudf.

[pxLi]

submodule sync failed new error with the latest changes #2677

[INFO] No Ant target defined - SKIPPED
...
10:32:07  + validate_status=0
10:32:07  + set -e
10:32:07  ++ mvn -Dmaven.wagon.http.retryHandler.count=3 -B help:evaluate -Dexpression=libcudf.build.path -q -DforceStdout
10:32:11  + LIBCUDF_BUILD_PATH=/home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4077/target/libcudf/cmake-build/
10:32:11  ++ git -C /home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4077/target/libcudf/cmake-build//_deps/rapids-cmake-src/ rev-parse HEAD
10:32:11  fatal: cannot change to '/home/jenkins/agent/workspace/jenkins-spark-rapids-jni_submodule-sync-dev-4077/target/libcudf/cmake-build//_deps/rapids-cmake-src/': No such file or directory
10:32:11  + rapids_cmake_sha=
10:32:11  [Pipeline] }

will make a PR to do a fix. merged #2684

[ttnghia]

Close as this was fixed.