[BUG] some RAPIDS repos are incompatible with CPM 0.40.2 #680

jameslamb · 2024-08-20T20:34:35Z

Describe the bug

When we tried upgrading to CPM 0.40.2 in #678, found that some RAPIDS repos had build failures.

Opening this to track the work of investigating that.

Steps/Code to reproduce bug

On latest cudf 24.10 (rapidsai/cudf@e450baf), from the root of the cudf repo, and with the changes from #678 applied.

docker run \
   --rm \
   -v $(pwd):/usr/local/src \
   -w /usr/local/src \
   -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.11 \
   ci/configure_cpp_static.sh

Those configure tests failed like this:

-- Using  approach to find dependencies
CMake Error at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:449 (file):
  file STRINGS file
  "/usr/local/src/build_static/_deps/arrow-src/thirdparty/versions.txt"
  cannot be read.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)

And later on:

-- Building xsimd from source
CMake Error at /pyenv/versions/3.11.9/lib/python3.11/site-packages/cmake/data/share/cmake-3.30/Modules/ExternalProject/shared_internal_commands.cmake:1099 (message):
  URL_HASH is set to

    SHA256=

  but must be ALGO=value where ALGO is

    MD5|SHA1|SHA224|SHA256|SHA384|SHA512|SHA3_224|SHA3_256|SHA3_384|SHA3_512

  and value is a hex string.
Call Stack (most recent call first):
  /pyenv/versions/3.11.9/lib/python3.11/site-packages/cmake/data/share/cmake-3.30/Modules/ExternalProject.cmake:3035 (_ep_add_download_command)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2465 (externalproject_add)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:226 (build_xsimd)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:304 (build_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2491 (resolve_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)

full logs (click me)


==========
== CUDA ==
==========

CUDA Version 12.5.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .


RAPIDS logger » [08/20/24 20:31:11]
┌──────────────────────────────────┐
|    Configure static cpp build    |
└──────────────────────────────────┘

# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`.
cmake>=3.26.4,!=3.30.0
ninja

Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/rapidsai-wheels-nightly/simple, https://pypi.nvidia.com
Collecting cmake!=3.30.0,>=3.26.4 (from -r /tmp/tmp.IfrYPE84N0/requirements.txt (line 3))
  Downloading cmake-3.30.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB)
Collecting ninja (from -r /tmp/tmp.IfrYPE84N0/requirements.txt (line 4))
  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Downloading cmake-3.30.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.9/26.9 MB 80.0 MB/s eta 0:00:00
Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.2/307.2 kB 57.2 MB/s eta 0:00:00
Installing collected packages: ninja, cmake
Successfully installed cmake-3.30.2 ninja-1.11.1.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip
-- CPM: Adding package [email protected] (jitify2)
-- Populating jitify
-- Configuring done (0.1s)
-- Generating done (0.1s)
-- Build files have been written to: /usr/local/src/build_static/_deps/jitify-subbuild
[0/7] Performing update step for 'jitify-populate'
-- Fetching latest from the remote origin
[1/7] No patch step for 'jitify-populate'
[3/7] No configure step for 'jitify-populate'
[4/7] No build step for 'jitify-populate'
[5/7] No install step for 'jitify-populate'
[6/7] No test step for 'jitify-populate'
[7/7] Completed 'jitify-populate'
-- CPM: Adding package [email protected] (v3.1.0)
-- CPM: Using local package [email protected]
-- CPM: Adding package [email protected] (e21d607157218540cd7c45461213fb96adf720b7)
-- rapids-cmake [CCCL]: applied diff cccl_symbol_visibility.diff to fix issue: 'Correct symbol visibility issues in libcudacxx [https://github.com/NVIDIA/cccl/pull/1832/]'
-- rapids-cmake [CCCL]: applied diff thrust_disable_64bit_dispatching.diff to fix issue: 'Remove 64bit dispatching as not needed by libcudf and results in compiling twice as many kernels [https://github.com/rapidsai/cudf/pull/11437]'
-- rapids-cmake [CCCL]: applied diff thrust_faster_sort_compile_times.diff to fix issue: 'Improve Thrust sort compile times by not unrolling loops for inlined comparators [https://github.com/rapidsai/cudf/pull/10577]'
-- rapids-cmake [CCCL]: applied diff thrust_faster_scan_compile_times.diff to fix issue: 'Improve Thrust scan compile times by reducing the number of kernels generated [https://github.com/rapidsai/cudf/pull/8183]'
-- CPM: Adding package [email protected] (branch-24.10)
-- RMM: RMM_LOGGING_LEVEL = 'INFO'
-- CPM: Adding package [email protected] (10.1.1)
-- Version: 10.1.1
-- Build type: Release
-- rapids-cmake [fmt]: applied diff fix_10_1_1_version.diff to fix issue: 'fmt 10.1.1 produces a CMake package with version 10.1.0'
-- CPM: Adding package [email protected] (v1.12.0)
-- Build spdlog: 1.12.0
-- Build type: Release
-- Generating install
-- rapids-cmake [spdlog]: applied diff nvcc_constexpr_fix.diff to fix issue: 'Fix constexpr mismatch between spdlog and fmt [https://github.com/gabime/spdlog/issues/2856]'
-- CPM: Adding package [email protected] (apache-arrow-16.1.0)
-- Building using CMake version: 3.30.2
-- Arrow version: 16.1.0 (full: '16.1.0')
-- Arrow SO version: 1601 (full: 1601.0.0)
-- clang-tidy 14 not found
-- clang-format 14 not found
-- Could NOT find ClangTools (missing: CLANG_FORMAT_BIN CLANG_TIDY_BIN) 
-- infer not found
-- Found cpplint executable at CPPLINT_BIN-NOTFOUND
-- System processor: x86_64
-- Arrow build warning level: PRODUCTION
-- Using ld linker
-- Build Type: RELEASE
-- Using  approach to find dependencies
CMake Error at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:449 (file):
  file STRINGS file
  "/usr/local/src/build_static/_deps/arrow-src/thirdparty/versions.txt"
  cannot be read.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


CMake Error at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:604 (string):
  string sub-command REPLACE requires at least four arguments.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


CMake Error at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:764 (string):
  string sub-command SUBSTRING requires four arguments.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


-- Building without OpenSSL support. Minimum OpenSSL version 1.0.2 required.
CMake Warning at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:291 (find_package):
  By not providing "Findxsimd.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "xsimd", but
  CMake did not find one.

  Could not find a package configuration file provided by "xsimd" with any of
  the following names:

    xsimdConfig.cmake
    xsimd-config.cmake

  Add the installation prefix of "xsimd" to CMAKE_PREFIX_PATH or set
  "xsimd_DIR" to a directory containing one of the above files.  If "xsimd"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2491 (resolve_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


-- Building xsimd from source
CMake Error at /pyenv/versions/3.11.9/lib/python3.11/site-packages/cmake/data/share/cmake-3.30/Modules/ExternalProject/shared_internal_commands.cmake:1099 (message):
  URL_HASH is set to

    SHA256=

  but must be ALGO=value where ALGO is

    MD5|SHA1|SHA224|SHA256|SHA384|SHA512|SHA3_224|SHA3_256|SHA3_384|SHA3_512

  and value is a hex string.
Call Stack (most recent call first):
  /pyenv/versions/3.11.9/lib/python3.11/site-packages/cmake/data/share/cmake-3.30/Modules/ExternalProject.cmake:3035 (_ep_add_download_command)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2465 (externalproject_add)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:226 (build_xsimd)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:304 (build_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2491 (resolve_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


-- Configuring incomplete, errors occurred!

Pinning rapids-cmake back to 0e65a84 resolves those failures.

diff --git a/rapids_config.cmake b/rapids_config.cmake
index 3a88769f6e..60aef012ff 100644
--- a/rapids_config.cmake
+++ b/rapids_config.cmake
@@ -26,11 +26,14 @@ else()
   )
 endif()
 
-if(NOT EXISTS "${CMAKE_CURRENT_BINARY_DIR}/CUDF_RAPIDS-${RAPIDS_VERSION_MAJOR_MINOR}.cmake")
-  file(
-    DOWNLOAD
-    "[https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-${RAPIDS_VERSION_MAJOR_MINOR}/RAPIDS.cmake](https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-$%7BRAPIDS_VERSION_MAJOR_MINOR%7D/RAPIDS.cmake)"
-    "${CMAKE_CURRENT_BINARY_DIR}/CUDF_RAPIDS-${RAPIDS_VERSION_MAJOR_MINOR}.cmake"
-  )
-endif()
+include(FetchContent)
+# GIT_TAG = last commit prior to https://github.com/rapidsai/rapids-cmake/pull/678
+FetchContent_Declare(
+  rapids-cmake
+  GIT_REPOSITORY https://github.com/rapidsai/rapids-cmake.git
+  GIT_TAG        0e65a8481b9895b5075d2b601690540a21b45deb
+)
+file(DOWNLOAD https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-24.10/RAPIDS.cmake
+  ${CMAKE_CURRENT_BINARY_DIR}/CUDF_RAPIDS-${RAPIDS_VERSION_MAJOR_MINOR}.cmake)
+

Expected behavior

N/A

Environment details (please complete the following information):

CMake

Additional context

N/A

The text was updated successfully, but these errors were encountered:

…678)" (#679) Proposes reverting #678. See #680 for context. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #679

robertmaynard · 2024-08-22T17:06:07Z

Given that the computed path for the STRINGS read is "/usr/local/src/build_static/_deps/arrow-src/thirdparty/versions.txt" we can infer that the arrow_SOURCE_DIR variable isn't being computed properly. Which indicates that the SOURCE_SUBDIR argument to CPMFindPackage is being ignored when computing the value.

Will continue to dig into this.

robertmaynard · 2024-08-22T18:05:20Z

Minimal reproducer: https://github.com/robertmaynard/cpm_40.2_repro

Going to look at necessary CPM changes next.

robertmaynard · 2024-08-22T19:39:55Z

~~Hole goes deeper. Looks to not be an issue with CPM but a CMake FetchContent bug~~

robertmaynard · 2024-08-23T15:03:37Z

Reproduced to a FetchContent + Policy126 bug in CMake.

Upstream PR: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9770

robertmaynard · 2024-08-29T17:13:43Z

Upstream PR was revised to: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9779

Which has been merged into the release branch, and will be part of CMake 3.30.3

…712) The `rapids_export_write_language` function would generate CMake code that would try and do `set(var PARENT_SCOPE)` in the root directory. This would cause CMake developer warnings to appear. This resolves the issues by adding root level guards around the `set` commands, and setting up each rapids-cmake test to have developer warnings as errors. Due to #680 we mark some tests with `NO_DEV_ERRORS` so that they continue to pass while we wait to 2025 to bump the minimum required CMake version. Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Bradley Dice (https://github.com/bdice) URL: #712

jameslamb added bug Something isn't working ? - Needs Triage Need team to review and classify labels Aug 20, 2024

jameslamb mentioned this issue Aug 20, 2024

Revert "Update to CPM 0.40.2 to fix CMake 3.30 deprecation warnings (#678)" #679

Merged

7 tasks

robertmaynard self-assigned this Aug 22, 2024

robertmaynard closed this as completed Aug 29, 2024

robertmaynard reopened this Aug 29, 2024

robertmaynard mentioned this issue Nov 4, 2024

rapids_export_write_language no longer generates cmake dev warnings #712

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] some RAPIDS repos are incompatible with CPM 0.40.2 #680

[BUG] some RAPIDS repos are incompatible with CPM 0.40.2 #680

jameslamb commented Aug 20, 2024 •

edited

Loading

robertmaynard commented Aug 22, 2024

robertmaynard commented Aug 22, 2024

robertmaynard commented Aug 22, 2024 •

edited

Loading

robertmaynard commented Aug 23, 2024

robertmaynard commented Aug 29, 2024

[BUG] some RAPIDS repos are incompatible with CPM 0.40.2 #680

[BUG] some RAPIDS repos are incompatible with CPM 0.40.2 #680

Comments

jameslamb commented Aug 20, 2024 • edited Loading

robertmaynard commented Aug 22, 2024

robertmaynard commented Aug 22, 2024

robertmaynard commented Aug 22, 2024 • edited Loading

robertmaynard commented Aug 23, 2024

robertmaynard commented Aug 29, 2024

jameslamb commented Aug 20, 2024 •

edited

Loading

robertmaynard commented Aug 22, 2024 •

edited

Loading