Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround to apply cudf's thirdparty patches for cudf-pins build [skip ci] #2588

Merged
merged 1 commit into from
Nov 14, 2024

Conversation

pxLi
Copy link
Collaborator

@pxLi pxLi commented Nov 13, 2024

workaround of #2582

This one is trying to update submodule sync to support applying patches,
we should expect rapids-cmake could provide a better way to deal with patches when using pinned versions later

The change would update the ${current_json_dir} to also include the relative path in the generated versions.json, this has been verified internally as submodule sync script would not be covered in pre-merge CI.

...
[2024-11-13T01:29:46.349Z] + echo 'Workaround for https://github.com/NVIDIA/spark-rapids-jni/issues/2582'
[2024-11-13T01:29:46.349Z] Workaround for https://github.com/NVIDIA/spark-rapids-jni/issues/2582
[2024-11-13T01:29:46.349Z] + cudf_patch_path=cudf/cpp/cmake/thirdparty/patches
[2024-11-13T01:29:46.349Z] + sed -i 's|${current_json_dir}|${current_json_dir}/../cudf/cpp/cmake/thirdparty/patches|g' thirdparty/cudf-pins/versions.json
...
[2024-11-13T01:31:20.903Z] [INFO]      [exec] -- Found CCCL: /home/jenkins/agent/workspace/jenkins-peixinl-spark-rapids-jni_submodule-sync-dev-1/target/libcudf/cmake-build/_deps/cccl-src/lib/cmake/cccl/cccl-config.cmake (found version "2.5.0.0") 
[2024-11-13T01:31:20.903Z] [INFO]      [exec] -- rapids-cmake [CCCL]: applied diff cccl_symbol_visibility.diff to fix issue: 'Correct symbol visibility issues in libcudacxx [https://github.com/NVIDIA/cccl/pull/1832/]'
[2024-11-13T01:31:20.903Z] [INFO]      [exec] -- rapids-cmake [CCCL]: applied diff thrust_disable_64bit_dispatching.diff to fix issue: 'Remove 64bit dispatching as not needed by libcudf and results in compiling twice as many kernels [https://github.com/rapidsai/cudf/pull/11437]'
[2024-11-13T01:31:20.903Z] [INFO]      [exec] -- rapids-cmake [CCCL]: applied diff thrust_faster_sort_compile_times.diff to fix issue: 'Improve Thrust sort compile times by not unrolling loops for inlined comparators [https://github.com/rapidsai/cudf/pull/10577]'
[2024-11-13T01:31:20.903Z] [INFO]      [exec] -- rapids-cmake [CCCL]: applied diff thrust_faster_scan_compile_times.diff to fix issue: 'Improve Thrust scan compile times by reducing the number of kernels generated [https://github.com/rapidsai/cudf/pull/8183]'

@pxLi pxLi requested review from ttnghia and jlowe November 13, 2024 02:57
@pxLi pxLi added bug Something isn't working build labels Nov 13, 2024
@pxLi pxLi changed the title Workaournd to apply cudf's thirdparty patches for cudf-pins build Workaournd to apply cudf's thirdparty patches for cudf-pins build [skip ci] Nov 13, 2024
@pxLi
Copy link
Collaborator Author

pxLi commented Nov 13, 2024

build

@pxLi pxLi self-assigned this Nov 13, 2024
@jlowe jlowe changed the title Workaournd to apply cudf's thirdparty patches for cudf-pins build [skip ci] Workaround to apply cudf's thirdparty patches for cudf-pins build [skip ci] Nov 13, 2024
@pxLi pxLi merged commit 84e08e2 into NVIDIA:branch-24.12 Nov 14, 2024
4 checks passed
@jlowe
Copy link
Member

jlowe commented Nov 15, 2024

Note that this workaround doesn't fix everything. It allows the CCCL patches in the cudf directory to work, but not the same patches in the spark-rapids-jni build when it tries to pull CCCL after building libcudf and libcudfjni.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working build
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants