Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmake option: CUDF_KVIKIO_REMOTE_IO #17291

Merged
merged 7 commits into from
Nov 12, 2024
Merged

cmake option: CUDF_KVIKIO_REMOTE_IO #17291

merged 7 commits into from
Nov 12, 2024

Conversation

madsbk
Copy link
Member

@madsbk madsbk commented Nov 9, 2024

Compile flag to enable/disable remote IO through KvikIO: CUDF_KVIKIO_REMOTE_IO

@madsbk madsbk added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Nov 9, 2024
@github-actions github-actions bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue labels Nov 9, 2024
ttnghia
ttnghia reviewed Nov 9, 2024
View reviewed changes
cpp/CMakeLists.txt Outdated
@@ -90,6 +90,12 @@ option(
mark_as_advanced(CUDF_BUILD_STREAMS_TEST_UTIL)
option(CUDF_STATIC_LINTERS "Enable static linters during compilation" OFF)

option(
CUDF_KVIKIO_REMOTE_IO
"Enable remote IO (e.g. AWS S3) support through KvikIO. If disabled, cudf-python will still be able to do remote IO."
Copy link
Contributor

@ttnghia ttnghia Nov 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is not enabled, the C++ code will not support it thus I don't think Python will be able to do it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated:

option(
  CUDF_KVIKIO_REMOTE_IO
  "Enable remote IO (e.g. AWS S3) support through KvikIO. If disabled, cudf-python will still be able to do remote IO through fsspec."
  ON
)

@madsbk madsbk force-pushed the remote_io_compile_guard branch from b08a15b to 90afe1d Compare November 10, 2024 10:58
@madsbk
Copy link
Member Author

madsbk commented Nov 10, 2024

@ttnghia, can you check if setting -DCUDF_KVIKIO_REMOTE_IO=OFF fixes fixes NVIDIA/spark-rapids-jni#2580 ?

@madsbk madsbk marked this pull request as ready for review November 10, 2024 13:25
@madsbk madsbk requested review from a team as code owners November 10, 2024 13:25
@pxLi
Copy link
Member

pxLi commented Nov 11, 2024 •
edited
Loading

it would be nice to also include an update to cudf JNI build script, thanks!
https://github.com/rapidsai/cudf/blob/branch-24.12/java/ci/build-in-docker.sh#L68

I will file one after merge later #17293

@pxLi pxLi mentioned this pull request Nov 11, 2024
Merged
@ttnghia
Copy link
Contributor

ttnghia commented Nov 11, 2024

@ttnghia, can you check if setting -DCUDF_KVIKIO_REMOTE_IO=OFF fixes fixes NVIDIA/spark-rapids-jni#2580 ?

Indeed NVIDIA/spark-rapids-jni#2580 should be fixed by rapidsai/kvikio#542.
For this PR, I confirm that it fixes the linking issue NVIDIA/spark-rapids-jni#2581. Thanks for working on this.

kingcrimsontianyu
kingcrimsontianyu approved these changes Nov 11, 2024
View reviewed changes
Copy link
Contributor

@kingcrimsontianyu kingcrimsontianyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm. Thanks for the fix!

@pxLi
Copy link
Member

pxLi commented Nov 11, 2024 •
edited
Loading

Still requires approval from cudf-cmake-codeowners @rapidsai/cudf-cmake-codeowners, thanks

This was referenced Nov 12, 2024
Merged
Merged
Merged
@pxLi
Copy link
Member

pxLi commented Nov 12, 2024

/merge

@rapids-bot rapids-bot bot merged commit bdddab3 into rapidsai:branch-24.12 Nov 12, 2024
102 checks passed
rapids-bot bot pushed a commit that referenced this pull request Nov 12, 2024
@pxLi
Replace workaround of JNI build with CUDF_KVIKIO_REMOTE_IO=OFF (#17293)
202c231 
JNI build does not require kvikIO, to unblock the build use `CUDF_KVIKIO_REMOTE_IO=OFF` in cpp build phase.

this should be merged after #17291

Authors:
  - Peixin (https://github.com/pxLi)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)

URL: #17293
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ttnghia ttnghia ttnghia left review comments

@kingcrimsontianyu kingcrimsontianyu kingcrimsontianyu approved these changes

@bdice bdice bdice approved these changes

@pmattione-nvidia pmattione-nvidia Awaiting requested review from pmattione-nvidia pmattione-nvidia is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@madsbk madsbk

Labels
CMake CMake build issue improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@madsbk @pxLi @ttnghia @bdice @kingcrimsontianyu