Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Forward-merge branch-24.12 to branch-25.02 #17379

Merged
merged 25 commits into from
Nov 20, 2024
Merged

Forward-merge branch-24.12 to branch-25.02 #17379

merged 25 commits into from
Nov 20, 2024

Conversation

bdice
Copy link
Contributor

@bdice bdice commented Nov 20, 2024 •
edited
Loading

Manual forward merge from 24.12 to 25.02. This PR should not be squashed.

Closes #17343.

msarahan and others added 25 commits November 15, 2024 20:54
@msarahan
add telemetry setup to test (rapidsai#16924)
8664fad 
This is a prototype implementation of rapidsai/build-infra#139

The work that this builds on:
* rapidsai/gha-tools#118, which adds a shell wrapper that automatically creates spans for the commands that it wraps. It also uses the `opentelemetry-instrument` command to set up monkeypatching for supported Python libraries, if the command is python-based
* https://github.com/rapidsai/shared-workflows/tree/add-telemetry, which installs the gha-tools work from above and sets necessary environment variables. This is only done for the conda-cpp-build.yaml shared workflow at the time of submitting this PR.

The goal of this PR is to observe telemetry data sent from a GitHub Actions build triggered by this PR as a proof of concept. Once it all works, the remaining work is:

* merge rapidsai/gha-tools#118
* Move the opentelemetry-related install stuff in https://github.com/rapidsai/shared-workflows/compare/add-telemetry?expand=1#diff-ca6188672785b5d214aaac2bf77ce0528a48481b2a16b35aeb78ea877b2567bcR118-R125 into https://github.com/rapidsai/ci-imgs, and rebuild ci-imgs
* expand coverage to other shared workflows
* Incorporate the changes from this PR to other jobs and to other repos

Authors:
  - Mike Sarahan (https://github.com/msarahan)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: rapidsai#16924
@jlowe
Update cmake to 3.28.6 in JNI Dockerfile (rapidsai#17342)
e683647 
Updates cmake to 3.28.6 in the JNI Dockerfile used to build the cudf jar.  This helps avoid a bug in older cmake where FindCUDAToolkit can fail to find cufile libraries.

Authors:
  - Jason Lowe (https://github.com/jlowe)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)
  - Gera Shegalov (https://github.com/gerashegalov)

URL: rapidsai#17342
@Matt711
Use pylibcudf contiguous split APIs in cudf python (rapidsai#17246)
9cc9071 
Apart of rapidsai#15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: rapidsai#17246
@davidwendt
Move strings translate benchmarks to nvbench (rapidsai#17325)
e4de8e4 
Moves `cpp/benchmarks/string/translate.cpp` implementation from google-bench to nvbench.
This is benchmark for the `cudf::strings::translate` API.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Nghia Truong (https://github.com/ttnghia)

URL: rapidsai#17325
@mroeschke
Move cudf._lib.unary to cudf.core._internals (rapidsai#17318)
aeb6a30 
Contributes to rapidsai#17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: rapidsai#17318
@shrshi
Reading multi-source compressed JSONL files (rapidsai#17161)
03ac845 
Fixes rapidsai#17068 
Fixes rapidsai#12299

This PR introduces a new datasource for compressed inputs which enables batching and byte range reading of multi-source JSONL files using the reallocate-and-retry policy. Moreover. instead of using a 4:1 compression ratio heuristic, the device buffer size is estimated accurately for GZIP, ZIP, and SNAPPY compression types. For remaining types, the files are first decompressed then batched.

~~TODO: Reuse existing JSON tests but with an additional compression parameter to verify correctness.~~
~~Handled by rapidsai#17219, which implements compressed JSON writer required for the above test.~~
Multi-source compressed input tests added!

Authors:
  - Shruti Shivakumar (https://github.com/shrshi)

Approvers:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)
  - Karthikeyan (https://github.com/karthikeyann)

URL: rapidsai#17161
@vyasr
Test the full matrix for polars and dask wheels on nightlies (rapidsa…
d514517 
…i#17320)

This PR ensures that we have nightly coverage of more of the CUDA/Python/arch versions that we claim to support for dask-cudf and cudf-polars wheels.

In addition, this PR ensures that we do not attempt to run the dbgen executable in the Polars repository on systems with too old of a glibc to support running them.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: rapidsai#17320
@mhaseeb123
Fix reading Parquet string cols when nrows and input_pass_limit >…
43f2f68 
… 0 (rapidsai#17321)

This PR fixes reading string columns in Parquet using chunked parquet reader when `nrows` and `input_pass_limit` are > 0.

Closes rapidsai#17311

Authors:
  - Muhammad Haseeb (https://github.com/mhaseeb123)

Approvers:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Ed Seidl (https://github.com/etseidl)
  - Lawrence Mitchell (https://github.com/wence-)
  - Bradley Dice (https://github.com/bdice)
  - https://github.com/nvdbaranec
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: rapidsai#17321
@mroeschke
Remove cudf._lib.hash in favor of inlining pylibcudf (rapidsai#17345)
18b40dc 
Contributes to rapidsai#17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: rapidsai#17345
@mroeschke
Remove cudf._lib.concat in favor of inlining pylibcudf (rapidsai#17344)
ba21673 
Contributes to rapidsai#17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: rapidsai#17344
@mroeschke
Remove cudf._lib.quantiles in favor of inlining pylibcudf (rapidsai#1…
02c35bf 
…7347)

Contributes to rapidsai#17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: rapidsai#17347
@mroeschke
Remove cudf._lib.labeling in favor of inlining pylibcudf (rapidsai#17346
302e625 
)

Contributes to rapidsai#17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: rapidsai#17346
@wence-
Support polars 1.14 (rapidsai#17355)
5f9a97f 
1.13 was yanked for some reason, but 1.14 doesn't bring anything new and difficult.

Authors:
  - Lawrence Mitchell (https://github.com/wence-)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - https://github.com/brandon-b-miller
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: rapidsai#17355
@shrshi
Writing compressed output using JSON writer (rapidsai#17323)
384abae 
Depends on rapidsai#17161 for implementations of compression and decompression functions (`io/comp/comp.cu`, `io/comp/comp.hpp`, `io/comp/io_uncomp.hpp` and `io/comp/uncomp.cpp`)

Adds support for writing GZIP- and SNAPPY-compressed JSON to the JSON writer.
Verifies correctness using a parameterized test in `tests/io/json/json_writer.cpp`

Authors:
  - Shruti Shivakumar (https://github.com/shrshi)
  - Vukasin Milovanovic (https://github.com/vuule)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)
  - Karthikeyan (https://github.com/karthikeyann)
  - Vukasin Milovanovic (https://github.com/vuule)

URL: rapidsai#17323
@jameslamb
fix library-loading issues in editable installs (rapidsai#17338)
9c5cd81 
Contributes to rapidsai/build-planning#118

The pattern introduced in rapidsai#17316 breaks editable installs in devcontainers. In that type of build, `libcudf.so` is built outside of the wheel but **not installed**, so it can't be found by `ld`. Extension modules in `cudf` and `pylibcudf` are able to find it via RPATHs instead.

This proposes:

* try-catching the entire library-loading attempt, to silently do nothing in cases like that
* ~adding imports of the `cudf` and `pylibcudf` libraries in the `devcontainers` CI job, as a smoke test to catch issues like this in the future~ *(edit: removed those, [`devcontainer` builds run on CPU nodes](https://github.com/rapidsai/shared-workflows/blob/4e84062f333ce5649bc65029d3979569e2d0a045/.github/workflows/build-in-devcontainer.yaml#L19))*

## Notes for Reviewers

### How I tested this

Tested this approach on rapidsai/kvikio#553

#

Authors:
  - James Lamb (https://github.com/jameslamb)
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Matthew Murray (https://github.com/Matt711)

URL: rapidsai#17338
@wence-
Fix integer overflow in compiled binaryop (rapidsai#17354)
c7bfa77 
For large columns, the computed stride might end up overflowing size_type. To fix this, use the grid_1d helper. See also rapidsai#10368.

- Closes rapidsai#17353

Authors:
  - Lawrence Mitchell (https://github.com/wence-)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - David Wendt (https://github.com/davidwendt)
  - Tianyu Liu (https://github.com/kingcrimsontianyu)
  - Muhammad Haseeb (https://github.com/mhaseeb123)
  - Nghia Truong (https://github.com/ttnghia)

URL: rapidsai#17354
@davidwendt
Move strings replace benchmarks to nvbench (rapidsai#17301)
03c055f 
Move `cpp/benchmark/string/replace.cpp` implementation from google-test to nvbench
This covers strings replace APIs:
- `cudf::strings::replace` scalar version
- `cudf::strings::replace_multiple` column version
- `cudf::strings::replace_slice`

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Yunsong Wang (https://github.com/PointKernel)
  - Shruti Shivakumar (https://github.com/shrshi)

URL: rapidsai#17301
@PointKernel
Optimize distinct inner join to use set find instead of retrieve (r…
56061bd 
…apidsai#17278)

This PR introduces a minor optimization for distinct inner joins by using the `find` results to selectively copy matches to the output. This approach eliminates the need for the costly `retrieve` operation, which relies on expensive atomic operations.

Authors:
  - Yunsong Wang (https://github.com/PointKernel)

Approvers:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Karthikeyan (https://github.com/karthikeyann)

URL: rapidsai#17278
@mroeschke
Add compute_column_expression to pylibcudf for transform.compute_colu…
7158ee0 
…mn (rapidsai#17279)

Follow up to rapidsai#16760

`transform.compute_column` (backing `.eval`) requires an `Expression` object created by a private routine in cudf Python. Since this routine will be needed for any user of the public `transform.compute_column`, moving it to pylibcudf.

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)
  - Lawrence Mitchell (https://github.com/wence-)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: rapidsai#17279
@a-hirota
Bug fix: restrict lines=True to JSON format in Kafka read_gdf method (r…
05365af 
…apidsai#17333)

This pull request modifies the read_gdf method in kafka.py to pass the lines parameter only when the message_format is "json". This prevents lines from being passed to other formats (e.g., CSV, Avro, ORC, Parquet), which do not support this parameter.

Authors:
  - Hirota Akio (https://github.com/a-hirota)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: rapidsai#17333
@kingcrimsontianyu
Adapt to KvikIO API change in the compatibility mode (rapidsai#17377)
6f83b58 
This PR adapts cuDF to a breaking API change in KvikIO (rapidsai/kvikio#547) introduced recently, which adds the `AUTO` compatibility mode to file I/O.

This PR causes no behavioral changes in cuDF: If the environment variable `KVIKIO_COMPAT_MODE` is left unset, cuDF by default still enables the compatibility mode in KvikIO. This is the same with the previous behavior (rapidsai#17185).

Authors:
  - Tianyu Liu (https://github.com/kingcrimsontianyu)

Approvers:
  - Vukasin Milovanovic (https://github.com/vuule)

URL: rapidsai#17377
@shrshi
Benchmarking JSON reader for compressed inputs (rapidsai#17219)
fc08fe8 
Depends on rapidsai#17161 for implementations of compression and decompression functions (`io/comp/comp.cu`, `io/comp/comp.hpp`, `io/comp/io_uncomp.hpp` and `io/comp/uncomp.cpp`)\
Depends on rapidsai#17323 for compressed JSON writer implementation.

Adds benchmark to measure performance of the JSON reader for compressed inputs.

Authors:
  - Shruti Shivakumar (https://github.com/shrshi)
  - Muhammad Haseeb (https://github.com/mhaseeb123)

Approvers:
  - MithunR (https://github.com/mythrocks)
  - Vukasin Milovanovic (https://github.com/vuule)
  - Karthikeyan (https://github.com/karthikeyann)
  - Muhammad Haseeb (https://github.com/mhaseeb123)

URL: rapidsai#17219
@pentschev
Deselect failing polars tests (rapidsai#17362)
a2a62a1 
Deselect `test_join_4_columns_with_validity` which is failing in nightly CI tests and is reproducible in some systems (xref pola-rs/polars#19870), but apparently not all. Deselect `test_read_web_file` as well that fails on rockylinux8 due to SSL CA issues.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: rapidsai#17362
@rjzamora
Add new dask_cudf.read_parquet API (rapidsai#17250)
3111aa4 
It's time to clean up the `dask_cudf.read_parquet` API and prioritize GPU-specific optimizations. To this end, it makes sense to expose our own `read_parquet` API within Dask cuDF. 

**Notes**:

- The "new" `dask_cudf.read_parquet` API is only relevant when query-planning is enabled (the default).
- Using `filesystem="arrow"` now uses `cudf.read_parquet` when reading from local storage (rather than PyArrow).
- (specific to Dask cuDF): The default `blocksize` argument is now specific to the "smallest" NVIDIA device detected within the active dask cluster (or the first device visible to the the client). More specifically, we use `pynvml` to find this representative device size, and we set `blocksize` to be 1/32 this size.
  - The user may also pass in something like `blocksize=0.125` to use `1/8` the minimum device size (or `blocksize='1GiB'` to bypass the default logic altogether).
- (specific to Dask cuDF): When `blocksize` is `None`, we disable partition fusion at optimization time.
- (specific to Dask cuDF): When `blocksize` is **not** `None`, we use the parquet metadata from the first few files to inform partition fusion at optimization time (instead of a rough column-count ratio).

Authors:
  - Richard (Rick) Zamora (https://github.com/rjzamora)
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Lawrence Mitchell (https://github.com/wence-)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: rapidsai#17250
@bdice
Merge branch-24.12 into branch-25.02
698a716
@bdice bdice requested a review from a team as a code owner November 20, 2024 15:00
@bdice bdice added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Nov 20, 2024
@bdice bdice requested review from a team as code owners November 20, 2024 15:00
@bdice bdice requested review from a team as code owners November 20, 2024 15:00
@bdice bdice requested a review from msarahan November 20, 2024 15:00
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Nov 20, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions bot added libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API. CMake CMake build issue Java Affects Java cuDF API. cudf.polars Issues specific to cudf.polars pylibcudf Issues specific to the pylibcudf package labels Nov 20, 2024
@raydouglass raydouglass merged commit 664fd87 into rapidsai:branch-25.02 Nov 20, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@msarahan msarahan Awaiting requested review from msarahan msarahan is a code owner automatically assigned from rapidsai/packaging-codeowners

@wence- wence- Awaiting requested review from wence- wence- is a code owner automatically assigned from rapidsai/cudf-python-codeowners

@Matt711 Matt711 Awaiting requested review from Matt711 Matt711 is a code owner automatically assigned from rapidsai/cudf-python-codeowners

@mythrocks mythrocks Awaiting requested review from mythrocks mythrocks is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

@ttnghia ttnghia Awaiting requested review from ttnghia ttnghia is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@bdice bdice

Labels
CMake CMake build issue cudf.polars Issues specific to cudf.polars improvement Improvement / enhancement to an existing function Java Affects Java cuDF API. libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
cuDF Python
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.