Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] cudf v21.12 #9689

Merged
merged 252 commits into from
Dec 3, 2021
Merged

[RELEASE] cudf v21.12 #9689

merged 252 commits into from
Dec 3, 2021

Conversation

GPUtester
Copy link
Collaborator

❄️ Code freeze for branch-21.12 and v21.12 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-21.12 until release (merging of this PR).

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-21.12 into main for the release

pxLi and others added 30 commits September 22, 2021 15:03
@pxLi
Update cudf java bindings to 21.12.0-SNAPSHOT (#9248)
a4771b3 
Signed-off-by: Peixin Li <[email protected]>

update cudf JNI version to 21.12.0

Authors:
  - Peixin (https://github.com/pxLi)

Approvers:
  - Jason Lowe (https://github.com/jlowe)
  - Robert (Bobby) Evans (https://github.com/revans2)

URL: #9248
@galipremsagar
Merge branch-21.10 into branch-21.12
3ed97af
@ajschmidt8
Merge pull request #9285 from galipremsagar/branch-21.12-merge-21.10
e0cf38b 
Fix Automerger for `Branch-21.12` from `branch-21.10`
@GPUtester
Merge pull request #9288 from rapidsai/branch-21.10
e2098e5 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@GPUtester
Merge pull request #9293 from rapidsai/branch-21.10
65850e2 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@GPUtester
Merge pull request #9294 from rapidsai/branch-21.10
7277443 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@GPUtester
Merge pull request #9295 from rapidsai/branch-21.10
8d4c523 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@mayankanand007
Added deprecation warning for .label_encoding() (#9289)
20498f7 
This PR addresses issue #8608 by adding a deprecation warning before we remove the functionality entirely.

Authors:
  - Mayank Anand (https://github.com/mayankanand007)

Approvers:
  - Ashwin Srinath (https://github.com/shwina)

URL: #9289
@GPUtester
Merge pull request #9297 from rapidsai/branch-21.10
a269820 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@GPUtester
Merge pull request #9298 from rapidsai/branch-21.10
2718443 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@GPUtester
Merge pull request #9302 from rapidsai/branch-21.10
908c130 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@bdice
Pin mypy in .pre-commit-config.yaml to match conda environment pinnin…
165cdac 
…g. (#9300)

Currently mypy set as is a "local" hook in pre-commit, meaning that it uses `mypy` from the user's environment. I have mypy 0.910 installed (released Jun 22, 2021) locally and it fails with the following error:
```
mypy.....................................................................Failed
- hook id: mypy
- exit code: 2
python/cudf/cudf/tests/conftest.py: error: Duplicate module named "conftest" (also at "python/cudf/cudf/benchmarks/conftest.py")
python/cudf/cudf/tests/conftest.py: note: Are you missing an __init__.py? Alternatively, consider using --exclude to avoid checking one of them.
Found 1 error in 1 file (errors prevented further checking)
```

The conda environment pinning is [currently 0.782](https://github.com/rapidsai/cudf/blob/ba763105e006494a536c1a2fafc5112ab3dae362/conda/environments/cudf_dev_cuda11.2.yml#L39) (released Jun 23, 2020). Newer versions of mypy (around 0.900) have substantial behavior changes, so my newer version throws errors that are not seen in the older mypy that is pinned for CI checks.

This PR changes the behavior of `.pre-commit-config.yml` to use mypy from a mirror repository, at the same pinning used in the conda environment, instead of the local executable.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Ashwin Srinath (https://github.com/shwina)

URL: #9300
@codereport
Potential overflow of decimal32 when casting to int64_t (#9287)
b4560f4 
This bug fixes #9281.

The issue is that we should be using `std::common_type_t` for explicit casting between different types to avoid overflow. That fix has been applied.

Authors:
  - Conor Hoekstra (https://github.com/codereport)

Approvers:
  - Jake Hemstad (https://github.com/jrhemstad)
  - Nghia Truong (https://github.com/ttnghia)
  - Robert Maynard (https://github.com/robertmaynard)
  - https://github.com/nvdbaranec
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #9287
@rongou
add a new memory buffer that directly allocates with cudaMalloc (#9311)
fd77ca6 
Fixes #9270

Authors:
  - Rong Ou (https://github.com/rongou)

Approvers:
  - Jason Lowe (https://github.com/jlowe)
  - Alessandro Bellina (https://github.com/abellina)

URL: #9311
@galipremsagar
Unpin dask and distributed in CI (#9307)
4de968e 
This PR unpins `dask` and `distributed` in CI.

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Charles Blackmon-Luca (https://github.com/charlesbluca)
  - AJ Schmidt (https://github.com/ajschmidt8)
  - https://github.com/jakirkham

URL: #9307
@devavret
Use nvcomp's snappy decompressor in avro reader (#9181)
42ffe78 
Depends on rapidsai/integration#352

Authors:
  - Devavret Makkar (https://github.com/devavret)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)
  - https://github.com/nvdbaranec
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #9181
@karthikeyann
Add shallow hash function and shallow equality comparison for column_…
0b007a5 
…view (#9312)

Fixes #9140 
Added `shallow_hash(column_view)`
Added unit tests
SWIPAT approval complete

It computes hash values based on the shallow states of `column_view`:
type, size, data pointer, null_mask pointer,  offset, and the hash value of the children. 
`null_count` is not used since it is a cached value and it may vary based on contents of `null_mask`, and may be pre-computed or not.

Fixes #9139
Added `is_shallow_equivalent(column_view, column_view)` ~shallow_equal~
Added unit tests

It compares two column_views based on the shallow states of column_view:
type, size, data pointer, null_mask pointer, offset, and the column_view of the children.
null_count is not used since it is a cached value and it may vary based on contents of null_mask, and may be pre-computed or not.

Authors:
  - Karthikeyan (https://github.com/karthikeyann)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #9312
@vyasr
Consolidate more methods in Frame (#9305)
e245cd5 
This PR standardizes some more methods that were previously duplicated between different Frame classes into Frame. Many of the changes should be straightforward to review, with the main possible exception being the changes to serialization logic. Note that because the format of the serialized data has been standardized across different Frame types, objects serialized with old versions of cudf will be incompatible with the new format. I've included some backwards compatibility layers to be removed in a future release. Additionally, this PR adds some methods to Frame that are really only applicable to DataFrame and Series. Since some such methods already exist in Frame, I think it's fine to continue adding to those pending the eventual creation of a common IndexedFrame class.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Ashwin Srinath (https://github.com/shwina)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #9305
@GPUtester
Merge pull request #9323 from rapidsai/branch-21.10
6484e2a 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@GPUtester
Merge pull request #9335 from rapidsai/branch-21.10
cafd943 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@GPUtester
Merge pull request #9336 from rapidsai/branch-21.10
fd0b710 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@bdice
Use f-string in join helper warning message. (#9325)
bef2c35 
Tiny fix to use an f-string in a warning message.

Current output:
```
.../cudf/python/cudf/cudf/core/join/_join_helpers.py:126: UserWarning: Can't safely cast column from float32 to int8, upcasting to {common_type}.
```

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)
  - https://github.com/brandon-b-miller

URL: #9325
@davidwendt
Fix some unused variable warnings in libcudf (#9326)
fdb9e3b 
Some unused variable warnings are appearing in 11.4 builds from recent PR merges.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Nghia Truong (https://github.com/ttnghia)

URL: #9326
@brandon-b-miller
Pure-python masked UDFs (#9174)
f9ce870 
Replaces C++ implementation of masked UDF pipeline with a pure python version which compiles and launches the entire kernel using numba. This solves a bunch of problems:

- CUDA 11.0 support is now available since the impl no longer needs `cuda::std::tuple` to work with NVRTC 11.0. 
- Support for special functions which compile to multiple function definitions, such as `pow`, `sin`, and `cos` is now provided since all the PTX is compiled and linked inside numba (Fixes #8470)
- Allows us to support this corner case which would require a separate c++ kernel in previous implementation
```python
def f(x):
    return 42
```

- Makes developing/adding features to the impl much easier

Authors:
  - https://github.com/brandon-b-miller

Approvers:
  - Robert Maynard (https://github.com/robertmaynard)
  - GALI PREM SAGAR (https://github.com/galipremsagar)
  - Graham Markall (https://github.com/gmarkall)
  - Ashwin Srinath (https://github.com/shwina)

URL: #9174
@bdice
Fix Cython compilation warnings. (#9327)
8935f6d 
This PR fixes three small issues relating to uninitialized variables in Cython code. This removes warnings from the build log and fixes potential bugs.

- Parquet code had a boolean variable `is_range_index` that was not initialized if `json_str != ""`.
- Reduction scan type wasn't correctly initialized (`if True:` / `elif False:` pattern)
- Undesired comparison of a signed int and an unsigned value in a utility function.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Ashwin Srinath (https://github.com/shwina)

URL: #9327
@GPUtester
Merge pull request #9342 from rapidsai/branch-21.10
e63f455 
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
@rjzamora
Avoid casting to list or struct dtypes in dask_cudf.read_parquet (#9314)
840faf5 
A similar fix for this problem was recently submitted in #9159 and closed in favor of #9203. It seems that the test added in the latter PR was not actually capturing the original problem. However, after [dask#8072](dask/dask#8072) is merged, the new test will certainly start failing.

Authors:
  - Richard (Rick) Zamora (https://github.com/rjzamora)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)
  - Benjamin Zaitlen (https://github.com/quasiben)

URL: #9314
@marlenezw
Add isocalendar API support (#9169)
ef50796 
Closes #8896 #4908.

 It creates a new `isocalendar` method in cudf python, and solves related datetime issues for obtaining week, year and day in isocalendar form.

Authors:
  - Marlene  (https://github.com/marlenezw)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Ram (Ramakrishna Prabhu) (https://github.com/rgsl888prabhu)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #9169
@mythrocks
Optionally nullify out-of-bounds indices in segmented_gather(). (#9318)
5cea6b5 
The behaviour of `cudf::lists::segmented_gather()` is currently undefined for any
index value `i` that falls outside the range `[-n, n)`, where `n` is the number of
elements in the list row.

This commit adds support to explicitly specify an `out_of_bounds_policy`, like in
`cudf::gather()`. The erstwhile behaviour is retained when the bounds policy is set
to `DONT_CHECK`. If the bounds policy is specified as `NULLIFY`, then for any
index falling outside the range `[-n, n)`, the list element is set to `null`.

E.g.
```c++
auto source_column = [{"a", "b", "c", "d"}, {"1", "2", "3", "4"}, {"x", "y", "z"}];
auto gather_map    = [{0, -1, 4, -5}, {1, 3, 5}, {}];
auto result = segmented_gather(source_column, gather_map, NULLIFY);
result == [{"a", "d", null, null}, {"2", "4", null}, {}];
```

Authors:
  - MithunR (https://github.com/mythrocks)

Approvers:
  - Conor Hoekstra (https://github.com/codereport)
  - Karthikeyan (https://github.com/karthikeyann)
  - Nghia Truong (https://github.com/ttnghia)

URL: #9318
@isVoid
Use Default Memory Resource for Temporaries in reduction.cpp (#9344)
ffb3814 
This PR replaces the memory resource for temporaries in `reduction.cpp` with default memory resources.

Authors:
  - Michael Wang (https://github.com/isVoid)

Approvers:
  - David Wendt (https://github.com/davidwendt)
  - Bradley Dice (https://github.com/bdice)
  - MithunR (https://github.com/mythrocks)

URL: #9344
ajschmidt8 and others added 9 commits November 19, 2021 12:30
@ajschmidt8
Fix dask-cudf recipe for Enhanced Compatibility (#9733)
cb894e0 
The `dask-cudf` recipe was changed after #9456 was opened, so the Enhanced Compatibility changes never made it into this recipe. This PR fixes that.
@ajschmidt8
update changelog
e05bd4b
@ajschmidt8
Merge remote-tracking branch 'upstream/branch-21.10' into fix-changelog
dbf3c1c
@ajschmidt8
fix merge issues
0906a09
@ajschmidt8
Merge pull request #9739 from ajschmidt8/fix-changelog
ff5d987 
Fix Changelog Merge Conflicts for `branch-21.12`
@ajschmidt8
Update DEFAULT_CUDA_VER in ci/cpu/prebuild.sh (#9749)
9fc35b7 
Now that we only do `11.5` builds for RAPIDS, the `DEFAULT_CUDA_VER` variable in `ci/cpu/prebuild.sh` should be set to `11.5` so that the rest of the logic in the file works correctly.
@galipremsagar @bdice @jakirkham
[REVIEW] Pin max dask & distributed versions (#9734)
8e2ac44 
* pin max dask version

* Update conda/recipes/dask-cudf/meta.yaml

Co-authored-by: Bradley Dice <[email protected]>

* Apply suggestions from code review

Co-authored-by: jakirkham <[email protected]>

Co-authored-by: Bradley Dice <[email protected]>
Co-authored-by: jakirkham <[email protected]>
@trxcllnt
[FIX] Add arrow_dataset and parquet targets to build exports (#9491)
8d9d222 
This PR adds the `arrow_dataset` and `parquet` targets to the build export when Arrow is built from source by CPM, similar to what we have to do today for `arrow` and `arrow_cuda` targets.
@shwina
Use ptxcompiler to patch Numba at runtime to support CUDA enhanced co…
a1ca8c1 
…mpatibility. (#9687)
robertmaynard and others added 17 commits November 29, 2021 15:35
@robertmaynard
Only run runtime jit tests with CUDA 11.5 runtime
0ebeffa 
CI runs all tests over a variety of different hardware. Tests
that have to use NVRTC to re-compile cudf C++ code are only
supported on CUDA 11.5+.
@robertmaynard
Fix style issues found by CI
dfcb48d
@robertmaynard
WIP: disable csv test
bbf137e
@robertmaynard
WIP: disable all io tests
a24d2a8
@karthikeyann @robertmaynard
remove jit integration tests
f614395
@karthikeyann @robertmaynard
remove jit code which are supported by compiled binops
16fcf48
@karthikeyann @robertmaynard
remove jit benchmark
1b9d624
@karthikeyann @robertmaynard
skip generic op udf (jit ptx) in pytest CUDA<11.5
e49a343
@karthikeyann @robertmaynard
add deleted UserDefinedOp
8f64086
@karthikeyann @robertmaynard
fix missing includes
efb203b
@karthikeyann @robertmaynard
fix segfault by nullptr check in cufile_shim dtor
011fb48
@karthikeyann @robertmaynard
enable cuio tests again
9bdc28b
@karthikeyann @robertmaynard
addres review comments
a3ba687
@jjacobelli
Merge pull request #9794 from robertmaynard/Remove_jit_code_of_binary…
2c6f919 
…_operation

Remove jit code of binary operation
@sperlingxx
fix make_empty_scalar_like (#9782)
74ac6ed 
Signed-off-by: sperlingxx <[email protected]>
@robertmaynard @karthikeyann
Move the binary_ops common dispatcher logic to be executed on the CPU (…
69e6dbb 
…#9816)

* move NullEquals to separate file

* To improve runtime performance move more binary_ops dispatch to host

* make sure to forceinline the operator_dispatcher

* Correct style issues found by ci

* Expand the binary-op compiled benchmark suite

* Ensure forceinline is on binary ops device dispatch functions

* Correct style issues found by ci

Co-authored-by: Karthikeyan Natarajan <[email protected]>
Co-authored-by: Karthikeyan <[email protected]>
@vuule @codereport
Fix ORC writer crash with empty input columns (#9831)
a93d333 
* skip some parts of writing if no rows; test

* expand test

* Apply suggestions from code review

Co-authored-by: Conor Hoekstra <[email protected]>

* fix

* remove integer-suffix

Co-authored-by: Conor Hoekstra <[email protected]>
@raydouglass raydouglass merged commit 3ceb0c0 into main Dec 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ttnghia ttnghia ttnghia approved these changes

@vyasr vyasr vyasr approved these changes

@harrism harrism harrism approved these changes

@davidwendt davidwendt Awaiting requested review from davidwendt davidwendt is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

@skirui-source skirui-source Awaiting requested review from skirui-source skirui-source is a code owner automatically assigned from rapidsai/cudf-python-codeowners

Assignees
No one assigned
Labels
CMake CMake build issue Java Affects Java cuDF API. libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.