Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scan min/max support for chrono types to libcudf reduction-scan (not groupby scan) #9518

Merged

Conversation

davidwendt
Copy link
Contributor

@davidwendt davidwendt commented Oct 25, 2021

Reference #8709

Enable chrono types (timestamp, duration) support in cudf::scan with inclusive scan-type. The only requirement for min/max scan is for the type to be relationally-comparable. The code already supported string type for min/max so the change here only required generalizing that approach.

New gtests were added to scan_tests.cpp for all chrono types. Also, the to_strings specialization in the column test utilities was enhanced to show timestamp-type-specific details which is helpful when using cudf::test::print on timestamp types.

Note that this does not include groupby scan support but only reduction's scan.

@davidwendt davidwendt added feature request New feature or request 2 - In Progress Currently a work in progress libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change labels Oct 25, 2021
@davidwendt davidwendt self-assigned this Oct 25, 2021
@codecov
Copy link

codecov bot commented Oct 25, 2021

Codecov Report

Merging #9518 (1401417) into branch-21.12 (ab4bfaa) will decrease coverage by 0.12%.
The diff coverage is n/a.

Impacted file tree graph

@@               Coverage Diff                @@
##           branch-21.12    #9518      +/-   ##
================================================
- Coverage         10.79%   10.66%   -0.13%     
================================================
  Files               116      117       +1     
  Lines             18869    19733     +864     
================================================
+ Hits               2036     2104      +68     
- Misses            16833    17629     +796     
Impacted Files Coverage Δ
python/dask_cudf/dask_cudf/sorting.py 92.90% <0.00%> (-1.21%) ⬇️
python/cudf/cudf/io/csv.py 0.00% <0.00%> (ø)
python/cudf/cudf/io/hdf.py 0.00% <0.00%> (ø)
python/cudf/cudf/io/orc.py 0.00% <0.00%> (ø)
python/cudf/cudf/__init__.py 0.00% <0.00%> (ø)
python/cudf/cudf/_version.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/abc.py 0.00% <0.00%> (ø)
python/cudf/cudf/api/types.py 0.00% <0.00%> (ø)
python/cudf/cudf/io/dlpack.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/frame.py 0.00% <0.00%> (ø)
... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e632f97...1401417. Read the comment docs.

@davidwendt
Copy link
Contributor Author

Looks like we should be able to support the following for scan (inclusive):

type(s) MIN MAX SUM PRODUCT
INT8, INT16, INT32, INT64, UINT8, UINT16, UINT32, UINT64, FLOAT32, FLOAT64, BOOL8 x x x x
TIMESTAMP: DAYS, SECONDS, MILLISECONDS, MICROSECONDS, NANOSECONDS x x
DURATION: DAYS, SECONDS, MILLISECONDS, MICROSECONDS, NANOSECONDS x x x
STRING x x
DECIMAL32, DECIMAL64 x x x

All other types (DICTIONARY32, LIST, STRUCT) are not supported. The std::is_arithmetic() covers the first and last row because the type-dispatcher is called using the dispatch-storage-type and there is a special if-statement throwing an error for fixed-point and PRODUCT. The STRING row is already supported. This PR adds support for TIMESTAMP and DURATION for MIN and MAX only. That leaves one element not accounted for: DURATION and SUM.

I could include support for DURATION/SUM in this PR (including the gtests). I did not realize we were just missing one case. But I'm inclined to adding into a follow up PR instead to keep this one focused on min/max support for which the target issue is requesting.

@davidwendt davidwendt added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Oct 27, 2021
@davidwendt davidwendt marked this pull request as ready for review October 27, 2021 13:36
@davidwendt davidwendt requested a review from a team as a code owner October 27, 2021 13:36
@jrhemstad
Copy link
Contributor

But I'm inclined to adding into a follow up PR instead

I'm fine with that. I would really like to see this code moved away from using a handrolled type/operator support matrix to using something like std::is_invocable to programmatically query what operations are supported on what types so we can avoid this issue where we don't support certain operations on certain types because the handrolled is_supported check was incorrect.

@davidwendt
Copy link
Contributor Author

But I'm inclined to adding into a follow up PR instead

I'm fine with that. I would really like to see this code moved away from using a handrolled type/operator support matrix to using something like std::is_invocable to programmatically query what operations are supported on what types so we can avoid this issue where we don't support certain operations on certain types because the handrolled is_supported check was incorrect.

This one is particularly offensive because the is_supported logic is coded into the gtests too.
https://github.com/rapidsai/cudf/blob/branch-21.12/cpp/tests/reductions/scan_tests.cpp#L148-L167
I tried fixing the tests first but gave up after a few hours. I need to go back and re-watch the is_invocable talk.

@ttnghia
Copy link
Contributor

ttnghia commented Nov 1, 2021

So this is only reduction scan, right? And for groupby scan it is not yet supported? If so then we should clarify that in the PR title/description.

@davidwendt davidwendt changed the title Add scan min/max support for chrono types to libcudf Add scan min/max support for chrono types to libcudf reduction-scan (not groupby scan) Nov 1, 2021
@davidwendt
Copy link
Contributor Author

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 1a2c347 into rapidsai:branch-21.12 Nov 3, 2021
@davidwendt davidwendt deleted the fea-min-max-scan-timestamps branch November 3, 2021 13:50
rapids-bot bot pushed a commit that referenced this pull request Nov 8, 2021
Depends on #9518 
Reference #9518 (comment)

Add support for inclusive scan of duration types with SUM aggregation.
Also refactor the `scan_tests.cpp` to put rank tests into `rank_tests.cpp`, common utilities into `scan_tests.hpp`, and move the actual unit test logic out of `cudf::test` namespace.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Jake Hemstad (https://github.com/jrhemstad)
  - Nghia Truong (https://github.com/ttnghia)

URL: #9536
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants