Update groupby result_cache to allow sharing intermediate results based on column_view instead of requests. #9195
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
karthikeyann
added
2 - In Progress
karthikeyann
changed the title
Co-authored-by: Jake Hemstad <[email protected]>
…/cudf into fea-shallow_equal_columnview
Co-authored-by: Jake Hemstad <[email protected]>
…/cudf into enh-groupby_cache_hashed
…low_hash_columnview
…/cudf into enh-groupby_cache_hashed
Co-authored-by: Jake Hemstad <[email protected]>
…nview' of github.com:karthikeyann/cudf into enh-groupby_cache_hashed
Codecov Report
@@ Coverage Diff @@
## branch-21.12 #9195 +/- ##
================================================
- Coverage 10.79% 10.76% -0.03%
================================================
Files 116 116
Lines 18869 19467 +598
================================================
+ Hits 2036 2096 +60
- Misses 16833 17371 +538
Continue to review full report at Codecov.
|
…/cudf into enh-groupby_cache_hashed
Co-authored-by: David Wendt <[email protected]>
…/cudf into enh-groupby_cache_hashed
jrhemstad
approved these changes
Oct 1, 2021
vyasr
requested changes
Oct 4, 2021
Co-authored-by: Vyas Ramasubramani <[email protected]>
vyasr
approved these changes
Oct 4, 2021
|
@gpucibot merge |
rapids-bot bot
pushed a commit
that referenced
this pull request
Oct 18, 2021
Add sort-groupby covariance and Pearson correlation in libcudf Addresses part of #1268 (groupby covariance) Addresses part of #8691 (groupby Pearson correlation) depends on PR #9195 For both covariance and Pearson correlation, the input column pair should be represented as 2 child columns of non-nullable struct column (`aggregation_request::values` = `struct_column_view{x, y}`) ``` covariance = Sum((x-mean_x)*(y-mean_y)) / (group_size-ddof) Pearson correlation = covariance/ xstddev / ystddev ``` x, y values both should be non-null. mean, stddev, count should be calculated on only common non-null values of both columns. mean, stddev, count of child columns are cached. One limitation is when both null columns has non-identical null masks, the cached result (mean, stddev, count) of common valid rows can not be reused because bitmask_and result nullmask goes out of scope and new nullmask is created for another set of columns (even if they are same). Unit tests for covariance and pearson correlation added. Authors: - Karthikeyan (https://github.com/karthikeyann) - Sheilah Kirui (https://github.com/skirui-source) Approvers: - Robert Maynard (https://github.com/robertmaynard) - https://github.com/nvdbaranec URL: #9154
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates groupby result_cache to use
pair<column_view, aggregation>as key to unordered_map.This allows to cache intermediate results based on the column view. So, it is possible to cache children column_view results and can be resused in other aggregation_request.
Depends on #9185
shallow_hash and is_shallow_equivalent are used for column_view.
Additional context:
This change is required to cache children column intermediate results in #9154 and allows to be shared across multiple aggregation requests.