Add Covariance, Pearson correlation for sort groupby (libcudf) #9154

karthikeyann · 2021-08-31T18:09:55Z

Add sort-groupby covariance and Pearson correlation in libcudf
Addresses part of #1268 (groupby covariance)
Addresses part of #8691 (groupby Pearson correlation)
depends on PR #9195

For both covariance and Pearson correlation, the input column pair should be represented as 2 child columns of non-nullable struct column (aggregation_request::values = struct_column_view{x, y})

covariance = Sum((x-mean_x)*(y-mean_y)) / (group_size-ddof)
Pearson correlation = covariance/ xstddev / ystddev

x, y values both should be non-null.
mean, stddev, count should be calculated on only common non-null values of both columns.

mean, stddev, count of child columns are cached.
One limitation is when both null columns has non-identical null masks, the cached result (mean, stddev, count) of common valid rows can not be reused because bitmask_and result nullmask goes out of scope and new nullmask is created for another set of columns (even if they are same).

Unit tests for covariance and pearson correlation added.

cpp/include/cudf/aggregation.hpp

codecov · 2021-08-31T19:48:42Z

Codecov Report

Merging #9154 (8426f56) into branch-21.12 (ab4bfaa) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@               Coverage Diff                @@
##           branch-21.12    #9154      +/-   ##
================================================
- Coverage         10.79%   10.75%   -0.04%     
================================================
  Files               116      116              
  Lines             18869    19482     +613     
================================================
+ Hits               2036     2096      +60     
- Misses            16833    17386     +553

Impacted Files	Coverage Δ
python/cudf/cudf/__init__.py	`0.00% <0.00%> (ø)`
python/cudf/cudf/_lib/__init__.py	`0.00% <ø> (ø)`
python/cudf/cudf/core/_base_index.py	`0.00% <0.00%> (ø)`
python/cudf/cudf/core/column/categorical.py	`0.00% <0.00%> (ø)`
python/cudf/cudf/core/column/column.py	`0.00% <0.00%> (ø)`
python/cudf/cudf/core/column/datetime.py	`0.00% <0.00%> (ø)`
python/cudf/cudf/core/column/lists.py	`0.00% <0.00%> (ø)`
python/cudf/cudf/core/column/numerical.py	`0.00% <0.00%> (ø)`
python/cudf/cudf/core/column/string.py	`0.00% <0.00%> (ø)`
python/cudf/cudf/core/column/timedelta.py	`0.00% <0.00%> (ø)`
... and 77 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1424a2d...8426f56. Read the comment docs.

…pearson_coeff

This reverts commit ba6e50a.

This reverts commit b198a51.

…groupby_corr

robertmaynard

CMake changes LGTM

karthikeyann · 2021-10-05T18:31:17Z

rerun tests

…ed on column_view instead of requests. (#9195) This PR updates groupby result_cache to use `pair<column_view, aggregation>` as key to unordered_map. This allows to cache intermediate results based on the column view. So, it is possible to cache children column_view results and can be resused in other aggregation_request. Depends on #9185 shallow_hash and is_shallow_equivalent are used for column_view. Additional context: This change is required to cache children column intermediate results in #9154 and allows to be shared across multiple aggregation requests. Authors: - Karthikeyan (https://github.com/karthikeyann) Approvers: - Jake Hemstad (https://github.com/jrhemstad) - Vyas Ramasubramani (https://github.com/vyasr) URL: #9195

jrhemstad · 2021-10-05T20:31:32Z

cpp/src/groupby/hash/groupby.cu

+    std::vector<std::unique_ptr<aggregation>> aggs;
+    aggs.push_back(make_sum_aggregation());
+    // COUNT_VALID
+    aggs.push_back(make_count_aggregation());
+
+    return aggs;


If you want to be fancy.

Suggested change

std::vector<std::unique_ptr<aggregation>> aggs;

aggs.push_back(make_sum_aggregation());

// COUNT_VALID

aggs.push_back(make_count_aggregation());

return aggs;

return {make_sum_aggregation(), make_count_aggregation()};

std::initializer_list only allows access to const elements. When unique_ptr is given in initializer list, std::vector tries to copy unique_ptr. copy constructor of unique_ptr is deleted.
So, this fails to compile.

Ah, sure, that makes sense. This should work though?

Suggested change

std::vector<std::unique_ptr<aggregation>> aggs;

aggs.push_back(make_sum_aggregation());

// COUNT_VALID

aggs.push_back(make_count_aggregation());

return aggs;

return std::vector({make_sum_aggregation(), make_count_aggregation()});

This has same effect as above.
Anything involving {} initializer_list on a object cannot be moved. (probably const_cast inside constructor might work, but that's not useful here.)

https://stackoverflow.com/a/8469002/1550940
Surprisingly array works.

std::unique_ptr<aggregation> init[] = {make_sum_aggregation(), make_count_aggregation()}; return std::vector<std::unique_ptr<aggregation>>{std::make_move_iterator(std::begin(init)), std::make_move_iterator(std::end(init))};

cpp/include/cudf/detail/aggregation/aggregation.hpp

cpp/src/groupby/sort/aggregate.cpp

cpp/src/groupby/sort/group_correlation.cu

Co-authored-by: nvdbaranec <[email protected]>

karthikeyann · 2021-10-18T09:45:58Z

@gpucibot merge

This PR adds the functionality to perform `.cov()` on a `GroupBy` object and completes #1268 Related issue: #1268 Related PRs: #9154, #9166, #9492 Next steps: - [ ] Fix Symmetry problem [PR 10098](#10098 (comment)): avoid computing the covariance/ correlation between the same colums twice - [ ] Consolidate both `cov()` and `corr()` - [ ] Fix #10303 - [ ] Add `cov `bindings in `aggregation.pyx` (separate PR): [comment](#9889 (comment)) - [ ] Simplify `combine_columns` after #10153 covers `interleave_columns`: [comment](#9889 (comment)) Authors: - Mayank Anand (https://github.com/mayankanand007) - Michael Wang (https://github.com/isVoid) - Sheilah Kirui (https://github.com/skirui-source) Approvers: - Bradley Dice (https://github.com/bdice) - Michael Wang (https://github.com/isVoid) - Vyas Ramasubramani (https://github.com/vyasr) URL: #9889

karthikeyann added 3 commits August 31, 2021 23:32

add CORR aggregation to groupby, headers, classes, visitor(sort)

763c53a

add group_corr.cu

4c989a9

add unit test temporarily

015795c

github-actions bot added CMake CMake build issue libcudf Affects libcudf (C++/CUDA) code. labels Aug 31, 2021

karthikeyann added 2 - In Progress Currently a work in progress feature request New feature or request non-breaking Non-breaking change and removed CMake CMake build issue labels Aug 31, 2021

This was linked to issues Aug 31, 2021

[FEA] rolling correlation #7666

Open

[FEA] Groupby correlation (Pearson) #8691

Closed

skirui-source removed a link to an issue Aug 31, 2021

[FEA] rolling correlation #7666

Open

jrhemstad reviewed Aug 31, 2021

View reviewed changes

cpp/include/cudf/aggregation.hpp Outdated Show resolved Hide resolved

jrhemstad reviewed Aug 31, 2021

View reviewed changes

cpp/include/cudf/aggregation.hpp Outdated Show resolved Hide resolved

skirui-source added 3 commits September 1, 2021 18:02

create new PR for pearson groupby correlation

ba6e50a

adding corr. func in python

b198a51

Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into …

b7464a2

…pearson_coeff

github-actions bot added CMake CMake build issue Python Affects Python cuDF API. labels Sep 4, 2021

karthikeyann added 2 commits September 6, 2021 09:40

Revert "create new PR for pearson groupby correlation"

3d00307

This reverts commit ba6e50a.

Revert "adding corr. func in python"

1200437

This reverts commit b198a51.

github-actions bot removed the Python Affects Python cuDF API. label Sep 6, 2021

karthikeyann added 6 commits September 6, 2021 11:03

Merge branch 'branch-21.10' of github.com:rapidsai/cudf into fea-sort…

178f28a

…groupby_corr

rename CORR to CORRELATION, added correlation_type as arg

60293cc

add shallow_hash(column_view)

d421d6d

add CompoundTypes to type_lists

9c4a9f3

add shallow_hash tests

a3dd235

add column copy test

2365d07

karthikeyann added the 3 - Ready for Review Ready for review by team label Oct 4, 2021

karthikeyann changed the title ~~Add Pearson correlation for sort groupby (libcudf)~~ Add Covariance, Pearson correlation for sort groupby (libcudf) Oct 4, 2021

karthikeyann requested review from beckernick and jrhemstad October 4, 2021 19:45

Merge branch 'branch-21.12' into fea-sortgroupby_corr

6703533

karthikeyann marked this pull request as ready for review October 4, 2021 19:47

karthikeyann requested review from a team as code owners October 4, 2021 19:47

karthikeyann requested a review from nvdbaranec October 4, 2021 19:47

robertmaynard approved these changes Oct 5, 2021

View reviewed changes

Merge branch 'branch-21.12' into fea-sortgroupby_corr

dd00e0d

jrhemstad reviewed Oct 5, 2021

View reviewed changes

karthikeyann requested a review from jrhemstad October 6, 2021 17:21

nvdbaranec requested changes Oct 7, 2021

View reviewed changes

Apply suggestions from code review

8426f56

Co-authored-by: nvdbaranec <[email protected]>

karthikeyann requested review from robertmaynard and nvdbaranec October 8, 2021 17:27

beckernick removed their request for review October 14, 2021 13:05

nvdbaranec approved these changes Oct 14, 2021

View reviewed changes

karthikeyann requested a review from a team October 15, 2021 03:14

rapids-bot bot merged commit 410efd9 into rapidsai:branch-21.12 Oct 18, 2021

mayankanand007 mentioned this pull request Dec 10, 2021

Add covariance for sort groupby (python) #9889

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Covariance, Pearson correlation for sort groupby (libcudf) #9154

Add Covariance, Pearson correlation for sort groupby (libcudf) #9154

karthikeyann commented Aug 31, 2021 •

edited

Loading

codecov bot commented Aug 31, 2021 •

edited

Loading

robertmaynard left a comment

karthikeyann commented Oct 5, 2021

jrhemstad Oct 5, 2021

karthikeyann Oct 6, 2021

jrhemstad Oct 7, 2021 •

edited

Loading

karthikeyann Oct 8, 2021

karthikeyann commented Oct 18, 2021

Add Covariance, Pearson correlation for sort groupby (libcudf) #9154

Add Covariance, Pearson correlation for sort groupby (libcudf) #9154

Conversation

karthikeyann commented Aug 31, 2021 • edited Loading

codecov bot commented Aug 31, 2021 • edited Loading

Codecov Report

robertmaynard left a comment

Choose a reason for hiding this comment

karthikeyann commented Oct 5, 2021

jrhemstad Oct 5, 2021

Choose a reason for hiding this comment

karthikeyann Oct 6, 2021

Choose a reason for hiding this comment

jrhemstad Oct 7, 2021 • edited Loading

Choose a reason for hiding this comment

karthikeyann Oct 8, 2021

Choose a reason for hiding this comment

karthikeyann commented Oct 18, 2021

karthikeyann commented Aug 31, 2021 •

edited

Loading

codecov bot commented Aug 31, 2021 •

edited

Loading

jrhemstad Oct 7, 2021 •

edited

Loading