-
Notifications
You must be signed in to change notification settings - Fork 915
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEA] support collect aggregations in reduction #7807
Comments
A collect reduction on a whole column doesn't make much sense to me. Turn an entire column into a list column with a single list? |
@jrhemstad Yes, it basically turns an entire column into a list column with a single list. We need this feature because we want to provide GPU support of spark built-in function > SELECT collect_list(col) FROM VALUES (1), (2), (1) AS tab(col);
[1,2,1]
> SELECT collect_set(col) FROM VALUES (1), (2), (1) AS tab(col);
[1,2] So, we need a method to produce scalars (in ListType) from corresponding input columns. It matches to the semantic of reduce. Or is there alternative approach to achieve this? Perhaps we can achieve the goal more directly with a method like |
This issue has been labeled |
This issue has been labeled |
Closes #7807 Curreent PR is to support the collect aggregation family in reduction context, which includes collect_list, collect_set, merge_lists, and merge_sets. The implementations are inspired by corresponding collect aggregations in groupby context. Authors: - Alfred Xu (https://github.com/sperlingxx) Approvers: - Jake Hemstad (https://github.com/jrhemstad) URL: #10353
Is your feature request related to a problem? Please describe.
Currently, cuDF supports collect aggregations in rolling windows and groupBy context (though the support is not complete). But, collect aggregations in reduction context is still missing.
Additional context
I believe we can not support this feature until #5887 got solved, since scalar of ListType is essential in reduction to collect.
The text was updated successfully, but these errors were encountered: