Skip to content

Commit

Permalink
Enable automatic column projection in groupby().agg (#12124)
Browse files Browse the repository at this point in the history
This PR corresponds to the Dask-cudf version of dask/dask#9442, which was found to improve the performance of many groupby-based workflows. After this PR, 

```python
import dask_cudf

path = "/criteo-dataset/day_0.parquet"
ddf = dask_cudf.read_parquet(path, split_row_groups=10)

# The following takes <2s with this PR, and fails with
# an OOM error on main (using a 32GB GPU):
ddf.groupby("C1").agg({"C2": "mean"}).compute()
```

Authors:
  - Richard (Rick) Zamora (https://github.com/rjzamora)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #12124
  • Loading branch information
rjzamora authored Nov 14, 2022
1 parent 825f049 commit 5081fb1
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 1 deletion.
7 changes: 6 additions & 1 deletion python/dask_cudf/dask_cudf/groupby.py
Original file line number Diff line number Diff line change
Expand Up @@ -685,8 +685,13 @@ def groupby_agg(
"with `sort=False`, or set `shuffle=True`."
)

# Determine required columns to enable column projection
required_columns = list(
set(gb_cols).union(aggs.keys()).intersection(ddf.columns)
)

return aca(
[ddf],
[ddf[required_columns]],
chunk=chunk,
chunk_kwargs=chunk_kwargs,
combine=combine,
Expand Down
4 changes: 4 additions & 0 deletions python/dask_cudf/dask_cudf/tests/test_groupby.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,10 @@ def test_groupby_agg(func, aggregation, pdf):

assert_cudf_groupby_layers(actual)

# groupby.agg should add an explicit getitem layer
# to improve/enable column projection
assert hlg_layer(actual.dask, "getitem")

dd.assert_eq(expect, actual, check_names=False, check_dtype=check_dtype)


Expand Down

0 comments on commit 5081fb1

Please sign in to comment.