[BUG] Groupby Mean Performance Regression

[quasiben]

There may be a performance regression when running groupby mean in cudf when executing with many columns. When cuDF performs multiple aggregations, it should be more performant to do this with the agg operator rather than each aggregation expressed individually. This is not the case the reproducible below (note this is built with 260 columns and 2592001 row)

In [1]: import cudf

In [2]: alphabets = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']

In [3]: prefixes = alphabets[:10]

In [4]: coll_dict = dict()
   ...: for prefix in prefixes:
   ...:     for this_name in alphabets:
   ...:         coll_dict[prefix + this_name] = float
   ...:

In [5]: coll_dict['id'] = int

In [6]: cdf = cudf.datasets.timeseries(start='2000',
   ...:                      end='2000-01-31',
   ...:                      dtypes=coll_dict,
   ...:                      freq='1s'
   ...:                      seed=1,
   ...:                     ).reset_index(drop=True)

In [7]: %timeit _ = cdf.groupby('id').agg(['min', 'max', 'count', 'mean'])
3.5 s ± 12.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [8]: %timeit _ = (cdf.groupby('id').min(), cdf.groupby('id').max(), cdf.groupby('id').count(), cdf.groupby('id').mean())
1.99 s ± 42.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [9]: %timeit _ = cdf.groupby('id').mean()
1.18 s ± 7.39 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

cc @pentschev @zronaghi @shwina @kkraus14

I'm collecting NVTX info now in hopes that this helps

[jrhemstad]

From the profile @quasiben shared, it shows how the MEAN aggregation is taking much longer than the others. Based on the kernel activity in the MEAN aggregation it is taking the sort-based path, which explains why it is much slower.

@devavret confirmed that currently calling cudf::groupby with MEAN results in using the sort-based path.

I believe this explains the performance regression as we used to compute the MEAN aggregation using the hash-based path. This is still possible today, it just wasn't added when groupby was ported to libcudf++.

[karthikeyann]

With PR #6392

In [10]: %timeit _ = cdf.groupby('id').agg(['min', 'max', 'count', 'mean'])
886 ms ± 34.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [11]: %timeit _ = (cdf.groupby('id').min(), cdf.groupby('id').max(), cdf.groupby('id').count(), cdf.groupby('id').mean())
909 ms ± 11.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[jrhemstad]

@karthikeyann what are all those tiny ranges before and after the cudaStreamSynchronize?

[karthikeyann]

before & after first cudaStreamSynchronize are make_fixed_width_column calls. (for columns pushed into sparse_cache).
After the last cudaStreamSynchronize are null_count calls.
Red ones are cudaFree calls.

Added a TODO for make_fixed_width_column calls as room for improvement in comment now.
It could use similar approach as PR #6605 and PR #6615