Update @by/@based_on macros to avoid constructing DataFrames

[jacobadenbaum]

Currently, both the @by and the @based_on macros construct DataFrames on each anonymous function call. When aggregating over a large number of groups, this can become very costly. This PR changes the helper functions to return NamedTuples instead.

Before this PR:

julia> using DataFrames, Random, DataFramesMeta
julia> Random.seed!(1234);
julia> df = DataFrame(id = repeat(1:1_000_000, 10), X = randn(10_000_000), Y = randn(10_000_000));
julia> @btime @by($df, :id, X = mean(:X), Y = mean(:Y));
  8.706 s (73999190 allocations: 4.39 GiB)

After:

julia> using DataFrames, Random, DataFramesMeta
julia> Random.seed!(1234);
julia> df = DataFrame(id = repeat(1:1_000_000, 10), X = randn(10_000_000), Y = randn(10_000_000));
julia> @btime @by($df, :id, X = mean(:X), Y = mean(:Y));
  952.237 ms (16999142 allocations: 936.71 MiB)

[tshort]

Good idea.

[nalimilan]

Thanks, but I think it would be even faster to use the new by API from JuliaData/DataFrames.jl#1601, which ensures full type stability (instead of passing a SubDataFrame). Maybe you can take inspiration from transform_helper?

(I also have a branch locally to transform single-argument calls like sum(x) into just sum so that optimized algorithm for reductions can be used, but I initially wanted to use them with transform, which doesn't support them yet. I can push the branch if you want to experiment with reusing that code for @by.)

[nalimilan]

AFAICT it would be more efficient to use combine instead of DataFrame(map(...)).

[jacobadenbaum]

If you wouldn't mind pushing the branch, that would be helpful.

[nalimilan]

I've pushed the nl/transform branch. It's not correct (since it uses combine inside transform despite these two functions having different behaviors in some cases), but the helpers machinery worked IIRC.

[pdeffebach]

Closing this as all the @by calls are based on DataFrames.combine now.