unexpected results from aggregation query

[jangorecki]

Sorry for lack of minimal reproducible example but issues like that are quite time consuming to track down. To produce data use R:

wget https://raw.githubusercontent.com/h2oai/db-benchmark/master/groupby-datagen.R
Rscript groupby-datagen.R 1e7 1e2 0 0

On G1_1e7_1e2_0_0.csv generated from that script pandas returns the following

ans = x.groupby(['id6']).agg({'v1':'sum', 'v2':'sum', 'v3':'sum'})
print(ans.shape, flush=True)
#(100000, 3)
print(ans.head(3), flush=True)
#      v1   v2         v3
#id6                     
#1    266  256  4503.0951
#2    334  316  5294.6590
#3    276  283  5166.5957

while cudf 0.5 from pip returns the following

ans = x.groupby(['id6']).agg({'v1':'sum', 'v2':'sum', 'v3':'sum'})
print(ans.shape, flush=True)
#(99998, 4)
print(ans.head(3), flush=True)
#   id6 sum_v1 sum_v2   sum_v3
#0    1    809    801 13638.96
#1    2    334    316  5294.66
#2    3    276    283 5166.595

Already shape of answer doesn't match. There are no NA/NaN in that data.

[kkraus14]

I'd guess that this is a CSV reader issue, have you checked the pandas dataframe matches the cudf dataframe after the read_csv call?

[jangorecki]

@kkraus14 possible, but from what I was able to check it looks fine.
The most important check to see number of groups in id6 raised error.
Difference in sum(v3) is probably from the fact that pandas store it as float64 while cudf as float32.

• head of data

# pandas
>>> print(x[['id6','v1','v2','v3']].head())
     id6  v1  v2       v3
0  59276   1   1  96.8126
1  78315   4   1  83.5654
2  27300   4   5  44.6464
3  65416   2   3  29.9499
4  19046   4   3  51.4899

# cudf
>>> print(x[['id6','v1','v2','v3']].head())
    id6   v1   v2        v3
0 59276    1    1  96.81261
1 78315    4    1  83.56539
2 27300    4    5   44.6464
3 65416    2    3 29.949902
4 19046    4    3 51.489902

• columns to aggregate

# pandas
>>> x['v1'].sum()
30001016
>>> x['v2'].sum()
29998058
>>> x['v3'].sum()
506570021.4995007

# cudf
>>> x['v1'].sum()
30001016
>>> x['v2'].sum()
29998058
>>> x['v3'].sum()
506570050.0

• columns to group by

# pandas
>>> x['id6'].nunique()
100000

# cudf
>>> x['id6'].unique_count()
Traceback (most recent call last):
  File "/home/jan/git/db-benchmark/cudf/py-cudf/lib/python3.6/site-packages/numba/cuda/cudadrv/nvvm.py", line 111, in __new__
    inst.driver = open_cudalib('nvvm', ccc=True)
  File "/home/jan/git/db-benchmark/cudf/py-cudf/lib/python3.6/site-packages/numba/c
uda/cudadrv/libs.py", line 48, in open_cudalib
    raise OSError('library %s not found' % lib)
OSError: library nvvm not found

[mjsamoht]

This doesn't look like a CSV reader issue. I'm removing the cuIO label but let me know if someone from cuIO team needs to take a look.

[VibhuJawa]

This seems to be resolved in 0.6.1 and 0.7 . Reproduction Gist:
https://gist.github.com/VibhuJawa/d3354dca525e0699d33aa0e286d2a717

[randerzander]

Closing per @VibhuJawa's tests. @jangorecki please re-open if there's still a problem for you.