[FEA] Combine categories on merge

[jangorecki]

I would expect to get same output for string and categorical columns used in join

import cudf as cu
df1 = cu.DataFrame({'id1':['a','b','c','b'], 'v1':[1,2,3,2]})
df2 = cu.DataFrame({'id1':['b','d'], 'v2':[2,4]})
ans1 = df1.merge(df2, on='id1')
df1['id1'] = df1['id1'].astype('category')
df2['id1'] = df2['id1'].astype('category')
ans2 = df1.merge(df2, on='id1')
ans1
#  id1  v1  v2
#0   b   2   2
#1   b   2   2
ans2
#  id1  v1  v2
#0   b   1   2
#1   d   2   4
#2   d   2   4
cu.__version__
#'0.10.0'

[kkraus14]

Able to repro on latest 0.12, investigating.

[kkraus14]

@brandon-b-miller I think your join implicit typecasting PR may take care of this? If not I think we should just do the minimal amount of work to throw an exception if the categories aren't synchronized for the time being and then wait for libcudf support for categorical columns.

[brandon-b-miller]

@kkraus14 It doesn't yet, but shouldn't be hard to tweak so that it raises in this case, I agree it doesn't really make sense to merge two categorical columns that specify different categories.

[kkraus14]

I agree it doesn't really make sense to merge two categorical columns that specify different categories.

I actually think it does make sense to merge them, it's just that with categorical columns being implemented in libcudf we shouldn't spend the effort which will just get ripped out in the near future anyway.

[brandon-b-miller]

I think there's only a way of making sense of it in strictly the unordered case. In the ordered case we'd have to somehow determine the ordering for the superset containing all the possible categories.

[kkraus14]

I think there's only a way of making sense of it in strictly the unordered case. In the ordered case we'd have to somehow determine the ordering for the superset containing all the possible categories.

Yea, good point. Pandas implicitly converts categoricals to object dtype when trying to run a merge against them so we have flexibility to implement our own logic here. Alternatively we do have the option of materializing the categorical encoded column to the dictionary's dtype, but that feels very unnecessary and expensive.

[kkraus14]

@brandon-b-miller is this still an issue after the porting?

[brandon-b-miller]

This should raise for the any code that includes the implicit typecasting.

TypeError: Left and right categories must be the same.

We should be seeing that instead of incorrect results.

[jangorecki]

Error would be definitely better than current expected results. Yet I think it make sense to keep this issue as FR to support categorical variables (when their categories does not match). Join might be actually faster, especially when there are very few categories and merging categories would be cheap.

[kkraus14]

Error would be definitely better than current expected results. Yet I think it make sense to keep this issue as FR to support categorical variables (when their categories does not match). Join might be actually faster, especially when there are very few categories and merging categories would be cheap.

issue updated to reflect the feature request