Avoid overflow

[matthieugomez]

Correct the following bug :

df = DataFrame(v1 = pool(1:1000), v2 = pool(fill(1, 1000))) 
groupby(df, [:v1, :v2])

(initially part of pull request #815)

[alyst]

That wouldn't fully resolve the overflow problem. If the frame is grouped by, say, 4 columns with >2^16 unique elements each (or 8 columns with >2^8 uniques), UInt64 would be overflown as well.
One possibility is to periodically call PooledDataArray(x) inside the columns loop to "compactify" the group indices. I've tried implementing something along these lines, but abandoned the idea, because it's just an ugly way to reinvent hashing.

An alternative solution would be to use Dict with DataFrameRow that already has contents-based hash() and isequal(). It's implemented in #850 for both grouping and joining. It would be nice, if somebody reviews it. :)

[matthieugomez]

What is panda / dplyr / data table doing ? Surely they solved this.
Anyway maybe the best thing for now would be to return an error if this happens ? This fix should still cover most of the bad situations.

[alyst]

dplyr has a lot of C++/Rcpp code (however, it also offers more, e.g. joining on columns with different types), but AFAICS in the core it uses dplyr_hash_map, so it's closer to what #850 proposes. Don't know about panda.

PooledDataArray optimization in #850 is not a big deal, it's just about avoiding setindex!() operation that is very slow. As I state there, it's more about maintaining left frame rows order and not doing unnecessary indexing for certain joins.

[alyst]

Throwing something more informative than current InexactError (also, for some reason, it has no stacktrace) would be a nice quickfix.
The problem is that even with UInt64 conversion the overflow would happen quite often even for medium-sized frames and reasonable number of join/group columns. For the user the workaround would be to concatenate "on"/"by" columns and avoid using multicolumn joins/groups, but that is rather ugly.

[matthieugomez]

Ok. Your fix looks better. Could you add the groupby test from my pull request in yours to make sure this problem is not rediscovered at some point?
Out of curiosity, could you also add a timing for simple grouping operations?