[BUG] "RuntimeError: Total number of concatenated rows exceeds size_type range" for 2.2 million row series

[marco-ve]

Describe the bug
A cudf.Series containing lists will throw a RuntimeError if one tries to access a certain number of rows, depending on the length of the contained lists.

Steps/Code to reproduce bug

gsr = cudf.Series([arr for arr in np.random.uniform(low=-0.2, high=0.2, size=(2000000,512))])
gsr

This will display just fine (2 million rows).

gsr = cudf.Series([arr for arr in np.random.uniform(low=-0.2, high=0.2, size=(2200000,512))])
gsr

2.2 million rows. This will throw a "RuntimeError: cuDF failure at: ../src/copying/concatenate.cu:359: Total number of concatenated rows exceeds size_type range". However, accesssing an element directly by index will work fine. Also, accessing elements [0:60] will work, but [0:61] will throw the same exception.

gsr = cudf.Series([arr for arr in np.random.uniform(low=-0.2, high=0.2, size=(2200000,256))])
gsr

This will display just fine (2.2 million rows, but shorter lists).

Expected behavior
Expected behavior would be for the series to access rows without throwing an error, independent of the dimensions of the object contained, same as in e.g. a pandas series.

Environment overview (please complete the following information)
Environment is a jupyterlab notebook hosted in the rapidsai docker container, using an RTX3090 with 24GB VRAM. cuml version is 21.06.01+2.g101fc0fda4.

[shwina]

List columns are limited by number of elements (not rows). The total number of elements in a list column cannot exceed INT32_MAX. We should raise a better error message here.

Internally, a list column holds a column composed of all its elements, i.e., the following list column:

[1, 2, 3]
[4, 5]
[6, None, 8]

internally holds the column [1, 2, 3, 4, 5, 6, None, 8]. The size of that column cannot exceed INT32_MAX.

[marco-ve]

Thanks for the clarification. But isn't the max value of an int32 around 2 billion, whereas 2,200,000 * 512 is only 1.1 billion?

Also why does accessing rows [0:60] not throw an error, but [0:61] does?

[shwina]

Thanks -- you're right, that is a bug. Investigating.

[shwina]

Slightly simpler, more revealing reproducer (although I still haven't root-caused this):

s = cudf.Series([arr for arr in np.ones((2_200_000, 512))]) # a list column of 2_200_000 rows and 512 elements per row
top = s.iloc[:31] # slice of first 31 rows
bottom = s.iloc[31:62]  # slice of rows 31-62
cudf.concat([top, bottom])  # errors
cudf.concat([top.copy(), bottom.copy()])  # deep copy before concat, no error

cc: @nvdbaranec