[FEA] Limitation on amount of Strings that can be loaded into the memory

[mlahir1]

It seems like there is a limitation of around 2B total chars that can be present in a string column even though we don't max out on the GPU memory.

Here is a repro for it.

import cudf
import cupy as cp
df =cudf.DataFrame()
df['a'] = cp.random.randint(0,999,500_000_000)
df = df.to_pandas()
df['b'] = 'Hi'
df['c'] = df.b+df.a.astype('str')
df.to_parquet('test-str.pq')

import cudf
df = cudf.read_parquet('test-str.pq', columns=['a'])

Reading the int64 col with 0.5B rows works just fine. Memory print: (8bytes x 5,000,000,000)

But when i try to read the same number of columns with string of max 5 chars each (max of 2.5B chars) Memory print: (5 bytes x 5,000,000,000) which is less than above

import cudf
df = cudf.read_parquet('test-str.pq', columns=['c'])
df.dtypes, df.head()

cudf throws me an exception.
RuntimeError: cuDF failure at: /workspace/.conda-bld/work/cpp/include/cudf/strings/detail/strings_column_factories.cuh:86: total size of strings is too large for cudf column

[vyasr]

This is expected. A strings column cannot contain more than INT_MAX=2^31 characters. See #3958 for a longer discussion. The usual solution for this problem in cudf is to use dask, which will handle chunking that column into multiple pieces for you.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.