[FEA] libcudf string split returning list of strings

[jlowe]

Is your feature request related to a problem? Please describe.
Spark supports a string split function that takes a string and delimiter and returns a list of strings. We would like to support this operation in the RAPIDS Accelerator for Apache Spark.

Describe the solution you'd like
A form of libcudf's cudf::strings::split that instead of returning separate columns for the fields it returns a single, list-of-strings column where each row contains a list of all the fields produced by the split. Ideally the split function would support a regular expression to identify the delimiter, but we can still support many common queries with a function that only allows an exact-match, scalar delimiter string.

Describe alternatives you've considered
Theoretically we could work with the existing cudf::strings::split but trying to map multiple columns to Spark's ArrayType is messy in practice and would be specific to this operation. It's not inline with the straightforward mapping of ArrayType to the new list type currently being added to libcudf.

[kkraus14]

Pandas similarly supports this if expand=False: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.split.html so we'd want this for cuDF Python as well.

[davidwendt]

Take a look at cudf::strings::contiguous_split_record which is intended to support the expand=False scenario.
It returns a column_view of split fields for each input string. Instead of individual columns, a single memory block holds the entire split result with the column_views identifying the results for each row.

[beckernick]

Perhaps cuDF Python may want to also add list support to the output of a few other string methods, too (find_multiple (#4569), extract, and findall)

[kkraus14]

extract / extractall both always return a DataFrame with separate columns per capture group, so no need for list support there.

findall will need list support

@galipremsagar I remember we did something in porting the nvstrings find_multiple to merge it into another API, do you remember?

[galipremsagar]

@galipremsagar I remember we did something in porting the nvstrings find_multiple to merge it into another API, do you remember?

We have a cython plumbing only for find_multiple: cudf._lib.strings.find_multiple.find_multiple: https://github.com/rapidsai/cudf/blob/branch-0.15/python/cudf/cudf/_lib/strings/find_multiple.pyx#L14

So we did the merging of API for replace which supports scalar & list-like inputs: https://docs.rapids.ai/api/cudf/nightly/api.html#cudf.core.column.string.StringMethods.replace

[davidwendt]

The libcudf part of this was completed in PR #5687
I've left this open for any required cudf Python/Cython plumbing.

[kkraus14]

This was handled in Python / Cython as well. Closing.