Add JNI for strings::code_points #14533
Conversation
Signed-off-by: Haoyang Li <[email protected]>
davidwendt
added
3 - Ready for Review
|
/ok to test |
| @@ -373,6 +373,16 @@ public final ColumnVector getByteCount() { | |||
| return new ColumnVector(byteCount(getNativeView())); | |||
| } | |||
|
|
|||
| /** | |||
| * Get the code point values (integers) for each character of each string. | |||
There was a problem hiding this comment.
This API seems very problematic in light of the effort to move to large strings. A strings column will soon support more than 2^31 characters. Calling this API on such a column will crash since it cannot manifest an INT32 column with more than 2^31 entries.
It also seems problematic from a usability point of view. Since it returns only a column of INT32 instead of LIST(INT32), it's not straightforward to figure out where the code points of one string stops and another starts. We can't use the offset column of the original string, since that's byte offsets instead of character offsets. I guess one would need to get the character lengths of the original string (converting nulls to zereoes) and then do a prefix scan to compute the code point offsets to know where one string's codepoints are in the result.
It also seems very wasteful for what NVIDIA/spark-rapids#9585 needs if called directly, since it will explode the memory of many string columns by 4X. We should first slice the original string column to only select the first character of each string. That would work around the large strings issue, the "where does a string start" issue, as well as the waste, since we only need the codepoint of the first character for that Spark feature.
There was a problem hiding this comment.
Thanks for the review and analysis! I'm trying the 'only select the first character' way in the plugin.
Another problem with the spark issue is that the results of Latin-1 Supplement chars are mismatched between spark and code_points. For example é is 50089 for code_points and utf-8, and 233 for spark and Unicode (and Latin-1 and utf-16?), I'm trying to work around it but it is possible that we need a custom kernel for ascii.
Co-authored-by: Nghia Truong <[email protected]>
|
/merge |
|
/ok to test |
Signed-off-by: Haoyang Li <[email protected]>
|
/ok to test |
Description
This implements JNI work for strings::code_points to expose the API to Java usage.
It will be useful for NVIDIA/spark-rapids#9585
Checklist