Large strings support in cudf::concatenate #15195
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidwendt
added
2 - In Progress
davidwendt
added
3 - Ready for Review
3 tasks
bdice
reviewed
Apr 1, 2024
| @@ -226,6 +197,53 @@ TEST_F(StringColumnTest, ConcatenateTooLarge) | |||
| EXPECT_THROW(cudf::concatenate(input_cols), std::overflow_error); | |||
| } | |||
|
|
|||
| TEST_F(StringColumnTest, ConcatenateLargeStrings) | |||
| { | |||
| CUDF_TEST_ENABLE_LARGE_STRINGS(); | |||
There was a problem hiding this comment.
What's the long term plan for CUDF_TEST_ENABLE_LARGE_STRINGS? Will we need this forever or is it only until we turn on long strings by default?
There was a problem hiding this comment.
This will be temporary until long strings are turned on by default.
It will be useful until then in case specific tests require them.
Also, a future PR will introduce a new test fixture where this feature is turned on by default.
And then this test may be moved to use that fixture at that time.
hyperbolic2346
requested changes
Apr 2, 2024
bdice
approved these changes
Apr 3, 2024
There was a problem hiding this comment.
Beautiful. Also I would echo @hyperbolic2346's comment, I appreciate the work that is going into making this project digestible in small PRs.
hyperbolic2346
approved these changes
Apr 3, 2024
|
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Enables
cudf::concatenateto create and return a large strings column (offsets are INT64).This also introduces the
LIBCUDF_LARGE_STRINGS_ENABLEDenvironment variable and utilities around it.One internal utility checks the value so appropriate logic can either throw an overflow exception or build INT64 offsets as appropriate.
The
cudf::test::large_strings_enableris introduced to set/unset the env var for individual tests are needed.A follow on PR will attempt to consolidate these kinds of tests with a specialized test fixture using this utility class.
Checklist