byte_range support for multibyte_split/read_text #10150
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cwharris
requested review from
karthikeyann,
davidwendt,
galipremsagar and
charlesbluca
github-actions
bot
added
Python
cwharris
added
3 - Ready for Review
davidwendt
reviewed
Feb 28, 2022
davidwendt
approved these changes
Feb 28, 2022
robertmaynard
approved these changes
Feb 28, 2022
vyasr
requested changes
Feb 28, 2022
vyasr
reviewed
Feb 28, 2022
vyasr
approved these changes
Mar 1, 2022
cpp/src/io/text/multibyte_split.cu
Outdated
| return static_cast<int32_t>(offset - relevant_offset_first); | ||
| }); | ||
|
|
||
| stream.synchronize(); |
There was a problem hiding this comment.
I'm confused, did you mean to delete this?
|
@gpucibot merge |
This was referenced Mar 1, 2022
rapids-bot bot
pushed a commit
that referenced
this pull request
Mar 2, 2022
#10150 broke compiler support for GCC 11 (built locally) because it was missing `#include <optional>` in a couple files. This fixes it. cc: @cwharris Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Karthikeyan (https://github.com/karthikeyann) URL: #10385
rapids-bot bot
pushed a commit
that referenced
this pull request
Mar 22, 2022
This uses the `byte_range` argument added to read_text with #10150 to create a dask version of `read_text` Authors: - https://github.com/ChrisJar Approvers: - Benjamin Zaitlen (https://github.com/quasiben) URL: #10407
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding byte_range support to multibyte_split/read_text.
Closes #9655
providing a byte range in terms of
(offset, size)allows multibyte_split to read a whole file, but only return the offsets within those ranges as well as one additional offset (unless it's the end of the file). If thinking in terms of "records", where each delimiter dictates the end of a record, we effectively return all records which begin within the byte range provided, and ignore all other records, including any record which may end (but not begin) within the range, and including any record which may begin in the range but end outside of the range.examples: