Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use offsetalator in nvtext tokenize functions #14783

Merged
merged 9 commits into from
Feb 6, 2024
Merged

Use offsetalator in nvtext tokenize functions #14783

merged 9 commits into from
Feb 6, 2024

Conversation

davidwendt
Copy link
Contributor

@davidwendt davidwendt commented Jan 18, 2024 •
edited
Loading

Description

Adds offsetalator in place of hardcoded offset type arrays to the strings split functions:

  • nvtext::tokenize()
  • nvtext::count_tokens()
  • nvtext::character_tokenize()
  • nvtext::ngrams_tokenize()
  • nvtext::tokenize_with_vocabulary()

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@davidwendt davidwendt added 2 - In Progress Currently a work in progress libcudf Affects libcudf (C++/CUDA) code. improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jan 18, 2024
@davidwendt davidwendt self-assigned this Jan 18, 2024
@GregoryKimball GregoryKimball mentioned this pull request Jan 23, 2024
Closed
@davidwendt davidwendt changed the base branch from branch-24.02 to branch-24.04 January 23, 2024 19:25
@davidwendt davidwendt added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Jan 30, 2024
@davidwendt davidwendt marked this pull request as ready for review January 30, 2024 20:48
@davidwendt davidwendt requested a review from a team as a code owner January 30, 2024 20:48
bdice
bdice approved these changes Feb 1, 2024
View reviewed changes
cpp/src/text/ngrams_tokenize.cu Show resolved Hide resolved
@davidwendt
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 72ecbe9 into rapidsai:branch-24.04 Feb 6, 2024
69 checks passed
@davidwendt davidwendt deleted the tokenize-offsetalator branch February 6, 2024 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bdice bdice bdice approved these changes

@karthikeyann karthikeyann karthikeyann approved these changes

@mythrocks mythrocks Awaiting requested review from mythrocks mythrocks is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

@nvdbaranec nvdbaranec Awaiting requested review from nvdbaranec nvdbaranec is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@davidwendt davidwendt

Labels
3 - Ready for Review Ready for review by team improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@davidwendt @bdice @karthikeyann