Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Deprecate ser.str.subword_tokenize API in favor of cudf.core.subword_tokenizer.SubwordTokenizer #8604

Closed
VibhuJawa opened this issue Jun 24, 2021 · 3 comments · Fixed by #9968
Closed

[FEA] Deprecate ser.str.subword_tokenize API in favor of cudf.core.subword_tokenizer.SubwordTokenizer #8604

VibhuJawa opened this issue Jun 24, 2021 · 3 comments · Fixed by #9968
Assignees
@VibhuJawa
Labels
feature request New feature or request Performance Performance related issue Python Affects Python cuDF API.

Comments

@VibhuJawa
Copy link
Member

VibhuJawa commented Jun 24, 2021 •
edited by harrism
Loading

Is your feature request related to a problem? Please describe.
Now that we have added cudf.core.subword_tokenizer.SubwordTokenizer with PR. We should deprecate the ser.str.subword_tokenize API in favor of it in the future release.

Reasons to switch to this API are:

  1. Equivalent to HuggingFace so makes switching easier
  2. Faster (13x+ at smaller batch sizes)
  3. Fixes discrepancies in parameters like stride etc. that can cause difficult bugs to find.

Additional context
Have added PR to provide warnings for users

Also have added a PR to clx to use the new tokenizer. (rapidsai/clx#430 )
@VibhuJawa VibhuJawa added feature request New feature or request Needs Triage Need team to review and classify labels Jun 24, 2021
@VibhuJawa VibhuJawa added Python Affects Python cuDF API. and removed Needs Triage Need team to review and classify labels Jun 24, 2021
@VibhuJawa VibhuJawa self-assigned this Jun 24, 2021
@harrism harrism changed the title [FEA] Depreciate ser.str.subword_tokenize API in favor of cudf.core.subword_tokenizer.SubwordTokenizer [FEA] Deprecate ser.str.subword_tokenize API in favor of cudf.core.subword_tokenizer.SubwordTokenizer Jun 24, 2021
@beckernick beckernick added Performance Performance related issue tech debt labels Jun 28, 2021
@galipremsagar galipremsagar mentioned this issue Jun 28, 2021
Merged
rapids-bot bot pushed a commit that referenced this issue Jul 8, 2021
@VibhuJawa
Add DeprecationWarning to ser.str.subword_tokenize (#8603)
a5427d2 
This PR adds deprecationWarning to `ser.str.subword_tokenize` .

Checkout related issue for details:  #8604

Authors:
  - Vibhu Jawa (https://github.com/VibhuJawa)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #8603
@vyasr
Copy link
Contributor

vyasr commented Sep 21, 2021

@VibhuJawa can we close this issue after #8603?

@VibhuJawa
Copy link
Member Author

@VibhuJawa can we close this issue after #8603?

Nope, we still need to depreciate it. I just added that warning last release so that users have a 2 release window to fix it . Will depreciate next release.

@VibhuJawa VibhuJawa mentioned this issue Sep 30, 2021
Merged
@github-actions
Copy link

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

@VibhuJawa VibhuJawa mentioned this issue Jan 4, 2022
Merged
@rapids-bot rapids-bot bot closed this as completed in #9968 Jan 5, 2022
rapids-bot bot pushed a commit that referenced this issue Jan 5, 2022
@VibhuJawa
Remove str.subword_tokenize (#9968)
3e893a6 
This PR resolves #8604 and #9447

Authors:
  - Vibhu Jawa (https://github.com/VibhuJawa)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)
  - David Wendt (https://github.com/davidwendt)
  - Christopher Harris (https://github.com/cwharris)

URL: #9968
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VibhuJawa VibhuJawa

Labels
feature request New feature or request Performance Performance related issue Python Affects Python cuDF API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[REVIEW]Remove str.subword_tokenize VibhuJawa/cudf
3 participants
@vyasr @VibhuJawa @beckernick