Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Add explicit errors for BPE and unigram, return tokenizer without cha… #54

Merged
merged 6 commits into from
Oct 3, 2024
Merged

fix: Add explicit errors for BPE and unigram, return tokenizer without cha… #54

merged 6 commits into from
Oct 3, 2024

Conversation

stephantul
Copy link
Member

…nges if tokens_to_remove is empty

@stephantul stephantul self-assigned this Oct 3, 2024
@stephantul stephantul requested a review from Pringled October 3, 2024 04:55
@stephantul stephantul mentioned this pull request Oct 3, 2024
Closed
@codecov Codecov
Copy link

codecov bot commented Oct 3, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 56.09756% with 18 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
model2vec/distill/tokenizer.py 54.05% 17 Missing ⚠️
model2vec/distill/distillation.py 75.00% 1 Missing ⚠️
Files with missing lines Coverage Δ
model2vec/distill/distillation.py 96.29% <75.00%> (-1.14%) ⬇️
model2vec/distill/tokenizer.py 71.18% <54.05%> (-25.88%) ⬇️

@stephantul stephantul changed the title Add explicit errors for BPE and unigram, return tokenizer without cha… fix: Add explicit errors for BPE and unigram, return tokenizer without cha… Oct 3, 2024
@stephantul stephantul changed the title fix: Add explicit errors for BPE and unigram, return tokenizer without cha… fix: Add explicit errors for BPE and unigram, return tokenizer without cha… Oct 3, 2024
@stephantul stephantul merged commit c34d862 into main Oct 3, 2024
2 of 3 checks passed
@stephantul stephantul deleted the unigram_bpe_support branch October 3, 2024 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Pringled Pringled Pringled approved these changes

Assignees

@stephantul stephantul

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@stephantul @Pringled