Skip to content

Issues: karpathy/minbpe

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

30 Open 8 Closed
30 Open 8 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

One problem in the annotations of test_wikipedia_example in the tests/test_tokenizer file
#93 opened Nov 18, 2024 by donglinkang2021
LLM is worse at non-English languages
#92 opened Nov 10, 2024 by 7CD
Question about Encoder Logic
#87 opened Jul 20, 2024 by JackxTong
5
Python API with C extensions for faster training and encoding
#85 opened Jun 27, 2024 by benarnav
LLM as calc
#81 opened Jun 6, 2024 by michaelshekasta
OSS-Fuzz Integration
#80 opened May 30, 2024 by ennamarie19
BPE in Haskell
#79 opened May 24, 2024 by BobMcDear
What to support GPT-4O tokenizer？
#77 opened May 15, 2024 by echo-valor
Notebook Issue In Google Colab
#74 opened May 13, 2024 by kelixirr
The regular expressions break all scripts with combining marks in the middle of the syllable
#73 opened May 12, 2024 by ajaykg
3
Amplifying your courses with my digital notes
#70 opened Apr 30, 2024 by AayushSameerShah
Instead of finding the one pair with the highest frequency and merging it at each step, do the highest N pairs
#69 opened Apr 23, 2024 by hippietrail
2
minbpe-rs: A pure Rust implementation of minbpe
#66 opened Apr 21, 2024 by shubham0204
2
decode() method in GPT4Tokenizer does not handle special tokens
#64 opened Apr 7, 2024 by Vakarva
Would using prompts that contain concatenated words to reduce token count negatively affect results
#61 opened Mar 28, 2024 by hatgit
Implementation of LlamaTokenizer (without sentencepiece)
#60 opened Mar 26, 2024 by MaveriQ
"regex.py" file name conflict
#59 opened Mar 26, 2024 by mogomaa79
Huggingface already has an efficient implementation of this?
#58 opened Mar 19, 2024 by laurislopata
3
Using minBPE token encoded sentence vectors need to be padded
#56 opened Mar 19, 2024 by elevateclub
1
counting pairs is inaccurate for repeating tokens?
#51 opened Feb 28, 2024 by JohannesVod
5
Alternative to bpe
#50 opened Feb 28, 2024 by marcov-dart
16
A thanks from self-learners community
#45 opened Feb 24, 2024 by IamExperimenting
how to deal with special tokens for multiple files
#44 opened Feb 24, 2024 by IamExperimenting
Byte2Byte Tokenizer
#37 opened Feb 21, 2024 by loretoparisi
Alternative to „Vector Representation Pre-training“ possible?
#30 opened Feb 21, 2024 by Heavy02011
1
ProTip! Type g i on any issue or pull request to go back to the issue listing page.