Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Readme Tokenizer Table #55

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Update Readme Tokenizer Table #55

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
be9307e
Update Readme Tokenizer Table
pandora-s-git Sep 21, 2024
bb7b29f
Update README.md
pandora-s-git Nov 6, 2024
bd599ca
Update README.md
pandora-s-git Nov 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,34 +11,45 @@ We are releasing three versions of our tokenizer powering different sets of mode
<tr>
<td>

| Open Model | Tokenizer |
| Weights Available Models | Tokenizer |
|------------|-----------|
| Mistral 7B v0.1 | v1 |
| Mistral 7B v0.2 | v1 |
| Mistral 7B v0.3 | v3 |
| Mistral 7B Instruct v0.1 | v1 |
| Mistral 7B Instruct v0.2 | v1 |
| Mistral 7B Instruct v0.3 | v3 |
| Mixtral 8x7B v0.1 | v1 |
| Mixtral 8x7B Instruct v0.1 | v1 |
| Mixtral 8x22B v0.1 | v1 |
| Mixtral 8x22B Instruct v0.1 | v3 |
| Mixtral 8x22B Instruct v0.3 | v3 |
| Codestral 22B v0.1 | v3 |
| Codestral Mamba 7B v0.1 | v3 |
| Mathstral 7B v0.1 | v3 |
| Nemo 12B 2407 | v3 - Tekken |
| Large 123B 2407 | v3 |
| Nemo 12B Instruct 2407 | v3 - Tekken |
| Large 123B Instruct 2407 | v3 |
| Small 22B Instruct 2409 | v3 |
| Pixtral 12B 2409 | v3 - Tekken |
| Ministral 8B Instruct 2410 | v3 - Tekken |

</td>
<td>

| Endpoint Model | Tokenizer |
| Endpoint Models | Tokenizer |
|---------------|-----------|
| mistral-embed | v1 |
| open-mistral-7b | v3 |
| open-mixtral-8x7b | v1 |
| open-mixtral-8x22b | v3 |
| mistral-small-latest | v2 |
| mistral-small-latest | v3 |
| mistral-large-latest | v3 |
| codestral-22b | v3 |
| open-codestral-mamba | v3 |
| open-mistral-nemo | v3 - Tekken |
| pixtral-12b-2409 | v3 - Tekken |
| ministral-8b-latest | v3 - Tekken |

</td>
</tr>
Expand Down