Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pr 33 (Feature/performance: This PR introduces a high number of performance improvements) #36

Merged
merged 48 commits into from
Mar 26, 2024
Merged

Pr 33 (Feature/performance: This PR introduces a high number of performance improvements) #36

merged 48 commits into from
Mar 26, 2024

Conversation

dmitry-brazhenko
Copy link
Owner

@dmitry-brazhenko dmitry-brazhenko commented Mar 26, 2024 •
edited
Loading

Initial PR: #33
@r-Larch implemented significant speed improvement for the lib.

I applied minor changes and now merging it

r-Larch added 30 commits March 22, 2024 20:20
@r-Larch
feat(performance): add benchmark project
02aa357
@r-Larch
feat(performance): reduce public API surface
aa4cf54
@r-Larch
feat(performance): reduce allocations
e9fdce3
@r-Larch
feat(performance): use compiled regex for better performance
3d8565c
@r-Larch
feat(performance): run benchmark
659f450
@r-Larch
chore: move reorganize files
076843f
@r-Larch
feat(performance): replace SpecialTokenPatternRegex with faster alloc…
7a5c48f 
… free solution
@r-Larch
feat(performance): reduce string allocations
87a19d6
@r-Larch
feat(performance): run benchmark
f2f739f
@r-Larch
feat(performance): add BytePairIndex class to support faster implemen…
a76a9cf 
…tation in net8.0
@r-Larch
feat(performance): cache model parameters to do params preparation on…
c27a616 
…ly once
@r-Larch
feat(performance): run benchmark
098524b
@r-Larch
feat(performance): improve allowedSpecialTokens handling + bug fix
32b95a8
@r-Larch
feat(performance): add support for ReadOnlySpan<char> in net8.0
d29f88a
@r-Larch
feat(performance): run benchmark
390a693
@r-Larch
feat(performance): use compile time generated regex in net8.0
b396e64
@r-Larch
feat: run tests for netstandard2.0 over net471
3a40044
@r-Larch
feat(performance): run benchmark
cdcc0b3
@r-Larch
feat: remove unused lastTokenLength + refactoring
71a2661
@r-Larch
feat(performance): implement fast MultiBytePairEncoder with almost ze…
9ae4222 
…ro allocations
@r-Larch
feat(performance): run benchmark
1856bc8
@r-Larch
feat(performance): reduce allocations
3ab2251
@r-Larch
feat(performance): backport some optimizations to net6.0 and netstandard
2bd47c3
@r-Larch
feat(performance): run benchmark
181cf52
@r-Larch
chore: cosmetics
8215e50
@r-Larch
feat(refactor): extract Encoding.Encode() logic into priv EncodeCore …
2146f79 
…to support broader use
@r-Larch
feat(token-count): implement low allocation token count public method
4f8df6e
@r-Larch
feat(performance): run benchmark
0c8cc8d
@r-Larch
feat(benchmark): don't make allocations in benchmark methods
a4a0203
@r-Larch
feat(performance): reduce minor allocations
ec640b1
r-Larch and others added 18 commits March 23, 2024 17:44
@r-Larch
feat(benchmark): add benchmark for large file token count
f30608b
@r-Larch
feat(performance): run benchmark
2b97c3d
@r-Larch
fix: add test for allowedSpecialTokens and fix code
2962218
@r-Larch
chore: fix naming
3a1523e
@r-Larch
feat(benchmark): add another benchmark to show in README.md
07586b0
@r-Larch
feat(readme): add benchmark to README.md and add docs for TokenCount …
1713302 
…method
@r-Larch
feat(performance): use multibyte cpu instructions for FastPartitionLi…
e0b6410 
…st.RemoveAt
@r-Larch
feat(performance): re-add fast path - got lost in refactoring
4c33ca2
@r-Larch
feat(benchmark): add comparison to other tokenizer
7ab1b9d
@r-Larch
feat(performance): improve ByteArrayEqualityComparer
964c571
@r-Larch
feat(performance): small improvement
584ee05
@r-Larch
chore: update benchmarks in README.md
0917f5a
@dmitry-brazhenko
Merge branch 'main' into pr-33
2b187f2 
# Conflicts:
#	SharpToken.Tests/SharpToken.Tests.cs
#	SharpToken.Tests/SharpToken.Tests.csproj
#	SharpToken.Tests/TestHelpers.cs
@dmitry-brazhenko
fix
836264a
@dmitry-brazhenko
improved pipleline
0c0e505
@dmitry-brazhenko
switch (key)
ab3b321
@dmitry-brazhenko
i += size - 1;
386ca41
@dmitry-brazhenko
CountTokens renamed
f64efd8
@dmitry-brazhenko dmitry-brazhenko changed the title Pr 33 Pr 33 (Feature/performance: This PR introduces a high number of performance improvements) Mar 26, 2024
@dmitry-brazhenko dmitry-brazhenko mentioned this pull request Mar 26, 2024
Closed
@dmitry-brazhenko dmitry-brazhenko merged commit 086544d into main Mar 26, 2024
4 checks passed
@dmitry-brazhenko dmitry-brazhenko deleted the pr-33 branch March 26, 2024 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dmitry-brazhenko @r-Larch