Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor to byte[] version #7

Merged
merged 68 commits into from
Oct 11, 2023
Merged

Refactor to byte[] version #7

merged 68 commits into from
Oct 11, 2023

Conversation

kortemik
Copy link
Member

No description provided.

MoonBow-1 and others added 30 commits September 12, 2023 15:55
@MoonBow-1
use MajorTokenStream class to collect major tokens
fddc145
@MoonBow-1
add Token class
ee5c578
@MoonBow-1
Find permutations inside Token class
11cd04a
@MoonBow-1
find permutations when splitter is at the edge of input string
2701d24
@MoonBow-1
Use a Set of splitter indexes to find permutations
cb146a3
@MoonBow-1
handle major token splitters at the edges
dcc92c7
@MoonBow-1
Add performance test
67929cb
@MoonBow-1
add Stream with mark and reset methods
2984604
@MoonBow-1
remove code from Token constructor use matcher
4f13cd7
@MoonBow-1
use compiled regex matcher
d735f85
@MoonBow-1
start work on TokenizeFunction
d3f08f4
@MoonBow-1
TokenizeFunction finished and more testing
413d9fe
@MoonBow-1
Token class minor token permutations
2334622
@kortemik
starts
9f17a44
@kortemik
sync
41e92e3
@kortemik
fast
8ebeb4e
@kortemik
perhaps run
4a84d25
@kortemik
before bytes
aa52782
@kortemik
after bytes
0c49830
@kortemik
some comments
87e892d
@kortemik
sout out from test
dfc4c88
@kortemik
last one did not get caught, testcased and fixed
3816d7d
@kortemik
enhance test casing and fix found bugs
b758f03
@kortemik
add info to tokens that are they delimiters or not
08d0db6
@kortemik
more fixes
c9e1407
@kortemik
fix Entaglement plz
13b4dac
@kortemik
fix testConcatenation
27d9329
@kortemik
test pass
44e0cce
@kortemik
rat shall pass
1b9e426
@kortemik
remove deprecated methods
1d7fa89
StrongestNumber9 and others added 18 commits October 9, 2023 18:13
@StrongestNumber9
position fix
873257b
@kortemik
Merge pull request #3 from StrongestNumber9/refactor-3
85e9e1a 
Fixes extendBuffer
@StrongestNumber9
Refactor delimiters
410fdd3
@StrongestNumber9
Optimize delimiterSets a bit
914cb00
@StrongestNumber9
Fixes FIXME for max size
6c6a4df
@kortemik
Merge pull request #4 from StrongestNumber9/refactor_delimiters
0ee2979 
Refactor delimiters
@StrongestNumber9
Optimizations
4b674ff
@kortemik
Merge pull request #5 from StrongestNumber9/optimizations
9eeb0c4 
Optimizations
@MoonBow-1
add Tokenizer
71524a9
@StrongestNumber9
More optimizations
e6fda21
@kortemik
allocate more for concatenation buffer if necessary
be29b73
@kortemik
remove fixme from ConcatenatedToken
afc1be7
@kortemik
fix condition for extending the buffer
f97ae2f
@MoonBow-1
use class members and return list
b99ad15
@MoonBow-1
Tokenizer returns list of byte[]
0102387
@MoonBow-1
Tokenizer returns list of Tokens
8451106
@kortemik
Merge pull request #6 from elliVM/refactor-1
c8529db 
add Tokenizer
@kortemik
Merge pull request #7 from StrongestNumber9/optimizations-2
03041aa 
More optimizations
@CLAassistant
Copy link

CLAassistant commented Oct 10, 2023 •
edited
Loading

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ kortemik
✅ StrongestNumber9
❌ MoonBow-1
You have signed the CLA already but the status is still pending? Let us recheck it.

StrongestNumber9
StrongestNumber9 approved these changes Oct 11, 2023
View reviewed changes
Copy link
Contributor

@StrongestNumber9 StrongestNumber9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

StrongestNumber9
StrongestNumber9 approved these changes Oct 11, 2023
View reviewed changes
Copy link
Contributor

@StrongestNumber9 StrongestNumber9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM#2

@kortemik kortemik merged commit 524d517 into teragrep:main Oct 11, 2023
@elliVM elliVM mentioned this pull request Oct 11, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@StrongestNumber9 StrongestNumber9 StrongestNumber9 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kortemik @CLAassistant @StrongestNumber9 @MoonBow-1