You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the tokenizer api generates a lot of allocations.
For every Text encountered text_analyzer::token_stream() is called
implTextAnalyzer{/// Creates a token stream for a given `str`.pubfntoken_stream<'a>(&self,text:&'a str) -> BoxTokenStream<'a>{self.tokenizer.box_token_stream(text)}}
This PR #2062 fixes this mostly.
Only allocation is now the BoxTokenStream per text, which could be avoided with some lifetime hacks (and unsafe probably).
It would be nice to remove the BoxTokenStream allocation per text and use the Tokenizer directly. e.g. set_text on the Tokenizer and then get the tokens from Tokenizer directly
Currently the tokenizer api generates a lot of allocations.
For every Text encountered
text_analyzer::token_stream()
is calledA boxed token stream typically creates a Token:
The text was updated successfully, but these errors were encountered: