External Tokenizers/Pre-tokenized text

[acertain]

It'd be nice to have a easy way to load pre-tokenized text with (optional?) offset data.

e.g. solr has PreAnalyzedField.

Would also be nice if it was accessible from the python API, since it's easy to get really good tokenization in python.

EDIT: the problem with this is queries.
solr let's you specify an analyzer chain or (determined at query time) give it a token stream for queries: https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/schema/PreAnalyzedFieldTest.java

But that's pretty unsatisfying, since you either can't use the solr query parsers as well (you need to implement your own query parser to determine what parts of the query to analyze) or get sub-optimal results.

Maybe this is a bad idea, and the right thing is just to register tokenizers from python?

[fulmicoton]

This seems like a good request.

Ideally we would like to handle the structured version of it too, so that people can handle proper synonyms, and keep track of the original offsets.

Also assuming you (or someone else) are facing today a use case and want a workaround:

• if you do not care about positions, tantivy handles multi valued fields. Using a raw tokenizer, and passing a list of tokens ["hello","happy", "tax", "payer" ] should work.
• You could create your own DSL to represent your token list, and implement a custom tokenizer to parse it.

[fulmicoton]

@petr-tik Would you like to mentor someone on that one?

[petr-tik]

Would be happy to. To clarify 2 things:

1. We need to add a rust tantivy API, which another PR to tantivy-py will then expose to users?
2. What is a typical use case for pre-tokenising outside the main search/indexing library? Would be great to either see some code or read a description of such a pipeline

[acertain]

The use case I have in mind is using a python linguistics library for lemmatization & tokenization, e.g. spacy (https://spacy.io/api/token / https://spacy.io/usage/linguistic-features#tokenization). Simple use of spacy might look like

nlp = spacy.load("en_core_web_sm")
def tokenize(text):
  return [(x.lemma_, x.offset, len(x)) for x in nlp(text)]