Add multi language tokenizer

[fmassot]

Add a simple multi-language tokenizer with language detection.

Fix #3055

TODO

• Check the size of dictionaries. Should we include them in the binary? => yes, 22MB seems fine.
• Add the possibility to bypass the language autodetection with a prefix of type {language-identifier}:
• test on a Wikipedia dataset
• bench tokenizer
• choose the tokenizer name: multi_language or multilanguage
• Expose SimpleTokenStream in tantivy main branch
• Add korean support
• Add docs (I will open a dedicated issue)
• Add a feature multilanguage?

Binary size

On macOS, using Lindera with the compress features adds 44 MB (97MB -> 141MB).

Benchmarks

Tokenizer	Throughput
Default short	64 MiB/s
Default long	100 MiB/s
Multilanguage default short	41 MiB/s
Multilanguage default long	55 MiB/s
Multilanguage jpn short	6 MiB/s
Multilanguage jpn long	7.2 MiB/s
Multilanguage cmn long	8 MiB/s
Multilanguage cmn long	12 MiB/s
chinese-compatible short	31 MiB/s
chinese-compatible long	36 MiB/s
jpn (lindera) short	6.8 MiB/s
jpn (lindera) medium	8.5 MiB/s

Mapping example:

version: 0.5

index_id: multilang

doc_mapping:
  field_mappings:
    - name: body
      type: text
      tokenizer: multilanguage
      record: position

[fmassot]

The license is clear: (Apache-2.0 OR MIT) AND BSD-3-Clause so no pb here.

[fmassot]

Closing, will open a new PR soon.