Code

Add multilang tokenizer#3608

Code

Merged

fmassot merged 12 commits intomainquickwit-oss/quickwit:mainfrom fmassot/multilang-v2quickwit-oss/quickwit:fmassot/multilang-v2Copy head branch name to clipboardJul 17, 2023

+1,488-378

Commits

Commits on Jul 17, 2023

• Add multilang tokenizer.

  fmassot

  committed
  b4b393e

  Copy full SHA for b4b393e

• Bump lindera and tantivy version.

  fmassot

  committed
  78468b7

  Copy full SHA for 78468b7

• Fix cargo and fmt.

  fmassot

  committed
  751d464

  Copy full SHA for 751d464

• Add tokenizers bench.

  fmassot

  committed
  d02b2a3

  Copy full SHA for d02b2a3

• Take into account comments from review. Bump lindera version to 2.6.0.

  fmassot

  committed
  d06af6f

  Copy full SHA for d06af6f

• Increase postgresql timeout for tests.

  fmassot

  committed
  7f95695

  Copy full SHA for 7f95695

• Put multilang in custom tokenizer, avoid copying lindera dictionaries.

  fmassot

  committed
  a52575c

  Copy full SHA for a52575c

• Remove lindera tantivy dep, clean.

  fmassot

  committed
  40c2611

  Copy full SHA for 40c2611

• Fix test.

  fmassot

  committed
  0da9e38

  Copy full SHA for 0da9e38

• Fix build.

  fmassot

  committed
  c7475e0

  Copy full SHA for c7475e0

• Add multilang in tokenizer manager.

  fmassot

  committed
  da127ae

  Copy full SHA for da127ae

• Fix lindera features.

  fmassot

  committed
  ebb1df0

  Copy full SHA for ebb1df0