Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multilang tokenizer #3608

Merged
merged 12 commits into from
Jul 17, 2023
Merged

Add multilang tokenizer #3608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b4b393e
Add multilang tokenizer.
fmassot Jun 26, 2023
78468b7
Bump lindera and tantivy version.
fmassot Jul 5, 2023
751d464
Fix cargo and fmt.
fmassot Jul 5, 2023
d02b2a3
Add tokenizers bench.
fmassot Jul 6, 2023
d06af6f
Take into account comments from review. Bump lindera version to 2.6.0.
fmassot Jul 9, 2023
7f95695
Increase postgresql timeout for tests.
fmassot Jul 9, 2023
a52575c
Put multilang in custom tokenizer, avoid copying lindera dictionaries.
fmassot Jul 10, 2023
40c2611
Remove lindera tantivy dep, clean.
fmassot Jul 11, 2023
0da9e38
Fix test.
fmassot Jul 11, 2023
c7475e0
Fix build.
fmassot Jul 17, 2023
da127ae
Add multilang in tokenizer manager.
fmassot Jul 17, 2023
ebb1df0
Fix lindera features.
fmassot Jul 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions config/tutorials/wikipedia/multilang-index-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#
# Index config file for multilang wikipedia datasets.
#

version: 0.6

index_id: multilang-wikipedia

doc_mapping:
tokenizers:
- name: multilang
type: multilang
field_mappings:
- name: title
type: text
tokenizer: multilang
record: position
stored: true
fieldnorms: true
- name: body
type: text
tokenizer: multilang
record: position
stored: true
fieldnorms: true
- name: url
type: text
tokenizer: raw

search_settings:
default_search_fields: [title, body]

indexing_settings:
commit_timeout_secs: 10
Loading