fix: #2078 return error when tokenizer not found while indexing #2093
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PSeitz
reviewed
Jun 15, 2023
src/indexer/segment_writer.rs
Outdated
Comment on lines
933
to
936
| let index = Index::create_in_dir(&tempdir_path, schema).unwrap(); | ||
| index | ||
| .tokenizers() | ||
| .register("custom_en", custom_en_tokenizer); |
There was a problem hiding this comment.
I think that can be removed, it's unused
There was a problem hiding this comment.
I guess we can remove the tokenizer registration but we may need the index creation else open_in_dir will fail
PSeitz
reviewed
Jun 15, 2023
src/indexer/segment_writer.rs
Outdated
| let mut document = Document::default(); | ||
| document.add_text(title, "The Old Man and the Sea"); | ||
| index_writer.add_document(document).unwrap(); | ||
| match index_writer.commit() { |
There was a problem hiding this comment.
you can use unwrap_err() which will panic if it's not an error
PSeitz
reviewed
Jun 15, 2023
src/indexer/segment_writer.rs
Outdated
| @@ -900,4 +911,42 @@ mod tests { | |||
| postings.positions(&mut positions); | |||
| assert_eq!(positions, &[4]); //< as opposed to 3 if we had a position length of 1. | |||
| } | |||
|
|
|||
| // ISSUE-#2078 - writing and searching shall throw error when the field tokenizer is missing | |||
There was a problem hiding this comment.
the comment implies this also tests search which is not the case
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. @@ Coverage Diff @@
## main #2093 +/- ##
=======================================
Coverage 94.38% 94.38%
=======================================
Files 321 321
Lines 60583 60612 +29
=======================================
+ Hits 57179 57210 +31
+ Misses 3404 3402 -2
|
PSeitz
approved these changes
Jun 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is to fix #2078