Skip to content

Commit

Permalink
Speed up parsing of large terms queries. (#24210)
Browse files Browse the repository at this point in the history
The addition of the normalization feature on keywords slowed down the parsing
of large `terms` queries since all terms now have to go through normalization.
However this can be avoided in the default case that the analyzer is a
`keyword` analyzer since all that normalization will do is a UTF8 conversion.
Using `Analyzer.normalize` for that is a bit overkill and could be skipped.
  • Loading branch information
jpountz authored Apr 21, 2017
1 parent a436597 commit f322f53
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,15 @@ public Object valueForDisplay(Object value) {

@Override
protected BytesRef indexedValueForSearch(Object value) {
if (searchAnalyzer() == Lucene.KEYWORD_ANALYZER) {
// keyword analyzer with the default attribute source which encodes terms using UTF8
// in that case we skip normalization, which may be slow if there many terms need to
// parse (eg. large terms query) since Analyzer.normalize involves things like creating
// attributes through reflection
// This if statement will be used whenever a normalizer is NOT configured
return super.indexedValueForSearch(value);
}

if (value == null) {
return null;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,4 +150,13 @@ public void testFuzzyQuery() {
() -> ft.fuzzyQuery("foo", Fuzziness.fromEdits(2), 1, 50, true));
assertEquals("Cannot search on field [field] since it is not indexed.", e.getMessage());
}

public void testNormalizeQueries() {
MappedFieldType ft = createDefaultFieldType();
ft.setName("field");
ft.setSearchAnalyzer(Lucene.KEYWORD_ANALYZER);
assertEquals(new TermQuery(new Term("field", new BytesRef("FOO"))), ft.termQuery("FOO", null));
ft.setSearchAnalyzer(Lucene.STANDARD_ANALYZER);
assertEquals(new TermQuery(new Term("field", new BytesRef("foo"))), ft.termQuery("FOO", null));
}
}

0 comments on commit f322f53

Please sign in to comment.