search as you type fieldmapper #35600

andyb-elastic · 2018-11-15T19:26:56Z

Adds a new field type, search_as_you_type, that acts like a text field optimized for as-you-type search completion. It creates a couple subfields that analyze the indexed terms as shingles, against which full terms are queried, and a prefix subfield that analyze terms as the largest shingle size used and edge-ngrams, against which partial terms are queried

Adds a boolean_prefix query type that creates a boolean clause of a term query for each term except the last, for which a boolean clause with a prefix query is created. This will be used as the recommended query type for querying search_as_you_type fields, although other text queries will be supported as well

This is for #33160

this verifies that all the fields are correctly configured in the default case and that all the fields are accounted for

max shingles = 3 by default, and min gram and max gram are no longer configurable

elasticmachine · 2018-11-15T19:26:58Z

Pinging @elastic/es-search-aggs

andyb-elastic · 2018-11-15T19:33:31Z

@jimczi from our discussion yesterday I

set default max_shingle_size to 3
capped it at 5
removed configurability for min_gram and max_gram
added tests for the configuration options exposed

I think this is ready for a first look unless there's anything else you'd like me to add

...rch-as-you-type/src/main/java/org/elasticsearch/index/mapper/SearchAsYouTypeFieldMapper.java

jimczi

It's a great start @andyb-elastic . I left some comments regarding how we should interact with this new mapper.

server/src/main/java/org/apache/lucene/index/FreqIsScoreSimilarity.java

server/src/main/java/org/elasticsearch/index/mapper/MapperService.java

server/src/main/java/org/elasticsearch/index/mapper/OldSearchAsYouTypeFieldMapper.java

...rch-as-you-type/src/main/java/org/elasticsearch/index/mapper/SearchAsYouTypeFieldMapper.java

...s-you-type/src/test/java/org/elasticsearch/index/mapper/SearchAsYouTypeFieldMapperTests.java

...rch-as-you-type/src/main/java/org/elasticsearch/index/mapper/SearchAsYouTypeFieldMapper.java

jimczi · 2018-11-16T17:36:18Z

...rch-as-you-type/src/main/java/org/elasticsearch/index/mapper/SearchAsYouTypeFieldMapper.java

+                    maxShingleSize = XContentMapValues.nodeIntegerValue(fieldNode);
+                    iterator.remove();
+                }
+            }


Should we throw an error if the map is not empty (has some fields that we don't support) ?

I assumed whatever calls this method did that afterwards, it looks like in at least some places they do. Should I throw one here too?

elasticsearch/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java

Lines 276 to 284 in 3c03975

Mapper.Builder<?,?> fieldBuilder = typeParser.parse(realFieldName, propNode, parserContext);

for (int i = fieldNameParts.length - 2; i >= 0; --i) {

ObjectMapper.Builder<?, ?> intermediate = new ObjectMapper.Builder<>(fieldNameParts[i]);

intermediate.add(fieldBuilder);

fieldBuilder = intermediate;

}

objBuilder.add(fieldBuilder);

propNode.remove("type");

DocumentMapperParser.checkNoRemainingFields(fieldName, propNode, parserContext.indexVersionCreated());

tests pass

server/src/main/java/org/elasticsearch/index/query/MultiMatchQueryBuilder.java

jimczi

Thanks for writing the docs @andyb-elastic , I left some comments.

When going through what multi match parameters are supported for bool_prefix, it looks like the operator, auto_generate_synonyms_phrase_query, and the fuzziness family of settings are applied to query in the multi_match context, so it seems like we should support them in the single-query form too

I agree, we should at least support operator and fuzziness.

server/src/main/java/org/elasticsearch/index/query/MultiMatchQueryBuilder.java

docs/reference/mapping/types/search-as-you-type.asciidoc

docs/reference/query-dsl/multi-match-query.asciidoc

server/src/main/java/org/elasticsearch/index/query/BoolPrefixQueryBuilder.java

andyb-elastic · 2019-03-22T22:38:30Z

I agree, we should at least support operator and fuzziness.

Sounds good, I think I'd like to add them in a follow up

jimczi · 2019-03-25T11:03:22Z

Sounds good, I think I'd like to add them in a follow up

Sure as you like. Let's not forget it though ;)

andyb-elastic · 2019-03-26T15:01:52Z

I'm adding those parameters here since it's not a big change

andyb-elastic · 2019-03-27T00:57:13Z

I pushed changes that rename the query to match_bool_prefix and provide support for operator and fuzziness and its related parameters

jimczi

This change looks good to me, let's merge !

Adds the search_as_you_type field type that acts like a text field optimized for as-you-type search completion. It creates a couple subfields that analyze the indexed terms as shingles, against which full terms are queried, and a prefix subfield that analyze terms as the largest shingle size used and edge-ngrams, against which partial terms are queried Adds a match_bool_prefix query type that creates a boolean clause of a term query for each term except the last, for which a boolean clause with a prefix query is created. The match_bool_prefix query is the recommended way of querying a search as you type field, which will boil down to term queries for each shingle of the input text on the appropriate shingle field, and the final (possibly partial) term as a term query on the prefix field. This field type also supports phrase and phrase prefix queries however

andyb-elastic · 2019-03-27T20:29:50Z

master - 6bba9fc
7.x - 23395a9

telendt · 2020-03-18T14:01:19Z

...s/mapper-extras/src/test/resources/rest-api-spec/test/search-as-you-type/20_highlighting.yml

+  - match: { hits.total: 1 }
+  - match: { hits.hits.0._source.a_field: "quick brown fox jump lazy dog" }
+  - match: { hits.hits.0._source.text_field: "quick brown fox jump lazy dog" }
+  - match: { hits.hits.0.highlight.a_field.0: "quick <em>brown</em> fox jump lazy dog" }


@andyb-elastic Is it unreasonable to expect quick <em>brown</em> <em>fox</em> jump lazy dog highlight here?

(That's what match_bool_prefix: "brown fo" on text_field would give).

I know it's not nice to comment on already merged PR, that's why I opened this issue:
#53744

jimczi and others added 11 commits November 2, 2018 13:03

Add SearchAsTypeFieldMapper

cd8de86

wip compiles now

6935480

wip default configuration test passes

6d79b6e

wip passes basic test

6443ed9

wip move to own module, passes

024e546

wip passes

9582375

this verifies that all the fields are correctly configured in the default case and that all the fields are accounted for

wip put all suggestrized field types into searchasyoutypefieldtype

4bcc6af

wip change defaults

cc6fb91

max shingles = 3 by default, and min gram and max gram are no longer configurable

wip remove commented code

15ef530

wip limit max shingle size

a8897d6

wip test configuration

db7bbc1

andyb-elastic added >feature WIP :Search Relevance/Suggesters "Did you mean" and suggestions as you type labels Nov 15, 2018

andyb-elastic requested a review from jimczi November 15, 2018 19:26

andyb-elastic mentioned this pull request Nov 15, 2018

search as you type field type #35548

Closed

andyb-elastic commented Nov 15, 2018

View reviewed changes

...rch-as-you-type/src/main/java/org/elasticsearch/index/mapper/SearchAsYouTypeFieldMapper.java Outdated Show resolved Hide resolved

jimczi reviewed Nov 16, 2018

View reviewed changes

andyb-elastic added 10 commits November 16, 2018 16:59

wip remove multifield support

0d1a506

wip remove original field mapper

3701310

wip move root field mapper class down a little tests pass

bb21fa0

wip structure subfield in a container, tests pass

5171d6e

wip put analysis details into field type

e1c8493

tests pass

wip added references to ngram subfields

501f3a0

tests pass

wip method name consistency

b2360d5

wip disable integTest bc we dont have any

c4b5e10

wip add test for doc parsing

a3eae0e

wip IT smoketest

9c334a7

andyb-elastic commented Mar 22, 2019

View reviewed changes

server/src/main/java/org/elasticsearch/index/query/MultiMatchQueryBuilder.java Show resolved Hide resolved

jimczi requested changes Mar 22, 2019

View reviewed changes

andyb-elastic added 2 commits March 22, 2019 15:14

default max shingle size in docs

4fa005c

fix reference to scoring mode

2c10385

andyb-elastic added 2 commits March 22, 2019 15:54

account for disallowed params

02383c8

Merge branch 'master' into feature-search-as-you-type-fieldmapper

9691f4a

andyb-elastic added 4 commits March 25, 2019 09:35

Merge branch 'master' into feature-search-as-you-type-fieldmapper

5ff97b1

rename server

a9abdff

rename module

f36d9c1

rename docs and some fixes

5003046

andyb-elastic added 4 commits March 26, 2019 11:40

support operator

52d4442

support fuzziness

5d891a0

fix searchmoduletests

41a7530

Merge branch 'master' into feature-search-as-you-type-fieldmapper

a146fac

jimczi approved these changes Mar 27, 2019

View reviewed changes

andyb-elastic changed the title ~~Feature search as you type fieldmapper~~ search as you type fieldmapper Mar 27, 2019

andyb-elastic merged commit 6bba9fc into elastic:master Mar 27, 2019

andyb-elastic added the backport pending label Mar 27, 2019

andyb-elastic mentioned this pull request Mar 27, 2019

backport 7.x - search as you type fieldmapper #40545

Merged

andyb-elastic removed the backport pending label Mar 27, 2019

jimczi mentioned this pull request Apr 3, 2019

Postings based completion suggester #33160

Closed

jrodewig mentioned this pull request Nov 11, 2019

Improve documentation on search_as_you_type datatype #48862

Closed

telendt reviewed Mar 18, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

search as you type fieldmapper #35600

search as you type fieldmapper #35600

andyb-elastic commented Nov 15, 2018 •

edited

Loading

elasticmachine commented Nov 15, 2018

andyb-elastic commented Nov 15, 2018

jimczi left a comment

jimczi Nov 16, 2018

andyb-elastic Nov 17, 2018

jimczi left a comment

andyb-elastic commented Mar 22, 2019

jimczi commented Mar 25, 2019

andyb-elastic commented Mar 26, 2019

andyb-elastic commented Mar 27, 2019

jimczi left a comment

andyb-elastic commented Mar 27, 2019

telendt Mar 18, 2020 •

edited

Loading

telendt Mar 18, 2020

	Mapper.Builder<?,?> fieldBuilder = typeParser.parse(realFieldName, propNode, parserContext);
	for (int i = fieldNameParts.length - 2; i >= 0; --i) {
	ObjectMapper.Builder<?, ?> intermediate = new ObjectMapper.Builder<>(fieldNameParts[i]);
	intermediate.add(fieldBuilder);
	fieldBuilder = intermediate;
	}
	objBuilder.add(fieldBuilder);
	propNode.remove("type");
	DocumentMapperParser.checkNoRemainingFields(fieldName, propNode, parserContext.indexVersionCreated());

search as you type fieldmapper #35600

search as you type fieldmapper #35600

Conversation

andyb-elastic commented Nov 15, 2018 • edited Loading

elasticmachine commented Nov 15, 2018

andyb-elastic commented Nov 15, 2018

jimczi left a comment

Choose a reason for hiding this comment

jimczi Nov 16, 2018

Choose a reason for hiding this comment

andyb-elastic Nov 17, 2018

Choose a reason for hiding this comment

jimczi left a comment

Choose a reason for hiding this comment

andyb-elastic commented Mar 22, 2019

jimczi commented Mar 25, 2019

andyb-elastic commented Mar 26, 2019

andyb-elastic commented Mar 27, 2019

jimczi left a comment

Choose a reason for hiding this comment

andyb-elastic commented Mar 27, 2019

telendt Mar 18, 2020 • edited Loading

Choose a reason for hiding this comment

telendt Mar 18, 2020

Choose a reason for hiding this comment

andyb-elastic commented Nov 15, 2018 •

edited

Loading

telendt Mar 18, 2020 •

edited

Loading