Skip to content

Commit

Permalink
Expose Lucene's FeatureField. (#30618)
Browse files Browse the repository at this point in the history
Lucene has a new `FeatureField` which gives the ability to record numeric
features as term frequencies. Its main benefit is that it allows to boost
queries with the values of these features and efficiently skip non-competitive
documents at the same time using block-max WAND and indexed impacts.
  • Loading branch information
jpountz authored May 23, 2018
1 parent 739bb4f commit 886db84
Show file tree
Hide file tree
Showing 14 changed files with 1,616 additions and 2 deletions.
4 changes: 3 additions & 1 deletion docs/reference/mapping/types.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ string:: <<text,`text`>> and <<keyword,`keyword`>>

<<parent-join>>:: Defines parent/child relation for documents within the same index

<<feature>>:: Record numeric features to boost hits at query time.

[float]
=== Multi-fields

Expand Down Expand Up @@ -86,6 +88,6 @@ include::types/percolator.asciidoc[]

include::types/parent-join.asciidoc[]


include::types/feature.asciidoc[]


59 changes: 59 additions & 0 deletions docs/reference/mapping/types/feature.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
[[feature]]
=== Feature datatype

A `feature` field can index numbers so that they can later be used to boost
documents in queries with a <<query-dsl-feature-query,`feature`>> query.

[source,js]
--------------------------------------------------
PUT my_index
{
"mappings": {
"_doc": {
"properties": {
"pagerank": {
"type": "feature" <1>
},
"url_length": {
"type": "feature",
"positive_score_impact": false <2>
}
}
}
}
}
PUT my_index/_doc/1
{
"pagerank": 8,
"url_length": 22
}
GET my_index/_search
{
"query": {
"feature": {
"field": "pagerank"
}
}
}
--------------------------------------------------
// CONSOLE
<1> Feature fields must use the `feature` field type
<2> Features that correlate negatively with the score need to declare it

NOTE: `feature` fields only support single-valued fields and strictly positive
values. Multi-valued fields and negative values will be rejected.

NOTE: `feature` fields do not support querying, sorting or aggregating. They may
only be used within <<query-dsl-feature-query,`feature`>> queries.

NOTE: `feature` fields only preserve 9 significant bits for the precision, which
translates to a relative error of about 0.4%.

Features that correlate negatively with the score should set
`positive_score_impact` to `false` (defaults to `true`). This will be used by
the <<query-dsl-feature-query,`feature`>> query to modify the scoring formula
in such a way that the score decreases with the value of the feature instead of
increasing. For instance in web search, the url length is a commonly used
feature which correlates negatively with scores.
181 changes: 181 additions & 0 deletions docs/reference/query-dsl/feature-query.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
[[query-dsl-feature-query]]
=== Feature Query

The `feature` query is a specialized query that only works on
<<feature,`feature`>> fields. Its goal is to boost the score of documents based
on the values of numeric features. It is typically put in a `should` clause of
a <<query-dsl-bool-query,`bool`>> query so that its score is added to the score
of the query.

Compared to using <<query-dsl-function-score-query,`function_score`>> or other
ways to modify the score, this query has the benefit of being able to
efficiently skip non-competitive hits when
<<search-uri-request,`track_total_hits`>> is set to `false`. Speedups may be
spectacular.

Here is an example:

[source,js]
--------------------------------------------------
PUT test
{
"mappings": {
"_doc": {
"properties": {
"pagerank": {
"type": "feature"
},
"url_length": {
"type": "feature",
"positive_score_impact": false
}
}
}
}
}
PUT test/_doc/1
{
"pagerank": 10,
"url_length": 50
}
PUT test/_doc/2
{
"pagerank": 100,
"url_length": 20
}
POST test/_refresh
GET test/_search
{
"query": {
"feature": {
"field": "pagerank"
}
}
}
GET test/_search
{
"query": {
"feature": {
"field": "url_length"
}
}
}
--------------------------------------------------
// CONSOLE

[float]
=== Supported functions

The `feature` query supports 3 functions in order to boost scores using the
values of features. If you do not know where to start, we recommend that you
start with the `saturation` function, which is the default when no function is
provided.

[float]
==== Saturation

This function gives a score that is equal to `S / (S + pivot)` where `S` is the
value of the feature and `pivot` is a configurable pivot value so that the
result will be less than +0.5+ if `S` is less than pivot and greater than +0.5+
otherwise. Scores are always is +(0, 1)+.

If the feature has a negative score impact then the function will be computed as
`pivot / (S + pivot)`, which decreases when `S` increases.

[source,js]
--------------------------------------------------
GET test/_search
{
"query": {
"feature": {
"field": "pagerank",
"saturation": {
"pivot": 8
}
}
}
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

If +pivot+ is not supplied then Elasticsearch will compute a default value that
will be approximately equal to the geometric mean of all feature values that
exist in the index. We recommend this if you haven't had the opportunity to
train a good pivot value.

[source,js]
--------------------------------------------------
GET test/_search
{
"query": {
"feature": {
"field": "pagerank",
"saturation": {}
}
}
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

[float]
==== Logarithm

This function gives a score that is equal to `log(scaling_factor + S)` where
`S` is the value of the feature and `scaling_factor` is a configurable scaling
factor. Scores are unbounded.

This function only supports features that have a positive score impact.

[source,js]
--------------------------------------------------
GET test/_search
{
"query": {
"feature": {
"field": "pagerank",
"log": {
"scaling_factor": 4
}
}
}
}
--------------------------------------------------
// CONSOLE
// TEST[continued]

[float]
==== Sigmoid

This function is an extension of `saturation` which adds a configurable
exponent. Scores are computed as `S^exp^ / (S^exp^ + pivot^exp^)`. Like for the
`saturation` function, `pivot` is the value of `S` that gives a score of +0.5+
and scores are in +(0, 1)+.

`exponent` must be positive, but is typically in +[0.5, 1]+. A good value should
be computed via traning. If you don't have the opportunity to do so, we recommend
that you stick to the `saturation` function instead.

[source,js]
--------------------------------------------------
GET test/_search
{
"query": {
"feature": {
"field": "pagerank",
"sigmoid": {
"pivot": 7,
"exponent": 0.6
}
}
}
}
--------------------------------------------------
// CONSOLE
// TEST[continued]
7 changes: 7 additions & 0 deletions docs/reference/query-dsl/special-queries.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@ This query allows a script to act as a filter. Also see the
This query finds queries that are stored as documents that match with
the specified document.

<<query-dsl-feature-query,`feature` query>>::

A query that computes scores based on the values of numeric features and is
able to efficiently skip non-competitive hits.

<<query-dsl-wrapper-query,`wrapper` query>>::

A query that accepts other queries as json or yaml string.
Expand All @@ -29,4 +34,6 @@ include::script-query.asciidoc[]

include::percolate-query.asciidoc[]

include::feature-query.asciidoc[]

include::wrapper-query.asciidoc[]
Loading

0 comments on commit 886db84

Please sign in to comment.