Skip to content

Commit

Permalink
"CONTAINS" support for BKD-backed geo_shape and shape fields (#50141) (…
Browse files Browse the repository at this point in the history 
…#50213)

Lucene 8.4 added support for "CONTAINS", therefore in this commit those
changes are integrated in Elasticsearch. This commit contains as well a
bug fix when querying with a geometry collection with "DISJOINT" relation.
  • Loading branch information
@iverase
iverase authored Dec 16, 2019
1 parent c732d99 commit 3717c73
Show file tree
Hide file tree
Showing 9 changed files with 250 additions and 27 deletions.
5 changes: 2 additions & 3 deletions docs/reference/mapping/types/geo-shape.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,9 +142,8 @@ The following features are not yet supported with the new indexing approach:
using a `bool` query with each individual point.

* `CONTAINS` relation query - when using the new default vector indexing strategy, `geo_shape`
queries with `relation` defined as `contains` are not yet supported. If this query relation
is an absolute necessity, it is recommended to set `strategy` to `quadtree` and use the
deprecated PrefixTree strategy indexing approach.
queries with `relation` defined as `contains` are supported for indices created with
ElasticSearch 7.5.0 or higher.

[[prefix-trees]]
[float]
Expand Down
6 changes: 3 additions & 3 deletions docs/reference/mapping/types/shape.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ The following features are not yet supported:
over each individual point. For now, if this is absolutely needed, this can be achieved
using a `bool` query with each individual point. (Note: this could be very costly)

* `CONTAINS` relation query - `shape` queries with `relation` defined as `contains` are not
yet supported.
* `CONTAINS` relation query - `shape` queries with `relation` defined as `contains` are supported
for indices created with ElasticSearch 7.5.0 or higher.

[float]
===== Example
Expand Down Expand Up @@ -445,4 +445,4 @@ POST /example/_doc
Due to the complex input structure and index representation of shapes,
it is not currently possible to sort shapes or retrieve their fields
directly. The `shape` value is only retrievable through the `_source`
field.
field.
3 changes: 1 addition & 2 deletions docs/reference/query-dsl/geo-shape-query.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -151,8 +151,7 @@ has nothing in common with the query geometry.
* `WITHIN` - Return all documents whose `geo_shape` field
is within the query geometry.
* `CONTAINS` - Return all documents whose `geo_shape` field
contains the query geometry. Note: this is only supported using the
`recursive` Prefix Tree Strategy deprecated[6.6]
contains the query geometry.

[float]
==== Ignore Unmapped
Expand Down
8 changes: 5 additions & 3 deletions docs/reference/query-dsl/shape-query.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -171,12 +171,14 @@ GET /example/_search

The following is a complete list of spatial relation operators available:

* `INTERSECTS` - (default) Return all documents whose `geo_shape` field
* `INTERSECTS` - (default) Return all documents whose `shape` field
intersects the query geometry.
* `DISJOINT` - Return all documents whose `geo_shape` field
* `DISJOINT` - Return all documents whose `shape` field
has nothing in common with the query geometry.
* `WITHIN` - Return all documents whose `geo_shape` field
* `WITHIN` - Return all documents whose `shape` field
is within the query geometry.
* `CONTAINS` - Return all documents whose `shape` field
contains the query geometry.

[float]
==== Ignore Unmapped
Expand Down
1 change: 1 addition & 0 deletions server/src/main/java/org/elasticsearch/common/geo/ShapeRelation.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ public QueryRelation getLuceneRelation() {
case INTERSECTS: return QueryRelation.INTERSECTS;
case DISJOINT: return QueryRelation.DISJOINT;
case WITHIN: return QueryRelation.WITHIN;
case CONTAINS: return QueryRelation.CONTAINS;
default:
throw new IllegalArgumentException("ShapeRelation [" + this + "] not supported");
}
Expand Down
29 changes: 23 additions & 6 deletions server/src/main/java/org/elasticsearch/index/query/VectorGeoShapeQueryProcessor.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,14 @@
package org.elasticsearch.index.query;

import org.apache.lucene.document.LatLonShape;
import org.apache.lucene.document.ShapeField;
import org.apache.lucene.geo.Line;
import org.apache.lucene.geo.Polygon;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.MatchNoDocsQuery;
import org.apache.lucene.search.Query;
import org.elasticsearch.Version;
import org.elasticsearch.common.geo.GeoShapeType;
import org.elasticsearch.common.geo.ShapeRelation;
import org.elasticsearch.geometry.Circle;
Expand All @@ -49,10 +51,10 @@ public class VectorGeoShapeQueryProcessor implements AbstractGeometryFieldMapper

@Override
public Query process(Geometry shape, String fieldName, ShapeRelation relation, QueryShardContext context) {
// CONTAINS queries are not yet supported by VECTOR strategy
if (relation == ShapeRelation.CONTAINS) {
// CONTAINS queries are not supported by VECTOR strategy for indices created before version 7.5.0 (Lucene 8.3.0)
if (relation == ShapeRelation.CONTAINS && context.indexVersionCreated().before(Version.V_7_5_0)) {
throw new QueryShardException(context,
ShapeRelation.CONTAINS + " query relation not supported for Field [" + fieldName + "]");
ShapeRelation.CONTAINS + " query relation not supported for Field [" + fieldName + "].");
}
// wrap geoQuery as a ConstantScoreQuery
return getVectorQueryFromShape(shape, fieldName, relation, context);
Expand Down Expand Up @@ -95,12 +97,21 @@ public Query visit(GeometryCollection<?> collection) {
}

private void visit(BooleanQuery.Builder bqb, GeometryCollection<?> collection) {
BooleanClause.Occur occur;
if (relation == ShapeRelation.CONTAINS || relation == ShapeRelation.DISJOINT) {
// all shapes must be disjoint / must be contained in relation to the indexed shape.
occur = BooleanClause.Occur.MUST;
} else {
// at least one shape must intersect / contain the indexed shape.
occur = BooleanClause.Occur.SHOULD;
}
for (Geometry shape : collection) {
if (shape instanceof MultiPoint) {
// Flatten multipoints
// Flatten multi-points
// We do not support multi-point queries?
visit(bqb, (GeometryCollection<?>) shape);
} else {
bqb.add(shape.visit(this), BooleanClause.Occur.SHOULD);
bqb.add(shape.visit(this), occur);
}
}
}
Expand Down Expand Up @@ -144,7 +155,13 @@ public Query visit(MultiPolygon multiPolygon) {
@Override
public Query visit(Point point) {
validateIsGeoShapeFieldType();
return LatLonShape.newBoxQuery(fieldName, relation.getLuceneRelation(),
ShapeField.QueryRelation luceneRelation = relation.getLuceneRelation();
if (luceneRelation == ShapeField.QueryRelation.CONTAINS) {
// contains and intersects are equivalent but the implementation of
// intersects is more efficient.
luceneRelation = ShapeField.QueryRelation.INTERSECTS;
}
return LatLonShape.newBoxQuery(fieldName, luceneRelation,
point.getY(), point.getY(), point.getX(), point.getX());
}

Expand Down
99 changes: 96 additions & 3 deletions server/src/test/java/org/elasticsearch/search/geo/GeoShapeQueryTests.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -483,10 +483,30 @@ public void testPointQuery() throws Exception {
public void testContainsShapeQuery() throws Exception {
// Create a random geometry collection.
Rectangle mbr = xRandomRectangle(random(), xRandomPoint(random()), true);
GeometryCollectionBuilder gcb = createGeometryCollectionWithin(random(), mbr);
boolean usePrefixTrees = randomBoolean();
GeometryCollectionBuilder gcb;
if (usePrefixTrees) {
gcb = createGeometryCollectionWithin(random(), mbr);
} else {
// vector strategy does not yet support multipoint queries
gcb = new GeometryCollectionBuilder();
int numShapes = RandomNumbers.randomIntBetween(random(), 1, 4);
for (int i = 0; i < numShapes; ++i) {
ShapeBuilder shape;
do {
shape = RandomShapeGenerator.createShapeWithin(random(), mbr);
} while (shape instanceof MultiPointBuilder);
gcb.shape(shape);
}
}

client().admin().indices().prepareCreate("test").addMapping("type", "location", "type=geo_shape,tree=quadtree" )
.get();
if (usePrefixTrees) {
client().admin().indices().prepareCreate("test").addMapping("type", "location", "type=geo_shape,tree=quadtree")
.execute().actionGet();
} else {
client().admin().indices().prepareCreate("test").addMapping("type", "location", "type=geo_shape")
.execute().actionGet();
}

XContentBuilder docSource = gcb.toXContent(jsonBuilder().startObject().field("location"), null).endObject();
client().prepareIndex("test", "type", "1").setSource(docSource).setRefreshPolicy(IMMEDIATE).get();
Expand Down Expand Up @@ -763,4 +783,77 @@ public void testEnvelopeSpanningDateline() throws IOException {
assertNotEquals("1", response.getHits().getAt(0).getId());
assertNotEquals("1", response.getHits().getAt(1).getId());
}

public void testGeometryCollectionRelations() throws IOException {
XContentBuilder mapping = XContentFactory.jsonBuilder().startObject()
.startObject("doc")
.startObject("properties")
.startObject("geo").field("type", "geo_shape").endObject()
.endObject()
.endObject()
.endObject();

createIndex("test", Settings.builder().put("index.number_of_shards", 1).build(), "doc", mapping);

EnvelopeBuilder envelopeBuilder = new EnvelopeBuilder(new Coordinate(-10, 10), new Coordinate(10, -10));

client().index(new IndexRequest("test")
.source(jsonBuilder().startObject().field("geo", envelopeBuilder).endObject())
.setRefreshPolicy(IMMEDIATE)).actionGet();

{
// A geometry collection that is fully within the indexed shape
GeometryCollectionBuilder builder = new GeometryCollectionBuilder();
builder.shape(new PointBuilder(1, 2));
builder.shape(new PointBuilder(-2, -1));
SearchResponse response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.CONTAINS))
.get();
assertEquals(1, response.getHits().getTotalHits().value);
response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.INTERSECTS))
.get();
assertEquals(1, response.getHits().getTotalHits().value);
response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.DISJOINT))
.get();
assertEquals(0, response.getHits().getTotalHits().value);
}
// A geometry collection that is partially within the indexed shape
{
GeometryCollectionBuilder builder = new GeometryCollectionBuilder();
builder.shape(new PointBuilder(1, 2));
builder.shape(new PointBuilder(20, 30));
SearchResponse response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.CONTAINS))
.get();
assertEquals(0, response.getHits().getTotalHits().value);
response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.INTERSECTS))
.get();
assertEquals(1, response.getHits().getTotalHits().value);
response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.DISJOINT))
.get();
assertEquals(0, response.getHits().getTotalHits().value);
}
{
// A geometry collection that is disjoint with the indexed shape
GeometryCollectionBuilder builder = new GeometryCollectionBuilder();
builder.shape(new PointBuilder(-20, -30));
builder.shape(new PointBuilder(20, 30));
SearchResponse response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.CONTAINS))
.get();
assertEquals(0, response.getHits().getTotalHits().value);
response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.INTERSECTS))
.get();
assertEquals(0, response.getHits().getTotalHits().value);
response = client().prepareSearch("test")
.setQuery(geoShapeQuery("geo", builder.buildGeometry()).relation(ShapeRelation.DISJOINT))
.get();
assertEquals(1, response.getHits().getTotalHits().value);
}
}
}
31 changes: 24 additions & 7 deletions ...patial/src/main/java/org/elasticsearch/xpack/spatial/index/query/ShapeQueryProcessor.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
*/
package org.elasticsearch.xpack.spatial.index.query;

import org.apache.lucene.document.ShapeField;
import org.apache.lucene.document.XYShape;
import org.apache.lucene.geo.XYLine;
import org.apache.lucene.geo.XYPolygon;
Expand All @@ -13,6 +14,7 @@
import org.apache.lucene.search.ConstantScoreQuery;
import org.apache.lucene.search.MatchNoDocsQuery;
import org.apache.lucene.search.Query;
import org.elasticsearch.Version;
import org.elasticsearch.common.geo.GeoShapeType;
import org.elasticsearch.common.geo.ShapeRelation;
import org.elasticsearch.geometry.Circle;
Expand All @@ -38,14 +40,14 @@ public class ShapeQueryProcessor implements AbstractGeometryFieldMapper.QueryPro

@Override
public Query process(Geometry shape, String fieldName, ShapeRelation relation, QueryShardContext context) {
// CONTAINS queries are not yet supported by VECTOR strategy
if (relation == ShapeRelation.CONTAINS) {
throw new QueryShardException(context,
ShapeRelation.CONTAINS + " query relation not supported for Field [" + fieldName + "]");
}
if (shape == null) {
return new MatchNoDocsQuery();
}
// CONTAINS queries are not supported by VECTOR strategy for indices created before version 7.5.0 (Lucene 8.3.0);
if (relation == ShapeRelation.CONTAINS && context.indexVersionCreated().before(Version.V_7_5_0)) {
throw new QueryShardException(context,
ShapeRelation.CONTAINS + " query relation not supported for Field [" + fieldName + "].");
}
// wrap geometry Query as a ConstantScoreQuery
return new ConstantScoreQuery(shape.visit(new ShapeVisitor(context, fieldName, relation)));
}
Expand Down Expand Up @@ -76,12 +78,21 @@ public Query visit(GeometryCollection<?> collection) {
}

private void visit(BooleanQuery.Builder bqb, GeometryCollection<?> collection) {
BooleanClause.Occur occur;
if (relation == ShapeRelation.CONTAINS || relation == ShapeRelation.DISJOINT) {
// all shapes must be disjoint / must be contained in relation to the indexed shape.
occur = BooleanClause.Occur.MUST;
} else {
// at least one shape must intersect / contain the indexed shape.
occur = BooleanClause.Occur.SHOULD;
}
for (Geometry shape : collection) {
if (shape instanceof MultiPoint) {
// Flatten multipoints
// We do not support multi-point queries?
visit(bqb, (GeometryCollection<?>) shape);
} else {
bqb.add(shape.visit(this), BooleanClause.Occur.SHOULD);
bqb.add(shape.visit(this), occur);
}
}
}
Expand Down Expand Up @@ -128,7 +139,13 @@ private Query visitMultiPolygon(XYPolygon... polygons) {

@Override
public Query visit(Point point) {
return XYShape.newBoxQuery(fieldName, relation.getLuceneRelation(),
ShapeField.QueryRelation luceneRelation = relation.getLuceneRelation();
if (luceneRelation == ShapeField.QueryRelation.CONTAINS) {
// contains and intersects are equivalent but the implementation of
// intersects is more efficient.
luceneRelation = ShapeField.QueryRelation.INTERSECTS;
}
return XYShape.newBoxQuery(fieldName, luceneRelation,
(float)point.getX(), (float)point.getX(), (float)point.getY(), (float)point.getY());
}

Expand Down
Loading

0 comments on commit 3717c73

Please sign in to comment.