geohash_grid aggregation: Add support for geo_shape

[nreese]

Would it be possible to enhance the geohash_grid aggregation to support geo_shape indexes. The functionality would be identical to geo_point with the only difference being that a document would get placed into each grid bucket that intersects the shape.

[clintongormley]

@nknize would this be feasible?

Related to #7574

[nknize]

I'll update the code for this to show how this can be done in the aggregation. Below is a code snippet for computing the geohash cells for a shape. It can be done for quad tree as well for natural mercator matching.

    // build poly
    ctx = JtsSpatialContext.GEO;
    Shape shape = ctx.readShapeFromWkt(wkt);

    // build geohash prefix tree and iterate intersecting cells
    CellIterator iter = new GeohashPrefixTree(ctx, 6).getTreeCellIterator(shape, 6);
    while (iter.hasNext()) {
      Cell c = iter.next();
      System.out.println(c);
    }

I am a little concerned about the complexity inside the aggregation framework.

[mikeurbach]

we talked about this at elasticon, it would be an awesome feature that my team would use a lot!

[rtrujill007]

👍

This would also greatly benefit us (Esri) as well!

[markharwood]

cc @elastic/es-search-aggs

[refaelasraf]

Have any updates about this question?

[cbuescher]

I am a little concerned about the complexity inside the aggregation framework.

This looks like we don't have agreement yet on whether this is feasible or not. I'll tag this for team discussion to see how we can move this forward or if we don't want to pursue this idea.

[nknize]

We revisited this topic in the weekly team discussion. The main blocker is that we do not yet have DocValues support for shapes. And we will likely not have doc value support until BKD Backed Shapes is complete.

DocValues for shapes is a bit more tricky than it might seem because there is no support for multivalue DocValues. If there were, we could index a docvalue for each decomposed shape term created by the index (either quad cell or tessellated triangle) and use them to compute intersection with the grid bucket(s). Alternatively we could create a single docvalue for each shape that represents the shape in some standard format (e.g., WKB), but then we not only have to unmarshall the docvalue but also recompute the relation between the shape and the aggregation grid cell(s). A linear scan issue that would likely not scale.

Since we can't take either of the above approaches we could, alternatively, create a single docvalue for each shape that represents the tessellation of the shape and then compute the intersection of the grid bucket(s) with that tessellation. While this is doable, it is also a linear scan issue that may not scale. Additionally, DocValues have a cap on storage size that would substantially limit the size of the shape in terms of number of vertices.

Bottom line, we need to determine an efficient docvalue approach (or alternative) that helps take advantage of the decomposed terms in the index and does not over constrain us on size of the shape.

[imotov]

Depends on #37206

[thomasneirynck]

To give some context from the Kibana Maps app perspective, and why this remains relevant.

The UX currently disables indices with geo_shape fields from being added as a grid or heatmap. It would help improve the symmetry between these data sources, and reduce some of the confusion as to why users cannot make heatmaps from e.g. points stored as a geo_shape. Adding this would also simplify the data modeling step up front, as users would not need to take into account whether they store their documents as geo_point or geo_shape, since there would be no limitations to choosing the latter at query-time or in the UX in Kibana. The different nuances between the two aren't always clear for the beginner user.

This difference in support between geo_point and geo_shape also impacts geojson data upload (or any format for that matter), as the upload will need to inspect what geometries are in the file, before the index is created (cf. elastic/kibana#28765).

(A similar issue with the difference in agg support between geo_point and geo_shape would be #7574. This is affecting the UX in the Kibana Maps app in a similar way, where the app disables functionality when the geo_shape field is used).

[csoulios]

Issue closed by PR #55966, released in v7.8.0