Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"exists" query on geo_point property with doc_values=false incorrectly matches documents with a null value #65306

Closed
yrodiere opened this issue Nov 20, 2020 · 2 comments · Fixed by #65307
Labels
:Analytics/Geo Indexing, search aggregations of geo points and shapes >bug :Search/Search Search-related issues that do not fall into other categories Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Meta label for search team

Comments

@yrodiere
Copy link
Contributor

Elasticsearch version (bin/elasticsearch --version): docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0

Plugins installed: []

JVM version (java -version): N/A (using container)

OS version (uname -a if on a Unix-like system): N/A (using container)

Description of the problem including expected versus actual behavior:

The exists query does not behave properly for geo_point properties when doc_values are set to false.

It will match all documents that have a value set for this property, even if that value is null.

I would expect documents whose value is null not to be matched by the exists query.

This is inconsistent in multiple ways:

  • ES 7.10 considers null as non-existing for geo_point properties with doc_values set to true (the default).
  • ES 7.10 considers null as non-existing for other property types.
  • Previous versions of Elasticsearch consider null as non-existing for geo_point properties. I tested 7.9.2 with the reproducer below in order to confirm this.

Steps to reproduce:

docker run -it --rm=true --name elasticsearch_test -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0
curl -X PUT "localhost:9200/test-geopoint?pretty" -H 'Content-Type: application/json' -d'                                                             
{
  "mappings": {
    "properties": {
      "geopoint": {
        "type": "geo_point", "doc_values": false
      }
    }
  }
}
'
curl -X PUT "localhost:9200/test-geopoint/_doc/existing" -H 'Content-Type: application/json' -d'{"geopoint":{"lat": 42.0, "lon": 43.0}}'
curl -X PUT "localhost:9200/test-geopoint/_doc/empty" -H 'Content-Type: application/json' -d'{}'
curl -X PUT "localhost:9200/test-geopoint/_doc/null" -H 'Content-Type: application/json' -d'{"geopoint":null}'
curl -X POST "localhost:9200/test-geopoint/_search/?pretty" -H 'Content-Type: application/json' -d'{ "query": { "exists": { "field": "geopoint" } } }'

The search query returns the document that has a null value for its geo_point property:

{
  "took" : 50,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-geopoint",
        "_type" : "_doc",
        "_id" : "existing",
        "_score" : 1.0,
        "_source" : {
          "geopoint" : {
            "lat" : 42.0,
            "lon" : 43.0
          }
        }
      },
      {
        "_index" : "test-geopoint",
        "_type" : "_doc",
        "_id" : "null",
        "_score" : 1.0,
        "_source" : {
          "geopoint" : null
        }
      }
    ]
  }
}
@yrodiere yrodiere added >bug needs:triage Requires assignment of a team area label labels Nov 20, 2020
@romseygeek romseygeek added the :Search/Search Search-related issues that do not fall into other categories label Nov 20, 2020
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Nov 20, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@romseygeek romseygeek removed the needs:triage Requires assignment of a team area label label Nov 20, 2020
jimczi added a commit to jimczi/elasticsearch that referenced this issue Nov 20, 2020
A bug was introduced in 7.10 that causes explicit `null` values to be indexed in the _field_names
field. This change fixes this bug for newly ingested data but `null` values ingested with 7.10 will
continue to match `exists` query so a reindex is required.

Fixes elastic#65306
@jtibshirani jtibshirani added the :Analytics/Geo Indexing, search aggregations of geo points and shapes label Nov 20, 2020
@elasticmachine elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Nov 20, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

jimczi added a commit that referenced this issue Nov 24, 2020
A bug was introduced in 7.10 that causes explicit `null` values to be indexed in the _field_names
field. This change fixes this bug for newly ingested data but `null` values ingested with 7.10 will
continue to match `exists` query so a reindex is required.

Fixes #65306
jimczi added a commit that referenced this issue Nov 24, 2020
A bug was introduced in 7.10 that causes explicit `null` values to be indexed in the _field_names
field. This change fixes this bug for newly ingested data but `null` values ingested with 7.10 will
continue to match `exists` query so a reindex is required.

Fixes #65306
jimczi added a commit that referenced this issue Nov 24, 2020
A bug was introduced in 7.10 that causes explicit `null` values to be indexed in the _field_names
field. This change fixes this bug for newly ingested data but `null` values ingested with 7.10 will
continue to match `exists` query so a reindex is required.

Fixes #65306
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/Geo Indexing, search aggregations of geo points and shapes >bug :Search/Search Search-related issues that do not fall into other categories Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Meta label for search team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants