Range query broken on dates

[motherhubbard]

Elasticsearch version (bin/elasticsearch --version):

{
"cluster_uuid" : "P_sQQBFfQ2Wt_Cn_rwz8Fg",
"version" : {
"number" : "7.4.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "22e1767283e61a198cb4db791ea66e3f11ab9910",
"build_date" : "2019-09-27T08:36:48.569419Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

OS version (uname -a if on a Unix-like system):
Linux SNIP 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

When performing a range query with an lte value and an ISO Date I get an error parsing the date. I don't get these errors on the gte field. Example queries and results shown below:

curl localhost:9208/MYINDEX/_search -d '{"query":{"range":{"core.date.timestamp":{"gte":"2018-01-01T00:00:00Z", "lte":"2018-01-02T00:00:00.000Z"}}}}' -H "Content-Type: application/json"
{"error":{"root_cause":[{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-02T00:00:00.000Z' could not be parsed, unparsed text found at index 4"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"MYINDEX","node":"YBAtWyr8QHiqxM5rqlx1eg","reason":{"type":"parse_exception","reason":"failed to parse date field [2018-01-02T00:00:00.000Z] with format [epoch_millis||strict_date_optional_time]: [Text '2018-01-02T00:00:00.000Z' could not be parsed, unparsed text found at index 4]","caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-02T00:00:00.000Z' could not be parsed, unparsed text found at index 4"}}}],"caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-02T00:00:00.000Z' could not be parsed, unparsed text found at index 4"}},"status":400}

In fact just lte fails:

curl localhost:9208/MYINDEX/_search -d '{"query":{"range":{"core.date.timestamp":{"lte":"2018-01-01T00:00:00Z"}}}}' -H "Content-Type: application/json"
{"error":{"root_cause":[{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-01T00:00:00Z' could not be parsed, unparsed text found at index 4"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"MYINDEX","node":"YBAtWyr8QHiqxM5rqlx1eg","reason":{"type":"parse_exception","reason":"failed to parse date field [2018-01-01T00:00:00Z] with format [epoch_millis||strict_date_optional_time]: [Text '2018-01-01T00:00:00Z' could not be parsed, unparsed text found at index 4]","caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-01T00:00:00Z' could not be parsed, unparsed text found at index 4"}}}],"caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-01T00:00:00Z' could not be parsed, unparsed text found at index 4"}},"status":400}

curl localhost:9208/MYINDEX/_search -d '{"query":{"range":{"core.date.timestamp":{"lte":"2018-01-02T00:00:00.000Z"}}}}' -H "Content-Type: application/json"
{"error":{"root_cause":[{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-02T00:00:00.000Z' could not be parsed, unparsed text found at index 4"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"MYINDEX","node":"YBAtWyr8QHiqxM5rqlx1eg","reason":{"type":"parse_exception","reason":"failed to parse date field [2018-01-02T00:00:00.000Z] with format [epoch_millis||strict_date_optional_time]: [Text '2018-01-02T00:00:00.000Z' could not be parsed, unparsed text found at index 4]","caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-02T00:00:00.000Z' could not be parsed, unparsed text found at index 4"}}}],"caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '2018-01-02T00:00:00.000Z' could not be parsed, unparsed text found at index 4"}},"status":400}

The following queries work:

curl localhost:9208/MYINDEX/_search -d '{"query":{"range":{"core.date.timestamp":{"gte":"2018-01-01T00:00:00Z", "lte":"now"}}}}' -H "Content-Type: application/json"

curl localhost:9208/MYINDEX/_search -d '{"query":{"range":{"core.date.timestamp":{"gte":"2018-01-01T00:00:00Z"}}}}' -H "Content-Type: application/json"

curl localhost:9208/MYINDEX/_search -d '{"query":{"range":{"core.date.timestamp":{"lte":1546300800000}}}}' -H "Content-Type: application/json"

All of these queries work on an ES5 system.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[not-napoleon]

This sounds like it might be a case of #40913 ?

[motherhubbard]

Thank you for responding so quickly. Its worth saying this was not an upgraded system from 6. We installed 7 onto a clean system.

I did a quick search for date_time_parse_exception but couldnt find anything relevant.

Cheers

[motherhubbard]

Here is the mapping for the field also:

"timestamp" : {
                  "type" : "date",
                  "format" : "epoch_millis||strict_date_optional_time"
                },

[gaobinlong]

In 7.5, the problem doesn't reproduce. It's a bug and fixed in 7.5. see #47423

[motherhubbard]

Awesome thanks!