Composite aggregation iterates forever with date_histogram format

[MartinBurian]

(Apologies, we have a managed Elasticsearch instance, I can only get the data through the API)

Elasticsearch version (bin/elasticsearch --version):
"version" : "7.10.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",

Plugins installed: []

JVM version (java -version):
"version" : "15.0.1",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "15.0.1+9"

OS version (uname -a if on a Unix-like system):
"name" : "Linux",
"pretty_name" : "Ubuntu 20.04.1 LTS",
"arch" : "amd64",
"version" : "5.4.0-58-generic",

Description of the problem including expected versus actual behavior:
When I run a composite aggregation with a certain format of date_histogram source, the pagination goes on infinitely.

Steps to reproduce:

1. Set up index

PUT /composite-reproduce
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic": "false",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "app": {
        "type": "keyword"
      },
      "result": {
        "type": "keyword"
      }
    }
  }
}

2. Add some test data

POST /composite-reproduce/_doc
{
  "@timestamp": "2021-04-27T14:38:51.789Z",
  "app": "f1rst 4pp",
  "result": "SUCCESS"
}


POST /composite-reproduce/_doc
{
  "@timestamp": "2021-04-27T14:38:51.789Z",
  "app": "f1rst 4pp",
  "result": "FAILURE"
}

POST /composite-reproduce/_doc
{
  "@timestamp": "2021-04-27T14:38:51.789Z",
  "app": "another app",
  "result": "SUCCESS"
}

3. Query the data - it returns the expected results

GET /composite-reproduce/_search
{
  "size": 0,
  "track_total_hits": -1,
  "aggregations": {
    "compositeAggregation": {
      "composite": {
        "size": 1000,
        "sources": [
          {
            "app": {
              "terms": {
                "field": "app"
              }
            }
          },
          {
            "date": {
              "date_histogram": {
                "field": "@timestamp",
                "value_type": "date",
                "order": "asc",
                "format": "YYYY-MM-dd",
                "calendar_interval": "1d",
                "time_zone": "Europe/Prague"
              }
            }
          }
        ]
      }
    }
  }
}

4. Grab the after_key and try to get the following page (it should be empty)

GET /composite-reproduce/_search
{
  "size": 0,
  "track_total_hits": -1,
  "aggregations": {
    "compositeAggregation": {
      "composite": {
        "size": 1000,
        "sources": [
          {
            "app": {
              "terms": {
                "field": "app"
              }
            }
          },
          {
            "date": {
              "date_histogram": {
                "field": "@timestamp",
                "value_type": "date",
                "order": "asc",
                "format": "YYYY-MM-dd",
                "calendar_interval": "1d",
                "time_zone": "Europe/Prague"
              }
            }
          }
        ],
        "after" : {
          "app" : "f1rst 4pp",
          "date" : "2021-04-27"
        }
      }
    }
  }
}

Actual response:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "compositeAggregation" : {
      "after_key" : {
        "app" : "f1rst 4pp",
        "date" : "2021-04-27"
      },
      "buckets" : [
        {
          "key" : {
            "app" : "f1rst 4pp",
            "date" : "2021-04-27"
          },
          "doc_count" : 2
        }
      ]
    }
  }
}

Subsequent calls with the after_key always return non-empty bucket list, resulting in infinite iteration.

Please note how I made a mistake and used YYYY in the format instead of yyyy (I wanted ordinary dates, not week-based year). When I changed the format to yyyy-MM-dd, the page iteration ends correctly.

May be related to #68963

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[qilingzhao]

it seems like some condition missing in DateFormatters, i try to fix it.

[not-napoleon]

So this is subtle. It looks like what's going on here is that the date formatter is happy to output mixed week-based and month-based values, but doesn't want to mix them in parsing. I'm looking at this post from Stack Overflow. The relevant excerpt from which says:

On parsing, SimpleDateFormat expects a matching set of values: either day, month, year or day of week, week in year, week-year. Since you supplied a week-year but did not supply day of week and week in year, those to values have been assumed as 1.

Based on that, my guess is the month and day are being ignored when we parse the after key, and so the date ends up as something like 2020-12-27 or some such. What ever the first day of the week-year was. And of course, the aggregation is correct in saying there are more buckets after that date, which is where you get the infinite looping behavior.

I need to do some more testing to make sure this is the problem, but so far this seems the most likely cause. Presuming that proves out, I think we should reject formats that can't be used for parsing correctly, at least for composite sources if not for aggs in general.

Relates to #47469

[not-napoleon]

@pgomulka - tagging you for feedback. Does my theory seem correct here? And if so, what options do we have for detecting formats that are going to lose data when parsing?

[pgomulka]

when parsing "2021-04-27" with "YYYY-MM-dd" elasticsearch will return 2021-01-04T00:00Z
Initial non resolved date is {WeekBasedYear[WeekFields[MONDAY,4]]=2021, MonthOfYear=4, DayOfMonth=27},ISO
We then want to resolve to Instant, and to do this we apply defaulting logic.
the reason is, that our parsing logic sees YYYY week based year, so it then expect week-of-a-week-based-year. As it is not present we would default to first day of the first week of the year. in 2021 - it was Monday, 4th of January.

basically any date from 2021 parsed with format "YYYY-MM-dd" will return 2021-01-04T00:00Z
I suppose this is the reason for an infinite loop

as suggested, instead of Y use y :)

date defaulting logic
https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/common/time/DateFormatters.java#L1909
date parsing logic
https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/search/DocValueFormat.java#L285
parser is JavaDateMathParser