Bugfix/202 transform date add date conversion #622

stevanbz · 2022-12-08T22:03:15Z

Issue #, if available:
#202

Description of changes:

This PR enables creation of target index mapping for a date fields by following next "rules":

If the date is used in a term aggregation/grouping, default format of a date (strict_date_optional_time||epoch_millis) will be used for mapping the target date field
If the date is used in metric aggregation(MIN, MAX, COUNT), default format of a date (strict_date_optional_time||epoch_millis) will be used for mapping the target date field
Once the term aggregation values for date fields are retrieved from a cluster (in epoch time millis format), automatically will be formatted to human readable format (ISO 8601 uuuu-MM-dd'T'HH:mm:ss.SSSZZ)
If the aggregations (MAX, MIN, COUNT) are applied on a date fields (in transform job), the same logic like in previous step will be applied when retrieving the values (since the cluster will return the epoch time millis once the agg function is applied on a given field)
Enabling specifying the format once the date_histogram is being used

@bowenlan-amzn
Here is some contextual background:

Once grouping by date field as term is executed, the request is being translated to composite aggregation request, that is executed against the cluster (on transform side) ie.:

"groups": [
    {
    "terms": {
      "source_field": "@timestamp",
      "target_field": "our_custom_target_date"
    }
    }
  ]...

is being translated to:

 "aggregations":{
      "some_test_agg":{
         "composite":{
            "size":10,
            "sources":[
               {
                  "our_custom_target_date":{
                     "terms":{
                        "field":"@timestamp",
                        "missing_bucket":true,
                        "order":"asc"
                     }
                  }
               }
            ]
         }

First thing noticed is that composite aggregation doesn't allow specifying a format of a date field that will be used for the purpose of grouping the data into buckets when using composite aggregation. On the other hand, cluster can return "original" field value as a string if regular terms grouping is being used or if the aggregation function (like min/max) is applied:

"aggs": {
    "custom_target_field": {
      "terms": {
        "field": "@timestamp",
        "size": 10
      }
    }
  }

Cluster, then returns:

"buckets" : [
        {
          "key" : 1661344590443,
          "key_as_string" : "2022-08-24T12:36:30.443Z",
          "doc_count" : 7
        }

So long story short, the cluster "flattened" the field values to epcoh millis and then did the grouping - in the case of composite aggregation the format can't be specified UNLESS the date_histogram is being used. In the case of "normal" term grouping, cluster by default returns the value _as_string.

Specifying a format is enabled on a date_histogram:

"date_histogram": {
     "source_field": "timestamp",
     "target_field": "hour_grouping",
     "calendar_interval": "hour",
     "format": "yyyy-MM-dd"
   }

Caveat:
When defining the interval and format (when date_histogram is used), user must be aware of their co-relation ie. since transform creates a buckets and once each result group is being returned it returns after_key (which is used in order to detect where the processing ended, in which bucket) which is used later for pagination, cluster won't know how to create the after key (since it did bucketing on the hour level but because of the format it flattens in a date format and after key that is being returned is in one moment always the same).

Beside the date format when using the date_histogram, the idea was to create target index mapping with the date field types used as a field definition, when grouping by a date is applied.

Further improvement would be - taking care about the date format that is used for a date field in a source index (user can specify multiple formats so, somehow once the execution of transform occurs, we would need to figure out in which format the date is stored and maybe use this format for storing and formatting the target index date field once the cluster retrieves the value)

CheckList:

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-09T19:52:17Z

Codecov Report

Merging #622 (e470657) into main (d921f0b) will decrease coverage by 0.02%.
The diff coverage is 74.37%.

❗ Current head e470657 differs from pull request most recent head 96af8b6. Consider uploading reports for the commit 96af8b6 to get more accurate results

@@             Coverage Diff              @@
##               main     #622      +/-   ##
============================================
- Coverage     75.90%   75.88%   -0.02%     
- Complexity     2811     2849      +38     
============================================
  Files           362      364       +2     
  Lines         16042    16176     +134     
  Branches       2301     2324      +23     
============================================
+ Hits          12176    12275      +99     
- Misses         2535     2563      +28     
- Partials       1331     1338       +7

Impacted Files	Coverage Δ
...pensearch/indexmanagement/IndexManagementPlugin.kt	`89.49% <ø> (+0.07%)`	⬆️
...h/indexmanagement/transform/util/TransformUtils.kt	`0.00% <0.00%> (ø)`
...earch/indexmanagement/transform/TransformRunner.kt	`82.24% <66.66%> (+0.32%)`	⬆️
.../action/preview/TransportPreviewTransformAction.kt	`85.71% <72.00%> (-1.25%)`	⬇️
...xmanagement/transform/TargetIndexMappingService.kt	`72.34% <72.34%> (ø)`
...management/common/model/dimension/DateHistogram.kt	`77.55% <77.77%> (-0.48%)`	⬇️
...arch/indexmanagement/transform/TransformIndexer.kt	`69.09% <85.71%> (ø)`
...ndexmanagement/transform/TransformSearchService.kt	`71.42% <100.00%> (+1.17%)`	⬆️
...indexmanagement/transform/util/TransformContext.kt	`80.00% <100.00%> (+5.00%)`	⬆️

... and 9 files with indirect coverage changes