Downsample action can attempt to downsample on the same interval and delete data

[andreidan]

Elasticsearch Version

8.10, 8.11, 8.12

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin

Problem Description

When an ILM policy is created we don't allow the same donwsample action (same interval) to be configured in subsequent phases.

This can be achieved though by updating the ILM policy by removing it from the hot phase and configuring it with the same fixed_interval in the warm phase.

This can cause some in-flight indices that were downsampled in hot to now be downsampled in warm for the same interval.
ILM will go through the warm/downsample action, skip the downsample operation (as the source index is donwsampled already) and delete the source index. This leads to data loss.

The exact use case is “update the next phase to contain the downsample action from the previous phase, whilst the managed index is waiting to enter the next phase”.

Steps to Reproduce

PUT _ilm/policy/ds-experiment-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "5m",
            "max_primary_shard_size": "10gb"
          },
          "downsample": {
            "fixed_interval": "1m"
          }
        }
      },
      "warm": {
        "min_age": "10m",
        "actions": {
        }
      }
    }
  }
}

POST my-ds-experiment/_rollover


PUT _component_template/ds-experiment-component
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "ds-experiment-policy"
        },
        "look_ahead_time": "4m",
        "codec": "best_compression"
      }
    },
    "mappings": {
      "runtime": {},
      "properties": {
        "@timestamp": {
          "format": "strict_date_optional_time_nanos",
          "type": "date_nanos"
        },
        "networkName": {
          "time_series_dimension": true,
          "type": "keyword"
        }
      }
    }
  }
}

PUT _index_template/my-ds-experiment-template
{
  "index_patterns": [
    "my-ds-experiment*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series",
        "routing_path": [
          "networkName"
        ]
      }
    }
  },
  "composed_of": [
    "ds-experiment-component"
  ],
  "data_stream": {
    "allow_custom_routing": false,
    "hidden": false
  }
}

PUT _data_stream/my-ds-experiment

PUT my-ds-experiment/_bulk
{ "create":{ } }
{ "@timestamp": "<replace with current timestamp>", "networkName":  "test1"}

// update policy while index has been already created
PUT _ilm/policy/ds-experiment-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "5m",
            "max_primary_shard_size": "10gb"
          }
        }
      },
      "warm": {
        "min_age": "10m",
        "actions": {
          "downsample": {
            "fixed_interval": "1m"
          }
        }
      }
    }
  }
}

// when index is moved to warm and downsampling happens, it will be deleted

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)