Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downsample action can attempt to downsample on the same interval and delete data #102249

Closed
andreidan opened this issue Nov 15, 2023 · 1 comment · Fixed by #102250
Closed

Downsample action can attempt to downsample on the same interval and delete data #102249

andreidan opened this issue Nov 15, 2023 · 1 comment · Fixed by #102250
Labels
>bug :Data Management/ILM+SLM Index and Snapshot lifecycle management Team:Data Management Meta label for data/management team v8.11.2 v8.12.0

Comments

@andreidan
Copy link
Contributor

Elasticsearch Version

8.10, 8.11, 8.12

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin

Problem Description

When an ILM policy is created we don't allow the same donwsample action (same interval) to be configured in subsequent phases.

This can be achieved though by updating the ILM policy by removing it from the hot phase and configuring it with the same fixed_interval in the warm phase.

This can cause some in-flight indices that were downsampled in hot to now be downsampled in warm for the same interval.
ILM will go through the warm/downsample action, skip the downsample operation (as the source index is donwsampled already) and delete the source index. This leads to data loss.

The exact use case is “update the next phase to contain the downsample action from the previous phase, whilst the managed index is waiting to enter the next phase”.

Steps to Reproduce

PUT _ilm/policy/ds-experiment-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "5m",
            "max_primary_shard_size": "10gb"
          },
          "downsample": {
            "fixed_interval": "1m"
          }
        }
      },
      "warm": {
        "min_age": "10m",
        "actions": {
        }
      }
    }
  }
}

POST my-ds-experiment/_rollover


PUT _component_template/ds-experiment-component
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "ds-experiment-policy"
        },
        "look_ahead_time": "4m",
        "codec": "best_compression"
      }
    },
    "mappings": {
      "runtime": {},
      "properties": {
        "@timestamp": {
          "format": "strict_date_optional_time_nanos",
          "type": "date_nanos"
        },
        "networkName": {
          "time_series_dimension": true,
          "type": "keyword"
        }
      }
    }
  }
}

PUT _index_template/my-ds-experiment-template
{
  "index_patterns": [
    "my-ds-experiment*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series",
        "routing_path": [
          "networkName"
        ]
      }
    }
  },
  "composed_of": [
    "ds-experiment-component"
  ],
  "data_stream": {
    "allow_custom_routing": false,
    "hidden": false
  }
}

PUT _data_stream/my-ds-experiment

PUT my-ds-experiment/_bulk
{ "create":{ } }
{ "@timestamp": "<replace with current timestamp>", "networkName":  "test1"}

// update policy while index has been already created
PUT _ilm/policy/ds-experiment-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "5m",
            "max_primary_shard_size": "10gb"
          }
        }
      },
      "warm": {
        "min_age": "10m",
        "actions": {
          "downsample": {
            "fixed_interval": "1m"
          }
        }
      }
    }
  }
}

// when index is moved to warm and downsampling happens, it will be deleted

Logs (if relevant)

No response

@andreidan andreidan added >bug :Data Management/ILM+SLM Index and Snapshot lifecycle management v8.12.0 v8.11.2 labels Nov 15, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@elasticsearchmachine elasticsearchmachine added the Team:Data Management Meta label for data/management team label Nov 15, 2023
elasticsearchmachine pushed a commit that referenced this issue Nov 15, 2023

Verified

This commit was signed with the committer’s verified signature. The key has expired.
addaleax Anna Henningsen
This fixes a bug in ILM where if an ILM policy is updated such that the
next phase (e.g. warm) contains the downsample action from the previous
phase (e.g. hot), whilst the managed index is waiting to enter the next
phase (e.g. warm), if the managed index was already downsampled in hot
it would get deleted in the warm phase.

Fixes #102249
andreidan added a commit to andreidan/elasticsearch that referenced this issue Nov 15, 2023
)

This fixes a bug in ILM where if an ILM policy is updated such that the
next phase (e.g. warm) contains the downsample action from the previous
phase (e.g. hot), whilst the managed index is waiting to enter the next
phase (e.g. warm), if the managed index was already downsampled in hot
it would get deleted in the warm phase.

Fixes elastic#102249
elasticsearchmachine pushed a commit that referenced this issue Nov 15, 2023
…102251)

This fixes a bug in ILM where if an ILM policy is updated such that the
next phase (e.g. warm) contains the downsample action from the previous
phase (e.g. hot), whilst the managed index is waiting to enter the next
phase (e.g. warm), if the managed index was already downsampled in hot
it would get deleted in the warm phase.

Fixes #102249
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Data Management/ILM+SLM Index and Snapshot lifecycle management Team:Data Management Meta label for data/management team v8.11.2 v8.12.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants