Parse timestring from index name for comparing min_index_age condition

[spapadop]

Is your feature request related to a problem? Please describe.
Let's say I have a 3 months deletion policy for all my indices and now it is August 1st, 2021.
My simple policy has the following condition to transit from 'hot' to 'deleted' state: "min_index_age": "90d".

Now, I migrate some historical indices (e.g. July or June 2021) to the cluster.
As it is very common, the date is noted within the index name, e.g. myindex-2021-07 or myindex-2021-07-03.
Having just created those indices in my cluster, even though they refer to past data, the index creation time is naturally August 1st, 2021. As a result, given my applied policy those historical indices will be deleted as of 90 days after their creation date (August 1st, 2021), which is not correct.

Same problem potentially applies if I perform re-indexing, as the creation date of the new index will be 'today', even though the data are older than today's date.

Describe the solution you'd like
Introduce the option for obtaining the index age after parsing the index name, instead of always using the index creation date.
A simple approach could be:

"conditions": {
                "min_index_age": "90d",
                "source": "index_name",
                "timestring": "%Y.%m.%d"
              }

with the default value (if source is not defined) to be the index_creation_date, as it is now.
"timestring" can may be omitted and calculated instead, adding some further complication on the back-end. But I guess keeping it (at least optionally) is a more robust solution.

Describe alternatives you've considered
I can't think of a working alternative with the current plugin capabilities.

[dbbaughe]

The recommendation would then possibly have issues where you have a mix of indices, some which are from reindexing or restores where the name represents the date and the creation_date is wrong along w/ indices that should use creation date.

To resolve that.. might make more sense to instead have each index define where it wants the creation date source to be, i.e. an index level setting that says use creation date or index name or use this specific value (as you might reindex or restore an index called foo-index which doesn't have a date in it).

The downside being it's up to the user to populate this index setting so that ISM knows which one to use correctly.

@spapadop What do you think?

[spapadop]

Sounds like a better idea indeed! And for sure not having specified anything the default behavior should remain as-is now, i.e. taking the "now" time as creation_date.

[mikerebilly]

Hi, is this feature request being considered or prioritized? I'm currently porting a system from ES to OS and this is an obstacle I've also bumped into. I would imagine that this might be a somewhat common use case?

[bowenlan-amzn]

Under https://github.com/opensearch-project/index-management/projects/1#card-78383842