ILM: parse origination date from index name

[andreidan]

Introduce the index.lifecycle.parse_origination_date setting that
indicates if the origination date should be parsed from the index name.
If set to true an index which doesn't match the expected format (namely
indexName-{dateFormat}-optional_digits will fail before being created.
The origination date will be parsed when initialising a lifecycle for an
index and it will be set as the index.lifecycle.origination_date for
that index.

A user set value for index.lifecycle.origination_date will always
override a possible parsable date from the index name.

Together with #46561 closes #42449

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?
• If submitting code, have you built your formula locally prior to submission with gradle check?
• If submitting code, is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed.
• If submitting code, have you checked that your submission is for an OS and architecture that we support?
• If you are submitting this code for a class then read our policy for that.

[elasticmachine]

Pinging @elastic/es-core-features

[dakrone]

Thanks for opening this for discussion Andrei! I have some ideas for how to implement it.

the user simply enables/disables the parsing, without having the option to configure the date format in the index name. Should we be more configurable and allow the user to set the date format? Or even more to provide the full index name pattern? (personal opinion here is to be more restrictive as this feature is not about legacy indexes but newly created ones which must obey the pattern)

I think to start we should support the yyyy.MM.dd format that you mentioned. We can always expand later to customization of the format if we have to, but it would be harder to go the other way around.

One thing I notice from your PR is parsing ^.*-(.+)$, which would support index-2019.09.18, but I think we will also want to support index-2019.09.18-000231 for indices that are using rollover (so the same, just ^.*-(.+)(-[\d]+)?$ I think (I did not double check the regex))

the way the "parse origination date" and explicit "origination date" interact is that the parse setting has priority if both are configured. I think if we're going this way it should actually be the other way around and have the explicit setting supersede the parse setting. Or maybe having both settings configured should yield an error?

I agree that the origination_date set in the settings should always have priority. If it's not set, and parse_origination_date is set, we should attempt to automatically set the origination date setting to what the parsed value is. We can do this I think in InitializePolicyContextStep#performAction when we update the cluster state with the new index metadata (we can update the settings at the same time)

How lenient should we be to parsing errors (ie. can't find "-" in the index name or the date parsing fails)?

We should be extremely strict about parsing errors, so that we avoid any unintended behavior around the age of the index, especially since this can lead to data being deleted with an unintended retention period.

We can do this by registering an IndexEventListener that implements the beforeIndexAddedToCluster method. In this method, we could validate that the index name correctly matched the regex for parsing the date out of the name. If it doesn't match, we can throw an exception and prevent the index from being created (rather than leniently creating it without parsing the date).

[andreidan]

@elasticmachine update branch

[andreidan]

Adding the initial draft PR description here for completeness (and to provide context to @dakrone 's comment)

Initial draft for parsing the origination date from the index name.

This is meant solely as support for discussion on how to implement
this feature.

A few assumptions that we should discuss:

• the user simply enables/disables the parsing, without having the
  option to configure the date format in the index name. Should we
  be more configurable and allow the user to set the date format? Or
  even more to provide the full index name pattern? (personal opinion
  here is to be more restrictive as this feature is not about legacy
  indexes but newly created ones which must obey the pattern)
• the way the "parse origination date" and explicit "origination date"
  interact is that the parse setting has priority if both are configured.
  I think if we're going this way it should actually be the other way
  around and have the explicit setting supersede the parse setting.
  Or maybe having both settings configured should yield an error?
• How lenient should we be to parsing errors (ie. can't find "-" in the
  index name or the date parsing fails)?

[andreidan]

@elasticmachine update branch

[dakrone]

Thank Andrei, I think this will be a great feature to have! I left some comments and questions for you on this.

[andreidan]

@elasticmachine update branch

[dakrone]

LGTM, thanks for iterating on this Andrei! I played with it locally and it works very well.