Allow ILM to transition to implicit cached steps

[andreidan]

ILM tries to honour the cached phase however, some steps are implicit (e.g. injected actions or the terminal policy/phase step) Currently, ILM would throw an exception if the currently cached phase was removed:

step [{"phase":"warm","action":"complete","name":"complete"}] for index [index]
with policy [my-policy] does not exist

ILM would currently also throw an exception if the next step was an implicit one and the phase was removed from the underlying policy e.g. if the index is on the migrate step and looking to transition to the check-migration step, whilt the warm phase doesn't exist in the policy anymore

step [{"phase":"warm","action":"migrate","name":"check-migration"}] for index
[index] with policy [my-policy] does not exist

This fixes these scenarios by enhancing the
PolicyStepsRegistry#parseStepKeysFromPhase method to compute all the steps in the phase (including all implicit steps)

Fixes #91749

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @andreidan, I've created a changelog YAML for you.

[andreidan]

Note: there might be a slight performance penalty here due to LifecyclePolicy#toSteps( https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/LifecyclePolicy.java#L215 )
now executing on every transition

[andreidan]

Potentially superseding #91754 (pending discussion on the potential performance penalty)

[dakrone]

@joegallo I think you introduced the PhaseCacheManagement piece, which was to increase the performance of this specific method if I recall correctly. Do you have thoughts on how this will affect performance?

[andreidan]

@dakrone I introduced the PhaseCacheManagement to handle the phase manipulations we perform when migrating a deployment to data tiers (ie. we potentially remove the allocate action from phases). The reason this class exists is for encapsulating the interaction and management of the cached phase.

The change I propose here eludes the PhaseCacheManagement to keep the change as isolated as possible (ie. code affects only the validation of the transition to the next step) - hence the code is in PolicyStepsRegistry.

However, we'll continue to disregard implicit steps when parsing the cached phase (as we do today) when it comes to establishing if the current cached phase is safely updateable when updating a policy (ie. PhaseCacheManagement#isIndexPhaseDefinitionUpdatable will continue to work as it is today)

@joegallo what do you think ?

[joegallo]

I've assigned this to myself -- I want to do a little more looking at parseStepKeysFromPhase and parseStepsFromPhase. I'm not in love with the Stringly-typed phaseDef argument we're using for both. It can be null, a phase name like 'new' or 'complete' or a full blown json String that we parse. I'd like to see if I can tighten that up a bit.

[andreidan]

@joegallo do you have an update here? Could we try to benchmark this approach and see if it's feasible?

[andreidan]

@elasticmachine update branch

[joegallo]

The plan is to merge this and give the benchmarks a little while to show an effect, then consult with Armin later this week to determine the path forward (if there's no effect, then that would imply a short meeting 😉).