[ML] Possibility to end up with .ml-state-write alias on multiple indices

[droberts195]

During QA upgrade testing from 6.8.14 to 7.11.1 we saw an error occur when we tried to have the .ml-state-write alias pointing at both .ml-state and .ml-state-000001. This is the relevant section of the Elasticsearch log:

It looks like the code in MlIndexAndAlias.createIndexAndAliasIfNecessary is working on the basis that no indices exist that match the pattern .ml-state*. The reason for this is the resolver.concreteIndexNames(clusterState, IndicesOptions.lenientExpandOpen(), indexPattern) call combined with the fact that the .ml-state index is temporarily unavailable during the upgrade. Lenient expand open deliberately ignores unavailable indices. For the purposes of determining whether an alias is already present on an index we should not ignore unavailable indices.

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[droberts195]

The reason for this is the resolver.concreteIndexNames(clusterState, IndicesOptions.lenientExpandOpen(), indexPattern) call combined with the fact that the .ml-state index is temporarily unavailable during the upgrade. Lenient expand open deliberately ignores unavailable indices.

On further investigation this analysis is wrong.

The reason for the problem is that 3 threads simultaneously try to create the index, because 3 jobs need to be restarted on the node after the upgrade. This can be seen from the 3 About to create first concrete index log lines. We do account for this possibility by ignoring ResourceAlreadyExists exceptions. But after that the logic for adjusting the alias assumes that it doesn't need moving.