Better UX around migration to data tiers during 7.16->8.0 upgrade

[joegallo]

Related to #76147

In many circumstances, indices created prior to 7.10.0 won't have a _tier_preference. We're flagging that as a critical issue (preventing upgrades), but we're doing it for every such index, and there could be many of them. That doesn't result in an especially good user experience in the Upgrade Assistant, see screenshot:

Resolving this for individual indices is straightforward, for example:

PUT some-index-1/_settings
{
    "index.routing.allocation.include._tier_preference":"data_hot"
}

Multiple indices can be resolved at the same time, as well (e.g. PUT some-index-2,some-index-3/_settings, etc).

The global fix, however, is the execute ILM's Migrate to data tiers routing API, which will add a _tier_preference for all indices that do not currently have one (note: it will choose "data_hot" for datastream backing indices, and "data_content" otherwise). Invoking it looks like this:

POST _ilm/stop

POST _ilm/migrate_to_data_tiers

POST _ilm/start

Perhaps some better options?

Like with Migrate system indices, we could make this a single step earlier in the upgrade assistant -- these remain N critical issues, but they're displayed via a special Migrate indices to data tiers callout with a button to execute the global fix.

We could switch from a critical issue per index found to instead have a single over-arching critical issue if any such indices are found (ideally with a list of the indices referenced off that one critical issue). That would reduce the noise and "wow, that's a lot of issues" effect. This would be best if paired with a button to execute the global fix.

We could keep N critical issues, but have a 'Quick resolve' button that fixes it for all indices -- there'd still be the "wow, that's a lot of issues" effect, but at least they'd all get solved in one button click.

Reproduction

I spun up a 7.5.0 cluster and created five some-index-N indices, then upgraded to 7.15.2 and created a some-index-6 index -- in all six cases without a body, just a plain PUT index call. Resulting in this collection of indices:

some-index-1 through some-index-5 do not have a _tier_preference:

While some-index-6 (having been created on 7.15.2) does have a _tier_preference:

Then I upgraded to the most recent 7.16.0 BC and opened the Kibana Upgrade Assistant -- notice that there are 5 Critical deprecation issues:

Drilling down into the deprecation issues, there's one for each of the indices that doesn't have a _tier_preference:

Kudos

Hat tip to @LeeDr for catching this!

/cc @cjcenizal @dakrone @henningandersen @jakelandis (edit: @leehinman sorry I pinged you by mistake!)

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[LeeDr]

Any progress on this?

[joegallo]

#80971 is merged, which rewords the deprecation issue and crucially downgrades it from 'critical' to 'warning', so this is no longer a blocker I'm going to close it.