Add index.priority setting to indices created by Kibana

[joshdover]

If Elasticsearch has shard allocation failures (eg. unallocated shards due to a node leaving the cluster), Kibana's core indices should be prioritized over other indices in recovering.

According to the index recovery prioritization docs, indices are recovered by this priority ranking:

• the optional index.priority setting (higher before lower)
• the index creation date (higher before lower)
• the index name (higher before lower)

Newer indices created since Kibana's last upgrade (which may have triggered migrations and creating a new index) will be recovered before Kibana's indices.

By setting the index.priority setting we can ensure that Kibana's indices recover before other, more recently created data indices.

I think we should probably do this for all indices created by Kibana, but the primary SavedObject index (.kibana_n) should have the highest priority.

CC'ing teams with indices this may affect:

• @elastic/kibana-alerting-services (task manager, event log)
• @elastic/kibana-reporting-services (reporting index)
• @elastic/siem (value list & exception list indices)

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[mikecote]

@joshdover is the scope of work of this issue to include task manager and event log or should I create another issue for the alerting team to work on those?

[joshdover]

TBD, but I suspect doing this for task manager (or any SO-managed index) would be trivial or automatic if we do this for the main SO index.

For other indices, let's see if there's a real need. This is sort of a nice-to-have that keeps Kibana responsive enough to make it useful for diagnosing the issues with the ES cluster. I don't think the auxiliary indices should have the same priority as the primary .kibana index, and maybe we only need this on SO indices.

[rudolf]

Will be done as part of #75780