Kibana migrations potentially fail if there's a running ES snapshot

[rudolf]

From https://github.com/elastic/cloud/issues/41742

When Kibana runs it's saved object migrations it's possible that it migrates a concrete index into an alias which performs the following steps:

1. Reindex.kibana into .kibana_n
2. Delete .kibana index
3. Create .kibana alias pointing to .kibana_n

If a snapshot is currently running step (2) will throw an exception "Cannot delete indices that are being snapshotted" and cause the migration to fail requiring manual intervention.

This is more likely to happen during 7.3 -> 7.4 since we perform a migration of .kibana_task_manager into an alias and .kibana_task_manager_n

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[spalger]

We had to implement retry logic for this in the esArchiver since snapshots are running pretty frequently on cloud... We should probably do something similar in the migrations

kibana/src/es_archiver/lib/indices/delete_index.js

Lines 49 to 56 in 3613174

	if (retryIfSnapshottingCount > 0 && isDeleteWhileSnapshotInProgressError(error)) {
	stats.waitingForInProgressSnapshot(index);
	await waitForSnapshotCompletion(client, index, log);
	return await deleteIndex({
	...options,
	retryIfSnapshottingCount: retryIfSnapshottingCount - 1
	});
	}

[rudolf]

Thanks, this is exactly what we need.

[tylersmalley]

There was also retry logic added to the check for if the migrations are up-to-date:

https://github.com/elastic/kibana/blob/master/src/core/server/saved_objects/migrations/core/elastic_index.ts#L152-L201