8.8.0 migrations temporary ES failures could cause permanent migration failure

[rudolf]

Usually, when Kibana migrations fail due to a temporary problem in Elasticsearch, Kibana is able to automatically succeed in finishing the migration when the failure condition is resolved.

When upgrading to 8.8.0 it's possible that a temporary Elasticsearch error (high disk watermark or circuit breaker exceptions) causes Kibana to permanently fail with an error like:

  {
    "index": ".kibana_alerting_cases_8.8.0_001",
    "id": "action:90cb3e60-3fdb-11ed-b808-09c70a0298e6",
    "cause": {
      "type": "cluster_block_exception",
      "reason": "index [.kibana_alerting_cases_8.8.0_001] blocked by: [FORBIDDEN/8/index write (api)];"
    },
    "status": 403
  }

This can happen if:

1. the [.kibana] migrator finishes it's migration and completes the UPDATE_TARGET_MAPPINGS_META step
2. one or more of the other index migrators e.g. [.kibana_alerting_cases] are unable to successfully complete the CLONE_TEMP_TO_TARGET step

This causes an inconsistent state where the metadata in .kibana suggests the splitting migration had completed but it in fact had not.

Mitigation

1. Ensure the underlying Elasticsearch health condition has been resolved
2. Stop all Kibana instances
3. Restore the Kibana feature state from the last snapshot before the upgrade https://www.elastic.co/guide/en/kibana/8.8/upgrade-migrations-rolling-back.html#_roll_back_by_restoring_the_kibana_feature_state_from_a_snapshot
4. Start Kibana.

Potential data loss

Failed 8.8.0 migrations could lead to data loss under some very specific circumstances.

Detection

If the following criteria apply your cluster might have lost data, please contact support or revert to a snapshot from before the upgrade:

1. You're upgrading an existing Kibana cluster
2. The upgrade to 8.8.0 failed and then eventually succeeded
3. The Kibana server logs during the upgrade contain INIT -> CREATE_NEW_TARGET entries.

Overview

When upgrading to a new stack version, Kibana runs a migration logic to upgrade saved objects documents. Eventually, these objects are copied over to new version indices.

As part of the dot kibana split, when upgrading to 8.8.0 (or later), the migration logic will create a bunch of new indices, and distribute Saved Objects stored in .kibana across these new indices. If the migration succeeds only partially (e.g. some indices are completely migrated and others aren't), we can be in a situation where some of the saved objects from .kibana index aren't properly copied over to the new version indices.

Scenario A. The write_blocked indices

• First migration attempt:
  • One of the newer indices, e.g. .kibana_alerting_cases, is correctly created and contains all the saved objects that are intended to go into that index.
  • It fails due to some external factors (e.g. shard_limit_exceptions from ES), and .kibana index is NOT migrated.
• Subsequent attempts:
  • .kibana index is on a "before split" state, so Kibana determines it needs to do the split.
  • .kibana_alerting_cases migrator will run a reindex flow, locking the existing .kibana_alerting_cases_8.8.0_001.
  • That very same migrator will attempt to update documents on the same index on a later step, failing to do so due to the write_block.

Kibana upgrade process gets stuck on a bootloop.
Removing the write_block is pointless, as it will be re-created at each restart.
The bootloop is not completely hard-locked though. If .kibana migrator manages to complete the migration process before the .kibana_alerting_cases fails, Kibana will believe it is on an "after split" state, and the other indices won't be write_blocked again. Manually removing the write_block at this point:

• Will allow Kibana to start normally.
• However, if any of the other indices' migrators did not complete the migration process, their corresponding SO will be missing. ⚠️ That's the reason why removing these write_blocks might be a bad idea.

Scenario B. The silent data loss

• First migration attempt:
  • The .kibana migrator finishes dispatching all the saved objects to their corresponding indices, which also includes the newer version of the.kibana index itself, e.g. .kibana_8.8.0_001.
  • Another index migrator, e.g. .kibana_alerting_cases fails to clone from the temporary index .kibana_alerting_cases_8.8.0_reindex_temp into the target index .kibana_alerting_cases_8.8.0_001.
  • The migration attempt fails, but the saved object indices are left on a "post split" state.
• Next migration attempt:
  • Kibana sees that the .kibana index is aligned with current stack version, and it determines that there is no need to split.
  • .kibana_alerting_cases migrator does not see its own index (it did not get to create the entrypoint aliases, aka .kibana_alerting_cases and .kibana_alerting_cases_8.8.0), so it assumes it is on a fresh deployment scenario. Depending on whether the first attempt failed:
    • Before the clone operation. It creates an empty .kibana_alerting_cases_8.8.0_001 and completes the migration.
      • ⚠️ All saved objects intended to go to that index are not copied over from .kibana_<previousVersion>_001, so from the SavedObjects API standpoint, they are effectively lost.
    • After the clone operation. It will attempt to create a .kibana_alerting_cases_8.8.0_001 which already exists (no-op), perform a few updates, and finally create the entrypoint aliases. This scenario has 2 possible sub-branches:
      • Before the _mappings are updated (most likely). The index created in the previous attempt won't have any mappings on it. The stored saved objects won't be indexed by any fields, and thus they won't be searchable ⚠️. Restarting Kibana again in this scenario should trigger a "compatible mappings" migration, which should then update ALL the documents in the index, so that ES can properly index them and they become searchable.
      • Ater the _mappings are updated (less likely). In this scenario, the documents should already be searchable, without any impact ✅ .

Technical details

• The saved object relocation (into multiple indices) requires reindexing, and it is performed in multiple steps:
  • First, SO documents are transformed and added to temporary indices (e.g. .kibana_analytics_8.8.0_reindex_temp).
  • Then, these indices are cloned to the target indices (e.g. .kibana_analytics_8.8.0_001).
  • Finally, 2 aliases are created for each new index (e.g. .kibana_analytics and .kibana_analytics_8.8.0).

In order to perform migrations, Kibana launches a "migrator" instance for each of the SO indices. These migrators run in parallel, and they handle the upgrade process described above.

Since the migrators run "independently", it might happen during an upgrade that one migrator succeeds and another one does not. This was an acceptable scenario up until 8.8.0, cause each index was truly independent of each other. If a migrator failed to migrate a specific index, it would simply retry next start.

In 8.8.0, we introduce dependencies between migrators:

• The .kibana migrator must dispatch SO documents to other indices, and to do so, it must wait for other migrators to create their corresponding .kibana_<domain>_8.8.0_reindex_temp temporary indices.
• Once all migrators have finished transforming and adding documents to the temporary indices, they can all proceed to clone and update aliases.

There's one particularity that makes .kibana index special: it stores information about the type => index breakdown in the .kibana.mapping._meta.indexTypesMap property. At startup, this information allows Kibana to determine if some types must be relocated into other indices during an upgrade.

This is where the current issue lays: .kibana migrator is currently dispatching SO documents to other indices and then completing the rest of the migration process for its own index. However, upon successful migration, it will condition the behaviour of the migration logic on subsequent attempts (by storing the _meta.indexTypesMap). Thus, it should make sure that all migrators have finished cloning + migrating their own indices before considering itself successful.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

Updated the issue to include the log entries that if present, would indicate potential data loss.

[Bamieh]

@rudolf; @gsoldevila and I discussed that the count might be false in case some documents are deleted during migration (disabled plugins, unused, etc)

[jrodewig]

@rudolf Should we add this to the 8.8 known issues? ¹ If so, do you mind drafting an update for @elastic/kibana-docs to review?

Footnotes

1. https://www.elastic.co/guide/en/kibana/8.8/release-notes-8.8.0.html ↩

[lukeelmers]

Should we add this to the 8.8 known issues?

I think we should, but we might want to wait a few more days until we know with 100% certainty that the fix will land in 8.8.1, that way we can give clear direction in the docs.

[gsoldevila]

@rudolf Should we add this to the 8.8 known issues? 1 If so, do you mind drafting an update for @elastic/kibana-docs to review?

@jrodewig @elastic/kibana-docs I've created a PR that adds a description for the issue in the Known issues, against the 8.8 branch:

#159197