Skip to content

Commit

Permalink
Update saved objects migrations documentation for 8.8.0 (#158241)
Browse files Browse the repository at this point in the history
Documentation must reflect changes brought by the [dot kibana
split](#154888).

Our public facing documentation just mentions two indices e.g.

https://www.elastic.co/guide/en/kibana/8.8/saved-object-migrations.html

https://www.elastic.co/guide/en/kibana/8.8/resolve-migrations-failures.html#upgrade-migrations-old-indices
this one in particular mentions that "all other indices are safe to
delete".
  • Loading branch information
gsoldevila authored May 23, 2023
1 parent 7a62628 commit 70eb9d3
Show file tree
Hide file tree
Showing 2 changed files with 49 additions and 25 deletions.
12 changes: 2 additions & 10 deletions docs/setup/upgrade/resolving-migration-failures.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ with the new version.
==== Saved object migration failures

If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
perform the migration again when the process restarts. Do not delete any saved objects indices to
fix a failed migration. Unlike previous versions, {kib} 7.12.0 and later does not require deleting
perform the migration again when the process restarts. Do not delete any saved objects indices to
fix a failed migration. Unlike previous versions, {kib} 7.12.0 and later does not require deleting
indices to release a failed migration lock.

If upgrade migrations fail repeatedly, refer to
Expand All @@ -19,14 +19,6 @@ When you address the root cause for the migration failure,
If you're unable to resolve a failed migration, contact Support.


[float]
[[upgrade-migrations-old-indices]]
==== Old `.kibana_N` indices

After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.

[float]
==== Known issues with {fleet} beta
If you see a`timeout_exception` or `receive_timeout_transport_exception` error,
Expand Down
62 changes: 47 additions & 15 deletions docs/setup/upgrade/saved-objects-migration.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,34 +8,66 @@ To access the assistant, go to *Stack Management > Upgrade Assistant*.

WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.

WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.
WARNING: The `kibana.index` and `xpack.tasks.index` configuration settings are obsolete and no longer taken into account in 8.x. If you are using custom index names, please perform the necessary adaptations before attempting to upgrade to 8.x.

[float]
[[upgrade-migrations-process]]
==== How saved objects migrations work

Saved objects are stored in two indices:

* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.

The index aliases `.kibana` and `.kibana_task_manager` always point to
the most up-to-date saved object indices.

When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
adds a write block to the outdated index.

The following tables lists the saved objects indices used by previous {kib} versions.
Saved objects are stored in multiple indices. Whilst all of them start with the `.kibana*` prefix, other `.kibana*` indices exist, which are not used to store saved objects. The following tables lists the saved objects indices used by each {kib} version.

.Saved object indices and aliases per {kib} version
[options="header"]
|=======================
|Upgrading from version | Outdated index (alias)
| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias)
|Upgrading from version | Index | Aliases
| 6.5.0 through 7.3.x
| `.kibana_N`
| `.kibana`
| 7.4.0 through 7.11.x
| `.kibana_N` (`.kibana` alias)

`.kibana_task_manager_N` (`.kibana_task_manager` alias)
| `.kibana_N` +
`.kibana_task_manager_N`
| `.kibana` +
`.kibana_task_manager`
| 7.11.x through 8.7.x
| `.kibana_{kibana_version}_001` +
`.kibana_task_manager_{kibana_version}_001`
| `.kibana`, `.kibana_{kibana_version}` +
`.kibana_task_manager`, `.kibana_task_manager_{kibana_version}`
| 8.8.0+
| `.kibana_{kibana_version}_001` +
`.kibana_alerting_cases_{kibana_version}_001`
`.kibana_analytics_{kibana_version}_001`
`.kibana_ingest_{kibana_version}_001`
`.kibana_task_manager_{kibana_version}_001`
`.kibana_security_solution_{kibana_version}_001`
| `.kibana`, `.kibana_{kibana_version}` +
`.kibana_alerting_cases`, `.kibana_alerting_cases_{kibana_version}`
`.kibana_analytics`, `.kibana_analytics_{kibana_version}`
`.kibana_ingest`, `.kibana_ingest_{kibana_version}`
`.kibana_task_manager`, `.kibana_task_manager_{kibana_version}`
`.kibana_security_solution`, `.kibana_security_solution_{kibana_version}`
|=======================

Starting on 7.11.0, each of the saved objects indices has a couple of aliases, e.g. the `.kibana_8.8.0_001` index has a _default_ alias `.kibana` and a _version_ alias `.kibana_8.8.0`. The _default_ aliases (e.g. `.kibana` and `.kibana_task_manager`) always point to
the most up-to-date saved object indices. Then, _version_ aliases are aligned with the deployed {kib} version.


Starting on 8.6.0, index names aren't necessarily aligned with the deployed {kib} version. When updates on a certain index are compatible, {kib} will keep the existing index instead of creating a new one. This allows for a more efficient upgrade process. The following example illustrates a completely valid state for a 8.8.0 deployment:

* `.kibana_8.8.0_001` index, with `.kibana` and `.kibana_8.8.0` aliases.
* `.kibana_task_manager_8.7.0_001` index, with `.kibana_task_manager` and `.kibana_task_manager_8.8.0` aliases.

Starting on 8.8.0, {kib} splits the main saved object index into multiple ones, as depicted on the table above. When upgrading from a previous version, the {kib} migration process will reindex some saved objects from the `.kibana` index into the new indices, depending on their types. Note that the `.kibana` index still exists, and it continues to store multiple saved object types.

[float]
[[upgrade-migrations-old-indices]]
==== Old {kib} indices

As a deployment is gradually upgraded, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0_001`, `.kibana_7.13.0_001`, `.kibana_8.0.0_001` etc).
{kib} only uses those indices that the _default_ and _version_ aliases point to.
The other, older {kib} saved object indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.

0 comments on commit 70eb9d3

Please sign in to comment.