Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.8] Update saved objects migrations documentation for 8.8.0 (#158241) #158270

Merged
merged 1 commit into from
May 23, 2023
Merged

[8.8] Update saved objects migrations documentation for 8.8.0 (#158241) #158270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 2 additions & 10 deletions docs/setup/upgrade/resolving-migration-failures.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ with the new version.
==== Saved object migration failures

If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
perform the migration again when the process restarts. Do not delete any saved objects indices to
fix a failed migration. Unlike previous versions, {kib} 7.12.0 and later does not require deleting
perform the migration again when the process restarts. Do not delete any saved objects indices to
fix a failed migration. Unlike previous versions, {kib} 7.12.0 and later does not require deleting
indices to release a failed migration lock.

If upgrade migrations fail repeatedly, refer to
Expand All @@ -19,14 +19,6 @@ When you address the root cause for the migration failure,
If you're unable to resolve a failed migration, contact Support.


[float]
[[upgrade-migrations-old-indices]]
==== Old `.kibana_N` indices

After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.

[float]
==== Known issues with {fleet} beta
If you see a`timeout_exception` or `receive_timeout_transport_exception` error,
Expand Down
62 changes: 47 additions & 15 deletions docs/setup/upgrade/saved-objects-migration.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,34 +8,66 @@ To access the assistant, go to *Stack Management > Upgrade Assistant*.

WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.

WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.
WARNING: The `kibana.index` and `xpack.tasks.index` configuration settings are obsolete and no longer taken into account in 8.x. If you are using custom index names, please perform the necessary adaptations before attempting to upgrade to 8.x.

[float]
[[upgrade-migrations-process]]
==== How saved objects migrations work

Saved objects are stored in two indices:

* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.

The index aliases `.kibana` and `.kibana_task_manager` always point to
the most up-to-date saved object indices.

When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
adds a write block to the outdated index.

The following tables lists the saved objects indices used by previous {kib} versions.
Saved objects are stored in multiple indices. Whilst all of them start with the `.kibana*` prefix, other `.kibana*` indices exist, which are not used to store saved objects. The following tables lists the saved objects indices used by each {kib} version.

.Saved object indices and aliases per {kib} version
[options="header"]
|=======================
|Upgrading from version | Outdated index (alias)
| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias)
|Upgrading from version | Index | Aliases
| 6.5.0 through 7.3.x
| `.kibana_N`
| `.kibana`
| 7.4.0 through 7.11.x
| `.kibana_N` (`.kibana` alias)

`.kibana_task_manager_N` (`.kibana_task_manager` alias)
| `.kibana_N` +
`.kibana_task_manager_N`
| `.kibana` +
`.kibana_task_manager`
| 7.11.x through 8.7.x
| `.kibana_{kibana_version}_001` +
`.kibana_task_manager_{kibana_version}_001`
| `.kibana`, `.kibana_{kibana_version}` +
`.kibana_task_manager`, `.kibana_task_manager_{kibana_version}`
| 8.8.0+
| `.kibana_{kibana_version}_001` +
`.kibana_alerting_cases_{kibana_version}_001`
`.kibana_analytics_{kibana_version}_001`
`.kibana_ingest_{kibana_version}_001`
`.kibana_task_manager_{kibana_version}_001`
`.kibana_security_solution_{kibana_version}_001`
| `.kibana`, `.kibana_{kibana_version}` +
`.kibana_alerting_cases`, `.kibana_alerting_cases_{kibana_version}`
`.kibana_analytics`, `.kibana_analytics_{kibana_version}`
`.kibana_ingest`, `.kibana_ingest_{kibana_version}`
`.kibana_task_manager`, `.kibana_task_manager_{kibana_version}`
`.kibana_security_solution`, `.kibana_security_solution_{kibana_version}`
|=======================

Starting on 7.11.0, each of the saved objects indices has a couple of aliases, e.g. the `.kibana_8.8.0_001` index has a _default_ alias `.kibana` and a _version_ alias `.kibana_8.8.0`. The _default_ aliases (e.g. `.kibana` and `.kibana_task_manager`) always point to
the most up-to-date saved object indices. Then, _version_ aliases are aligned with the deployed {kib} version.


Starting on 8.6.0, index names aren't necessarily aligned with the deployed {kib} version. When updates on a certain index are compatible, {kib} will keep the existing index instead of creating a new one. This allows for a more efficient upgrade process. The following example illustrates a completely valid state for a 8.8.0 deployment:

* `.kibana_8.8.0_001` index, with `.kibana` and `.kibana_8.8.0` aliases.
* `.kibana_task_manager_8.7.0_001` index, with `.kibana_task_manager` and `.kibana_task_manager_8.8.0` aliases.

Starting on 8.8.0, {kib} splits the main saved object index into multiple ones, as depicted on the table above. When upgrading from a previous version, the {kib} migration process will reindex some saved objects from the `.kibana` index into the new indices, depending on their types. Note that the `.kibana` index still exists, and it continues to store multiple saved object types.

[float]
[[upgrade-migrations-old-indices]]
==== Old {kib} indices

As a deployment is gradually upgraded, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0_001`, `.kibana_7.13.0_001`, `.kibana_8.0.0_001` etc).
{kib} only uses those indices that the _default_ and _version_ aliases point to.
The other, older {kib} saved object indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.