Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs][migrations v2] Update SO migration docs to include removal of index write block when handling corrupt SOs. #103014

Merged
merged 3 commits into from
Jun 24, 2021
Merged

[docs][migrations v2] Update SO migration docs to include removal of index write block when handling corrupt SOs. #103014

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
93758c2
Update corrupt SO docs to include removal of index write blocks.
lukeelmers Jun 22, 2021
7e90e1e
Update based on initial feedback.
lukeelmers Jun 23, 2021
b497d5a
Update based on feedback.
lukeelmers Jun 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 39 additions & 8 deletions docs/setup/upgrade/upgrade-migrations.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,22 +55,53 @@ This section highlights common causes of {kib} upgrade failures and how to preve
There is a known issue in v7.12.0 for users who tried the fleet beta. Upgrade migrations fail because of a large number of documents in the `.kibana` index.

This can cause Kibana to log errors like:
> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]

[source,sh]
--------------------------------------------
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]

Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
lukeelmers marked this conversation as resolved.
Show resolved Hide resolved
--------------------------------------------

See https://github.com/elastic/kibana/issues/95321 for instructions to work around this issue.

[float]
===== Corrupt saved objects
We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.
We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment.

Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.

For example, given the following error message:
> Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.

The following steps must be followed to allow the upgrade migration to succeed.
Please be aware the Dashboard having ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` belonging to the space `marketing_space` will no more be available:
1. Delete the corrupt document with `DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275`
2. Restart {kib}
[source,sh]
--------------------------------------------
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
--------------------------------------------

The following steps must be followed to allow the upgrade migration to succeed:
lukeelmers marked this conversation as resolved.
Show resolved Hide resolved

. Remove the write block which the migration system has placed on the index:
lukeelmers marked this conversation as resolved.
Show resolved Hide resolved
+
[source,sh]
--------------------------------------------
PUT .kibana_7.12.1_001/_settings
{
"index": {
"blocks.write": false
}
}
--------------------------------------------

. Delete the corrupt document:
+
[source,sh]
--------------------------------------------
DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275
--------------------------------------------

. Restart {kib}.

Please be aware the Dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **will no longer be available**.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When restoring a snapshot is not an option, it's also recommended to delete the indices the migration created (the temp and target indices) before retrying the migration


[float]
===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings
Expand Down