From 3c7b79ed9ae7ac75e4219df6d99e746af19bdc37 Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Thu, 31 Mar 2022 10:24:53 +0200 Subject: [PATCH 01/10] Docs for archival --- docs/reference/archive.asciidoc | 80 +++++++++++++++++++ .../cluster-index-compat.asciidoc | 19 ++--- docs/reference/upgrade.asciidoc | 15 ++-- 3 files changed, 99 insertions(+), 15 deletions(-) create mode 100644 docs/reference/archive.asciidoc diff --git a/docs/reference/archive.asciidoc b/docs/reference/archive.asciidoc new file mode 100644 index 0000000000000..8485cb5a1f689 --- /dev/null +++ b/docs/reference/archive.asciidoc @@ -0,0 +1,80 @@ +[[archive-indices]] += Reading indices from older {es} versions + +{es} has full query and write support for indices created in the previous major +version. If you have indices created in {es} versions 5 or 6, you can now use +the archive functionality to import them into newer {es} versions as well. + +The archive functionality provides slower read-only access to older {es} data, +for compliance or regulatory reasons, the occasional lookback or investigation, +or to rehydrate parts of it. Access to the data is expected to be infrequent, +and can therefore happen with limited performance and query capabilities. + +For this, {es} now has the ability to access older snapshot repositories +(back to version 5). The legacy indices in the <> +can either be <>, or can be directly accessed +via <> so that the archived data +won't even need to fully reside on local disks for access. + +[[archive-indices-supported-field-types]] +== Supported field types + +Old mappings are imported as much "as-is" as possible into {es} 8, but only +provide regular query capabilities on a select subset of fields: + +- <> +- <> +- <> +- <> +- <>: the date `format` setting on date fields is supported + in so far as it behaves similarly across these versions. In case it is not, + this field can be updated on legacy indices so that it can be changed by a + user if need be. +- <>: the `normalizer` setting on keyword + fields is supported in so far as it behaves similarly across these versions. + In case it is not, this field can be updated on legacy indices if need be. +- <>: scoring capabilities are limited, and all + queries return constant scores that are equal to 1.0. The `analyzer` + settings on text fields are supported in so far as they behave similarly + across these versions. In case they do not, they can be updated on legacy + indices if need be. +- <> +- <> +- <> fields +// - <> fields in limited form, not supporting any type of nested queries. +- some basic metadata fields, e.g. `_type` for querying {es} 5 indices +- <> +- <> + +{es} 5 indices with mappings that have {ref-7x}/removal-of-types.html[multiple mapping types] +are collapsed together on a best-effort basis before they are imported. + +In case the auto-import of mappings does not work, or the new version +can't make sense of the mapping, it falls back to a lightweight import of +the mapping where the original mapping is stored in the _meta section of +the imported index's mapping, and relies on the user to put the relevant +mapping parts manually in place. + +== Supported APIs + +Archive indices are read-only, and provide data access via the +<> and <> APIs. +They do not support the <> nor any write APIs. + +Archive indices allow running queries as well as aggregations in so far as +they are <>. + +Due to `_source` access the data can also be [[docs-reindex,reindexed]] +to a new index that has full compatibility with the current {es} version. + +== How to upgrade older {es} clusters? + +Take a snapshot of the indices in the old cluster, delete indices that are not +directly supported by ES 8 (i.e. indices older than 7.0), upgrade the cluster +without the old indices, and then <> the legacy +indices from the snapshot or <> +them via searchable snapshots. + +In the future, we plan on streamlining the upgrade process going forward, +making it easier to take legacy indices along when going to future major +{es} versions. diff --git a/docs/reference/snapshot-restore/cluster-index-compat.asciidoc b/docs/reference/snapshot-restore/cluster-index-compat.asciidoc index 24e348105ee46..3fdc5c38e51df 100644 --- a/docs/reference/snapshot-restore/cluster-index-compat.asciidoc +++ b/docs/reference/snapshot-restore/cluster-index-compat.asciidoc @@ -1,12 +1,13 @@ -[cols="^,^,^,^,^"] +[cols="^,^,^,^,^,^"] |==== -| 4+^h| Cluster version -h| Index creation version | 6.8 | 7.0–7.1 | 7.2–{prev-major-last} | 8.0–{minor-version} -| 5.0–5.6 | {yes-icon} | {no-icon} | {no-icon} | {no-icon} -| 6.0–6.7 | {yes-icon} | {yes-icon} | {yes-icon} | {no-icon} -| 6.8 | {yes-icon} | {no-icon} | {yes-icon} | {no-icon} -| 7.0–7.1 | {no-icon} | {yes-icon} | {yes-icon} | {yes-icon} -| 7.2–{prev-major-last} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} -| 8.0–{minor-version} | {no-icon} | {no-icon} | {no-icon} | {yes-icon} +| 5+^h| Cluster version +h| Index creation version | 6.8 | 7.0–7.1 | 7.2–{prev-major-last} | 8.0–8.2 | 8.3-{minor-version} +| 5.0–5.6 | {yes-icon} | {no-icon} | {no-icon} | {no-icon} | {yes-icon}footnote:archive[Using <> functionality] +| 6.0–6.7 | {yes-icon} | {yes-icon} | {yes-icon} | {no-icon} | {yes-icon}footnote:archive[] +| 6.8 | {yes-icon} | {no-icon} | {yes-icon} | {no-icon} | {yes-icon}footnote:archive[] +| 7.0–7.1 | {no-icon} | {yes-icon} | {yes-icon} | {yes-icon} | {yes-icon} +| 7.2–{prev-major-last} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} | {yes-icon} +| 8.0–8.2 | {no-icon} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} +| 8.3–{minor-version} | {no-icon} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} |==== diff --git a/docs/reference/upgrade.asciidoc b/docs/reference/upgrade.asciidoc index 677877053147e..6b802aba00208 100644 --- a/docs/reference/upgrade.asciidoc +++ b/docs/reference/upgrade.asciidoc @@ -19,12 +19,15 @@ from 7.x]. [[upgrade-index-compatibility]] === Index compatibility -{es} can read indices created in the previous major version. If you -have indices created in 6.x or earlier, you must reindex or delete them -before upgrading to {version}. {es} nodes will fail to start if -incompatible indices are present. Snapshots of 6.x or earlier indices cannot be -restored to a 8.x cluster even if they were created by a 7.x cluster. -The **Upgrade Assistant** in {prev-major-last} identifies any indices +{es} has full query and write support for indices created in the previous major +version. If you have indices created in 6.x or earlier, you might use the +<> to import them into newer {es} +versions, or you must reindex or delete them before upgrading to {version}. +{es} nodes will fail to start if incompatible indices are present. +Snapshots of 6.x or earlier indices can only restored using the +<> to a 8.x cluster even if they +were created by a 7.x cluster. +The **Upgrade Assistant** in {prev-major-last} identifies any indices that need to be reindexed or removed. [discrete] From 7235fc2203775f40ca431929aa8af8f4706a7275 Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Thu, 28 Apr 2022 14:57:44 +0200 Subject: [PATCH 02/10] fixes link? --- docs/reference/index.asciidoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index 06b091bb35df6..d18a0b957245d 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -21,6 +21,8 @@ include::setup.asciidoc[] include::upgrade.asciidoc[] +include::archive.asciidoc[] + include::index-modules.asciidoc[] include::mapping.asciidoc[] From a042ae582be512ee3f247b4b9d652d68afe6503d Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Thu, 28 Apr 2022 15:06:29 +0200 Subject: [PATCH 03/10] lol --- docs/reference/archive.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/archive.asciidoc b/docs/reference/archive.asciidoc index 8485cb5a1f689..1ee25b3cdbd58 100644 --- a/docs/reference/archive.asciidoc +++ b/docs/reference/archive.asciidoc @@ -64,7 +64,7 @@ They do not support the <> nor any write APIs. Archive indices allow running queries as well as aggregations in so far as they are <>. -Due to `_source` access the data can also be [[docs-reindex,reindexed]] +Due to `_source` access the data can also be <> to a new index that has full compatibility with the current {es} version. == How to upgrade older {es} clusters? From dbeac620d206a4cec96577119bb3de56e0227305 Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Tue, 10 May 2022 11:29:35 +0200 Subject: [PATCH 04/10] fix --- docs/reference/archive.asciidoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/reference/archive.asciidoc b/docs/reference/archive.asciidoc index 1ee25b3cdbd58..13695973539a6 100644 --- a/docs/reference/archive.asciidoc +++ b/docs/reference/archive.asciidoc @@ -11,7 +11,7 @@ or to rehydrate parts of it. Access to the data is expected to be infrequent, and can therefore happen with limited performance and query capabilities. For this, {es} now has the ability to access older snapshot repositories -(back to version 5). The legacy indices in the <> +(going back to version 5). The legacy indices in the <> can either be <>, or can be directly accessed via <> so that the archived data won't even need to fully reside on local disks for access. @@ -41,7 +41,6 @@ provide regular query capabilities on a select subset of fields: - <> - <> - <> fields -// - <> fields in limited form, not supporting any type of nested queries. - some basic metadata fields, e.g. `_type` for querying {es} 5 indices - <> - <> From 185b73c9946ce28788de5375d0b7e5685692b7b3 Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Mon, 16 May 2022 15:54:02 +0200 Subject: [PATCH 05/10] rename / toc --- docs/reference/index.asciidoc | 2 -- docs/reference/upgrade.asciidoc | 2 ++ .../archive-indices.asciidoc} | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) rename docs/reference/{archive.asciidoc => upgrade/archive-indices.asciidoc} (96%) diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index 927cedf0befb5..fc4850315a906 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -21,8 +21,6 @@ include::setup.asciidoc[] include::upgrade.asciidoc[] -include::archive.asciidoc[] - include::index-modules.asciidoc[] include::mapping.asciidoc[] diff --git a/docs/reference/upgrade.asciidoc b/docs/reference/upgrade.asciidoc index 6b802aba00208..5bcdc9a4b7c94 100644 --- a/docs/reference/upgrade.asciidoc +++ b/docs/reference/upgrade.asciidoc @@ -45,3 +45,5 @@ the REST API. include::{xes-repo-dir}/security/fips-java17.asciidoc[] include::upgrade/archived-settings.asciidoc[] + +include::upgrade/archive-indices.asciidoc[] diff --git a/docs/reference/archive.asciidoc b/docs/reference/upgrade/archive-indices.asciidoc similarity index 96% rename from docs/reference/archive.asciidoc rename to docs/reference/upgrade/archive-indices.asciidoc index 13695973539a6..458ff5e5007c6 100644 --- a/docs/reference/archive.asciidoc +++ b/docs/reference/upgrade/archive-indices.asciidoc @@ -1,5 +1,5 @@ [[archive-indices]] -= Reading indices from older {es} versions +== Reading indices from older {es} versions {es} has full query and write support for indices created in the previous major version. If you have indices created in {es} versions 5 or 6, you can now use @@ -17,7 +17,7 @@ via <> so that the archived data won't even need to fully reside on local disks for access. [[archive-indices-supported-field-types]] -== Supported field types +=== Supported field types Old mappings are imported as much "as-is" as possible into {es} 8, but only provide regular query capabilities on a select subset of fields: @@ -54,7 +54,7 @@ the mapping where the original mapping is stored in the _meta section of the imported index's mapping, and relies on the user to put the relevant mapping parts manually in place. -== Supported APIs +=== Supported APIs Archive indices are read-only, and provide data access via the <> and <> APIs. @@ -66,7 +66,7 @@ they are <> to a new index that has full compatibility with the current {es} version. -== How to upgrade older {es} clusters? +=== How to upgrade older {es} 5 or 6 clusters? Take a snapshot of the indices in the old cluster, delete indices that are not directly supported by ES 8 (i.e. indices older than 7.0), upgrade the cluster From 43f285a8169a6dab0cc4c433c8c8fc08396b83eb Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Mon, 16 May 2022 16:08:15 +0200 Subject: [PATCH 06/10] discrete --- docs/reference/upgrade/archive-indices.asciidoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/upgrade/archive-indices.asciidoc b/docs/reference/upgrade/archive-indices.asciidoc index 458ff5e5007c6..a084af5d4b5bb 100644 --- a/docs/reference/upgrade/archive-indices.asciidoc +++ b/docs/reference/upgrade/archive-indices.asciidoc @@ -16,6 +16,7 @@ can either be <>, or can be directly accessed via <> so that the archived data won't even need to fully reside on local disks for access. +[discrete] [[archive-indices-supported-field-types]] === Supported field types @@ -54,6 +55,7 @@ the mapping where the original mapping is stored in the _meta section of the imported index's mapping, and relies on the user to put the relevant mapping parts manually in place. +[discrete] === Supported APIs Archive indices are read-only, and provide data access via the @@ -66,6 +68,7 @@ they are <> to a new index that has full compatibility with the current {es} version. +[discrete] === How to upgrade older {es} 5 or 6 clusters? Take a snapshot of the indices in the old cluster, delete indices that are not From cea246982e6db8dddacb8b1a10f835d94e2b7a06 Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Tue, 17 May 2022 11:56:45 +0200 Subject: [PATCH 07/10] add highlight --- docs/changelog/86261.yaml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 docs/changelog/86261.yaml diff --git a/docs/changelog/86261.yaml b/docs/changelog/86261.yaml new file mode 100644 index 0000000000000..1556152bcc26a --- /dev/null +++ b/docs/changelog/86261.yaml @@ -0,0 +1,18 @@ +pr: 86261 +summary: Snapshots as simple archives +area: Search +type: feature +issues: + - 81210 +highlight: + title: Reading indices from older Elasticsearch versions + body: |- + Elasticsearch has full query and write support for indices created in the previous major + version. If you have indices created in Elasticsearch versions 5 or 6, you can now use + the archive functionality to import and query these indices as well. + + The archive functionality provides slower read-only access to older data, + for compliance or regulatory reasons, the occasional lookback or investigation, + or to rehydrate parts of it. Access to the data is expected to be infrequent, + and can therefore happen with limited performance and query capabilities. + notable: true From 99fab8450a59dc8a0f206e3deee6f759bb782bfe Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Wed, 25 May 2022 23:38:23 +0200 Subject: [PATCH 08/10] Delete docs/changelog/86261.yaml --- docs/changelog/86261.yaml | 18 ------------------ 1 file changed, 18 deletions(-) delete mode 100644 docs/changelog/86261.yaml diff --git a/docs/changelog/86261.yaml b/docs/changelog/86261.yaml deleted file mode 100644 index 1556152bcc26a..0000000000000 --- a/docs/changelog/86261.yaml +++ /dev/null @@ -1,18 +0,0 @@ -pr: 86261 -summary: Snapshots as simple archives -area: Search -type: feature -issues: - - 81210 -highlight: - title: Reading indices from older Elasticsearch versions - body: |- - Elasticsearch has full query and write support for indices created in the previous major - version. If you have indices created in Elasticsearch versions 5 or 6, you can now use - the archive functionality to import and query these indices as well. - - The archive functionality provides slower read-only access to older data, - for compliance or regulatory reasons, the occasional lookback or investigation, - or to rehydrate parts of it. Access to the data is expected to be infrequent, - and can therefore happen with limited performance and query capabilities. - notable: true From ffc6112b7a05ddfc451710259d1ada9824ca75d3 Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Mon, 30 May 2022 11:12:57 +0200 Subject: [PATCH 09/10] review comments --- .../reference/upgrade/archive-indices.asciidoc | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/reference/upgrade/archive-indices.asciidoc b/docs/reference/upgrade/archive-indices.asciidoc index a084af5d4b5bb..de108645b2366 100644 --- a/docs/reference/upgrade/archive-indices.asciidoc +++ b/docs/reference/upgrade/archive-indices.asciidoc @@ -10,7 +10,7 @@ for compliance or regulatory reasons, the occasional lookback or investigation, or to rehydrate parts of it. Access to the data is expected to be infrequent, and can therefore happen with limited performance and query capabilities. -For this, {es} now has the ability to access older snapshot repositories +For this, {es} has the ability to access older snapshot repositories (going back to version 5). The legacy indices in the <> can either be <>, or can be directly accessed via <> so that the archived data @@ -29,6 +29,7 @@ provide regular query capabilities on a select subset of fields: - <> - <>: the date `format` setting on date fields is supported in so far as it behaves similarly across these versions. In case it is not, + for example {ref-7x}/migrate-to-java-time.html[when using custom date formats], this field can be updated on legacy indices so that it can be changed by a user if need be. - <>: the `normalizer` setting on keyword @@ -49,11 +50,16 @@ provide regular query capabilities on a select subset of fields: {es} 5 indices with mappings that have {ref-7x}/removal-of-types.html[multiple mapping types] are collapsed together on a best-effort basis before they are imported. -In case the auto-import of mappings does not work, or the new version -can't make sense of the mapping, it falls back to a lightweight import of -the mapping where the original mapping is stored in the _meta section of -the imported index's mapping, and relies on the user to put the relevant -mapping parts manually in place. +In case the auto-import of mappings does not work, or the new {es} version +can't make sense of the mapping, it falls back to importing the index without +the mapping, but stores the original mapping in the <> +section of the imported index. The legacy mapping can then be introspected +using the <> API and an updated mapping can +be manually put in place using the <> API, +copying and adapting relevant sections of the legacy mapping to work with the +current {es} version. While auto-import is expected to work in most cases, +failures of doing so should be raised with the Elastic team for future +improvements. [discrete] === Supported APIs From b58f463488a530bfb8f456be40b3a374377a8121 Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Mon, 30 May 2022 13:11:56 +0200 Subject: [PATCH 10/10] as long as + link --- docs/reference/upgrade/archive-indices.asciidoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference/upgrade/archive-indices.asciidoc b/docs/reference/upgrade/archive-indices.asciidoc index de108645b2366..5e905aa9215a4 100644 --- a/docs/reference/upgrade/archive-indices.asciidoc +++ b/docs/reference/upgrade/archive-indices.asciidoc @@ -28,16 +28,16 @@ provide regular query capabilities on a select subset of fields: - <> - <> - <>: the date `format` setting on date fields is supported - in so far as it behaves similarly across these versions. In case it is not, + as long as it behaves similarly across these versions. In case it is not, for example {ref-7x}/migrate-to-java-time.html[when using custom date formats], this field can be updated on legacy indices so that it can be changed by a user if need be. - <>: the `normalizer` setting on keyword - fields is supported in so far as it behaves similarly across these versions. + fields is supported as long as it behaves similarly across these versions. In case it is not, this field can be updated on legacy indices if need be. - <>: scoring capabilities are limited, and all queries return constant scores that are equal to 1.0. The `analyzer` - settings on text fields are supported in so far as they behave similarly + settings on text fields are supported as long as they behave similarly across these versions. In case they do not, they can be updated on legacy indices if need be. - <> @@ -58,8 +58,8 @@ using the <> API and an updated mapping can be manually put in place using the <> API, copying and adapting relevant sections of the legacy mapping to work with the current {es} version. While auto-import is expected to work in most cases, -failures of doing so should be raised with the Elastic team for future -improvements. +failures of doing so should be https://github.com/elastic/elasticsearch/issues/new/choose[raised] +with the Elastic team for future improvements. [discrete] === Supported APIs