Docs for snapshots as simple archives #86261
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,13 @@ | ||
|
|
||
| [cols="^,^,^,^,^,^"] | ||
| |==== | ||
| | 5+^h| Cluster version | ||
| h| Index creation version | 6.8 | 7.0–7.1 | 7.2–{prev-major-last} | 8.0–8.2 | 8.3-{minor-version} | ||
| | 5.0–5.6 | {yes-icon} | {no-icon} | {no-icon} | {no-icon} | {yes-icon}footnote:archive[Using <<archive-indices,archive>> functionality] | ||
| | 6.0–6.7 | {yes-icon} | {yes-icon} | {yes-icon} | {no-icon} | {yes-icon}footnote:archive[] | ||
| | 6.8 | {yes-icon} | {no-icon} | {yes-icon} | {no-icon} | {yes-icon}footnote:archive[] | ||
| | 7.0–7.1 | {no-icon} | {yes-icon} | {yes-icon} | {yes-icon} | {yes-icon} | ||
| | 7.2–{prev-major-last} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} | {yes-icon} | ||
| | 8.0–8.2 | {no-icon} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} | ||
| | 8.3–{minor-version} | {no-icon} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} | ||
| |==== |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| [[archive-indices]] | ||
| == Reading indices from older {es} versions | ||
|
|
||
| {es} has full query and write support for indices created in the previous major | ||
| version. If you have indices created in {es} versions 5 or 6, you can now use | ||
| the archive functionality to import them into newer {es} versions as well. | ||
|
|
||
| The archive functionality provides slower read-only access to older {es} data, | ||
| for compliance or regulatory reasons, the occasional lookback or investigation, | ||
| or to rehydrate parts of it. Access to the data is expected to be infrequent, | ||
| and can therefore happen with limited performance and query capabilities. | ||
|
|
||
| For this, {es} has the ability to access older snapshot repositories | ||
| (going back to version 5). The legacy indices in the <<snapshot-restore,snapshot repository>> | ||
| can either be <<restore-snapshot-api,restored>>, or can be directly accessed | ||
| via <<searchable-snapshots,searchable snapshots>> so that the archived data | ||
| won't even need to fully reside on local disks for access. | ||
|
|
||
| [discrete] | ||
| [[archive-indices-supported-field-types]] | ||
| === Supported field types | ||
|
|
||
| Old mappings are imported as much "as-is" as possible into {es} 8, but only | ||
| provide regular query capabilities on a select subset of fields: | ||
|
|
||
| - <<number,Numeric types>> | ||
| - <<boolean,`boolean` type>> | ||
| - <<ip,`ip` type>> | ||
| - <<geo-point,`geo_point` type>> | ||
| - <<date,`date` types>>: the date `format` setting on date fields is supported | ||
| in so far as it behaves similarly across these versions. In case it is not, | ||
| for example {ref-7x}/migrate-to-java-time.html[when using custom date formats], | ||
| this field can be updated on legacy indices so that it can be changed by a | ||
| user if need be. | ||
| - <<keyword-field-type,`keyword` type>>: the `normalizer` setting on keyword | ||
| fields is supported in so far as it behaves similarly across these versions. | ||
|
There was a problem hiding this comment. here too "is supported as long as..." |
||
| In case it is not, this field can be updated on legacy indices if need be. | ||
| - <<text-field-type,`text` type>>: scoring capabilities are limited, and all | ||
| queries return constant scores that are equal to 1.0. The `analyzer` | ||
| settings on text fields are supported in so far as they behave similarly | ||
|
|
||
| across these versions. In case they do not, they can be updated on legacy | ||
| indices if need be. | ||
| - <<multi-fields,Multi-fields>> | ||
| - <<field-alias,Field aliases>> | ||
| - <<object,`object`>> fields | ||
| - some basic metadata fields, e.g. `_type` for querying {es} 5 indices | ||
| - <<runtime-mapping-fields,runtime fields>> | ||
| - <<mapping-source-field,`_source` field>> | ||
|
|
||
| {es} 5 indices with mappings that have {ref-7x}/removal-of-types.html[multiple mapping types] | ||
| are collapsed together on a best-effort basis before they are imported. | ||
|
|
||
| In case the auto-import of mappings does not work, or the new {es} version | ||
| can't make sense of the mapping, it falls back to importing the index without | ||
| the mapping, but stores the original mapping in the <<mapping-meta-field,_meta>> | ||
| section of the imported index. The legacy mapping can then be introspected | ||
| using the <<indices-get-mapping,GET mapping>> API and an updated mapping can | ||
| be manually put in place using the <<indices-put-mapping,update mapping>> API, | ||
| copying and adapting relevant sections of the legacy mapping to work with the | ||
| current {es} version. While auto-import is expected to work in most cases, | ||
| failures of doing so should be raised with the Elastic team for future | ||
|
|
||
| improvements. | ||
|
|
||
| [discrete] | ||
| === Supported APIs | ||
|
|
||
| Archive indices are read-only, and provide data access via the | ||
| <<search-search,search>> and <<search-field-caps,field capabilities>> APIs. | ||
| They do not support the <<docs-get,Get API>> nor any write APIs. | ||
|
|
||
| Archive indices allow running queries as well as aggregations in so far as | ||
| they are <<archive-indices-supported-field-types,supported by the given field type>>. | ||
|
|
||
| Due to `_source` access the data can also be <<docs-reindex,reindexed>> | ||
| to a new index that has full compatibility with the current {es} version. | ||
|
|
||
| [discrete] | ||
| === How to upgrade older {es} 5 or 6 clusters? | ||
|
|
||
| Take a snapshot of the indices in the old cluster, delete indices that are not | ||
| directly supported by ES 8 (i.e. indices older than 7.0), upgrade the cluster | ||
| without the old indices, and then <<restore-snapshot-api,restore>> the legacy | ||
| indices from the snapshot or <<searchable-snapshots-api-mount-snapshot,mount>> | ||
| them via searchable snapshots. | ||
|
|
||
| In the future, we plan on streamlining the upgrade process going forward, | ||
| making it easier to take legacy indices along when going to future major | ||
| {es} versions. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what do you mean with "is supported in so far as it behaves similarly across these versions"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have changed the meaning of various custom date formats / date literals across versions (e.g. year of era becomes week-based year), see https://www.elastic.co/guide/en/elasticsearch/reference/7.17/migrate-to-java-time.html#java-time-migration-incompatible-date-formats
Perhaps I could link to the above for additional context, as well as limit the scope of the problem to "custom date formats".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sounds good to expand a bit, thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you mean "is supported as long as it behaves similarly across versions"?