Docs for snapshots as simple archives #86261
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,13 @@ | ||
|
|
||
| [cols="^,^,^,^,^,^"] | ||
| |==== | ||
| | 5+^h| Cluster version | ||
| h| Index creation version | 6.8 | 7.0–7.1 | 7.2–{prev-major-last} | 8.0–8.2 | 8.3-{minor-version} | ||
| | 5.0–5.6 | {yes-icon} | {no-icon} | {no-icon} | {no-icon} | {yes-icon}footnote:archive[Using <<archive-indices,archive>> functionality] | ||
| | 6.0–6.7 | {yes-icon} | {yes-icon} | {yes-icon} | {no-icon} | {yes-icon}footnote:archive[] | ||
| | 6.8 | {yes-icon} | {no-icon} | {yes-icon} | {no-icon} | {yes-icon}footnote:archive[] | ||
| | 7.0–7.1 | {no-icon} | {yes-icon} | {yes-icon} | {yes-icon} | {yes-icon} | ||
| | 7.2–{prev-major-last} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} | {yes-icon} | ||
| | 8.0–8.2 | {no-icon} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} | ||
| | 8.3–{minor-version} | {no-icon} | {no-icon} | {no-icon} | {yes-icon} | {yes-icon} | ||
| |==== |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| [[archive-indices]] | ||
| == Reading indices from older {es} versions | ||
|
|
||
| {es} has full query and write support for indices created in the previous major | ||
| version. If you have indices created in {es} versions 5 or 6, you can now use | ||
| the archive functionality to import them into newer {es} versions as well. | ||
|
|
||
| The archive functionality provides slower read-only access to older {es} data, | ||
| for compliance or regulatory reasons, the occasional lookback or investigation, | ||
| or to rehydrate parts of it. Access to the data is expected to be infrequent, | ||
| and can therefore happen with limited performance and query capabilities. | ||
|
|
||
| For this, {es} now has the ability to access older snapshot repositories | ||
| (going back to version 5). The legacy indices in the <<snapshot-restore,snapshot repository>> | ||
| can either be <<restore-snapshot-api,restored>>, or can be directly accessed | ||
| via <<searchable-snapshots,searchable snapshots>> so that the archived data | ||
| won't even need to fully reside on local disks for access. | ||
|
|
||
| [discrete] | ||
| [[archive-indices-supported-field-types]] | ||
| === Supported field types | ||
|
|
||
| Old mappings are imported as much "as-is" as possible into {es} 8, but only | ||
| provide regular query capabilities on a select subset of fields: | ||
|
|
||
| - <<number,Numeric types>> | ||
| - <<boolean,`boolean` type>> | ||
| - <<ip,`ip` type>> | ||
| - <<geo-point,`geo_point` type>> | ||
| - <<date,`date` types>>: the date `format` setting on date fields is supported | ||
| in so far as it behaves similarly across these versions. In case it is not, | ||
|
There was a problem hiding this comment. what do you mean with "is supported in so far as it behaves similarly across these versions"? There was a problem hiding this comment. We have changed the meaning of various custom date formats / date literals across versions (e.g. year of era becomes week-based year), see https://www.elastic.co/guide/en/elasticsearch/reference/7.17/migrate-to-java-time.html#java-time-migration-incompatible-date-formats Perhaps I could link to the above for additional context, as well as limit the scope of the problem to "custom date formats". There was a problem hiding this comment. sounds good to expand a bit, thanks. There was a problem hiding this comment. do you mean "is supported as long as it behaves similarly across versions"? |
||
| this field can be updated on legacy indices so that it can be changed by a | ||
| user if need be. | ||
| - <<keyword-field-type,`keyword` type>>: the `normalizer` setting on keyword | ||
| fields is supported in so far as it behaves similarly across these versions. | ||
|
There was a problem hiding this comment. here too "is supported as long as..." |
||
| In case it is not, this field can be updated on legacy indices if need be. | ||
| - <<text-field-type,`text` type>>: scoring capabilities are limited, and all | ||
| queries return constant scores that are equal to 1.0. The `analyzer` | ||
| settings on text fields are supported in so far as they behave similarly | ||
|
|
||
| across these versions. In case they do not, they can be updated on legacy | ||
| indices if need be. | ||
| - <<multi-fields,Multi-fields>> | ||
| - <<field-alias,Field aliases>> | ||
| - <<object,`object`>> fields | ||
| - some basic metadata fields, e.g. `_type` for querying {es} 5 indices | ||
| - <<runtime-mapping-fields,runtime fields>> | ||
| - <<mapping-source-field,`_source` field>> | ||
|
|
||
| {es} 5 indices with mappings that have {ref-7x}/removal-of-types.html[multiple mapping types] | ||
| are collapsed together on a best-effort basis before they are imported. | ||
|
|
||
| In case the auto-import of mappings does not work, or the new version | ||
| can't make sense of the mapping, it falls back to a lightweight import of | ||
| the mapping where the original mapping is stored in the _meta section of | ||
| the imported index's mapping, and relies on the user to put the relevant | ||
| mapping parts manually in place. | ||
|
There was a problem hiding this comment. I wonder if this is enough guidance for users to solve problems when importing the mappings. what's your thinking? There was a problem hiding this comment. I can clarify the process a bit more (e.g. user can inspect the original mapping using get mappings API, and use put mappings API to add selected portions back to index). Additionally I would like to ask users to report failures to auto-import mapping to us (but with the expectation that we will treat it as low-priority items to be fixed). There was a problem hiding this comment. sounds great, I was also wondering if users know what you mean with "_meta" section. I would have to look it up myself :) |
||
|
|
||
| [discrete] | ||
| === Supported APIs | ||
|
|
||
| Archive indices are read-only, and provide data access via the | ||
| <<search-search,search>> and <<search-field-caps,field capabilities>> APIs. | ||
| They do not support the <<docs-get,Get API>> nor any write APIs. | ||
|
|
||
| Archive indices allow running queries as well as aggregations in so far as | ||
| they are <<archive-indices-supported-field-types,supported by the given field type>>. | ||
|
|
||
| Due to `_source` access the data can also be <<docs-reindex,reindexed>> | ||
| to a new index that has full compatibility with the current {es} version. | ||
|
|
||
| [discrete] | ||
| === How to upgrade older {es} 5 or 6 clusters? | ||
|
|
||
| Take a snapshot of the indices in the old cluster, delete indices that are not | ||
| directly supported by ES 8 (i.e. indices older than 7.0), upgrade the cluster | ||
| without the old indices, and then <<restore-snapshot-api,restore>> the legacy | ||
| indices from the snapshot or <<searchable-snapshots-api-mount-snapshot,mount>> | ||
| them via searchable snapshots. | ||
|
|
||
| In the future, we plan on streamlining the upgrade process going forward, | ||
| making it easier to take legacy indices along when going to future major | ||
| {es} versions. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: remove "now"?