-
Notifications
You must be signed in to change notification settings - Fork 24.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add Snapshot Lifecycle Retention documentation (#47545)
* Add Snapshot Lifecycle Retention documentation This commits adds API and general purpose documentation for SLM retention. Relates to #43663 * Fix docs tests * Update default now that #47604 has been merged * Update docs/reference/ilm/apis/slm-api.asciidoc Co-Authored-By: Gordon Brown <[email protected]> * Update docs/reference/ilm/apis/slm-api.asciidoc Co-Authored-By: Gordon Brown <[email protected]> * Update docs with feedback
- Loading branch information
Showing
4 changed files
with
205 additions
and
21 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| [role="xpack"] | ||
| [testenv="basic"] | ||
| [[slm-retention]] | ||
| == Snapshot lifecycle management retention | ||
|
|
||
| Automatic deletion of older snapshots is an optional feature of snapshot lifecycle management. | ||
| Retention is run as a cluster level task that is not associated with a particular policy's schedule | ||
| (though the configuration of which snapshots to keep is done on a per-policy basis). Retention | ||
| configuration conists of two parts—The first a cluster-level configuration for when retention is | ||
| run and for how long, the second configured on a policy for which snapshots should be eligible for | ||
| retention. | ||
|
|
||
| The cluster level settings for retention are shown below, and can be changed dynamically using the | ||
| <<cluster-update-settings,cluster-update-settings>> API: | ||
|
|
||
| |===================================== | ||
| | Setting | Default value | Description | ||
|
|
||
| | `slm.retention_schedule` | `0 30 1 * * ?` | A periodic or absolute time schedule for when | ||
| retention should be run. Supports all values supported by the cron scheduler: <<schedule-cron,Cron | ||
| scheduler configuration>>. Retention can also be manually run using the | ||
| <<slm-api-execute-retention,Execute retention API>>. Defaults to daily at 1:30am in the master | ||
| node's timezone. | ||
|
|
||
| | `slm.retention_duration` | `"1h"` | A limit of how long SLM should spend deleting old snapshots. | ||
| |===================================== | ||
|
|
||
| Policy level configuration for retention is done inside the `retention` object when creating or | ||
| updating a policy. All of the retention configurations options are optional. | ||
|
|
||
| [source,console] | ||
| -------------------------------------------------- | ||
| PUT /_slm/policy/daily-snapshots | ||
| { | ||
| "schedule": "0 30 1 * * ?", | ||
| "name": "<daily-snap-{now/d}>", | ||
| "repository": "my_repository", | ||
| "retention": { <1> | ||
| "expire_after": "30d", <2> | ||
| "min_count": 5, <3> | ||
| "max_count": 50 <4> | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // TEST[setup:setup-repository] | ||
| <1> Optional retention configuration | ||
| <2> Keep snapshots for 30 days | ||
| <3> Always keep at least 5 successful snapshots | ||
| <4> Keep no more than 50 successful snapshots | ||
|
|
||
| Supported configuration for retention from within a policy are as follows. The default value for | ||
| each is unset unless specified by the user in the policy configuration. | ||
|
|
||
| NOTE: The oldest snapshots are always deleted first, in the case of a `max_count` of 5 for a policy | ||
| with 6 snapshots, the oldest snapshot will be deleted. | ||
|
|
||
| |===================================== | ||
| | Setting | Description | ||
| | `expire_after` | A timevalue for how old a snapshot must be in order to be eligible for deletion. | ||
| | `min_count` | A minimum number of snapshots to keep, regardless of age. | ||
| | `max_count` | The maximum number of snapshots to keep, regardless of age. | ||
| |===================================== | ||
|
|
||
| As an example, the retention setting in the policy configured about would read in English as: | ||
|
|
||
| ____ | ||
| Remove snapshots older than thirty days, but always keep the latest five snapshots. If there are | ||
| more than fifty snapshots, remove the oldest surplus snapshots until there are no more than fifty | ||
| successful snapshots. | ||
| ____ | ||
|
|
||
| If multiple policies are configured to snapshot to the same repository, or manual snapshots have | ||
| been taken without using the <<slm-api-execute,Execute Policy API>>, they are treated as not | ||
| eligible for retention, and do not count towards any limits. This allows multiple policies to have | ||
| differing retention configuration while using the same snapshot repository. | ||
|
|
||
| Statistics for snapshot retention can be retrieved using the <<slm-get-stats,Get Snapshot Lifecycle | ||
| Stats API>>: | ||
|
|
||
| [source,console] | ||
| -------------------------------------------------- | ||
| GET /_slm/stats | ||
| -------------------------------------------------- | ||
| // TEST[continued] | ||
|
|
||
| Which returns a response | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| { | ||
| "retention_runs": 13, <1> | ||
| "retention_failed": 0, <2> | ||
| "retention_timed_out": 0, <3> | ||
| "retention_deletion_time": "1.4s", <4> | ||
| "retention_deletion_time_millis": 1404, | ||
| "policy_stats": [ | ||
| { | ||
| "policy": "daily-snapshots", | ||
| "snapshots_taken": 1, | ||
| "snapshots_failed": 1, | ||
| "snapshots_deleted": 0, <5> | ||
| "snapshot_deletion_failures": 0 <6> | ||
| } | ||
| ], | ||
| "total_snapshots_taken": 1, | ||
| "total_snapshots_failed": 1, | ||
| "total_snapshots_deleted": 0, <7> | ||
| "total_snapshot_deletion_failures": 0 <8> | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE[skip:this is not actually running retention] | ||
| <1> Number of times retention has been run | ||
| <2> Number of times retention failed while running | ||
| <3> Number of times retention hit the `slm.retention_duration` time limit and had to stop before deleting all eligible snapshots | ||
| <4> Total time spent deleting snapshots by the retention process | ||
| <5> Number of snapshots created by the "daily-snapshots" policy that have been deleted | ||
| <6> Number of snapshots that failed to be deleted | ||
| <7> Total number of snapshots deleted across all policies | ||
| <8> Total number of snapshot deletion failures across all policies |