Support Snapshot Management (*SM*)

[bowenlan-amzn]

Snapshot Management

The purpose of this request for comments (RFC) is to introduce our plans of adding the Snapshot Management feature and collect feedback and discuss our plans with community. This RFC will cover the high-level functionality of the Snapshot Management and goes into some implementation details.

Overview

Problem Statement

OpenSearch provides an official backup mechanism which is the snapshot feature. The snapshots are incremental, i.e., new snapshot will only save the new change since the last run. Though the backup mechanism is provided, but we don’t provide an automated backup mechanism inside the OpenSearch cluster. So users have to rely on the external management tool like Curator. On the other hand, Index State Management (ISM) allows users to manage the index lifecycle, for example, user can use ISM to decrease the index replica count from 5 to 1 after 1 day, and take a snapshot before deleting the index after 30 days. However, snapshot feature is for backing up multiple indices but ISM is only scoped to work on single index. So we need a similar management feature, Snapshot Management (SM), but meant to work on the snapshot. This is a popular feature request from our repository issue and forum.

Desired State

We plan to provide a new plugin feature, snapshot management (SM), which can be directly installed to or bundled with the OpenSearch cluster. Users can create a SM policy to specify the time interval/schedule for creating a snapshot of certain group of indices into one specific snapshot repository. SM will then help to manage the created snapshots by allowing users to specify how many and/or how long they want to keep these snapshots. When the policy is running, user can see the creating or deleting snapshots. User should be able to manually trigger a one-time policy run, this is useful before a cluster maintenance or migration. Notification function will be integrated with SM so user can be notified when snapshots being created. New UI pages and APIs will be implemented to support the SM feature, moreover, we can add UI pages to show existing snapshots or even register snapshot repository and restore snapshots to provide a one-stop station for snapshot management.

Time based trigger (cron schedule) will be the only trigger we will deliver in the P0. In the future iterations, triggers like cluster status change event (cluster turning to unhealthy state) or index size can be added.

User Story

1. User uses the snapshot management UI or manually call REST API to register a repository repo1.
2. User creates a SM policy specifying snapshot index pattern log* to repo1 by a cron schedule 0 20 * * *, and delete condition 5, 3d.
3. User directly executes the policy to create a snapshot right away, to check if the snapshot is created as expected.
4. User monitors a snapshot is being created automatically at 20:00 on the same day.
5. After 3d, user will see 3 snapshots created by this SM policy, after 6d, since there were 6 snapshots created, the oldest one will be deleted.
6. User can stop or start this SM policy when facing a bad cluster or maintenance time.
7. User can see all the existing snapshots in the snapshot management UI.

Proposed Solution

Snapshot Management (SM) feature will be another feature inside Index Management (IM) plugin.

Users can define a SM policy daily-snapshot through REST APIs or frontend pages which also connects to REST APIs. The policy specifies the time interval/schedule of automated snapshot creation. Based on this policy, SM will create a corresponding scheduled running job daily-snapshot-slm that managed by job scheduler plugin. The IM plugin lives on all nodes in the cluster along with the job scheduler plugin, but only data nodes which hold the shards of IM system index will be used to execute the scheduled job.

Job Scheduler (JS): existing OpenSearch plugin which provides a spi to be extended by other plugins. Other plugin can then define the scheduled job that will be run by fixed interval or cron schedule.

Every time this scheduled job runs, it triggers the execution of the SM state machine. SM state machine contains states which execute the business logic like create snapshot API call, cron schedule checking etc. The metadata daily-snapshot-slm-metadata for the state machine including fields like current state, creating snapshot info and so on will be persisted into the IM system index. Persisted metadata will be read by the next scheduled job run, so that SM state machine can continue to execute from previous run end state.

State Machine

We choose to call create/delete snapshot API with wait_for_completion=false, which is asynchronously, and check the snapshot creation/deletion progress in the next scheduled job run.

The state machine runs as follows generally:
Taking 2 connected states like CREATE_CONDITION_MET and CREATING as example, SM state machine current state is CREATE_CONDITION_MET, then the state machine execution will try doing the actions inside the CREATING state, only when all the actions successfully performed then will the current state move forward and be persisted as CREATING instead.
If any action failed, like the pre-condition check failed because cluster is red, we will either stay in CREATE_CONDITION_MET state waiting for next scheduled job to retry (failed information will be saved in metadata), or directly skip to WAITING after retry exhausted. So we don’t need to have any FAILED states.
continue label in the graph above means state machine won’t wait for the next scheduled job to execute second state’s actions. And RUN label indicates waiting for the next.

After understanding how the state machine runs, let’s see how scheduled job triggers state machine running:
When a SM scheduled job like daily-snapshot-sm runs, SM runner will create a state machine context and initialized it with this job daily-snapshot-sm and the retrieved corresponding metadata daily-snapshot-sm-metadata. This context represents the state machine with the current running state from the metadata and execution details from the job.

For the fork of CREATE_CONDITION_MET and DELETE_CONDITION_MET, the create one will always be executed first. So if these 2 cron schedules met at the same time period, the check for deletion cron schedule will wait for a circle run this state machine. This is because of the idea that snapshot creation a higher priority than snapshot cleanup. Based on this, DELETING state actually has a line connected CREATE_CONDITION_MET, so even before the snapshot deletion finished, we can check if the create condition met and go to CREATING state.

Concurrent Snapshot

You may wonder if the snapshot deletion will block the creation. The answer is no for OpenSearch. Snapshot operations have become fully concurrent with a maximum ops limit 1000. We believe to have overlapping creating snapshot is not a good idea in SM since it increases the resource utilization snapshot take and could slow down other concurrent snapshots. So we don’t want the snapshot creation under the same SM policy to be concurrent unless there is a valid use case. For SM, under same policy, there won’t be overlapping snapshot creation because of this state machine design, we always wait the current creating snapshot to finish before checking the next creation condition met.

However, if user define several SM policy with same cron schedule, then they will all create snapshots at the same time. We will call this out in our documentation as a bad practice. And if they have multiple clusters using SM to create snapshot into one repository at the same time, this is also a bad practice. To define different cron schedules for SM policies would be a best practice to advocate.

Implementation

An example of snapshot management policy:

PUT _plugins/_sm/policy/<policy_name>

# policy_name cannot be updated, because it's used as id of SM policy/job

Payload:
{
    "description": <String. Description of the SM policy>,
    "enabled": <Optional boolean, defaults to true. To determine if this SM job is enabled at creation>,
    "schedule": <String. The cron schedule to auto create snapshot>,
    "snapshot_config": <Object. The configuration for SM-create snapshot>,
        "date_math": <Optional string, defaults to empty. Specify the date suffix of SM-create snapshot>,
        "indices": <String. The indices names that automated snapshot would include>,
        "repository": <String. The snapshot repository that automated snapshot would reside in>,
        "ignore_available": <Optional boolean, defaults to false>,
        "include_global_state": <Optional boolean, defaults to true>,
        "partial": <Optional boolean, defaults to false>,
        "metadata": <Optional object>,
        "timeout": <Optional string, defaults to 2h. Timeout for create snapshot>,
    "delete_condition": <Object. The condition to decide whether to delete the auto created snapshot>,
        "count": <Array of int, defaults to [5, 50]. The minimum and maximum kept snapshot number>,
        "age": <Optional string, defaults to empty. The oldest time to keep auto snapshot>,
        "timeout": <Optional string, defaults to 2h. Timeout for delete snapshot>
} 

${policy_name}-sm will be the document id
${policy_name}-${date_math}-${hash_of_time} will be the create snapshot name

---

• Will add more details as design and implementation going on...

Related issues:
#55
#56
#127

Related forum threads:
4815
7160
6292
5973

[sandervandegeijn]

Great feature, really would like this. I read the proposal:

"Notification function will be integrated with SM so user can be notified when snapshots being created. "

Really would like to be able to trigger alerts on every state, especially when snapshots fail.

[sandervandegeijn]

For usability of the snapshot and automations it would be nice if the limitations mentioned here https://opensearch.org/docs/latest/opensearch/snapshot-restore/#security-plugin-considerations

regarding the security module vs snapshots could be resolved.

Another thing that needs to be solved to fix the snapshot functionality: opensearch-project/security#1652

[bowenlan-amzn]

For usability of the snapshot and automations it would be nice if the limitations mentioned here https://opensearch.org/docs/latest/opensearch/snapshot-restore/#security-plugin-considerations

Happen to see this logic https://github.com/opensearch-project/security/blob/d676716e83d1ab387e9e6a0c0f3284e39ed967f5/src/main/java/org/opensearch/security/privileges/SnapshotRestoreEvaluator.java#L74C2-L79

Seems this setting plugins.security.unsupported.restore.securityindex.enabled can be used to override this problem