[DOC] Add Segment Replication Backpressure to documentation

[Rishikesh1159]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request. Provide a summary of the request and all versions that are affected.

OpenSearch is currently working on new segment replication feature. Segment Replication has been released as an experimental feature in opensearch 2.3. As part of Segment Replication feature we are introducing new Segment Replication Backpressure mechanism.

Implementation of this new segment replication backpressure mechanism is completed and more details can be found here. This feature will be released in opensearch 2.7.

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

[Rishikesh1159]

Overview:

• Segment Replication Backpressure is a rejection mechanism at a per-shard level that dynamically rejects indexing requests when replica shards in your cluster are falling behind primary shards.
• Segment Replication Backpressure mechanism starts rejecting indexing requests when:
  • More than half of the replication group is 'stale'. Defined by setting MAX_ALLOWED_STALE_SHARDS.
  • A replica is stale if it is behind more than MAX_INDEXING_CHECKPOINTS and its current replication lag is over MAX_REPLICATION_TIME_SETTING.
• With this mechanism we also monitor the replica shards stuck or lagging for long time. When a replica shards are stuck/lagging for more than double the MAX_REPLICATION_TIME_SETTING, we remove these lagging shards and replace them with new replica shards.

Settings:

All settings below are dynamic cluster setting and users can enable or disable settings using PUT _cluster/settings . SEGMENT_REPLICATION_INDEXING_PRESSURE_ENABLED setting must be set to true for rest of the backpressure settings to work.

• SEGMENT_REPLICATION_INDEXING_PRESSURE_ENABLED is a setting that enables segment replication backpressure mechanism. By default segment replication backpressure is false (disabled).
• MAX_REPLICATION_TIME_SETTING setting is the maximum time that a replica shard can take to copy from primary. Once MAX_REPLICATION_TIME_SETTING is breached along with MAX_INDEXING_CHECKPOINTS then segment replication backpressure mechanism gets triggered. The default value of this setting is 5 minutes.
• MAX_INDEXING_CHECKPOINTS setting is the maximum number of indexing checkpoints that a replica shard can fall behind when copying from primary. Once MAX_INDEXING_CHECKPOINTS is breached along with MAX_REPLICATION_TIME_SETTING then segment replication backpressure mechanism gets triggered. The default value of this setting is 4 checkpoints.
• MAX_ALLOWED_STALE_SHARDS setting is the maximum number of stale replica shards that can exist in a replication group. Once MAX_ALLOWED_STALE_SHARDS is breached then segment replication backpressure mechanism gets triggered. The default value of this setting is ‘.5’ which 50% of a replication group.

API:

GET _cat/segment_replication API is used to fetch metrics related to segment replication backpressure:

shardId       target_node    target_host   checkpoints_behind bytes_behind   current_lag   last_completed_lag   rejected_requests
[index-1][0]     runTask-1    127.0.0.1              0              0b           0s              7ms                    0

→ Parameters: checkpoints_behind and current_lag directly correlate with MAX_INDEXING_CHECKPOINTS and MAX_REPLICATION_TIME_SETTING .

→ These checkpoints_behind and current_lag metrics are taken into consideration when triggering segment replication backpressure mechanism.

[Naarcha-AWS]

Passing this to @ariamarble since this likely has some overlap with the Segment Replication GA PR. #3461