Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Update segment replication to include alerting plugin limitation #4976

Merged
merged 17 commits into from
Sep 13, 2023
Merged
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -12,29 +12,29 @@ redirect_from:

# Segment replication

With segment replication, segment files are copied across shards instead of documents being indexed on each shard copy. This improves indexing throughput and lowers resource utilization at the expense of increased network utilization.
Segment replication involves copying segment files across shards instead of indexing documents on each shard copy. This approach enhances indexing throughput and reduces resource utilization, but increases network utilization. Segment replication is the first feature in a series of features designed to decouple reads and writes in order to lower compute costs.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

When the primary shard sends a checkpoint to replica shards on a refresh, a new segment replication event is triggered on replica shards. This happens:

- When a new replica shard is added to a cluster.
- When there are segment file changes on a primary shard refresh.
- During peer recovery, such as replica shard recovery and shard relocation (explicit allocation using the `move` allocation command or automatic shard rebalancing).

Segment replication is the first feature in a series of features designed to decouple reads and writes in order to lower compute costs.

## Use cases

Segment replication can be applied in a variety of scenarios, including:

- Users who have high write loads but do not have high search requirements and are comfortable with longer refresh times.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Users" are not "scenarios", so the first two items do not complete the stem. The stem is fine, but I would rephrase the first two items.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised.

- Users with very high loads who want to add new nodes, as you do not need to index all nodes when adding a new node to the cluster.
- OpenSearch cluster deployments with low replica counts, such as those used for log analytics.

## Segment replication configuration

Setting the default replication type for a cluster affects all newly created indexes. However, you can specify a different replication type when creating an index. Index-level settings always override cluster-level settings.
Setting the default replication type for a cluster affects all newly created indexes. You can, however, specify a different replication type when creating an index. Index-level settings override cluster-level settings.

### Creating an index with the segment replication type
### Creating an index with segment replication

To set segment replication as the replication strategy for an index, create the index with `replication.type` set to `SEGMENT`:
To use segment replication as the replication strategy for an index, create the index with the `replication.type` parameter set to `SEGMENT` as follows:

```json
PUT /my-index1
Expand All @@ -50,13 +50,10 @@ PUT /my-index1

In segment replication, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/).

Segment replication currently does not support the `wait_for` value in the `refresh` query parameter.
{: .important }

For the best performance, we recommend enabling both of the following settings:
For the best performance, it is recommended that you enable the following settings:

1. [Segment replication backpressure]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/backpressure/).
2. Balanced primary shard allocation:
1. [Segment replication backpressure]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/segment-replication/backpressure/)
2. Balanced primary shard allocation, using the following command:

```json
curl -X PUT "$host/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
Expand All @@ -69,7 +66,7 @@ curl -X PUT "$host/_cluster/settings?pretty" -H 'Content-Type: application/json'
```
{% include copy-curl.html %}

### Setting the replication type on a cluster
### Setting the replication type for a cluster

You can set the default replication type for newly created cluster indexes in the `opensearch.yml` file as follows:

Expand All @@ -78,15 +75,12 @@ cluster.indices.replication.strategy: 'SEGMENT'
```
{% include copy.html %}

This cluster-level setting cannot be enabled through the REST API.
{: .note}

This setting is not applied to system indexes and hidden indexes. By default, all system and hidden indexes in OpenSearch will still use document replication even if this setting is enabled.
This cluster-level setting cannot be enabled through the [REST API]({{site.url}}{{site.baseurl}}/api-reference/index/). This setting is not applied to system indexes and hidden indexes. By default, all system and hidden indexes in OpenSearch use document replication even if this setting is enabled.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
{: .note}

### Creating an index with the document replication type
### Creating an index with document replication

Even when the default replication type is set to segment replication, you can create an index that uses document replication by setting `replication.type` to `DOCUMENT`:
Even when the default replication type is set to segment replication, you can create an index that uses document replication by setting `replication.type` to `DOCUMENT` as follows:

```json
PUT /my-index1
Expand All @@ -106,11 +100,13 @@ When using segment replication, consider the following:

1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685).
1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters.
1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) plugin. The Alerting plugin uses the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards.
1. Segment replication leads to increased network congestion on primary shards. See [Issue - Optimize network bandwidth on primary shards](https://github.com/opensearch-project/OpenSearch/issues/4245).
1. Integration with remote-backed storage as the source of replication is [currently not supported](https://github.com/opensearch-project/OpenSearch/issues/4448).
1. Read-after-write guarantees: The `wait_until` refresh policy is not compatible with segment replication. If you use the `wait_until` refresh policy while ingesting documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. We are exploring other mechanisms for providing read-after-write guarantees. For more information, see the corresponding [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/6046).
1. Read-after-write guarantees: The `wait_until` and `wait_for` refresh policy is not compatible with segment replication. If you use the `wait_until` refresh policy while ingesting documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. We are exploring other mechanisms for providing read-after-write guarantees. For more information, see the corresponding [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/6046).
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
1. System indexes will continue to use document replication internally until read-after-write guarantees are available. In this case, document replication does not hinder the overall performance because there are few system indexes.

1. Segment replication is not compatible with [security analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/). Security analytics relies on being able to access the latest data in an index, and segment replication can delay the availability of this data.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Benchmarks

During initial benchmarks, segment replication users reported 40% higher throughput than when using document replication with the same cluster setup.
Expand All @@ -120,22 +116,18 @@ The following benchmarks were collected with [OpenSearch-benchmark]({{site.url}}
The benchmarks demonstrate the effect of the following configurations on segment replication:

- [The workload size](#increasing-the-workload-size)

- [The number of primary shards](#increasing-the-number-of-primary-shards)

- [The number of replicas](#increasing-the-number-of-replicas)

Your results may vary based on the cluster topology, hardware used, shard count, and merge settings.
{: .note }

### Increasing the workload size
### Increasing workload size

The following table lists benchmarking results for the `nyc_taxi` dataset with the following configuration:

- 10 m5.xlarge data nodes

- 40 primary shards, 1 replica each (80 shards total)

- 4 primary shards and 4 replica shards per node

<table>
Expand Down Expand Up @@ -398,4 +390,4 @@ The benchmarking results show a non-zero error rate as the number of replicas in
## Next steps

1. Track [future enhancements to segment replication](https://github.com/orgs/opensearch-project/projects/99).
1. Read [this blog post about segment replication](https://opensearch.org/blog/segment-replication/).
1. Read this [blog post about segment replication](https://opensearch.org/blog/segment-replication/).
vagimeli marked this conversation as resolved.
Show resolved Hide resolved