From bed32c1a1be32e0da987fc8cc1a95f40e68094eb Mon Sep 17 00:00:00 2001 From: Aria Marble <111301581+ariamarble@users.noreply.github.com> Date: Fri, 21 Apr 2023 14:25:44 -0700 Subject: [PATCH] Content updates for segrep - benchmarking and recommended settings (#3836) * Content updates for segrep Signed-off-by: ariamarble * Apply suggestions from doc review Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * doc review changes Signed-off-by: ariamarble * small wording change Signed-off-by: ariamarble * small change Signed-off-by: ariamarble --------- Signed-off-by: ariamarble Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- .../segment-replication/backpressure.md | 1 + .../segment-replication/index.md | 182 +++++++++++------- 2 files changed, 110 insertions(+), 73 deletions(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/backpressure.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/backpressure.md index 92d5edb239..1ac4496e07 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/backpressure.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/backpressure.md @@ -31,6 +31,7 @@ You can use the segment replication API endpoint to retrieve segment replication ```bash GET _cat/segment_replication ``` +{% include copy-curl.html %} #### Example response diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index b1d1fcfbf8..2229366d81 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -41,19 +41,36 @@ PUT /my-index1 } } ```` +{% include copy-curl.html %} In segment replication, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). +Segment replication currently does not support the `wait_for` value in the `refresh` query parameter. +{: .important } + +For the best performance, we recommend enabling both of the following settings: + +1. [Segment replication backpressure]({{site.url}}{{site.baseurl}}tuning-your-cluster/availability-and-recovery/segment-replication/backpressure/). +2. Balanced primary shard allocation: + +```json +curl -X PUT "$host/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' + { + "persistent": { + "cluster.routing.allocation.balance.prefer_primary": true, + "segrep.pressure.enabled": true + } + } +``` +{% include copy-curl.html %} + ## Comparing replication benchmarks During initial benchmarks, segment replication users reported 40% higher throughput than when using document replication with the same cluster setup. -The following benchmarks were collected with [OpenSearch-benchmark](https://github.com/opensearch-project/opensearch-benchmark) using the [`stackoverflow`](https://www.kaggle.com/datasets/stackoverflow/stackoverflow) and [`nyc_taxi`](https://github.com/topics/nyc-taxi-dataset) datasets. - -Both test runs were performed on a 10-node (m5.xlarge) cluster with 10 shards and 5 replicas. Each shard was about 3.2GBs in size. The tests were run with the following settings: +The following benchmarks were collected with [OpenSearch-benchmark](https://github.com/opensearch-project/opensearch-benchmark) using the [`nyc_taxi`](https://github.com/topics/nyc-taxi-dataset) dataset. -- `indices.recovery.max_bytes_per_sec`: 10gb -- `indices.recovery.max_concurrent_file_chunks`: 5 +The test run was performed on a 10-node (m5.xlarge) cluster with 10 shards and 5 replicas. Each shard was about 3.2GBs in size. The benchmarking results are listed in the following table. @@ -61,135 +78,154 @@ The benchmarking results are listed in the following table. - Document Replication - Segment Replication - Percent difference + Document Replication + Segment Replication + Percent difference - Test execution time (minutes) - - 40.00 - 22.00 + Test execution time (minutes) + 118.00 + 98.00 + 27% - Throughput (number of requests per second) + Index Throughput (number of requests per second) p0 - 17553.90 - 28584.30 - 63% + 71917.20 + 105740.70 + 47.03% p50 - 20647.20 - 32790.20 - 59% + 77392.90 + 110988.70 + 43.41% p100 - 23209.00 - 34286.00 - 48% + 93032.90 + 123131.50 + 32.35% + + + Query Throughput (number of requests per second) + p0 + 1.748 + 1.744 + -0.23% - CPU (%) p50 - 65.00 - 30.00 - -54% + 1.754 + 1.753 + 0% + + + p100 + 1.769 + 1.768 + -0.06% + + + CPU (%) + p50 + 37.19 + 25.579 + -31.22% p90 - 79.00 - 35.00 - -56% + 94.00 + 88.00 + -6.38% p99 - 98.00 - 45.08 - -54% + 100 + 100 + 0% p100 - 98.00 - 59.00 - -40% + 100.00 + 100.00 + 0% - Memory (%) + Memory (%) p50 - 48.20 - 39.00 - -19% + 30 + 24.241 + -19.20% p90 - 62.00 61.00 - -2% + 55.00 + -9.84% p99 - 66.21 - 68.00 - 3% + 72 + 62 + -13.89%% p100 - 71.00 - 69.00 - -3% + 80.00 + 67.00 + -16.25% - IOPS + Index Latency (ms) p50 - - - + 803 + 647.90 + -19.32% p90 - - - + 1215.50 + 908.60 + -25.25% p99 - - - + 9738.70 + 1565 + -83.93% p100 - - - + 21559.60 + 2747.70 + -87.26% - Latency + Query Latency (ms) p50 - - - + 36.209 + 37.799 + 4.39% p90 - - - + 42.971 + 60.823 + 41.54% p99 - - - + 50.604 + 70.072 + 38.47% p100 - - - + 52.883 + 73.911 + 39.76%