Add remote store buffer and index interval settings #4973

Naarcha-AWS · 2023-09-06T17:45:26Z

Fixes [DOC] Add documentation for remote store cluster level buffer interval settings #4906
Fixes [DOC] Add documentation for cluster level default and minimum refresh interval settings #4905
Fixes [DOC] Update documentation for the index setting index.remote_store.translog.buffer_interval #4992

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS · 2023-09-06T17:45:50Z

@ashking94: Could I get details on the buffer level settings related to remote store?

ashking94 · 2023-09-08T07:50:28Z

@ashking94: Could I get details on the buffer level settings related to remote store?

Hi @Naarcha-AWS, thanks for getting started on this.

There are 3 cluster level settings that we have introduced and more details about them follow below -

Introduction of cluster default and minimum refresh interval settings

cluster.default.index.refresh_interval , Data type: Time unit, Dynamic NodeScope Cluster Setting

This setting is used to set the refresh interval when the index.refresh_interval index setting is not provided during index creation or when the existing index.refresh_interval index setting is set as null. This comes handy when the user wants to set a default refresh interval across all indexes created in a cluster which is different from 1s and also at the same time have searchIdle feature supported. The setting can only be as low as the cluster.minimum.index.refresh_interval cluster setting.

cluster.minimum.index.refresh_interval , Data type: Time unit, Dynamic NodeScope Cluster Setting

This setting is used to set the minimum refresh interval applicable for all indexes in a cluster. The cluster.default.index.refresh_interval cluster setting value needs to be higher than this setting's value. Index creation will fail if the index setting index.refresh_interval is supplied with a value lower than the cluster minimum refresh interval.

GH issue - opensearch-project/OpenSearch#9266

Introduction of cluster default remote translog buffer interval setting

cluster.remote_store.translog.buffer_interval, Data type: Time unit, Dynamic NodeScope Cluster Setting

This setting provides the default value of translog buffer interval that is used for performing periodic translog uploads. The setting is only effective if the index setting index.remote_store.translog.buffer_interval is not present with an index.

GH issue - opensearch-project/OpenSearch#9574

ashking94

Pls make changes as per the brief shared.

_api-reference/cluster-api/cluster-settings.md

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS · 2023-09-13T19:05:59Z

@ashking94: This is ready for your review.

Signed-off-by: Naarcha-AWS <[email protected]>

ashking94

Please do the necessary changes. Thanks!

_api-reference/cluster-api/cluster-settings.md

Signed-off-by: Naarcha-AWS <[email protected]>

ashking94 · 2023-09-20T17:10:32Z

_api-reference/cluster-api/cluster-settings.md

@@ -81,6 +82,9 @@ The following request field parameters are compatible with the cluster API.
 | network.breaker.inflight_requests.limit | String | The limit for the in-flight requests breaker. Default is `100%` of the JVM heap. |
 | network.breaker.inflight_requests.overhead | Integer/Time unit | The constant that all in-flight request estimations are multiplied by to determine a final estimation. Default is `2`. |
 | script.max_compilations_rate | String | The limit for the number of unique dynamic scripts within a defined interval that are allowed to be compiled. Default is 150 every 5 minutes: `150/5m`. |
+| cluster.default.index.refresh_interval | Time unit | Sets the refresh interval what the `index.refresh_interval` setting is not provided. This setting is useful when you want to set a default refresh interval across all indexes in a cluster and also support the `searchIdle` setting. You can only set the interval as low as the `cluster.minimum.index.refresh_interval` setting. |


what the -> when the

_api-reference/cluster-api/cluster-settings.md

Signed-off-by: Naarcha-AWS <[email protected]>

cwillum

Looks good.

cwillum · 2023-09-20T17:13:02Z

_api-reference/cluster-api/cluster-settings.md

@@ -81,6 +82,9 @@ The following request field parameters are compatible with the cluster API.
 | network.breaker.inflight_requests.limit | String | The limit for the in-flight requests breaker. Default is `100%` of the JVM heap. |
 | network.breaker.inflight_requests.overhead | Integer/Time unit | The constant that all in-flight request estimations are multiplied by to determine a final estimation. Default is `2`. |
 | script.max_compilations_rate | String | The limit for the number of unique dynamic scripts within a defined interval that are allowed to be compiled. Default is 150 every 5 minutes: `150/5m`. |
+| cluster.default.index.refresh_interval | Time unit | Sets the refresh interval what the `index.refresh_interval` setting is not provided. This setting is useful when you want to set a default refresh interval across all indexes in a cluster and also support the `searchIdle` setting. You can only set the interval as low as the `cluster.minimum.index.refresh_interval` setting. |


The first sentence didn't make sense. Check me on the suggested change. I'm not sure if that's the idea.

I would also break up the two ideas "set a default refresh interval across all indexes in a cluster" and "also support the searchIdle setting" into two separate sentences. It can do two things. But, since they seem not at all connected, having them in the same sentence reads awkwardly. Can you also link to a description of searchidle? It' not clear how the setting supports it.

Suggested change

| cluster.default.index.refresh_interval | Time unit | Sets the refresh interval what the `index.refresh_interval` setting is not provided. This setting is useful when you want to set a default refresh interval across all indexes in a cluster and also support the `searchIdle` setting. You can only set the interval as low as the `cluster.minimum.index.refresh_interval` setting. |

| cluster.default.index.refresh_interval | Time unit | Sets the refresh interval when the `index.refresh_interval` setting is not provided. This setting is useful when you want to set a default refresh interval across all indexes in a cluster and also support the `searchIdle` setting. You cannot set the interval lower than the `cluster.minimum.index.refresh_interval` setting. |

_api-reference/cluster-api/cluster-settings.md

Co-authored-by: Chris Moore <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

natebower

@Naarcha-AWS One minor change. Otherwise, LGTM!

_api-reference/cluster-api/cluster-settings.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

…ct#4973) * Add remote store buffer and index interval settings Signed-off-by: Naarcha-AWS <[email protected]> * Add corrected cluster settings. Signed-off-by: Naarcha-AWS <[email protected]> * Add new formatting settings. Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Chris Moore <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Chris Moore <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

* Add remote store buffer and index interval settings Signed-off-by: Naarcha-AWS <[email protected]> * Add corrected cluster settings. Signed-off-by: Naarcha-AWS <[email protected]> * Add new formatting settings. Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Chris Moore <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Chris Moore <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

Add remote store buffer and index interval settings

19ae9fe

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS added the 3 - Tech review PR: Tech review in progress label Sep 6, 2023

Naarcha-AWS requested review from cwillum, hdhalter, kolchfa-aws, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners September 6, 2023 17:45

Naarcha-AWS added Needs SME Waiting on input from subject matter expert v2.10.0 release-notes PR: Include this PR in the automated release notes labels Sep 6, 2023

ashking94 suggested changes Sep 8, 2023

View reviewed changes

_api-reference/cluster-api/cluster-settings.md Outdated Show resolved Hide resolved

_api-reference/cluster-api/cluster-settings.md Outdated Show resolved Hide resolved

ashking94 mentioned this pull request Sep 8, 2023

[DOC] Update documentation for the index setting index.remote_store.translog.buffer_interval #4992

Closed

4 tasks

Naarcha-AWS and others added 2 commits September 13, 2023 13:50

Merge branch 'main' into remote-store-cluster-settings

a6c4b91

Signed-off-by: Naarcha-AWS <[email protected]>

Add corrected cluster settings.

00e6c0d

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS requested a review from ashking94 September 13, 2023 19:05

Naarcha-AWS and others added 2 commits September 13, 2023 14:07

Add new formatting settings.

b24eabe

Signed-off-by: Naarcha-AWS <[email protected]>

Merge branch 'main' into remote-store-cluster-settings

5daabef

Naarcha-AWS added 4 - Doc review PR: Doc review in progress and removed 3 - Tech review PR: Tech review in progress Needs SME Waiting on input from subject matter expert labels Sep 15, 2023

Merge branch 'main' into remote-store-cluster-settings

cb8e057

ashking94 suggested changes Sep 20, 2023

View reviewed changes

_api-reference/cluster-api/cluster-settings.md Outdated Show resolved Hide resolved

_api-reference/cluster-api/cluster-settings.md Outdated Show resolved Hide resolved

Naarcha-AWS commented Sep 20, 2023

View reviewed changes

_api-reference/cluster-api/cluster-settings.md Outdated Show resolved Hide resolved

Naarcha-AWS commented Sep 20, 2023

View reviewed changes

_api-reference/cluster-api/cluster-settings.md Outdated Show resolved Hide resolved

Apply suggestions from code review

77111cf

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS requested a review from ashking94 September 20, 2023 16:53

ashking94 reviewed Sep 20, 2023

View reviewed changes

ashking94 approved these changes Sep 20, 2023

View reviewed changes

Naarcha-AWS commented Sep 20, 2023

View reviewed changes

_api-reference/cluster-api/cluster-settings.md Outdated Show resolved Hide resolved

Apply suggestions from code review

fcb7520

Signed-off-by: Naarcha-AWS <[email protected]>

cwillum approved these changes Sep 20, 2023

View reviewed changes

Apply suggestions from code review

c8f0f57

Co-authored-by: Chris Moore <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels Sep 20, 2023

natebower reviewed Sep 20, 2023

View reviewed changes

_api-reference/cluster-api/cluster-settings.md Outdated Show resolved Hide resolved

Apply suggestions from code review

5ba9e55

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS merged commit 70be12b into main Sep 20, 2023

Naarcha-AWS deleted the remote-store-cluster-settings branch September 20, 2023 17:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add remote store buffer and index interval settings #4973

Add remote store buffer and index interval settings #4973

Naarcha-AWS commented Sep 6, 2023 •

edited

Loading

Naarcha-AWS commented Sep 6, 2023

ashking94 commented Sep 8, 2023 •

edited

Loading

ashking94 left a comment

Naarcha-AWS commented Sep 13, 2023

ashking94 left a comment

ashking94 Sep 20, 2023

cwillum left a comment

cwillum Sep 20, 2023

natebower left a comment

	\| cluster.default.index.refresh_interval \| Time unit \| Sets the refresh interval what the `index.refresh_interval` setting is not provided. This setting is useful when you want to set a default refresh interval across all indexes in a cluster and also support the `searchIdle` setting. You can only set the interval as low as the `cluster.minimum.index.refresh_interval` setting. \|
	\| cluster.default.index.refresh_interval \| Time unit \| Sets the refresh interval when the `index.refresh_interval` setting is not provided. This setting is useful when you want to set a default refresh interval across all indexes in a cluster and also support the `searchIdle` setting. You cannot set the interval lower than the `cluster.minimum.index.refresh_interval` setting. \|

Add remote store buffer and index interval settings #4973

Add remote store buffer and index interval settings #4973

Conversation

Naarcha-AWS commented Sep 6, 2023 • edited Loading

Checklist

Naarcha-AWS commented Sep 6, 2023

ashking94 commented Sep 8, 2023 • edited Loading

Introduction of cluster default and minimum refresh interval settings

Introduction of cluster default remote translog buffer interval setting

ashking94 left a comment

Choose a reason for hiding this comment

Naarcha-AWS commented Sep 13, 2023

ashking94 left a comment

Choose a reason for hiding this comment

ashking94 Sep 20, 2023

Choose a reason for hiding this comment

cwillum left a comment

Choose a reason for hiding this comment

cwillum Sep 20, 2023

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Naarcha-AWS commented Sep 6, 2023 •

edited

Loading

ashking94 commented Sep 8, 2023 •

edited

Loading