Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cluster default remote store translog buffer interval setting #9574

Closed
ashking94 opened this issue Aug 28, 2023 · 0 comments · Fixed by #9584
Closed

Add cluster default remote store translog buffer interval setting #9574

ashking94 opened this issue Aug 28, 2023 · 0 comments · Fixed by #9584
Labels
enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework Storage Issues and PRs relating to data and metadata storage v2.10.0

Comments

@ashking94
Copy link
Member

ashking94 commented Aug 28, 2023

Is your feature request related to a problem? Please describe.
This proposes to introduce a dynamic NodeScope settings i.e. cluster default remote store translog buffer interval setting. The index.remote_store.translog.buffer_interval setting currently has a default value of 650 ms which is picked up when the user does not provide an explicit index setting for the same. The 650ms value is known to outperform document replication index in terms of indexing throughput and latency with lesser CPU and memory consumption under settings with so workload (3 shard 1 replica, 3 cluster manager + 3 data nodes, aws instance type - r5.xlarge). However, the same buffer interval might not be the most suitable value for other workloads and configuration. Hence, there arises a need to keep the default configurable to allow the user to tune this and set a default that is most suitable to their needs.

Expected behaviour with the new setting -

  • An index created without specifying the index.remote_store.translog.buffer_interval will use the cluster default.
  • We are going to keep the same cluster default as being used by the above index setting - 650ms.
  • If the cluster default is updated, then all the indexes that are using the cluster default will shift to the new updated value.
  • If an index is created with overridden index setting, then it will continue to use the same even if the cluster default setting is udpated.

Describe the solution you'd like
Introduce the setting as mentioned above.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@ashking94 ashking94 added enhancement Enhancement or improvement to existing feature or request untriaged Storage:Durability Issues and PRs related to the durability framework v2.10.0 Storage Issues and PRs relating to data and metadata storage and removed untriaged labels Aug 28, 2023
@ashking94 ashking94 changed the title Add cluster level remote store translog buffer interval setting Add cluster default remote store translog buffer interval setting Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework Storage Issues and PRs relating to data and metadata storage v2.10.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant