Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ergonomic settings for indices recovery based on node size for data_cold role #68134

Closed
dliappis opened this issue Jan 28, 2021 · 1 comment · Fixed by #68480
Closed

Ergonomic settings for indices recovery based on node size for data_cold role #68134

dliappis opened this issue Jan 28, 2021 · 1 comment · Fixed by #68480
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >enhancement Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@dliappis
Copy link
Contributor

To enhance the performance of the cold tier during failure scenarios or prewarming we could move away from the fixed 40MB/s value for indices.recovery/max_bytes_per_sec and instead ergonomically choose the value based on the node size.

So for nodes with the data_cold node_role we could implement a scheme like:

NOTE: the following table is referring to the available memory for the node (e.g. memory available inside a container) not jvm heap size

node size in GB indices.recovery.max_bytes_per_sec change from current default
node_size <= 1 40 MB/s unchanged
1 < node_size <= 2 40 MB/s unchanged
2 < node_size <= 4 40 MB/s unchanged
4 < node_size <= 8 60 MB/s increased
8 < node_size <= 16 90 MB/s increased
16 < node_size <= 32 125 MB/s increased
32 < node_size 250 MB/s increased
@dliappis dliappis added >enhancement needs:triage Requires assignment of a team area label labels Jan 28, 2021
@danielmitterdorfer danielmitterdorfer added :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. and removed needs:triage Requires assignment of a team area label labels Jan 29, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jan 29, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@dliappis dliappis added needs:triage Requires assignment of a team area label and removed :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. labels Jan 29, 2021
@elasticmachine elasticmachine removed the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jan 29, 2021
@dliappis dliappis added the :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. label Feb 3, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Feb 3, 2021
@dliappis dliappis removed the needs:triage Requires assignment of a team area label label Feb 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >enhancement Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants