Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve sharing and diffability of IndexRoutingTable #94933

Open
2 tasks
Tracked by #77466
DaveCTurner opened this issue Mar 31, 2023 · 1 comment
Open
2 tasks
Tracked by #77466

Improve sharing and diffability of IndexRoutingTable #94933

DaveCTurner opened this issue Mar 31, 2023 · 1 comment
Labels
:Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >enhancement Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >tech debt

Comments

@DaveCTurner
Copy link
Contributor

Today IndexRoutingTable$Builder#build constructs an all-new IndexShardRoutingTable for each shard on every change. High-traffic clusters may have large numbers of shards per index, and today's behaviour yields many duplicate IndexShardRoutingTable instances which together may consume a substantial amount of heap on the elected master.

Moreover IndexRoutingTable is a SimpleDiffable so we must transmit the whole table over the wire on any change, which prevents sharing on the receiving node and therefore yields many duplicate IndexShardRoutingTable instances on the other nodes in the cluster.

We should:

  • Re-use existing IndexShardRoutingTable instances where possible when adjusting the routing table on the elected master
  • Refine the diffability of the IndexRoutingTable so that it avoids sending unchanged IndexShardRoutingTable instances over the wire.

Relates #77466
@DaveCTurner DaveCTurner added >enhancement :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. labels Mar 31, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Mar 31, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@DaveCTurner DaveCTurner mentioned this issue Mar 31, 2023
Open
97 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >enhancement Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >tech debt
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@DaveCTurner @elasticsearchmachine