Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Group shard request per node in the can match phase #78164

Closed
Tracked by #77466
jimczi opened this issue Sep 22, 2021 · 2 comments
Closed
Tracked by #77466

Group shard request per node in the can match phase #78164

jimczi opened this issue Sep 22, 2021 · 2 comments
Labels
>enhancement :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team

Comments

@jimczi
Copy link
Contributor

jimczi commented Sep 22, 2021

The can match phase fans out the original search request to all shards.
These individual requests have a cost:

  • They serialize the same request multiple times.
  • They need to resolve the index permissions multiple times.

Additionally, the broadcasting of these shard request is done on the network thread without any throttling, so search requests that target a large number of shards can disturb the cluster coordination, leading to nodes leaving the cluster.

Similar to #74648, we should move to a per-node request model for this phase. We can group shard request per node and perform the local operations for multiple shard/index at once. That would reduce the cost of coordination drastically and would leave room on data nodes to perform the operations on a separate thread pool.

@jimczi jimczi added >enhancement :Search/Search Search-related issues that do not fall into other categories labels Sep 22, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Sep 22, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@jimczi
Copy link
Contributor Author

jimczi commented Oct 20, 2021

Implemented in #79344

@jimczi jimczi closed this as completed Oct 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team
Projects
None yet
Development

No branches or pull requests

2 participants