Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Field caps against many remote clusters consumes substantial heap #78665

Closed
DaveCTurner opened this issue Oct 5, 2021 · 3 comments · Fixed by #83494
Closed

Field caps against many remote clusters consumes substantial heap #78665

DaveCTurner opened this issue Oct 5, 2021 · 3 comments · Fixed by #83494
Assignees
@dnhatn
Labels
>bug :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team

Comments

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Oct 5, 2021 •
edited
Loading

Elasticsearch version (bin/elasticsearch --version): Seen in 7.14.0 but master looks affected still

Plugins installed: []

JVM version (java -version): Bundled

OS version (uname -a if on a Unix-like system): Cloud

Description of the problem including expected versus actual behavior:

When running field caps for CCS we broadcast a request to every remote cluster and only start to merge the results together once all responses are received. We have users with 150+ large remote clusters, and one such user reported a single field caps request which ended up consuming 10+GB of heap with the un-merged responses. It doesn't take many of them to reliably cause their CCS cluster to experience OOMs.

Can we merge the results from remote clusters incrementally, and limit the number of remote cluster requests in flight? Can we shift some of the merging work onto the remote clusters even?

Steps to reproduce:

  1. Add a few hundred remote clusters each with several hundred indices containing ~1000 fields.
  2. Request field caps against all the remote clusters at once.

Provide logs (if relevant):

Link from the internal case will appear below.
@DaveCTurner DaveCTurner added >bug :Search/Search Search-related issues that do not fall into other categories labels Oct 5, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Oct 5, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@dnhatn
Copy link
Member

dnhatn commented Oct 5, 2021

@DaveCTurner We are actively working on this issue.

@dnhatn dnhatn self-assigned this Oct 5, 2021
@jtibshirani jtibshirani mentioned this issue Oct 5, 2021
Closed
@jtibshirani
Copy link
Contributor

I added this to our "meta" tracking issue: #76509 (comment)

@dnhatn dnhatn mentioned this issue Feb 10, 2022
Merged
dnhatn added a commit that referenced this issue Feb 17, 2022
@dnhatn
Group field caps response by index mapping hash (#83494)
69e898d 
This commit utilizes the index mapping hash to share the fields-caps for 
indices with the same index mapping to reduce the memory usage and the
size of transport messages.

Closes #78665
Closes #82879
probakowski pushed a commit to probakowski/elasticsearch that referenced this issue Feb 23, 2022
@dnhatn @probakowski
Group field caps response by index mapping hash (elastic#83494)
31ead1f 
This commit utilizes the index mapping hash to share the fields-caps for 
indices with the same index mapping to reduce the memory usage and the
size of transport messages.

Closes elastic#78665
Closes elastic#82879
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
>bug :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Group field caps response by index mapping hash dnhatn/elasticsearch
4 participants
@DaveCTurner @jtibshirani @dnhatn @elasticmachine