Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Search: Do something different with many shard failures #103708

Closed
nik9000 opened this issue Dec 27, 2023 · 4 comments
Closed

Search: Do something different with many shard failures #103708

nik9000 opened this issue Dec 27, 2023 · 4 comments
Labels
>bug priority:normal A label for assessing bug priority to be used by ES engineers :Search Foundations/Search Catch all for Search Foundations Team:Search Meta label for search team

Comments

@nik9000
Copy link
Member

nik9000 commented Dec 27, 2023

Elasticsearch Version

all

Installed Plugins

No response

Java Version

bundled

OS Version

all

Problem Description

I'm looking at a cascading failure caused by the loss of a node hosting many, many, many frozen shards. Coodinating nodes receive a ~6kb shard failure per shard. If you have tens of thousands of shards failing on search request those failure can really add up to some serious memory usage. Maybe we should truncate these?

Steps to Reproduce

See above.

Logs (if relevant)

No response
@nik9000 nik9000 added >bug :Search/Search Search-related issues that do not fall into other categories needs:triage Requires assignment of a team area label labels Dec 27, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Search Meta label for search team label Dec 27, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@elasticsearchmachine elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Dec 27, 2023
@DaveCTurner
Copy link
Contributor

See also #99220.

@nik9000
Copy link
Member Author

nik9000 commented Dec 27, 2023

"if a substantial number of shards fail then its contents will become massive"

Yeah. That's the ticket.

@quux00 quux00 mentioned this issue Jan 9, 2024
Closed
@quux00 quux00 mentioned this issue Mar 8, 2024
Closed
@benwtrent benwtrent added the priority:normal A label for assessing bug priority to be used by ES engineers label Jul 9, 2024
@javanna
Copy link
Member

javanna commented Jul 17, 2024

Closing as a duplicate of #99220

@javanna javanna closed this as not planned Won't fix, can't repro, duplicate, stale Jul 17, 2024
@javanna javanna added :Search Foundations/Search Catch all for Search Foundations and removed :Search/Search Search-related issues that do not fall into other categories labels Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug priority:normal A label for assessing bug priority to be used by ES engineers :Search Foundations/Search Catch all for Search Foundations Team:Search Meta label for search team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@nik9000 @javanna @benwtrent @DaveCTurner @elasticsearchmachine