Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add circuit breaker on heap memory usage for file cache #6540

Closed
adnapibar opened this issue Mar 3, 2023 · 1 comment · Fixed by #6591
Closed

Add circuit breaker on heap memory usage for file cache #6540

adnapibar opened this issue Mar 3, 2023 · 1 comment · Fixed by #6591
Assignees
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request

Comments

@adnapibar
Copy link
Contributor

Coming from #6227
Implement a circuit breaker to prevent the file cache for searchable snapshot causing an OutOfMemoryError. The breaker will put a limit on the cache for how much memory it can use. CircuitBreakerException will be thrown for requests that will cause the cache to use more heap memory than specified.

@kotwanikunal
Copy link
Member

Two possible solutions were assessed -

  1. Add a new child memory breaker which will keep track of memory usage by the file cache
  2. Use a pre-existing child breaker to keep track of total memory usage (tracked by the parent breaker)

Approach 1 would be ideal if there was a fixed size of an entry for the cache, but it varies depending on the platform architecture, OS, JVM variations.
Approach 2 solves the problem by keeping tracking of total memory usage instead of individual entries, and checking the breaker as soon as an entry is added.

@github-project-automation github-project-automation bot moved this from In Progress to Done in Searchable Snapshots Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants