Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve circuit breaking on aggregations #14046

Closed
jpountz opened this issue Oct 9, 2015 · 5 comments
Closed

Improve circuit breaking on aggregations #14046

jpountz opened this issue Oct 9, 2015 · 5 comments
Labels
:Analytics/Aggregations Aggregations :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload >feature help wanted adoptme high hanging fruit

Comments

@jpountz
Copy link
Contributor

jpountz commented Oct 9, 2015

Unfortunately circuit breaking on aggregations only catches a small subset of out-of-memory errors. I was talking with @dakrone recently who had the interesting idea to also count how many aggregators we create and use it for circuit-breaking. I think it would be a good heuristic to avoid out-of-memory errors in the deeply-nested aggregation case that we often see.
@jpountz jpountz added :Analytics/Aggregations Aggregations :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload labels Oct 9, 2015
@dakrone
Copy link
Member

dakrone commented Oct 12, 2015

When I discussed this with @jpountz we talked about potentially adding this to the constructor of AggregatorBase to increment when it is created (since we can get the breaker from the aggregation context) and decrement it in the close method, which I think is a neat idea to try.

Part of this would be determining the "weight" of a single AggregatorBase, then we could have a separate BucketBreaker where the overhead could be tuned for specific use cases (as a mulitplier of the class size).

Thoughts and heap dumps welcome :)

@dakrone
Copy link
Member

dakrone commented Oct 28, 2015

If anyone is curious about this, I'm working on a branch with it here: master...dakrone:bucket-circuit-breaker

@clintongormley clintongormley mentioned this issue Oct 30, 2015
Closed
24 tasks
@clintongormley clintongormley mentioned this issue Jan 12, 2016
Closed
@dakrone dakrone mentioned this issue Jul 12, 2016
Merged
dakrone added a commit to dakrone/elasticsearch that referenced this issue Jul 25, 2016
@dakrone
Circuit break on aggregation bucket numbers with request breaker
124a9fa 
This adds new circuit breaking with the "request" breaker, which adds
circuit breaks based on the number of buckets created during
aggregations. It consists of incrementing during AggregatorBase creation

This also bumps the REQUEST breaker to 60% of the JVM heap now.

The output when circuit breaking an aggregation looks like:

```json
{
  "shard" : 0,
  "index" : "i",
  "node" : "a5AvjUn_TKeTNYl0FyBW2g",
  "reason" : {
    "type" : "exception",
    "reason" : "java.util.concurrent.ExecutionException: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]];",
    "caused_by" : {
      "type" : "execution_exception",
      "reason" : "QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [myagg]>] would be larger than limit of [104857600/100mb]];",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]",
        "bytes_wanted" : 104860781,
        "bytes_limit" : 104857600
      }
    }
  }
}
```

Relates to elastic#14046
@dakrone
Copy link
Member

dakrone commented Jul 25, 2016

Closed by #19394

@dakrone dakrone closed this as completed Jul 25, 2016
@haizaar
Copy link

haizaar commented Aug 30, 2016

Will this circuit breaker prevent me from running aggregation that returns zillion buckets? Like generating histogram on file size metrics with step of 5, which can easily result in trillion buckets.

@dakrone
Copy link
Member

dakrone commented Aug 30, 2016

@haizaar that's the intention, yes

@jimczi jimczi mentioned this issue Aug 2, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Analytics/Aggregations Aggregations :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload >feature help wanted adoptme high hanging fruit
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dakrone @clintongormley @haizaar @jpountz