Ergonomically set the search.max_buckets setting

[jpountz]

Description

search.max_buckets is similar to the maximum number of clauses in queries in that it is a proxy for resource usage, which is much easier to count than actual resource usage. Having a limit on the number of buckets produced helps ensure that aggregations do not exceed resource limits, such as heap.

But not all nodes have the same resources, and the fixed limit on search.max_buckets doesn't reflect this. In #46433 (comment) we decided to move from a fixed limit on the number of clauses in queries to a value that would be set ergonomically based on resources, let's do the same with the similar search.max_buckets setting?

Questions:

• What should be the inputs for the maximum number of buckets?
• Should we also no longer make it configurable, like we did for the maximum number of clauses of queries?

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[jpountz]

What should be the inputs for the maximum number of buckets?

My initial reaction to this question is that we'll want the maximum heap size to be an input to this: the more heap you have, the more buckets you can have. And search coordination queue size: the more concurrent requests, the less buckets per request you can have?

[martijnvg]

We discussed this during the our weekly sync. For now like to keep search.max_buckets as is until we have circuit breaker at reduce time. After that we will revisit what to do with search.max_buckets limit.

[martijnvg]

We're still relying on search.max_buckets in order to avoid memory pressure. Previously we had plans to redesign the internal aggregation responses (binary based), which then would allow us to account memory. It is unlikely that this will happen. So I think, we should look into ergonomically setting search.max_buckets cluster setting.