Add safeguards to prevent simple user errors

[clintongormley]

There are a number of places where a naive user can break Elasticsearch very easily. We should add more (dynamically overridable) safeguards that prevent users from hurting themselves.

Note:

• We are adding high limits to start so that we don't suddenly disable things that users already do today, but so that sysadmins have tools that they can use to protect their clusters. We can revisit the limits later on.
• All these settings should be prefixed by policy. to make them easier to document together and to understand their purpose.

Accepted limits:

• Hard limit on from/size #9311 Hard limit on from/size
• Add global search timeout setting #12149 Global default value for search timeouts (Could be ridiculously high like an hour and it would still help)
• Disable fielddata on text fields by defaults. #17386 Disable fielddata-loading on analyzed text fields by default (Adrien)
• Add a soft limit on the number of shards that can be queried in a single search request. #17396 Limit the max number of shards to 1000 (Adrien)
• Limit request size #17133 Limit the size of all in-flight requests (Daniel)
• Add limit to total number of fields in mapping #17357 Limit the number of fields that can be added to a mapping to 1000
• Add a soft limit on the mapping depth. #17400 Add maximum mapping depth to 20
• Add sane limits for thread size and queue size (Jim)
• Add a maximum search request size. #26423 Don't allow search requests greater than (eg) 10MB (Adrien)
• Set soft limit on the number of nested fields per index #14983 Limit the number of nested fields per index to 50 (Yannick)
• Very large windows_size can cause node to run OOM #17522 Limit window_size in rescore API (@nik9000)
• Disable script access to _source by default #17558 Disable script access to _source fields by default
• Relocating many shards at once results in canceled relocations #18739 Limit the number of shards that can be rerouted at the same time
• Add a limit to from + size in top_hits and inner hits. #26492 Hard limit on from/size in top hits and inner hits (much smaller than a normal query) (MVG)
• Circuit break the number of inline scripts compiled per minute #19694 Limit script compilation rate to avoid hard coding of params in scripts
• Hard limit on total number of shards in a cluster #20705 Max number of shards per node (enforced as total shards per cluster)
• Limit index creation rate #20760 Limit index creation rate
• Limit the timeout of scroll requests #23268 Add upper limit for scroll expiry time (Jim)
• Add a soft limit on the number of requested doc-value and script fields #26390 Add upper limit for the number of requested doc-values fields (Christoph)

---

For discussion:

• Add a switch to disallow slow queries #29050 Disable certain query types, eg wildcard, span etc?
• Improve circuit breaking on aggregations #14046 Limit on the number of buckets returned by aggs
• Include hits in request circuit breaker #9310 Limit the size of the response (eg for very large doc bodies)
• Kill slow scripts when search timeout has lapsed aka while(true) should not require a rolling restart to recover from Don't run a script a second time when the first execution takes longer than 1 second
• Disable searching across all indices #6470 Disable searching on all indices by default Handled by max number of shards
• Limit the number nested Lucene documents per document. Limit the number of nested documents #26962

Any other ideas?

[jpountz]

Limit the max number of shards

I'm wondering if we should do it per index or per cluster. If we do it per index, then we might also want to have a max number of indices per cluster.

Limit the size of a bulk request

I guess it would also apply to multi-get and multi-search.

[alexbrasetvik]

Some of this could go into a "sanity checker"-kind of plugin akin to the migration plugin that runs a bunch of tests as well.

That one could warn when e.g. minimum master nodes looks wrong, and when the number of shards/indexes/fields looks silly / approaches the above limits.

[clintongormley]

@alexbrasetvik the requires the user to actually run the check. Often poor sysadmins are at the mercy of their users. What I'd like to do is to prevent users from blowing things up by mistake.

[alexbrasetvik]

@clintongormley Agreed! I still think there's room for both, though such a tool should be another issue.

For example, a high number of indexes with few documents and identical mappings can be a sign that the user is doing per-user index partitioning when he shouldn't. That will turn into a problem, even if the current values are far from hitting above mentioned limits.

[pickypg]

Any other ideas?

• Limit the max number of indices
  • It's effectively covered by limiting by shards, but touching too many indices may indicate more of a logical issue than the shard count (e.g., with daily indices, it's much easier to realize that sending a request to 5 indices represents five days rather than 25 shards with default counts).
• Limit the concurrent request size
  • Request circuit breaker across all concurrent requests

[dakrone]

Limit the concurrent request size

This is already available with the thread pools and queue_sizes to limit the number of requests per-node and apply backpressure.

EDIT: I guess I am taking "size" as "count", is that what you mean?

[pickypg]

@dakrone Size of an actual request. For instance, if one request comes in with an aggregation that uses size: 0 at the same time as another, then maybe we should block the second one (or at least delay).

[jpountz]

Another protection to add: check mapping depth #14370

[ppf2]

Limit the max value that can be set for queue_size for our search, bulk, index, etc.. thread pools so users can't set them to unlimited, millions, etc..?

[s1monw]

@clintongormley I think we missed one rather important aspect when it comes to soft-limits. Today the user can override those limits via dynamic properties which is ok most of the time but in the case of a cloud hosting infrastructure where the org that runs the infrastructure needs to have full control over these limits they should be able to disable the dynamic property or should disable setting these settings entirely?

[jpountz]

Most of the work has been done, and items that have not been done have an assigned issue so I'll close this issue. Thanks everyone!