Should we have a default timeout?

[jpountz]

While discussing #26258 @jimczi asked whether we should set a default timeout. This proved a bit controversial: on the one hand it would reduce the impact that insane queries could have on clusters but on the other hand this might be a bit trappy as users might not be checking the timed_out field of search responses, which is the way to know that the response is partial due to a timeout. I'm opening this issue to get opinions from a wider audience.

[dakrone]

+1 to have a default timeout, return a 206 Partial Content for partial content (timed out responses) and set the timeout to something reasonable, like 5 minutes

[jasontedor]

206 status code is inappropriate. It's appropriate only if the client sends a range request (as indicated by a Range header field).

[dakrone]

Okay that makes sense then. Still +1 on the timeout though

[nik9000]

The thing that bothers me about the default timeout is the timeout reporting. Right now we return a 200 OK and you have to inspect the response to make sure it is 100% ok. Adding a default timeout adds another way that the response can be not ok without the user asking for it. I'd certainly prefer that we have a default timeout, but I'd prefer we tie it to some change in behavior around error reporting so it is obvious when you get partial results.

[jimczi]

Discussed it in FixItFriday, the partial result is a big concern so we decided to explore adding a new setting at the cluster level (or at the index level if needed) that would add an hard timeout to search request. It could be activated by default with a very high value like 5 minutes. The benefit would be to automatize the garbage collection of long running queries.
Though we still need to benchmark the cost of adding a timeout checks to every query:
#26258
@danielmitterdorfer @polyfractal can you rerun the tests you've setup for #26258 but with timeouts this time ? This should help decide if we activate timeouts by default.

[polyfractal]

/cc @dakrone since he ran the tests, I just played with some spreadsheets to generate the p-values :)

[elasticmachine]

Pinging @elastic/es-search-aggs

[javanna]

Should we close this in favour of #26258 ?

[jimczi]

If we close I think it should be in favor of #30897 and we can discuss the need for a default value there ?

[cbuescher]

Since #26258 is already closed now, closing this in favour of #30897 where we can continue the discussion.