Could _search sort on @timestamp by default when the index pattern only matches data streams?

[jpountz]

Not only is sorting by descending @timestamp what users usually want when querying data streams, but it's also faster than sorting by _score when querying large amounts of data thanks to optimizations to skip shards that only contain old data. Could we make it the default?

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jimczi]

We discussed offline and agreed that this is something we want to implement.
It will also benefit queries with size: 0 since we sort shards based on the primary sort.

[mayya-sharipova]

Is the plan to always add sort on @timestamp?

• Should we remove timestamp values from the "sort" section in the resulting hits?
• If a search contains a scoring query, should we rewrite to sort on [_score, @timestamp]?
• If a search contains other sorts, should we add an extra sort on @timestamp?

Or is this only for cases when no query and sort provided?

[jpountz]

These are good questions, I had not thought about that.

Should we remove timestamp values from the "sort" section in the resulting hits?

My initial take on this is that I'd rather keep them so that the response is the same whether explicitly sorting by @timestamp or relying on the default sort order.

If a search contains a scoring query, should we rewrite to sort on [_score, @timestamp]?
If a search contains other sorts, should we add an extra sort on @timestamp?

Since we don't automatically use _score as a tie-breaker when sorting by field today, I wouldn't always use @timestamp as a tie-breaker. That said I haven't thought much about it.

I think that there is another question that is important, which is should we only do this on data streams, or on any index that deals with time-based data? For instance, some users are still using regular time-based indices if they need updates, and these indices would also benefit from a different default sort. But I don't know how we can identify time-based indices from non-time-based indices.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)