EQL: Response format enhancements

[costin]

1. implicit sort

Currently, there's no sorting meaning the documents are returned based on their doc_id. Under the assumption that the docs are sorted based on @timestamp, the return order is correct however what if the index is not sorted?

Returning documents always in a predefined order gives predictability and the expense of some performance in case of misconfiguration (no insert time sorting).

2. Are fields ever needed in the response?

Events always return entire documents. So do sequences and joins and while matching on the field requires the engine to interpret that, response wise, do we ever need to process the returned data?
It seems always the search hits are returned verbatim - there's no way to perform projection and thus no extra processing is needed.
This is important since the engine can simply return the data as is.
A separate point is whether we need to care of whether a field is in the source or not for the join-key since when we do the querying, we can just simply return the _source or, if we have to do field extraction, use doc_values for exact fields.

Related to #49634

3. How is pagination suppose to work?

Currently our response returns only results but not information on resuming.
The default page seems to be 50 items - how do we handle multiple pages?
Do we want to support pagination or not? I suppose we do but that requires some kind of a cursor.

And search_after parameter is inadequate for sequences and joins since information such as what is the last key we matched on, how to extract such information from the response cannot be passed around.

See: #49594

[elasticmachine]

Pinging @elastic/es-search (:Search/EQL)

[costin]

Raised

• EQL: Mandatory sorting on @timestamp ascending #52910 for 1,
• EQL: Returning _source for matching documents #52912 for 2
• EQL: Decide on pagination mechanism and field #52913 for 3.

Closing this issue in favor of the 3 above.