[4.x] Kibana issuing multiple _msearches causing slow performance

[tsullivan]

Kibana initially sends multiple _msearch requests each with single search and single unique index name, instead of calling _msearch once with an array of multiple indices.

Kibana version: 4.5

Elasticsearch version: 2.3

Server OS version: Mac OS X 10.11.6

Browser version: Firefox v44

Browser OS version: Linux 64-bit

Original install method (e.g. download page, yum, from source, etc.): Downloads page

Description of the problem including expected versus actual behavior: Initial load of Discover page, after broadening the time frame to require multiple indices in the search,

Steps to reproduce:

1. Index 3+ consecutive days worth of log data into Elasticsearch
2. Create a time-based Index Pattern, where "Index contains time-based events"
3. Navigate to Discover and change the time range to "Last 7 days"
4. Observe multiple /elasticsearch/_msearch requests sent to Elasticsearch. ← Here is the performance problem.
5. Click the "add" button for _score in the field list, sort by the _score column, and refresh the page
6. Observe a single /elasticsearch/_msearch requests sent to Elasticsearch.
7. Remove the _score column, and refresh the page
8. Observe a single /elasticsearch/_msearch requests sent to Elasticsearch. ← This is how the request should have been crafted after first setting the time range to "Last 7 days" in step 3.

[pickypg]

Fixing this should be a pretty big performance win.

[rashidkpc]

This is not a bug, it is by design. One big _msearch is faster on tiny datasets, like the one on your laptop, but not on huge installations with long retention times. The behavior can be changed by checking Do not expand index pattern when searching (Not recommended) when creating the index pattern. The correct "fix" for this issue would be to remove (Not recommended) and replace it with a few sentences explaining the performance tradeoffs.

PS: The reason you get one request when adding _score is due to the relative nature of scoring.

[ebuildy]

Funny, I would like report the opposite, to me multi-search is more expensive than running individual widget searches (I can see in browser dev console, dashboard _msearch took 2.000ms where as running each widget queries individually took no more than a few millisecondes).

[digitalpacman]

I instantly ran into this after upgrading from ES 1 / kibana 2. My dashboards with 400k in the old system load in 100ms, while kibana 6.x takes 700-800ms just to run discovery on 5k records. This is all because of _msearch in my opinion. Horrible end point.

[lukasolson]

Closing as the discussion for this is continued here:

#11832
#14750