[Logs UI] Flyoutitemquery never completes and has consistent timeouts on small cluster with large number of indices

[driskell]

Kibana version: 6.8

Elasticsearch version: 6.8

Server OS version: CentOS 7 latest

Browser version: Safari

Browser OS version: 12.1.1

Original install method (e.g. download page, yum, from source, etc.): yum

Description of the problem including expected versus actual behavior:
When streaming logs on Logs everything is loading nicely and fast. However, once you click an item to view the log entry details, the slide out sticks on Loading and never loads. Inspecting network we can see the flyoutitemquery takes 30 seconds and fails and returns a timeout.

Expected:

1. If timeout happens, an error to display. Currently it shows nothing.
2. The log event to actually display - it seems Logs is capable of retrieving lots of logs and streaming live with no issue, yet retrieving a single item causes a timeout.

Thoughts:
Is it because there is no index mentioned in the flyoutitemquery, so ES has to scan all indices for the given ID?

Steps to reproduce:

1. Visit Logs
2. Stream Live
3. Click an event to show event
4. Examine Network in inspector whilst doing the above, and compare response times. Retrieving lots of logs to stream is fast. flyoutitemquery is slow or times out.

Errors in browser console (if relevant):
None

Provide logs and/or server output (if relevant):
N/A

[elasticmachine]

Pinging @elastic/infra-logs-ui

[weltenwort]

I couldn't reproduce it, because I probably don't have a sufficiently large dataset.

If timeout happens, an error to display. Currently it shows nothing.

A reasonable request. It's similar to #40882, but for the log item query.

Is it because there is no index mentioned in the flyoutitemquery, so ES has to scan all indices for the given ID?

That's a possibility, which I will investigate. Should be fixable.

[driskell]

Just to confirm - this is still an issue in 6.8.6

I have not yet considered 7.x

[driskell]

Reproduced in 7.x

[driskell]

Note that PR does not resolve the displaying of an error. It resolves the underlying conditions though.