Kibana dashboard crashes when requests time out

[Sagesh]

Kibana version: 6.5

Elasticsearch version: 6.5

Server OS version: Ubuntu

Browser version: Chrome

Describe the bug:
When we use scripted fields in Kibana elastic search is getting crashed
Steps to reproduce:

1. Create 2 scripted fields to extract. last value and first value from a field based on delimiter (ex: Electronics >> Mobile>>Apple>>IPhone 8). first column will extract "Electronics" and the other column will extract "IPhone 8"
2. Create a visualization using the scripted columns (2 columns) and few other columns, we have around 7 - 8 visualization in the dashboard
3. In dashboard keep clicking the visualization (faster) to filter the values and similarly delete the filter from dashboard. If we continue for few seconds the entire elastic will hang and start giving timeout errors
4. If we verify the long running search calls in elastic it will keep increasing and will start throwing timeout errors.

Expected behavior:
Elastic search shouldn't hung/ crash
Errors in browser console (if relevant):
Timeout error

[wylieconlon]

@Sagesh Can you clarify if you are seeing a Kibana crash when requests time out? While you may be able to reproduce most easily with scripted fields, that sounds like a more general issue.

[elasticmachine]

Pinging @elastic/kibana-platform

[wylieconlon]

I was able to reproduce this in 6.8. First I saw several toast messages, and then after leaving it in the background for a while I got it to crash:

[elasticmachine]

Pinging @elastic/kibana-app-arch

[wylieconlon]

We were able to find a potential cause: if the Elasticsearch request times out, it could cause our proxy to return a 200 response with an empty body due to this line in create_proxy.js:

      return callWithRequest(req, 'transport.request', {
        path: '/_msearch',
        method: 'POST',
        query,
        body: payload.toString('utf8')
      }, { signal }).finally(r => h.response(r));

https://github.com/elastic/kibana/blob/master/src/legacy/core_plugins/elasticsearch/lib/create_proxy.js#L42

[wylieconlon]

The timeout seems to cause an empty response to get sent back, which crashes Kibana:

[Sagesh]

Thanks for your response. It's good that you were able to reproduce the issue.
We were validating the issue through "

GET /_cat/thread_pool/search?v
POST /_tasks/_cancel?nodes=ES NODE NAME&actions=*search"
command. Once the issue happens "Search" count will not get reduced and we have to restart ES for resolving the issue.

Thanks,
Sagesh

[Sagesh]

@Sagesh Can you clarify if you are seeing a Kibana crash when requests time out? While you may be able to reproduce most easily with scripted fields, that sounds like a more general issue.

After the timeout happens,ES is not responding for any queries.So we have to restart ES. Once the ES is back we were able to use Kibana.

[wylieconlon]

Thanks for clarifying. The issue you're describing is being actively worked in by many different people across teams, but you may be interested in how the Task API works with long-running searches: elastic/elasticsearch#43105

In future releases of the stack, Kibana will be smarter about cancelling dashboard queries that are no longer being used, such as in the case you mentioned. The only workaround I can suggest is to improve your query performance by following some of our general tuning recommendations.

There is still a bug in Kibana that happens when queries get cancelled before returning results which I described earlier, so I intend to keep this issue open to discuss that bug.

[lukasolson]

I believe this has been fixed in 7.7. If you are still seeing this behavior, feel please re-open.