Kibana returns empty response after 120 seconds for a long-running query

[vkurdin]

Kibana version:
6.6.0, 6.6.2
Elasticsearch version:
6.6.0, 6.6.2
Server OS version:
Ubuntu 18.04.2 LTS
Browser version:
curl 7.58.0, Firefox 66
Browser OS version:
Ubuntu 18.04.2 LTS
Original install method (e.g. download page, yum, from source, etc.):
Docker: https://www.elastic.co/guide/en/kibana/6.6/docker.html
Describe the bug:
Kibana returns empty response after 120 seconds for a long-running query
Steps to reproduce:

1. Install ES, Kibana, populate with some demo(web logs) data
2. Set Kibana config values

elasticsearch.requestTimeout: 300000
elasticsearch.shardTimeout: 300000

3. To simulate long-running query behaviour install stu-elastic/slow-query-simulator plugin
   https://github.com/stu-elastic/slow-query-simulator with some modifications to work with 6.6.0 ES: https://gist.github.com/vkurdin/6ea8a35cfd003736f5d8a4b718f6bc41
4. Restart ES with installed plugin
5. Open Discover app in Kibana
6. Open dev-console in browser and hit Refresh button in Discover app
7. Capture request to /elasticsearch/_msearch, copy it as cURL
8. Change request body ("query" subtree) to

"query":{"slow":"sleepSec":200}

9. Execute in shell(bash):

time %curl_command% ....

Observe closed connection after 120 seconds.
Expected behavior:
Kibana should wait at least elasticsearch.requestTimeout milliseconds for a long-running query.
Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:
Kibana 6.5.4 does not have this issue, works as expected.

[elasticmachine]

Pinging @elastic/kibana-app

[timroes]

Hi, could you please clarify a bit, what part actually times out or rather which part works. You described using the same query as curl in your steps to reproduce. Does that curl query work fine and don't time out, or does that time out too? What happens if you send the same query not via discover but from Kibana's Dev Tools > Console to Elasticsearch?

[jbudz]

Node.js has a global timeout of 2 minutes for all sockets. I'm sorta surprised it worked prior to 6.6, but maybe the hapi proxy config that has been removed was letting it happen.

I have #31603 to make it configurable.

[vkurdin]

@timroes
Ok.
I have ES\Kibana installtation on small cloud instance with approx 300gb of data.
Some requests sent from Kibana (Discover, Dev Tools Сonsole, Visualizations etc.) have to search\aggregate across all documents, open frozen indexes etc, so it takes more than to 2 minutes to execute them.
Starting from Kibana 6.6.0+ that kind of requests are failing: connection\socket closed exactly after 120 seconds.
I agree that Steps to reproduce with Discover app are too complex. More simple way to reproduce:
Send request from Dev Tools Console and watch it hanged after 120 seconds:

GET kibana_sample_data_logs/_search
{
  "query" : {
     "slow": {
        "sleepSec": 200
     }
  }
}

Bare requests to ES sent by curl from shell execute as long as it takes.

[tsullivan]

Hi, if this is working in curl, then the "browser version" for this issue is not curl. In addition to what @jbudz wrote about Node having its own global timeout for connections, browsers also have their own timeouts. It's up to the browser vendors to decide how to limit wait times. Some browsers have preferences for users to change the timeouts.

[vkurdin]

@tsullivan It's not working(same 120 seconds timeout) with curl either. I provided steps with curl only to demonstrate that it's browser-independent.
UPD: changed report a little bit.

[timroes]

If curl against the Kibana server (as of your description) is not working but curl against ES directly according to your previous comment is working, this really sounds like a limitation of the node socket timeout, that @jbudz mentioned earlier. Jon, I think we might want to add something in the documentation when having those properties about that correlation, that users know that they might need to change that if their shardTimeout is higher than 2 minutes?

[ilmarh]

#14043 maybe merge these issues in one?

[lukasolson]

Duplicate of #31549.