Kibana times out after 120s

[champtar]

Kibana version:
5.6.0

Elasticsearch version:
5.6.0

Server OS version:
CentOS 7.4

Browser version:
curl 7.29 (CentOS 7.4)

Original install method (e.g. download page, yum, from source, etc.):
yum

Description of the problem including expected versus actual behavior:
Kibana timeout after 120s despite setting elasticsearch.requestTimeout to a larger value

Steps to reproduce:

1. set elasticsearch.requestTimeout: 30000 (30s) and restart kibana
2. run a long query (in my example aggregation with script reading _source to be sure it's slow enough)

time curl 'http://127.0.0.1:5601/api/console/proxy?path=vme-access-logs-*%2F_search&method=POST' -H 'Accept: text/plain, */*; q=0.01' -H 'kbn-version: 5.6.0' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36' -H 'Content-Type: application/json' --data-binary $'{\n  "aggregations" : {\n    "2" : {\n      "terms" : {\n        "field" : "hub",\n        "size" : 1000,\n        "min_doc_count" : 1,\n        "shard_min_doc_count" : 0,\n        "show_term_doc_count_error" : false,\n        "order" : [\n          {\n            "_count" : "desc"\n          },\n          {\n            "_term" : "asc"\n          }\n        ]\n      },\n      "aggregations" : {\n        "7" : {\n          "sum" : {\n            "script" : {\n              "inline" : "return params[\'_source\'][\'response_code\'] == \'400\' ? 1 : 0;",\n              "lang" : "painless"\n            }\n          }\n        }\n      }\n    }\n  }\n}\n' --compressed -v

kibana response (as expected non empty response / timeout after 30s)

< HTTP/1.1 504 Gateway Timeout
< kbn-name: kibana
< kbn-version: 5.6.0
< content-type: application/json; charset=utf-8
< cache-control: no-cache
< vary: accept-encoding
< content-encoding: gzip
< Date: Mon, 18 Sep 2017 19:07:04 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
< 
* Connection #0 to host 127.0.0.1 left intact
{"statusCode":504,"error":"Gateway Timeout","message":"Client request timeout"}
real	0m30.058s
user	0m0.006s
sys	0m0.004s

3. now set elasticsearch.requestTimeout: 300000 (5m) and restart kibana
4. run the same query, you get an "empty response" after 120s (kibana just close the tcp connection (confirmed with tcpdump))

* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server

real	2m0.029s
user	0m0.005s
sys	0m0.013s

Additional information:

https://nodejs.org/api/http.html#http_server_timeout
120s is the default nodejs timeout

[kimjoar]

Hi @champtar. Thanks for reporting. It looks like you're correct and that it's the default timeout for nodejs you're hitting. I can't immediately think of a way to get around this in the short-term, as it doesn't look like we expose any config value right now you can use to change this setting.

[didierb78]

Hello. This problem still occurs in 6.0.0 (install method: tar.gz archive). Please see https://discuss.elastic.co/t/kibana-6-timeout-below-2-minutes-every-settings-10-minutes-with-gateway-timeout/. Here is the last part of this message:

The problem was similar to this one, solved by this pull request. However all of this happened in Kibana 4. I still think it's a node.js configuration issue, and this issue (meaning the one you are reading right now at github), still open in 5.6.0, seems to agree.

[didierb78]

Problem Solved! It was, in fact, a timeout of a proxy between Kibana (firefox) and Kibana (node.js)!

[epixa]

@didierb78 Thanks for the heads up. I'm going to close this since it seems to have been an issue with the reverse proxy configuration in front of Kibana.

[cpointner]

@epixa please re-open the issue since the bug is in fact not fixed!
I have the same issue as @champtar and have confirmed that no proxy timeout is the reason for this behavior.

[epixa]

@cpointner Thanks for the heads up!

[epixa]

@elastic/kibana-platform

[messiahUA]

I'm using kibana 6.1.3 in k8s (nginx ingress) and experience this issue.

ELASTICSEARCH_REQUESTTIMEOUT set to 600000 and nginx proxy-read-timeout set to 600. Connection closes on long queries in 120 seconds.

[epixa]

Can someone that is still experiencing this provide concrete steps to reproduce? It seems to only be affecting an extremely small number of users which makes it very difficult to gather information about, and I can't reproduce it with Kibana alone.

[stavinsky]

I have the same problem. Exactly 2 minutes.
Only in kibana. Pure curl query to es or query though cerebro can wait forever.
Timeouts in kibana set to much bigger values.

elasticsearch.requestTimeout: 240000
elasticsearch.shardTimeout: 240000

Kibana 6.7.1

[ilmarh]

kibana/src/core/server/http/http_tools.ts

Line 84 in aaa180b

// Revert to previous 120 seconds keep-alive timeout in Node < 8.

Is it hardcoded here or somewhere else?

[landong1973]

I was hit by the same problem using Kibana 7.0.1.Kibana returned error 502 Bad Gate way exactly after120 seconds despite it connected directly to elasticsearch. A solution is needed!

[landong1973]

It seems to me this issue has been resolved 3 years ago, see the pull request
#6341

I am wondering how come it poped up again? @w33ble @spalger @epixa

[lukasolson]

Duplicate of #31549.