504 Gateway time out (if the backed takes more than 60 seconds)

[viveksinghggits]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):**

I searched for the exact error (error: 504 Gateway Time-out) that I got but didnt see any issue related to that.

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug Report

NGINX Ingress controller version: 0.24.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Bare metal
• OS (e.g. from /etc/os-release):

NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.5"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5"

• Kernel (e.g. uname -a): Linux ilcepoc504 3.10.0-862.20.2.el7.x86_64 #1 SMP Thu Oct 11 08:14:38 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

What happened:
I created a fission HT that is being served through the ingress controller, if I test that fission function using command fission fn test --name <fn-name> that worked as expected and I get the response. The function is taking 100seconds. But if I curl to the fission HTTP trigger that I have created I get below error
curl: (22) The requested URL returned error: 504 Gateway Time-out

What you expected to happen:
I am expecting to get a response no matter how much time my function takes.

How to reproduce it (as minimally and precisely as possible):
Create a fission function that takes more than 60 seconds to be completed expose that function using HTTP trigger and ingress, curl to the HTTP trigger that you have created.

[aledbf]

I am expecting to get a response no matter how much time my function takes.

That is not how nginx works. You need to set a timeout using the annotation nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts

[aakashrshah]

After adding

nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"

it throws a 502

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: Error reading from remote server

Using a reverse proxy to connect to api on a apache server image hosted on K8s cluster.

[m-dinesh-kumar]

I know this thread is closed, but for folks who has the same issue we used the following to resolve.

The timeout added was in seconds.

Nginx - Ingress for K8S
The following annotations were added

    nginx.ingress.kubernetes.io/proxy-connect-timeout: "180"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "180"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "180"

Once this was added the Apache had its own timeout of 60s by default
Apache - Used as reverse proxy server

NOTE: The below ProxyPass was used in <Location> directive. Hence no alias
ProxyPass http://<your-nginx-ingress-url> connectiontimeout=180 timeout=180

The issue got resolved.

[answerquest]

@m-dinesh-kumar thanks, can you specify if the time unit is in seconds or other?

[mrtrkmn]

None of suggestions work correctly. FYI

[answerquest]

Hi, just for benefit of folks working with Openshift : Instead of ingress, I created an openshift Route. No YAML needed, they had a straightforward wizard in the openshift console. After creating, in its settings there are "Annotations". I added: key: haproxy.router.openshift.io/timeout , value: 5m and saved. That increased my deployment's gateway timeout limit to 5 minutes instead of default 60 seconds.

This may not be applicable for non-openshift users but might help others like me who strayed here. We moved to Openshift recently so even our architects didn't know we could do this.
Related (but from commandline oc PoV) docs: https://docs.openshift.com/container-platform/3.5/install_config/configuring_routing.html

[stephen304]

Running into the same issue - the timeout seemed to completely bork a wordpress install which takes a while when you submit the setup wizard. I would like to raise the timeout globally, but the nginx-ingress docs linked above seem to omit which configmap the parameters go in. I have 4 nginx-ingress configmaps:

ingress       nginx-load-balancer-microk8s-conf    0      35h
ingress       nginx-ingress-tcp-microk8s-conf      0      35h
ingress       nginx-ingress-udp-microk8s-conf      0      35h
ingress       ingress-controller-leader-nginx      0      35h

[sanzenwin]

@m-dinesh-kumar 💯