308 redirect loop when setting "force-ssl-redirect" to "true" #1957

danielfm · 2018-01-23T19:00:02Z

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): force ssl redirect loop 308

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report

NGINX Ingress controller version: 0.10.0

Kubernetes version (use kubectl version): v1.8.5

Environment:

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release): Container Linux by CoreOS 1576.5.0 (Ladybug)
Kernel (e.g. uname -a): 4.14.11-coreos
Install tools: kube-aws
Others:

What happened:

The force-ssl-redirect flag seems not to be working anymore.

What you expected to happen:

This is what happened before I updated the nginx ingress controller (this is from version 0.9.0-beta.19):

$ kubectl -n kube-system port-forward hissing-wasp-nginx-ingress-controller-75b88b9b55-9vq8h 8080:80
Forwarding from 127.0.0.1:8080 -> 80

# In another terminal, redirect is expected
$ curl -I -H "Host: <ingress host>" -H "X-Forwarded-Proto: http" http://localhost:8080
HTTP/1.1 301 Moved Permanently
Server: nginx/1.13.7
Date: Tue, 23 Jan 2018 18:48:26 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://<ingress host>/

# Redirect is NOT expected
$ curl -I -H "Host: <ingress host>" -H "X-Forwarded-Proto: https" -H "X-Forwarded-Port: 443" http://localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.13.7
Date: Tue, 23 Jan 2018 18:48:31 GMT
Content-Type: text/html
Content-Length: 79850
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: Next.js 4.2.1

This is what happens in 0.10.0:

# Redirect is expected
$ curl -I -H "Host: <ingress host>" -H "X-Forwarded-Proto: http" http://localhost:8080
HTTP/1.1 308 Permanent Redirect
Server: nginx/1.13.8
Date: Tue, 23 Jan 2018 18:47:04 GMT
Content-Type: text/html
Content-Length: 187
Connection: keep-alive
Location: https://<ingress host>/

# Redirect is NOT expected
$ curl -I -H "Host: <ingress host>" -H "X-Forwarded-Proto: https" -H "X-Forwarded-Port: 443" http://localhost:8080
HTTP/1.1 308 Permanent Redirect
Server: nginx/1.13.8
Date: Tue, 23 Jan 2018 18:47:19 GMT
Content-Type: text/html
Content-Length: 187
Connection: keep-alive
Location: https://<ingress host>/

The text was updated successfully, but these errors were encountered:

grebois · 2018-01-24T02:41:12Z

Also happening with;

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
    ingress.kubernetes.io/secure-backends: "true"
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: "nginx"

spec:
  rules:
  - host: dashboard.<my_domain>
    http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 443

fyuvb · 2018-01-24T05:19:22Z

The same thing happens to me with config

nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

on version 0.10.0

JoelSpeed · 2018-01-24T09:27:22Z

Also seeing this problem. I believe it was introduced in this PR https://github.com/kubernetes/ingress-nginx/pull/1854/files

We have ELBs in front of our ingress which terminate the TLS connection and connect to the ingress controllers over http. Therefore $scheme=http and $pass_access_scheme=https so the rule is matched and $redirect_to_https is set to true.

I believe this to be a perfectly valid setup and I believe the intention of this change was to stop the case where an upstream load balancer was receiving an http connection but issuing https connections to the ingress controller, in which case the rule would be https:http not http:https

If the upstream proto is https then I don't think it should ever perform the redirect.

@aledbf Would you mind confirming my suspicions of the intention?

aledbf · 2018-01-24T12:05:18Z

@danielfm @JoelSpeed please use quay.io/aledbf/nginx-ingress-controller:0.318

JoelSpeed · 2018-01-24T12:07:50Z

@aledbf Can I ask what you've changed?

danielfm · 2018-01-24T12:07:51Z

@aledbf this image fixed the issue for me.

aledbf · 2018-01-24T12:08:31Z

@JoelSpeed you are right, that PR checks the variables $scheme and $pass_access_scheme
This is the table that shows when is required a redirect:

Scheme	X-Forwarded-Proto	Redirect?
http	http	yes
http	https	yes
https	http	no
https	https	no

aledbf · 2018-01-24T12:08:55Z

Can I ask what you've changed?

I removed the change in #1854

Edit: everything started with this issue #1841

Edit: with this change the table is:

Scheme	X-Forwarded-Proto	Redirect?
http	http	yes
http	https	no
https	http	no
https	https	no

JoelSpeed · 2018-01-24T12:14:48Z

@aledbf I think lines two and three of that table should be swapped.

If X-Forwarded-Proto is http that means the connection to the upstream load balancer was made by http which would be when we want to upgrade, but line three suggests otherwise?

Similarly if X-Forwarded-Proto is https the original connection to the upstream load balancer was https and then we don't care if the connection to nginx was http, the original client connected by https which is the desired effect right?

JoelSpeed · 2018-01-24T12:22:32Z

@aledbf quay.io/aledbf/nginx-ingress-controller:0.318 works for me

aledbf · 2018-01-24T12:24:35Z

If X-Forwarded-Proto is HTTP that means the connection to the upstream load balancer was made by HTTP which would be when we want to upgrade, but line three suggests otherwise?

Please keep in mind we cannot differentiate a connection from a client or a load balancer. With that in mind if you make a request to NGINX using https we should not redirect (again) to https.

JoelSpeed · 2018-01-24T12:33:27Z

Please keep in mind we cannot differentiate a connection from a client or a load balancer. With that in mind if you make a request to NGINX using https we should not redirect (again) to https.

I don't think I follow here.

If a client connects directly to the ingress controller, then the X-Forwarded-Proto header won't exist and so $pass_access_scheme=$scheme

ingress-nginx/rootfs/etc/nginx/template/nginx.tmpl

Lines 197 to 200 in 3bc2c68

    
           map $http_x_forwarded_proto $pass_access_scheme { 
        
               default          $http_x_forwarded_proto; 
        
               ''               $scheme; 
        
           }

The only case when you would get a https:http for your table is when there is a load balancer between the client and the ingress controller and the client made the connection to the load balancer by http?

If that isn't the case could you quickly explain a scenario where you'd get https:http from a client directly?

aledbf · 2018-01-24T12:36:08Z

If that isn't the case could you quickly explain a scenario where you'd get https:http from a client directly?

The load balancer uses HTTPS to connect to NGINX or a client sends the X-Forwarded-Proto header

JoelSpeed · 2018-01-24T12:46:17Z

The load balancer uses HTTPS to connect to NGINX

That's what I was suggesting, sorry, I didn't make that too clear

Client -- HTTP -- > LB -- HTTPS --> NGINX

Ends up with X-Forwarded-Proto=http therefore $pass_access_scheme=http but $scheme=https so the table entry is https:http and we should redirect to get

Client -- HTTPS -- > LB -- HTTPS --> NGINX

aledbf · 2018-01-24T12:55:15Z

Ends up with X-Forwarded-Proto=http therefore $pass_access_scheme=http but $scheme=https so the table entry is https:http and we should redirect to get

What's the difference handling a connection between a load balancer and direct client?
If we add a redirect for https:http then you can force a redirect using an HTTPS connection setting X-Forwarded-Proto=http which lead to a redirect loop.

aledbf · 2018-01-24T12:59:41Z

@JoelSpeed just to be clear, I think you are right and the table should be

    map "$scheme:$pass_access_scheme" $redirect_to_https {
        default          0;
        "http:http"      1;
        "https:http"     1;
    }

but this issue has already bit me too many times.

aledbf · 2018-01-24T13:03:46Z

Please test quay.io/aledbf/nginx-ingress-controller:0.319. It contains ^^

JoelSpeed · 2018-01-24T13:36:13Z

I'm getting Image pull failures, looks like that image didn't push to quay

erwbgy · 2018-01-24T16:58:04Z

The quay.io/aledbf/nginx-ingress-controller:0.318 image also fixed a problem I was having where:

nginx.ingress.kubernetes.io/ssl-passthrough: "true"

was being ignored.

aledbf · 2018-01-24T17:06:24Z

I'm getting Image pull failures, looks like that image didn't push to quay

Please try again

JoelSpeed · 2018-01-24T17:12:59Z

Works for me 👍

aledbf · 2018-01-24T17:23:49Z

@JoelSpeed are you ok with the change in the redirect map?

JoelSpeed · 2018-01-24T17:26:01Z

Yeah, I think this is a good solution

discostur · 2018-03-28T12:59:12Z

Hi,

just had a similary problem ... trying to configure a catch all ingress like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    releaseTime: 2018-03-28 12:14:07.305364616 +0000 UTC
  creationTimestamp: 2018-03-27T20:40:51Z
  generation: 11
  labels:
    heritage: Tiller
    release: int
  name: redirection-v1
  namespace: integration
spec:
  rules:
  - host: '*.de'
    http:
      paths:
      - backend:
          serviceName: redirection-v1
          servicePort: 8080

I'm trying to reach the service via plain HTTP and there should be NO redirect, but:

$curl -I -H "X-Forwarded-Proto: https" http://redirection.integration.de/?domain=xyz
HTTP/1.1 200 OK
Server: nginx/1.13.9
Date: Wed, 28 Mar 2018 12:51:58 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

$curl -I -H "X-Forwarded-Proto: http" http://redirection.integration.de/?domain=xyz
HTTP/1.1 308 Permanent Redirect
Server: nginx/1.13.9
Date: Wed, 28 Mar 2018 12:51:54 GMT
Content-Type: text/html
Content-Length: 187
Connection: keep-alive
Location: https://redirection.integration.de/?domain=xyz

I don't unterstand why i get a redirect if i'm connecting via HTTP ... it i set x-forwarded-for to https the request is plain http and i get an answer from my service. Is there anything else i have to configure?

adriancislo · 2018-08-07T11:34:54Z

Based on https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md
I set (changing default value)
nginx.ingress.kubernetes.io/ssl-redirect: "false" in ingress annotations and it started to behave as I expected. Call service via http returns 200 not 308

trueinviso · 2019-02-08T19:52:40Z

It appears that this may be an issue again.

I am using kubernetes on AWS and this is what my ingress looks like:

kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  rules:
  - host: "example.com"
    http:
      paths:
      - path: /
        backend:
          serviceName: app-cluster-ip-service
          servicePort: 80

I am terminating TLS at the load balancer and I was getting the infinite redirect loop when I turned on force-ssl-redirect="true", if i set it to false the loop stops but I don't get a redirect to https. When I update the image in the ingress-nginx-controller deployment file to the previous version then all is well. So in case anyone else is unfamiliar with this like I was:

kubectl edit deployment -n ingress-nginx nginx-ingress-controller

Look for:

image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:<version>

The newest version right now is 0.22.0:

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0

and I changed it to:

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0

It appears that this was fixed in the previous version, and then some new changes broke it again.

bfin · 2019-02-12T16:22:00Z

@trueinviso , in my case – also using the controller behind an AWS ELB – the infinite redirect loop was caused by a (documented) breaking change introduced in v0.22.0 regarding forwarded headers. Setting use-forwarded-headers: "true" in the configmap (docs) fixed the issue for me.

trueinviso · 2019-02-17T16:41:22Z

@bfin Thanks! That fixes it. I'm not sure I would have been able to figure that out on my own haha.

deliuth · 2019-05-08T17:56:56Z

Thanks @bfin ! works for me on 0.22.0 behind an AWS ELB

joshbranham · 2019-07-30T17:30:09Z

Thanks @bfin this fixed our issues as well

gdelgado · 2019-11-12T23:38:24Z

Tried the above use-forwarded-headers: "true" within the configmap and it did not work. Using nginx version 0.24.1. Any one else having the same issue with 0.24.1

juniorrocha · 2020-03-03T03:47:31Z

Tried the above use-forwarded-headers: "true" within the configmap and it did not work. Using nginx version 0.24.1. Any one else having the same issue with 0.24.1

I had the same problem and fixed with use-forwaded-headers: "true", within 0.24.1

BobbyJohansen · 2020-03-06T23:49:58Z

It would seem that when using:

http-snippet: |
    map true $pass_access_scheme {
      default "https";
    }
    map true $pass_port {
      default 443;
    }

    server {
      listen 8080 proxy_protocol;
      return 308 https://$host$request_uri;
    }

I do not get X-forwarded-for headers even when I have use-forwaded-headers: "true" set I am also using quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1

Does anyone know how to get the X-Forwarded-For headers to propagate over the http to https redirect?

vikas027 · 2024-04-15T02:07:49Z

use-forwarded-headers: "true" worked for me as well in my

Traffic Flow
Internet -> App Gateway -> Firewall -> Internal Load Balancer -> Nginx Ingress Controller Pods

ConfigMAp

apiVersion: v1
data:
  allow-snippet-annotations: "false"
  use-forwarded-headers: "true"
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.0
    helm.sh/chart: ingress-nginx-4.10.0
  name: ingress-nginx-controller
  namespace: ingress-nginx

aledbf added bug kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed bug labels Jan 23, 2018

aledbf mentioned this issue Jan 24, 2018

Rollback #1854 #1969

Merged

aledbf closed this as completed in #1969 Jan 24, 2018

JimtotheB mentioned this issue Jun 29, 2018

Http -> https redirect on TCP ELB terminating ssl, results in a 308 redirect loop. #2724

Closed

trueinviso mentioned this issue Feb 12, 2019

Ingress with app-root and force-ssl-redirect is not redirecting to HTTPS #3235

Closed

aledbf mentioned this issue Apr 14, 2020

Add port for plain HTTP to HTTPS redirection #5374

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

308 redirect loop when setting "force-ssl-redirect" to "true" #1957

308 redirect loop when setting "force-ssl-redirect" to "true" #1957

danielfm commented Jan 23, 2018

grebois commented Jan 24, 2018

fyuvb commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

danielfm commented Jan 24, 2018

aledbf commented Jan 24, 2018

aledbf commented Jan 24, 2018 •

edited

Loading

JoelSpeed commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018 •

edited

Loading

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

aledbf commented Jan 24, 2018 •

edited

Loading

JoelSpeed commented Jan 24, 2018

aledbf commented Jan 24, 2018

aledbf commented Jan 24, 2018

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

erwbgy commented Jan 24, 2018

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

discostur commented Mar 28, 2018

adriancislo commented Aug 7, 2018

trueinviso commented Feb 8, 2019

bfin commented Feb 12, 2019

trueinviso commented Feb 17, 2019

deliuth commented May 8, 2019

joshbranham commented Jul 30, 2019

gdelgado commented Nov 12, 2019

juniorrocha commented Mar 3, 2020

BobbyJohansen commented Mar 6, 2020 •

edited

Loading

vikas027 commented Apr 15, 2024

308 redirect loop when setting "force-ssl-redirect" to "true" #1957

308 redirect loop when setting "force-ssl-redirect" to "true" #1957

Comments

danielfm commented Jan 23, 2018

grebois commented Jan 24, 2018

fyuvb commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

danielfm commented Jan 24, 2018

aledbf commented Jan 24, 2018

aledbf commented Jan 24, 2018 • edited Loading

JoelSpeed commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018 • edited Loading

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

aledbf commented Jan 24, 2018 • edited Loading

JoelSpeed commented Jan 24, 2018

aledbf commented Jan 24, 2018

aledbf commented Jan 24, 2018

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

erwbgy commented Jan 24, 2018

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

aledbf commented Jan 24, 2018

JoelSpeed commented Jan 24, 2018

discostur commented Mar 28, 2018

adriancislo commented Aug 7, 2018

trueinviso commented Feb 8, 2019

bfin commented Feb 12, 2019

trueinviso commented Feb 17, 2019

deliuth commented May 8, 2019

joshbranham commented Jul 30, 2019

gdelgado commented Nov 12, 2019

juniorrocha commented Mar 3, 2020

BobbyJohansen commented Mar 6, 2020 • edited Loading

vikas027 commented Apr 15, 2024

aledbf commented Jan 24, 2018 •

edited

Loading

JoelSpeed commented Jan 24, 2018 •

edited

Loading

aledbf commented Jan 24, 2018 •

edited

Loading

BobbyJohansen commented Mar 6, 2020 •

edited

Loading