[nginx] Remote IP address not preserved with TLS in spite of externalTrafficPolicy: Local

[rolftimmermans]

I am using Kubernetes 1.7.2 with externalTrafficPolicy: Local for the nginx loadbalancer service.

I am running the nginx controller as a daemonset on select nodes.

Requests via HTTP are correctly logged with the remote IP address:
89.200.35.217 - [89.200.35.217] - - [03/Aug/2017:09:24:24 +0000] "GET / HTTP/1.1" 301 5 "-" "curl/7.53.1" 83 0.002 [upstream-default-backend] 10.64.80.50:80 5 0.002 301

However, requests over HTTPS always have the remote IP address set to 127.0.0.1:
127.0.0.1 - [127.0.0.1] - - [03/Aug/2017:09:24:42 +0000] "GET / HTTP/1.1" 301 5 "-" "-" 37 0.003 [upstream-default-backend] 10.64.80.50:80 5 0.003 301

I am using the image gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.11

[marcocaberletti]

Hi,
I've solved setting use-proxy-protocol: "true" with the configuration ConfigMap.

[rolftimmermans]

I am not using an additional HTTP load balancer in front of the nginx ingress controllers, so I cannot use the proxy protocol.

EDIT: Adding that setting use-proxy-protocol: "true" does indeed solve the problem! Which confused me immensely, since it appears to be completely unrelated to our situation. Is this a bug?

EDIT2: It turns out this is not a good solution and causes nginx to become unresponsive. See below for details.

[simonklb]

gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.8 seem to work fine without proxy protocol.

[aledbf]

@rolftimmermans where are you running your cluster? what version of k8s?

[rolftimmermans]

Our cluster is on version 1.7.2 at Google Cloud (GKE).

[aledbf]

@rolftimmermans how are you exposing the controller? Using a service type=Loadbalancer?
Please share the yaml files

[rolftimmermans]

Yes. Below is the configuration.

Service with type: LoadBalancer causes a region-wide TCP load balancer to be provisioned on GCP. As far as I'm aware this uses packet forwarding to forward TCP packets directly to the nodes of the Kubernetes cluster. With externalTrafficPolicy: Local that should mean that the TCP packets are routed directly to the node on which the nginx ingress controller runs. Please correct me if I'm wrong.

apiVersion: v1
  kind: Service
  metadata:
    name: loadbalancer
    labels:
      app: loadbalancer
  spec:
    type: LoadBalancer
    loadBalancerIP: $(API_IP)
    externalTrafficPolicy: Local
    ports:
    - port: 80
      name: http
    - port: 443
      name: https
    selector:
      app: loadbalancer-nginx

apiVersion: extensions/v1beta1
  kind: DaemonSet
  metadata:
    name: loadbalancer-nginx
    labels:
      app: loadbalancer-nginx
  spec:
    template:
      metadata:
        labels:
          app: loadbalancer-nginx
      spec:
        containers:
        - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.11
          name: nginx-ingress-controller
          readinessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
          ports:
          - containerPort: 80
            name: http
          - containerPort: 443
            name: https
          env:
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          args:
          - /nginx-ingress-controller
          - --watch-namespace=$(POD_NAMESPACE)
          - --default-backend-service=$(POD_NAMESPACE)/api
          - --default-ssl-certificate=$(POD_NAMESPACE)/api-tls
          - --publish-service=$(POD_NAMESPACE)/loadbalancer
          - --configmap=$(POD_NAMESPACE)/loadbalancer-nginx-conf

apiVersion: v1
  kind: ConfigMap
  metadata:
    name: loadbalancer-nginx-conf
  data:
    client-body-buffer-size: "32M"
    server-tokens: "false"
    # Strangely, using use-proxy-protocol: "true" seems to solve the problem.
    use-proxy-protocol: "true"
    proxy-buffering: "false"
    proxy-read-timeout: "600"
    proxy-send-timeout: "600"
    proxy-body-size: "1G"
    hsts: "false"
    upstream-keepalive-connections: "50"

The relevant ingress configuration:

apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: api
    labels:
      app: api
    annotations:
      kubernetes.io/tls-acme: "true"
      kubernetes.io/ingress.class: "nginx"
      ingress.kubernetes.io/service-upstream: "true"
      nginx.org/hsts: "false"
  spec:
    tls:
    - secretName: api-tls
      hosts:
      - $(API_HOST)
    backend:
      serviceName: api
      servicePort: 80

[rolftimmermans]

Ok, so setting use-proxy-protocol: "true" is NOT a good idea if the proxy protocol is not (always?) used.

Nginx at some point may become confused and stops responding to requests properly. It will cause connection timeouts for lots of clients. Logs are full of the following type of entries:

2017/08/11 15:04:48 [error] 10203#10203: *67930 broken header: "GET /robots.txt HTTP/1.1
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
Accept-Encoding: gzip,deflate

" while reading PROXY protocol, client: xxx.xxx.xxx.xxx, server: 0.0.0.0:80
2017/08/11 15:04:49 [error] 10440#10440: *67931 broken header: "GET /robots.txt HTTP/1.1
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
Accept-Encoding: gzip,deflate

" while reading PROXY protocol, client: xxx.xxx.xxx.xxx, server: 0.0.0.0:80
2017/08/11 15:08:20 [error] 10203#10203: *67932 broken header: "GET / HTTP/1.1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4

" while reading PROXY protocol, client: xxx.xxx.xxx.xxx, server: 0.0.0.0:80
2017/08/11 15:08:20 [error] 10410#10410: *67933 broken header: "GET / HTTP/1.1
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4

" while reading PROXY protocol, client: xxx.xxx.xxx.xxx, server: 0.0.0.0:80
2017/08/11 15:08:21 [error] 10341#10341: *67934 broken header: "GET / HTTP/1.1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4

[rolftimmermans]

Below are observations based on nginx-ingress-controller:0.9.0-beta.11

So it seems the following is the problem. This scenario happens

• if the nginx ingress controller handles both HTTP and HTTPS traffic; and
• if there is NO load balancer in front of the controller or if there is a TCP forwarder in front of the controller (like the GCP regional TCP load balancer)

By default the following configuration is generated (irrelevant parts omitted). Note that the real IP address is parsed from the X-Forwarded-For header (if that header is absent it uses the remote address). This works well for HTTP, but not for HTTPS. HTTPS traffic appears to be proxied by the golang controller itself. It uses the PROXY protocol to pass the source address (note listen 442 proxy_protocol), but the configuration never uses this information, it uses X-Forwarded-For!

# WITHOUT use-proxy-protocol: "true"
http {
    set_real_ip_from    0.0.0.0/0;
    real_ip_header      X-Forwarded-For; # <--- wrong for HTTPS!

    server {
        listen 80 default_server reuseport backlog=511;
        listen [::]:80 default_server reuseport backlog=511;

        listen 442 proxy_protocol default_server reuseport backlog=511 ssl http2;
        listen [::]:442 proxy_protocol  default_server reuseport backlog=511 ssl http2;
    }

Contrast this to the following scenario.

If use-proxy-protocol: "true" is set, the following configuration is generated (irrelevant parts omitted again). Now the source for all traffic is taken from the PROXY protocol. This works well for HTTPS! But now HTTP handling is broken because (see above) there is no load balancer in front of the ingress controller that actually uses the proxy protocol. Nginx will fail to correctly parse incoming HTTP requests!

# WITH use-proxy-protocol: "true"
http {
    set_real_ip_from    0.0.0.0/0;
    real_ip_header      proxy_protocol;

    server {
        listen 80 proxy_protocol default_server reuseport backlog=511; # <--- wrong!
        listen [::]:80 proxy_protocol default_server reuseport backlog=511; # <--- wrong!

        listen 442 proxy_protocol default_server reuseport backlog=511 ssl http2;
        listen [::]:442 proxy_protocol  default_server reuseport backlog=511 ssl http2;
    }

So it seems there is no way to serve HTTP & HTTPS and log the source IP for HTTPS if there is no load balancer in front of the nginx ingress controller.

Unfortunately I don't know why HTTPS traffic is proxied by golang. I imagine a different configuration would need to be generated for taking the source IP from HTTPS traffic – HTTPS source IP should apparently always use the PROXY protocol since this is how the golang proxy passes on the source IP. The HTTP traffic should be configured to use X-Forwarded-Proto or proxy_protocol depending on the setting of use-proxy-protocol: "true".

It seems this is a bug. I'd love to try to propose a patch, but I'm very unfamiliar with the principles behind the nginx controller (e.g.: why is HTTPS traffic proxied by the golang controller?). I hope this is useful for someone more familiar with the code base. Let me know if anyone needs more information.

[aledbf]

@rolftimmermans please use the image quay.io/aledbf/nginx-ingress-controller:0.191

Edit: this image contains current master where the ssl-passthrough feature is behind a flag and by default it's disabled.

[aledbf]

(e.g.: why is HTTPS traffic proxied by the golang controller?).

This is required to enable ssl-passthrough. The golang proxy allows the pipe of the connection to the backend exposing the SSL certificate. The proxy protocol in port 442 is required to not lose the source IP address (internet -> go proxy -> nginx upstream)

[aledbf]

Service with type: LoadBalancer causes a region-wide TCP load balancer to be provisioned on GCP

In GCP/GKE proxy protocol is available only in HTTPS

[rolftimmermans]

please use the image quay.io/aledbf/nginx-ingress-controller:0.191
this image contains current master where the ssl-passthrough feature is behind a flag and by default it's disabled.

Yes, this does seem to work fine. I can now see source addresses being correctly logged for both HTTP and HTTPS traffic. Excellent change, would love to see this in a stable(ish) release!

[aledbf]

@rolftimmermans #1249

[aledbf]

Closing. Fixed in master

[rushabhnagda11]

@rolftimmermans Can you share your nginx service file? I'm facing the exact same issue currently.

I'm directly exposing my nginx ingress service with type = loadbalancer, setting enable-proxy-protocl doesn't work for me and I see the same broken header issue.

[rolftimmermans]

@rushabhnagda11 I don't understand exactly what you mean with "nginx service file". My (simplified) configuration is listed here. Bottom line is that you should NOT set enable-proxy-protocol unless you have a load balancer in front of your cluster that actually uses the PROXY protocol.

Instead, upgrade to 0.9.0-beta.12.

[rushabhnagda11]

@rolftimmermans I've upgraded to 0.9.0-beta.12, however the ip being forwarded to my application server in both headers is the private ip of the machine in which the nginx pod is running.

"x-real-ip":"10.112.98.42","x-forwarded-for":"10.112.98.42"

I'm running 1.5.x, so I've enabled preserving source ip throug beta annotations.

Update : Setting this annotation :
annotations: service.beta.kubernetes.io/external-traffic: OnlyLocal

Makes nginx go unresponsive. Moment I remove it, everything seems fine. By fine I mean that traffic is routed to my application pod. Ip is still incorrect.

[rushabhnagda11]

Is there any other info/config that I can provide? seems like a solved issue from ingress side.

[aledbf]

@rushabhnagda11 what k8s are you using? where are you running the cluster? how the traffic reaches the cluster?

[rushabhnagda11]

k8s version -> k8s 1.5.6,
Platform -> cluster is running on IBM's bluemix
traffic reaching in the cluster -> Not really sure. I've just given my nginx-ingress service as type LoadBalancer. I'm using the external ip in my dns for routing. There is no external load balancer of mine or in my infra.

Seems like there is a public subnet that they've given me. And from there the traffic reaches my specific public IP.

[rushabhnagda11]

On further debugging it looks like cat /etc/nginx/ngin.conf had the set_real_ip_from conf as 0.0.0.0/0, which would mean that nginx would pick the last non-trusted ip in the x-forwarded-for chain, which would always be the pod servers' private ip, (0.0.0.0/0 -> all ips are non trusted)

in the --configmap option i've specified the proxy-real-ip-cidr:10.112.98.42 option which updates my /etc/nginx/nginx.conf to

    real_ip_header      X-Forwarded-For;

    real_ip_recursive   on;
    set_real_ip_from    10.112.98.0/24;

However nginx logs still show the follwing 2017-09-05T07:18:33.853129101Z 10.112.98.42 - [10.112.98.42] - - [05/Sep/2017:07:18:33 +0000] "GET /admin/users?count=20&page=1 HTTP/2.0" 304 0 "http://blahblahblah" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36" 70 0.026 [mypod] 172.30.155.46:3000 0 0.026 304

Shouldn't 10.112.98.42 be omitted from the x-forwarded-for chain?

Update 1: If I specify the x-forwarder-for header in my request as say 1.2.3.4, then the ip is being logged as 1.2.3.4 through nginx. Does this point to an issue from my client?

Update 2: I logged into my nginx pod using kubectl exec -it and kept playing around with the log_format line in/etc/nginx/nginx.conf and then doing a /usr/sbin/nginx -s reload -c /etc/nginx/nginx.conf

Observations:
log_format upstreaminfo '$remote_addr $realip_remote_addr logs 10.112.98.42 10.112.98.42 10.112.98.42 the same ip address in all cases. This is the private ip of the machine that the nginx pod is running on.

Which brings me to more fundamental questions :

1. Where is the ip address of a request stored?
2. Can I find this and send it in a separate header in nginx?

@aledbf Just checking if this info helps

[rushabhnagda11]

Had a chat with support and this seems like this is an issue from bluemix side. Currently there is no way to fetch client ip addresses

[Raichel-zz]

@rushabhnagda11 is there any progress with this issue? have you find a way to get the client ip?

[italianocaliente]

Hi there,
Still cannot get this to work, using Kubernetes 1.10 and the same configs as @rolftimmermans
use-proxy-protocol cannot be set to "true" otherwise no pages load, errors in the logs on the controller like this: Error while reading PROXY protocol, client: 10.244.1.1, server: 0.0.0.0:443

Load balancer does not support PROXY protocol, so leaving this disabled i get:

Client is always 10.244.x.x (internal Ip) or if I set: externalTrafficPolicy: Local in the Ingress Service, it shows an IP of my LoadBalancer (looks like an internal ip within Loadbalancer network but its not the real external IP of client) from DigitalOcean with Manual configuration with TCP Passthrough seperated for HTTPS and HTTP.

I've tried every image including: nginx-ingress-controller:0.9.0-beta.11 and quay.io/aledbf/nginx-ingress-controller:0.191 as well as the latest images on both Repos and the results are always the same.

Note: I get the Real client IP in HTTP mode. Can anyone help?

[ghost]

@attiinfante, see my solution in #808 (comment)

[dano0b]

In case somebody is still searching a solution: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip

[dharmendrakariya]

https://www.yellowduck.be/posts/k8s-getting-real-ip-digital-ocean/

[longwuyuan]

Please discuss on slack at kubernetes.slack.com in the Digitalocean-k8s channel. Thanks, ; Long

…

On Wed, 11 Aug, 2021, 1:17 PM dharmendra kariya, ***@***.***> wrote: https://www.yellowduck.be/posts/k8s-getting-real-ip-digital-ocean/ — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1067 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWUNF4QYDE2N6YNOKMTT4ITKNANCNFSM4DVQXLEQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .