Frequent Nginx reloads with rewrite-target

[yvespp]

Is this a request for help?: No

What keywords did you search in NGINX Ingress controller issues before filing this one? rewrite-target

---

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT
NGINX Ingress controller version: 0.23.0
Kubernetes version (use kubectl version): v1.13.4

Environment:

• Cloud provider or hardware configuration: VMs
• OS (e.g. from /etc/os-release): RHEL 7
• Kernel (e.g. uname -a): 3.10
• Install tools:
• Others:

What happened:
After updating form 0.21.0 to 0.23.0 I noticed that the ingress pods use more CPU and memory than before. I then turned the logging to --v=2 to see if full reloads are happening. I can see that rewrite rules are being changed frequently, examples:

W0304 09:39:15.737876       8 nginx.go:645] Failed to executing diff command: exit status 1
I0304 09:39:15.737957       8 nginx.go:648] NGINX configuration diff:
--- /etc/nginx/nginx.conf       2019-03-04 09:39:01.153547931 +0000
+++ /tmp/new-nginx-cfg216069577 2019-03-04 09:39:15.690763202 +0000
@@ -1,5 +1,5 @@

-# Configuration checksum: 1085924712731947086
+# Configuration checksum: 6267339863093917714

 # setup custom paths that do not require root access
 pid /tmp/nginx.pid;
@@ -176333,7 +176333,6 @@
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_tries               3;

-                       rewrite "(?i)/" /api break;
                        proxy_pass http://upstream_balancer;

                        proxy_redirect                          off;

----
W0304 09:44:12.602975       9 nginx.go:645] Failed to executing diff command: exit status 1
I0304 09:44:12.603060       9 nginx.go:648] NGINX configuration diff:
--- /etc/nginx/nginx.conf       2019-03-04 09:44:08.114979520 +0000
+++ /tmp/new-nginx-cfg957563659 2019-03-04 09:44:12.547044866 +0000
@@ -1,5 +1,5 @@

-# Configuration checksum: 14196874684224941827
+# Configuration checksum: 8439993306429807185

 # setup custom paths that do not require root access
 pid /tmp/nginx.pid;
@@ -176333,7 +176333,7 @@
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_tries               3;

-                       rewrite "(?i)/" /api break;
+                       rewrite "(?i)/" /admin break;
                        proxy_pass http://upstream_balancer;

                        proxy_redirect                          off;


0304 13:26:26.520745       8 nginx.go:645] Failed to executing diff command: exit status 1
I0304 13:26:26.520797       8 nginx.go:648] NGINX configuration diff:
--- /etc/nginx/nginx.conf       2019-03-04 13:25:58.844020512 +0000
+++ /tmp/new-nginx-cfg152306317 2019-03-04 13:26:26.470289369 +0000
@@ -1,5 +1,5 @@

-# Configuration checksum: 6736329955966515302
+# Configuration checksum: 14695381665522160294

 # setup custom paths that do not require root access
 pid /tmp/nginx.pid;
@@ -177049,7 +177049,6 @@
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_tries               3;

-                       rewrite "(?i)/" /admin break;
                        proxy_pass http://upstream_balancer;

                        proxy_redirect                          off;


W0304 13:42:36.448418       8 nginx.go:645] Failed to executing diff command: exit status 1
I0304 13:42:36.448495       8 nginx.go:648] NGINX configuration diff:
--- /etc/nginx/nginx.conf       2019-03-04 13:41:50.324299711 +0000
+++ /tmp/new-nginx-cfg328414343 2019-03-04 13:42:36.397750546 +0000
@@ -1,5 +1,5 @@

-# Configuration checksum: 1899003736800463012
+# Configuration checksum: 16444804011979257649

 # setup custom paths that do not require root access
 pid /tmp/nginx.pid;
@@ -177388,7 +177388,7 @@
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_tries               3;

-                       rewrite "(?i)/" /admin break;
+                       rewrite "(?i)/" /api break;
                        proxy_pass http://upstream_balancer;

                        proxy_redirect                          off;

What you expected to happen: No non dynamic reloads should happen as nothing changed.

How to reproduce it:
I'm not sure, we have a few Ingress resources that use nginx.ingress.kubernetes.io/rewrite-target.
On Ingress is a bit special as the same host has two rewrites (in two Ingress resources). This would also match with the diff from above.

- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /admin
    name: jap-featuretoggle-proxy-service-admin
  spec:
    rules:
    - host: split.kubedev.mobicorp.test
      http:
        paths:
        - backend:
            serviceName: jap-featuretoggle-proxy-service
            servicePort: 3010
          path: /split/admin
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /api
    name: jap-featuretoggle-proxy-service-api
  spec:
    rules:
    - host: split.kubedev.mobicorp.test
      http:
        paths:
        - backend:
            serviceName: jap-featuretoggle-proxy-service
            servicePort: 3000
          path: /split/api

Anything else we need to know:

[aledbf]

@yvespp please check the release notes for 0.22.0 https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.22.0
You need to change the paths with rewrite target.

- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /admin/$1
    name: jap-featuretoggle-proxy-service-admin
  spec:
    rules:
    - host: split.kubedev.mobicorp.test
      http:
        paths:
        - backend:
            serviceName: jap-featuretoggle-proxy-service
            servicePort: 3010
          path: /split/admin/?(.*)
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /api/$1
    name: jap-featuretoggle-proxy-service-api
  spec:
    rules:
    - host: split.kubedev.mobicorp.test
      http:
        paths:
        - backend:
            serviceName: jap-featuretoggle-proxy-service
            servicePort: 3000
          path: /split/api/?(.*)

[yvespp]

Sorry, didn't see that.
I removed all the old configs and I don't have any reloads now, thanks!

[paalkr]

I still encounter constant reloads using 0.24.1 in combination with ingress.kubernetes.io/rewrite-target and ingress.kubernetes.io/configuration-snippet. Our ingress rules are a little special, as all rewrites are handled in the ingress.kubernetes.io/configuration-snippet config.

version 0.21 did not have this issue.

Simple ingress example that will trigger the issue. We have several ingress rules with way more complex configuration snippets, so we basically cannot create our rules without using this approach.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/backend-protocol: HTTPS
    ingress.kubernetes.io/configuration-snippet: |
      rewrite (?i)/arcgis/rest/services/Utilities/Geometry/GeometryServer(.*)$ /arcgis/rest/services/Utilities/Geometry/GeometryServer$1 break;
      rewrite (?i)/arcgis/services/Utilities/Geometry/GeometryServer(.*)$ /arcgis/services/Utilities/Geometry/GeometryServer$1 break;
    ingress.kubernetes.io/proxy-body-size: 128m
    ingress.kubernetes.io/rewrite-target: /$1
    ingress.kubernetes.io/secure-backends: "true"
    ingress.kubernetes.io/service-upstream: "false"
    ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/ingress.class: nginx
  labels:
    name: geometry-togkart-prod
  name: geometry-togkart-prod
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: geometry
          servicePort: agshttps
        path: /arcgis/rest/services/Utilities/Geometry/GeometryServer(.*)
      - backend:
          serviceName: geometry
          servicePort: agshttps
        path: /arcgis/services/Utilities/Geometry/GeometryServer(.*)
  - host: foo2.bar.com
    http:
      paths:
      - backend:
          serviceName: geometry
          servicePort: agshttps
        path: /arcgis/rest/services/Utilities/Geometry/GeometryServer(.*)
      - backend:
          serviceName: geometry
          servicePort: agshttps
        path: /arcgis/services/Utilities/Geometry/GeometryServer(.*)
  tls:
  - hosts:
    - foo.bar.com
    - foo2.bar.com

-v=2 log entry

-			rewrite (?i)/arcgis(.*)$ /arcgis$1 break;
+			rewrite (?i)/arcgis/rest/services/Utilities/Geometry/GeometryServer(.*)$ /arcgis/rest/services/Utilities/Geometry/GeometryServer$1 break;
+			rewrite (?i)/arcgis/services/Utilities/Geometry/GeometryServer(.*)$ /arcgis/services/Utilities/Geometry/GeometryServer$1 break;    
 			
 			rewrite "(?i)/" /$1 break;
 			proxy_pass https://upstream_balancer;

[paalkr]

I looks like using this notation causes a constant reload

    ingress.kubernetes.io/configuration-snippet: |
      rewrite (?i)/arcgis/rest/services/Utilities/Geometry/GeometryServer(.*)$ /arcgis/rest/services/Utilities/Geometry/GeometryServer$1 break;
      rewrite (?i)/arcgis/services/Utilities/Geometry/GeometryServer(.*)$ /arcgis/services/Utilities/Geometry/GeometryServer$1 break;

while this does not (just pass the whole section in as a string, line breaks and white spaces does not matter)

    ingress.kubernetes.io/configuration-snippet: 'rewrite (?i)/arcgis/rest/services/Utilities/Geometry/GeometryServer(.*)$ /arcgis/rest/services/Utilities/Geometry/GeometryServer$1 break; 
												rewrite (?i)/arcgis/services/Utilities/Geometry/GeometryServer(.*)$ /arcgis/services/Utilities/Geometry/GeometryServer$1 break;'

[yoniranjan]

I see this issue on new install. No rules, Just added ingress deployment

I0524 19:16:42.743650 9 controller.go:170] Configuration changes detected, backend reload required.
I0524 19:16:42.746105 9 util.go:74] rlimit.max=1048576
I0524 19:16:42.747194 9 template.go:750] empty byte size, hence it will not be set
W0524 19:16:42.844590 9 nginx.go:673] Failed to executing diff command: exit status 1
I0524 19:16:42.844672 9 nginx.go:676] NGINX configuration diff: