Rewrite annotation breaks default backend paths

[weemel]

NGINX Ingress controller version: Latest (v0.41.2)

Kubernetes version (use kubectl version): 1.17 (but not really relevant)

Environment:

• Cloud provider or hardware configuration: Azure (but not really relevant)
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

Hello, i'm not sure if this is a bug exactly but am not sure where else to raise it. We have ingresses set up for multiple hostnames that do not tend to use the root path. We have been running under the assumption that anything unknown (not defined) will route through to the default backend as stated in the docs, and so we have been using the default backend for health checks under <hostname>/healthz. This has been fine for static paths but we recently started using the rewrite annotation which broke the health check for some hostnames, returning 404 instead of a 200.

What you expected to happen:

Undefined paths that fall through to the default backend not to get re-written.

We noticed that where the rewrite annotation is /$1 for the first ingress in a hostname, we get this config:

server {
    server_name <hostname> ;
    ...

    location ~* "^/" {
      ...
      rewrite "(?i)/" /$1 break;
      ...
    }
}

Since there is no capture group, the path /healthz reaches the default backend but gets rewritten as /, which for our default backend returns a 404 rather than a 200.

The controller sets the root location's rewrite annotation to the annotation of the ingress here - it would be nice for us if it didn't do that, but I'm not sure if we are just using the default backend in a way that is wrong or unintended since that rule has been there for a long time now. I noticed in the PR for that specific line, there was some discussion about how applying the ingress's rewrite/redirect annotations to the default backend could break the paths, but that was for a slightly different use case anyway.

We could manually add the /healthz path but it would be nice not to have to do this so that multiple hostnames could be set up dynamically and independently. I'm curious whether there are use cases where it is useful to have the rewrite annotation applied to the root location, given our situation where there is no capture group so the rewrite annotation can't apply?

Thanks a lot!

How to reproduce it:

Install minikube/kind

• Minikube https://minikube.sigs.k8s.io/docs/start/
• Kind https://kind.sigs.k8s.io/docs/user/quick-start/

Install the ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

Install an application that will act as default backend (is just an echo app)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/http-svc.yaml

Create an ingress with rewrite target and non-root app path

echo "
  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    name: foo-bar
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /$1
  spec:
    rules:
    - host: foo.bar
      http:
        paths:
        - backend:
            serviceName: http-svc
            servicePort: 80
          path: /myapp/?(.*)
" | kubectl apply -f -

Resulting Nginx config:

	## start server foo.bar
	server {
		server_name foo.bar ;
		...
		
		location ~* "^/myapp/?(.*)" {
		  ...
			
			rewrite "(?i)/myapp/?(.*)" /$1 break;
			proxy_pass http://upstream_balancer;
			
			proxy_redirect                          off;
			
		}
		
		location ~* "^/" {
			...
			
			rewrite "(?i)/" /$1 break;
			proxy_pass http://upstream_balancer;
			
			proxy_redirect                          off;
			
		}
		
	}
	## end server foo.bar

/kind bug
/kind support

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mneira10]

Facing this issue ATM. Does anyone have a workaround?

[ph4s3r]

Facing this issue ATM. Does anyone have a workaround?

I created a clusterIP svc for nginx and added a /healthz path to the ingress that resolves to the clusterIP. in AKS without some workaround you cannot get your nginx ingress controller work since if your health check does not respond Azure terminates connections (https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview#probe-down-behavior)