Ingress paths other than root does not work.

[minhptruong2010]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): ingress, nginx, multi path, doesn't work, root works.

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

I've looked into all other similar reports and tried all of the solutions but none works for me.

NGINX Ingress controller version:
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.16.2

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.13", GitCommit:"290fb182489a396dce5d136451388f9b12f29c94", GitTreeState:"clean", BuildDate:"2018-05-15T18:08:47Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  Azure ACS

• OS (e.g. from /etc/os-release):
  PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
  NAME="Debian GNU/Linux"
  VERSION_ID="9"
  VERSION="9 (stretch)"
  ID=debian
  HOME_URL="https://www.debian.org/"
  SUPPORT_URL="https://www.debian.org/support"
  BUG_REPORT_URL="https://bugs.debian.org/"

• Kernel (e.g. uname -a):
  Linux nginx-ingress-controller-758fd9d646-t82vl 4.13.0-1018-azure Refactor default backend handling and add better events #21-Ubuntu SMP Thu May 17 13:58:38 UTC 2018 x86_64 GNU/Linux

• Install tools:

• Others:

What happened:
I have a Kubernetes cluster on Azure ACS and deployed ingress-nginx with the following yaml files. The root path works and successfully routes me to serviceone but any other route does not work.
Making a request to 'External-IP/servicetwo' returns a 404 from the browser not from the default-backend 404.

I run all of the following with kubectl apply -f filename.yml

https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml

kind: Deployment
metadata: 
  name: servicetwo
  namespace: ingress-nginx
spec: 
  replicas: 1
  template: 
    metadata: 
      labels: 
        app: servicetwo
    spec: 
      containers: 
        - name: servicetwo
          image: wregistry.azurecr.io/linux_xorigin:v1
          imagePullPolicy: IfNotPresent
          ports: 
            - containerPort: 80
              name: servicetwo
      imagePullSecrets:
        - name: regcred
---
apiVersion: v1
kind: Service
metadata:
  name: servicetwo
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
  - port: 80
  selector:
    app: servicetwo
---
apiVersion: apps/v1beta1
kind: Deployment
metadata: 
  name: serviceone
  namespace: ingress-nginx
spec: 
  replicas: 1
  template: 
    metadata: 
      labels: 
        app: serviceone
    spec: 
      containers: 
        - name: serviceone
          image: wregistry.azurecr.io/api_xorigin:v1
          imagePullPolicy: IfNotPresent
          ports: 
            - containerPort: 80
              name: serviceone
      imagePullSecrets:
        - name: regcred
---
apiVersion: v1
kind: Service
metadata:
  name: serviceone
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
  - port: 80
  selector:
    app: serviceone
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/ingress.allow-http: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
  namespace: ingress-nginx
spec:
  rules:
  - http:
      paths:
        - path: /
          backend:
            serviceName: serviceone
            servicePort: 80
        - path: /servicetwo
          backend:
            serviceName: servicetwo
            servicePort: 80

I've also tried adding the wildcards on the routes. This causes no paths two work and returns a 404 from the default-backend from for External-IP/ and External-IP/servicetwo.

  rules:
  - http:
      paths:
        - path: /*
          backend:
            serviceName: serviceone
            servicePort: 80
        - path: /servicetwo/*
          backend:
            serviceName: servicetwo
            servicePort: 80

What you expected to happen:
I expected ingress to be able to route the requests to the correct services.

How to reproduce it (as minimally and precisely as possible):
Run an Azure ACS kubernetes cluster. Deploy and expose two services then run these yaml files:
https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/cloud-generic.yaml

next run the following with the correct service names substituted in

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/ingress.allow-http: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
  namespace: ingress-nginx
spec:
  rules:
  - http:
      paths:
        - path: /
          backend:
            serviceName: serviceone
            servicePort: 80
        - path: /servicetwo
          backend:
            serviceName: servicetwo
            servicePort: 80

Anything else we need to know:
there following is the nginx.conf from /etc/nginx/nginx.conf from the pod running my nginx controller

# setup custom paths that do not require root access
pid /tmp/nginx.pid;

daemon off;

worker_processes 2;

worker_rlimit_nofile 523264;

worker_shutdown_timeout 10s ;

events {
        multi_accept        on;
        worker_connections  16384;
        use                 epoll;
}

http {

        lua_package_cpath "/usr/local/lib/lua/?.so;/usr/lib/lua-platform-path/lua/5.1/?.so;;";
        lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;/usr/local/lib/lua/?.lua;;";

        init_by_lua_block {
                require("resty.core")
                collectgarbage("collect")

                local lua_resty_waf = require("resty.waf")
                lua_resty_waf.init()

                ok, res = pcall(require, "monitor")
                if not ok then
                error("require failed: " .. tostring(res))
                else
                monitor = res
                end
        }

        real_ip_header      X-Forwarded-For;

        real_ip_recursive   on;

        set_real_ip_from    0.0.0.0/0;

        geoip_country       /etc/nginx/geoip/GeoIP.dat;
        geoip_city          /etc/nginx/geoip/GeoLiteCity.dat;
        geoip_org           /etc/nginx/geoip/GeoIPASNum.dat;
        geoip_proxy_recursive on;

        aio                 threads;
        aio_write           on;

        tcp_nopush          on;
        tcp_nodelay         on;

        log_subrequest      on;

        reset_timedout_connection on;

        keepalive_timeout  75s;
        keepalive_requests 100;

        client_body_temp_path           /tmp/client-body;
        fastcgi_temp_path               /tmp/fastcgi-temp;
        proxy_temp_path                 /tmp/proxy-temp;

        client_header_buffer_size       1k;
        client_header_timeout           60s;
        large_client_header_buffers     4 8k;
        client_body_buffer_size         8k;
        client_body_timeout             60s;

        http2_max_field_size            4k;
        http2_max_header_size           16k;

        types_hash_max_size             2048;
        server_names_hash_max_size      1024;
        server_names_hash_bucket_size   32;
        map_hash_bucket_size            64;

        proxy_headers_hash_max_size     512;
        proxy_headers_hash_bucket_size  64;

        variables_hash_bucket_size      128;
        variables_hash_max_size         2048;

        underscores_in_headers          off;
        ignore_invalid_headers          on;

        limit_req_status                503;

        include /etc/nginx/mime.types;
        default_type text/html;

        gzip on;
        gzip_comp_level 5;
        gzip_http_version 1.1;
        gzip_min_length 256;
        gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
        gzip_proxied any;
        gzip_vary on;

        # Custom headers for response

        server_tokens on;

        # disable warnings
        uninitialized_variable_warn off;

        # Additional available variables:
        # $namespace
        # $ingress_name
        # $service_name
        # $service_port
        log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id';

        map $request_uri $loggable {

                default 1;
        }

        access_log /var/log/nginx/access.log upstreaminfo if=$loggable;

        error_log  /var/log/nginx/error.log notice;

        resolver 10.0.0.10 valid=30s;

        # Retain the default nginx handling of requests without a "Connection" header
        map $http_upgrade $connection_upgrade {
                default          upgrade;
                ''               close;
        }

        map $http_x_forwarded_for $the_real_ip {

                default          $remote_addr;

        }

        # trust http_x_forwarded_proto headers correctly indicate ssl offloading
        map $http_x_forwarded_proto $pass_access_scheme {
                default          $http_x_forwarded_proto;
                ''               $scheme;
        }

        # validate $pass_access_scheme and $scheme are http to force a redirect
        map "$scheme:$pass_access_scheme" $redirect_to_https {
                default          0;
                "http:http"      1;
                "https:http"     1;
        }

        map $http_x_forwarded_port $pass_server_port {
                default           $http_x_forwarded_port;
                ''                $server_port;
        }

        map $pass_server_port $pass_port {
                443              443;
                default          $pass_server_port;
        }

        # Obtain best http host
        map $http_host $this_host {
                default          $http_host;
                ''               $host;
        }

        map $http_x_forwarded_host $best_http_host {
                default          $http_x_forwarded_host;
                ''               $this_host;
        }

        # Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server.
        # If no such header is provided, it can provide a random value.
        map $http_x_request_id $req_id {
                default   $http_x_request_id;

                ""        $request_id;

        }

        server_name_in_redirect off;
        port_in_redirect        off;

        ssl_protocols TLSv1.2;

        # turn on session caching to drastically improve performance

        ssl_session_cache builtin:1000 shared:SSL:10m;
        ssl_session_timeout 10m;

        # allow configuring ssl session tickets
        ssl_session_tickets on;

        # slightly reduce the time-to-first-byte
        ssl_buffer_size 4k;

        # allow configuring custom ssl ciphers
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;

        ssl_ecdh_curve auto;

        proxy_ssl_session_reuse on;

        upstream ingress-nginx-serviceone-80 {
                least_conn;

                keepalive 32;

                server 10.244.4.152:80 max_fails=0 fail_timeout=0;

        }

        upstream ingress-nginx-servicetwo-80 {
                least_conn;

                keepalive 32;

                server 10.244.4.151:80 max_fails=0 fail_timeout=0;

        }

        upstream upstream-default-backend {
                least_conn;

                keepalive 32;

                server 10.244.1.29:8080 max_fails=0 fail_timeout=0;

        }

        ## start server _
        server {
                server_name _ ;

                listen 80 default_server  backlog=511;

                listen [::]:80 default_server  backlog=511;

                set $proxy_upstream_name "-";

                listen 443  default_server  backlog=511 ssl http2;

                listen [::]:443  default_server  backlog=511 ssl http2;

                # PEM sha: 871c97e097749986fb361505a184cefb35a4e8ab
                ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
                ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;

                location /servicetwo {

                        set $namespace      "ingress-nginx";
                        set $ingress_name   "my-ingress";
                        set $service_name   "servicetwo";
                        set $service_port   "80";
                        set $location_path  "/servicetwo";

                        rewrite_by_lua_block {

                        }

                        log_by_lua_block {

                                monitor.call()
                        }

                        if ($scheme = https) {
                                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
                        }

                        port_in_redirect off;

                        set $proxy_upstream_name "ingress-nginx-servicetwo-80";

                        # Cors Preflight methods needs additional options and different Return Code
                        if ($request_method = 'OPTIONS') {
                                more_set_headers 'Access-Control-Allow-Origin: *';
                                more_set_headers 'Access-Control-Allow-Credentials: true';
                                more_set_headers 'Access-Control-Allow-Methods: PUT, GET, POST, OPTIONS';
                                more_set_headers 'Access-Control-Allow-Headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';
                                more_set_headers 'Access-Control-Max-Age: 1728000';
                                more_set_headers 'Content-Type: text/plain charset=UTF-8';
                                more_set_headers 'Content-Length: 0';
                                return 204;
                        }

                        more_set_headers 'Access-Control-Allow-Origin: *';
                        more_set_headers 'Access-Control-Allow-Credentials: true';
                        more_set_headers 'Access-Control-Allow-Methods: PUT, GET, POST, OPTIONS';
                        more_set_headers 'Access-Control-Allow-Headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';

                        client_max_body_size                    "1m";

                        proxy_set_header Host                   $best_http_host;

                        # Pass the extracted client certificate to the backend

                        # Allow websocket connections
                        proxy_set_header                        Upgrade           $http_upgrade;

                        proxy_set_header                        Connection        $connection_upgrade;

                        proxy_set_header X-Request-ID           $req_id;
                        proxy_set_header X-Real-IP              $the_real_ip;

                        proxy_set_header X-Forwarded-For        $the_real_ip;

                        proxy_set_header X-Forwarded-Host       $best_http_host;
                        proxy_set_header X-Forwarded-Port       $pass_port;
                        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

                        proxy_set_header X-Original-URI         $request_uri;

                        proxy_set_header X-Scheme               $pass_access_scheme;

                        # Pass the original X-Forwarded-For
                        proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

                        # mitigate HTTPoxy Vulnerability
                        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                        proxy_set_header Proxy                  "";

                        # Custom headers to proxied server

                        proxy_connect_timeout                   5s;
                        proxy_send_timeout                      60s;
                        proxy_read_timeout                      60s;

                        proxy_buffering                         "off";
                        proxy_buffer_size                       "4k";
                        proxy_buffers                           4 "4k";
                        proxy_request_buffering                 "on";

                        proxy_http_version                      1.1;

                        proxy_cookie_domain                     off;
                        proxy_cookie_path                       off;

                        # In case of errors try the next upstream server before returning an error
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_tries               3;

                        proxy_pass http://ingress-nginx-servicetwo-80;

                        proxy_redirect                          off;

                }

                location / {

                        set $namespace      "ingress-nginx";
                        set $ingress_name   "my-ingress";
                        set $service_name   "serviceone";
                        set $service_port   "80";
                        set $location_path  "/";

                        rewrite_by_lua_block {

                        }

                        log_by_lua_block {

                                monitor.call()
                        }

                        if ($scheme = https) {
                                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
                        }

                        port_in_redirect off;

                        set $proxy_upstream_name "ingress-nginx-serviceone-80";

                        # Cors Preflight methods needs additional options and different Return Code
                        if ($request_method = 'OPTIONS') {
                                more_set_headers 'Access-Control-Allow-Origin: *';
                                more_set_headers 'Access-Control-Allow-Credentials: true';
                                more_set_headers 'Access-Control-Allow-Methods: PUT, GET, POST, OPTIONS';
                                more_set_headers 'Access-Control-Allow-Headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';
                                more_set_headers 'Access-Control-Max-Age: 1728000';
                                more_set_headers 'Content-Type: text/plain charset=UTF-8';
                                more_set_headers 'Content-Length: 0';
                                return 204;
                        }

                        more_set_headers 'Access-Control-Allow-Origin: *';
                        more_set_headers 'Access-Control-Allow-Credentials: true';
                        more_set_headers 'Access-Control-Allow-Methods: PUT, GET, POST, OPTIONS';
                        more_set_headers 'Access-Control-Allow-Headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';

                        client_max_body_size                    "1m";

                        proxy_set_header Host                   $best_http_host;

                        # Pass the extracted client certificate to the backend

                        # Allow websocket connections
                        proxy_set_header                        Upgrade           $http_upgrade;

                        proxy_set_header                        Connection        $connection_upgrade;

                        proxy_set_header X-Request-ID           $req_id;
                        proxy_set_header X-Real-IP              $the_real_ip;

                        proxy_set_header X-Forwarded-For        $the_real_ip;

                        proxy_set_header X-Forwarded-Host       $best_http_host;
                        proxy_set_header X-Forwarded-Port       $pass_port;
                        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

                        proxy_set_header X-Original-URI         $request_uri;

                        proxy_set_header X-Scheme               $pass_access_scheme;

                        # Pass the original X-Forwarded-For
                        proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

                        # mitigate HTTPoxy Vulnerability
                        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                        proxy_set_header Proxy                  "";

                        # Custom headers to proxied server

                        proxy_connect_timeout                   5s;
                        proxy_send_timeout                      60s;
                        proxy_read_timeout                      60s;

                        proxy_buffering                         "off";
                        proxy_buffer_size                       "4k";
                        proxy_buffers                           4 "4k";
                        proxy_request_buffering                 "on";

                        proxy_http_version                      1.1;

                        proxy_cookie_domain                     off;
                        proxy_cookie_path                       off;

                        # In case of errors try the next upstream server before returning an error
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_tries               3;

                        proxy_pass http://ingress-nginx-serviceone-80;

                        proxy_redirect                          off;

                }

                # health checks in cloud providers require the use of port 80
                location /healthz {

                        access_log off;
                        return 200;
                }

                # this is required to avoid error if nginx is being monitored
                # with an external software (like sysdig)
                location /nginx_status {

                        allow 127.0.0.1;

                        allow ::1;

                        deny all;

                        access_log off;
                        stub_status on;
                }

        }
        ## end server _

        # default server, used for NGINX healthcheck and access to nginx stats
        server {
                # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
                # Changing this value requires a change in:
                # https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
                listen 18080 default_server  backlog=511;
                listen [::]:18080 default_server  backlog=511;
                set $proxy_upstream_name "-";

                location /healthz {

                        access_log off;
                        return 200;
                }

                location /is-dynamic-lb-initialized {

                        access_log off;

                        content_by_lua_block {
                                local configuration = require("configuration")
                                local backend_data = configuration.get_backends_data()
                                if not backend_data then
                                ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
                                return
                                end

                                ngx.say("OK")
                                ngx.exit(ngx.HTTP_OK)
                        }
                }

                location /nginx_status {
                        set $proxy_upstream_name "internal";

                        access_log off;
                        stub_status on;
                }

                location / {

                        set $proxy_upstream_name "upstream-default-backend";

                        proxy_pass          http://upstream-default-backend;

                }

        }
}

stream {
        log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

        access_log /var/log/nginx/access.log log_stream;

        error_log  /var/log/nginx/error.log;

        # TCP services

        # UDP services

}

[aledbf]

@minhptruong this annotation is not supported kubernetes.io/ingress.allow-http: "true", this one only works if you have a TLS section in the ingress nginx.ingress.kubernetes.io/ssl-redirect: "false" and this one should have nginx as the prefix ingress.kubernetes.io/rewrite-target: /

[aledbf]

Closing. Please reopen if the issue persists after updating the annotations

[gouravbatra]

Still Not working.

My Ingress File:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-srv
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: auth.domain.com
http:
paths:
- path: /*
backend:
serviceName: auth-srv
servicePort: 3000
- host: api.domain.com
http:
paths:
- path: /feature1/*
backend:
serviceName: feature1-srv
servicePort: 3001

My Express Js Router Code:

router.get('/admin/getdata', async (req: Request, res: Response) => {
res.status(201).send({});
});

My Postman Request:
Method: Get
Url http://api.domain.com/feature1/admin/getdata

I am getting route not found error