Add CORS template check inside location for externalAuth.SignURL #8814

harry1064 · 2022-07-12T11:56:01Z

What this PR does / why we need it:

This PR add the CORS template while handling redirect location for externalAuth.SignURL.
Similar to add_header Set-Cookie $auth_cookie; need to again set inside the location block handling redirect for externalAuth.SignURL, we have added the CORS template again.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation only

Which issue/s this PR fixes

fixes #8786

How Has This Been Tested?

I've tested locally by setting up cluster using Kind and following instruction from the Issue reporter. After making this change, provided CORS related annotation are provide and as well nginx.ingress.kubernetes.io/auth-signin, response header contains the CORS related headers.

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.
I've read the CONTRIBUTION guide
I have added tests to cover my changes.
All new and existing tests passed.

k8s-ci-robot · 2022-07-12T11:56:08Z

@harry1064: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2022-07-12T11:56:09Z

Welcome @harry1064!

It looks like this is your first PR to kubernetes/ingress-nginx 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/ingress-nginx has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2022-07-12T11:56:09Z

Hi @harry1064. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

longwuyuan · 2022-07-12T12:03:14Z

Hi @harry1064 ,

Personally, I would like to see the info I list below before commenting. Any chance you can add this info ;

kubectl get all,ing -A -o wide
kubectl describe ing -A
kubectl describe svc
kubectl -n exec -- cat /etc/nginx/nginx.conf
Your complete and exact curl command with -v and other flags to show the headers
kubectl -n logs
Any other related info

harry1064 · 2022-07-12T13:11:32Z

Hi @longwuyuan
I have added the relevant information as requested.

kubectl get all,ing -A -o wide

NAMESPACE            NAME                                                          READY   STATUS      RESTARTS   AGE   IP            NODE                              NOMINATED NODE   READINESS GATES
default              pod/auth-service-7c844c8f64-4qnzs                             1/1     Running     0          8h    10.244.0.9    ingress-nginx-dev-control-plane   <none>           <none>
default              pod/protected-service-64c969cdd9-qsz95                        1/1     Running     0          8h    10.244.0.10   ingress-nginx-dev-control-plane   <none>           <none>
default              pod/public-service-c764b7d4d-f24mj                            1/1     Running     0          8h    10.244.0.8    ingress-nginx-dev-control-plane   <none>           <none>
ingress-nginx        pod/ingress-nginx-admission-create-5bx76                      0/1     Completed   0          22h   10.244.0.5    ingress-nginx-dev-control-plane   <none>           <none>
ingress-nginx        pod/ingress-nginx-admission-patch-m9qqf                       0/1     Completed   0          22h   10.244.0.6    ingress-nginx-dev-control-plane   <none>           <none>
ingress-nginx        pod/ingress-nginx-controller-5689d5bcd6-sq957                 1/1     Running     0          22h   10.244.0.7    ingress-nginx-dev-control-plane   <none>           <none>
kube-system          pod/coredns-558bd4d5db-hmlrn                                  1/1     Running     0          23h   10.244.0.4    ingress-nginx-dev-control-plane   <none>           <none>
kube-system          pod/coredns-558bd4d5db-v2hpv                                  1/1     Running     0          23h   10.244.0.3    ingress-nginx-dev-control-plane   <none>           <none>
kube-system          pod/etcd-ingress-nginx-dev-control-plane                      1/1     Running     0          23h   172.18.0.2    ingress-nginx-dev-control-plane   <none>           <none>
kube-system          pod/kindnet-nrqlb                                             1/1     Running     0          23h   172.18.0.2    ingress-nginx-dev-control-plane   <none>           <none>
kube-system          pod/kube-apiserver-ingress-nginx-dev-control-plane            1/1     Running     0          23h   172.18.0.2    ingress-nginx-dev-control-plane   <none>           <none>
kube-system          pod/kube-controller-manager-ingress-nginx-dev-control-plane   1/1     Running     1          23h   172.18.0.2    ingress-nginx-dev-control-plane   <none>           <none>
kube-system          pod/kube-proxy-wblfl                                          1/1     Running     0          23h   172.18.0.2    ingress-nginx-dev-control-plane   <none>           <none>
kube-system          pod/kube-scheduler-ingress-nginx-dev-control-plane            1/1     Running     1          23h   172.18.0.2    ingress-nginx-dev-control-plane   <none>           <none>
local-path-storage   pod/local-path-provisioner-6c9449b9dd-c7zzl                   1/1     Running     0          23h   10.244.0.2    ingress-nginx-dev-control-plane   <none>           <none>

NAMESPACE       NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
default         service/auth-service                         LoadBalancer   10.96.72.241    <pending>     8080:31512/TCP               8h    app=auth-service
default         service/kubernetes                           ClusterIP      10.96.0.1       <none>        443/TCP                      23h   <none>
default         service/protected-service                    LoadBalancer   10.96.225.22    <pending>     8080:30001/TCP               8h    app=protected-service
default         service/public-service                       LoadBalancer   10.96.186.164   <pending>     8080:32733/TCP               8h    app=public-service
ingress-nginx   service/ingress-nginx-controller             NodePort       10.96.6.142     <none>        80:31421/TCP,443:30521/TCP   22h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.96.30.163    <none>        443/TCP                      22h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     service/kube-dns                             ClusterIP      10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP       23h   k8s-app=kube-dns

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS    IMAGES                                          SELECTOR
kube-system   daemonset.apps/kindnet      1         1         1       1            1           <none>                   23h   kindnet-cni   docker.io/kindest/kindnetd:v20220510-4929dd75   app=kindnet
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   23h   kube-proxy    k8s.gcr.io/kube-proxy:v1.21.12                  k8s-app=kube-proxy

NAMESPACE            NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS               IMAGES                                                    SELECTOR
default              deployment.apps/auth-service               1/1     1            1           8h    auth-service             xvnpw/auth-service:0.0.4                                  app=auth-service
default              deployment.apps/protected-service          1/1     1            1           8h    protected-service        xvnpw/protected-service:0.0.1                             app=protected-service
default              deployment.apps/public-service             1/1     1            1           8h    public-service           xvnpw/public-service:0.0.1                                app=public-service
ingress-nginx        deployment.apps/ingress-nginx-controller   1/1     1            1           22h   controller               gcr.io/k8s-staging-ingress-nginx/controller:1.0.0-dev     app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system          deployment.apps/coredns                    2/2     2            2           23h   coredns                  k8s.gcr.io/coredns/coredns:v1.8.0                         k8s-app=kube-dns
local-path-storage   deployment.apps/local-path-provisioner     1/1     1            1           23h   local-path-provisioner   docker.io/kindest/local-path-provisioner:v0.0.22-kind.0   app=local-path-provisioner

NAMESPACE            NAME                                                  DESIRED   CURRENT   READY   AGE   CONTAINERS               IMAGES                                                    SELECTOR
default              replicaset.apps/auth-service-7c844c8f64               1         1         1       8h    auth-service             xvnpw/auth-service:0.0.4                                  app=auth-service,pod-template-hash=7c844c8f64
default              replicaset.apps/protected-service-64c969cdd9          1         1         1       8h    protected-service        xvnpw/protected-service:0.0.1                             app=protected-service,pod-template-hash=64c969cdd9
default              replicaset.apps/public-service-c764b7d4d              1         1         1       8h    public-service           xvnpw/public-service:0.0.1                                app=public-service,pod-template-hash=c764b7d4d
ingress-nginx        replicaset.apps/ingress-nginx-controller-5689d5bcd6   1         1         1       22h   controller               gcr.io/k8s-staging-ingress-nginx/controller:1.0.0-dev     app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5689d5bcd6
kube-system          replicaset.apps/coredns-558bd4d5db                    2         2         2       23h   coredns                  k8s.gcr.io/coredns/coredns:v1.8.0                         k8s-app=kube-dns,pod-template-hash=558bd4d5db
local-path-storage   replicaset.apps/local-path-provisioner-6c9449b9dd     1         1         1       23h   local-path-provisioner   docker.io/kindest/local-path-provisioner:v0.0.22-kind.0   app=local-path-provisioner,pod-template-hash=6c9449b9dd

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE   CONTAINERS   IMAGES                                                                                                                              SELECTOR
ingress-nginx   job.batch/ingress-nginx-admission-create   1/1           26s        22h   create       registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660   controller-uid=f1fb3741-fb51-4082-805f-8c3ff82997de
ingress-nginx   job.batch/ingress-nginx-admission-patch    1/1           27s        22h   patch        registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660   controller-uid=a51a51c9-e6c8-49af-8f02-5f79269c9834

NAMESPACE   NAME                                   CLASS    HOSTS      ADDRESS       PORTS   AGE
default     ingress.networking.k8s.io/my-ingress   <none>   app.test   10.96.6.142   80      8h

kubectl describe ing -A

Name:             my-ingress
Labels:           <none>
Namespace:        default
Address:          10.96.6.142
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  app.test
              /public-service/(.*)      public-service:8080 (10.244.0.8:5000)
              /protected-service/(.*)   protected-service:8080 (10.244.0.10:5000)
Annotations:  kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/auth-signin: https://api-auth.xyz.com/oauth2/start?rd=$scheme://$best_http_host$request_uri
              nginx.ingress.kubernetes.io/auth-url: http://auth-service.default.svc.cluster.local:8080/verify
              nginx.ingress.kubernetes.io/cors-allow-origin: https://allowed.origin.com
              nginx.ingress.kubernetes.io/enable-cors: true
              nginx.ingress.kubernetes.io/rewrite-target: /$1
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    2m9s (x6 over 7h53m)  nginx-ingress-controller  Scheduled for sync

kubectl describe svc

Name:                     auth-service
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=auth-service
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.72.241
IPs:                      10.96.72.241
Port:                     <unset>  8080/TCP
TargetPort:               5000/TCP
NodePort:                 <unset>  31512/TCP
Endpoints:                10.244.0.9:5000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.0.1
IPs:               10.96.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         172.18.0.2:6443
Session Affinity:  None
Events:            <none>


Name:                     protected-service
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=protected-service
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.225.22
IPs:                      10.96.225.22
Port:                     <unset>  8080/TCP
TargetPort:               5000/TCP
NodePort:                 <unset>  30001/TCP
Endpoints:                10.244.0.10:5000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


Name:                     public-service
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=public-service
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.186.164
IPs:                      10.96.186.164
Port:                     <unset>  8080/TCP
TargetPort:               5000/TCP
NodePort:                 <unset>  32733/TCP
Endpoints:                10.244.0.8:5000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

kubectl exec -it ingress-nginx-controller-5689d5bcd6-sq957 -n ingress-nginx -- cat nginx.conf


			set $namespace      "";
			set $ingress_name   "";
			set $service_name   "";
			set $service_port   "";
			set $location_path  "";
			set $global_rate_limit_exceeding n;

			rewrite_by_lua_block {
				lua_ingress.rewrite({
					force_ssl_redirect = false,
					ssl_redirect = false,
					force_no_ssl_redirect = false,
					preserve_trailing_slash = false,
					use_port_in_redirects = false,
					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
				})
				balancer.rewrite()
				plugins.run()
			}

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
			# other authentication method such as basic auth or external auth useless - all requests will be allowed.
			#access_by_lua_block {
			#}

			header_filter_by_lua_block {
				lua_ingress.header()
				plugins.run()
			}

			body_filter_by_lua_block {
				plugins.run()
			}

			log_by_lua_block {
				balancer.log()

				monitor.call()

				plugins.run()
			}

			access_log off;

			port_in_redirect off;

			set $balancer_ewma_score -1;
			set $proxy_upstream_name "upstream-default-backend";
			set $proxy_host          $proxy_upstream_name;
			set $pass_access_scheme  $scheme;

			set $pass_server_port    $server_port;

			set $best_http_host      $http_host;
			set $pass_port           $pass_server_port;

			set $proxy_alternative_upstream_name "";

			client_max_body_size                    1m;

			proxy_set_header Host                   $best_http_host;

			# Pass the extracted client certificate to the backend

			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;

			proxy_set_header                        Connection        $connection_upgrade;

			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $remote_addr;

			proxy_set_header X-Forwarded-For        $remote_addr;

			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			proxy_set_header X-Scheme               $pass_access_scheme;

			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";

			# Custom headers to proxied server

			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;

			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;

			proxy_max_temp_file_size                1024m;

			proxy_request_buffering                 on;
			proxy_http_version                      1.1;

			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;

			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_timeout             0;
			proxy_next_upstream_tries               3;

			proxy_pass http://upstream_balancer;

			proxy_redirect                          off;

		}

		# health checks in cloud providers require the use of port 80
		location /healthz {

			access_log off;
			return 200;
		}

		# this is required to avoid error if nginx is being monitored
		# with an external software (like sysdig)
		location /nginx_status {

			allow 127.0.0.1;

			allow ::1;

			deny all;

			access_log off;
			stub_status on;
		}

	}
	## end server _

	## start server app.test
	server {
		server_name app.test ;

		listen 80  ;
		listen [::]:80  ;
		listen 443  ssl http2 ;
		listen [::]:443  ssl http2 ;

		set $proxy_upstream_name "-";

		ssl_certificate_by_lua_block {
			certificate.call()
		}

		location = /_external-auth-L3Byb3RlY3RlZC1zZXJ2aWNlLyguKik-Prefix {
			internal;

			# Ensure that modsecurity will not run on an internal location as this is not accessible from outside

			# ngx_auth_request module overrides variables in the parent request,
			# therefore we have to explicitly set this variable again so that when the parent request
			# resumes it has the correct value set for this variable so that Lua can pick backend correctly
			set $proxy_upstream_name "default-protected-service-8080";

			proxy_pass_request_body     off;
			proxy_set_header            Content-Length          "";
			proxy_set_header            X-Forwarded-Proto       "";
			proxy_set_header            X-Request-ID            $req_id;

			proxy_set_header            Host                    auth-service.default.svc.cluster.local;
			proxy_set_header            X-Original-URL          $scheme://$http_host$request_uri;
			proxy_set_header            X-Original-Method       $request_method;
			proxy_set_header            X-Sent-From             "nginx-ingress-controller";
			proxy_set_header            X-Real-IP               $remote_addr;

			proxy_set_header            X-Forwarded-For        $remote_addr;

			proxy_set_header            X-Auth-Request-Redirect $request_uri;

			proxy_buffering                         off;

			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;
			proxy_request_buffering                 on;

			proxy_ssl_server_name       on;
			proxy_pass_request_headers  on;

			client_max_body_size        1m;

			# Pass the extracted client certificate to the auth provider

			proxy_http_version 1.1;
			set $target http://auth-service.default.svc.cluster.local:8080/verify;

			proxy_pass $target;
		}

		location @8638240922c3aa074994ed6194c30efb21bab86e {
			internal;

			# Cors Preflight methods needs additional options and different Return Code

			if ($http_origin ~* ((https://allowed\.origin\.com))$ ) { set $cors 'true'; }

			if ($request_method = 'OPTIONS') {
				set $cors ${cors}options;
			}

			if ($cors = "true") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
			}

			if ($cors = "trueoptions") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
				more_set_headers 'Content-Type: text/plain charset=UTF-8';
				more_set_headers 'Content-Length: 0';
				return 204;
			}

			add_header Set-Cookie $auth_cookie;

			# Ensure that modsecurity will not run on an internal location as this is not accessible from outside

			return 302 https://api-auth.xyz.com/oauth2/start?rd=$scheme://$best_http_host$request_uri;
		}

		location ~* "^/protected-service/(.*)" {

			set $namespace      "default";
			set $ingress_name   "my-ingress";
			set $service_name   "protected-service";
			set $service_port   "8080";
			set $location_path  "/protected-service/(.*)";
			set $global_rate_limit_exceeding n;

			rewrite_by_lua_block {
				lua_ingress.rewrite({
					force_ssl_redirect = false,
					ssl_redirect = true,
					force_no_ssl_redirect = false,
					preserve_trailing_slash = false,
					use_port_in_redirects = false,
					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
				})
				balancer.rewrite()
				plugins.run()
			}

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
			# other authentication method such as basic auth or external auth useless - all requests will be allowed.
			#access_by_lua_block {
			#}

			header_filter_by_lua_block {
				lua_ingress.header()
				plugins.run()
			}

			body_filter_by_lua_block {
				plugins.run()
			}

			log_by_lua_block {
				balancer.log()

				monitor.call()

				plugins.run()
			}

			port_in_redirect off;

			set $balancer_ewma_score -1;
			set $proxy_upstream_name "default-protected-service-8080";
			set $proxy_host          $proxy_upstream_name;
			set $pass_access_scheme  $scheme;

			set $pass_server_port    $server_port;

			set $best_http_host      $http_host;
			set $pass_port           $pass_server_port;

			set $proxy_alternative_upstream_name "";

			# this location requires authentication

			auth_request        /_external-auth-L3Byb3RlY3RlZC1zZXJ2aWNlLyguKik-Prefix;
			auth_request_set    $auth_cookie $upstream_http_set_cookie;

			add_header          Set-Cookie $auth_cookie;

			set_escape_uri $escaped_request_uri $request_uri;
			error_page 401 = @8638240922c3aa074994ed6194c30efb21bab86e;

			# Cors Preflight methods needs additional options and different Return Code

			if ($http_origin ~* ((https://allowed\.origin\.com))$ ) { set $cors 'true'; }

			if ($request_method = 'OPTIONS') {
				set $cors ${cors}options;
			}

			if ($cors = "true") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
			}

			if ($cors = "trueoptions") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
				more_set_headers 'Content-Type: text/plain charset=UTF-8';
				more_set_headers 'Content-Length: 0';
				return 204;
			}

			client_max_body_size                    1m;

			proxy_set_header Host                   $best_http_host;

			# Pass the extracted client certificate to the backend

			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;

			proxy_set_header                        Connection        $connection_upgrade;

			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $remote_addr;

			proxy_set_header X-Forwarded-For        $remote_addr;

			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			proxy_set_header X-Scheme               $pass_access_scheme;

			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";

			# Custom headers to proxied server

			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;

			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;

			proxy_max_temp_file_size                1024m;

			proxy_request_buffering                 on;
			proxy_http_version                      1.1;

			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;

			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_timeout             0;
			proxy_next_upstream_tries               3;

			rewrite "(?i)/protected-service/(.*)" /$1 break;
			proxy_pass http://upstream_balancer;

			proxy_redirect                          off;

		}

		location = /_external-auth-L3B1YmxpYy1zZXJ2aWNlLyguKik-Prefix {
			internal;

			# Ensure that modsecurity will not run on an internal location as this is not accessible from outside

			# ngx_auth_request module overrides variables in the parent request,
			# therefore we have to explicitly set this variable again so that when the parent request
			# resumes it has the correct value set for this variable so that Lua can pick backend correctly
			set $proxy_upstream_name "default-public-service-8080";

			proxy_pass_request_body     off;
			proxy_set_header            Content-Length          "";
			proxy_set_header            X-Forwarded-Proto       "";
			proxy_set_header            X-Request-ID            $req_id;

			proxy_set_header            Host                    auth-service.default.svc.cluster.local;
			proxy_set_header            X-Original-URL          $scheme://$http_host$request_uri;
			proxy_set_header            X-Original-Method       $request_method;
			proxy_set_header            X-Sent-From             "nginx-ingress-controller";
			proxy_set_header            X-Real-IP               $remote_addr;

			proxy_set_header            X-Forwarded-For        $remote_addr;

			proxy_set_header            X-Auth-Request-Redirect $request_uri;

			proxy_buffering                         off;

			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;
			proxy_request_buffering                 on;

			proxy_ssl_server_name       on;
			proxy_pass_request_headers  on;

			client_max_body_size        1m;

			# Pass the extracted client certificate to the auth provider

			proxy_http_version 1.1;
			set $target http://auth-service.default.svc.cluster.local:8080/verify;

			proxy_pass $target;
		}

		location @e884a17a0753be22d71e9d6bcafec6d2566b9f02 {
			internal;

			# Cors Preflight methods needs additional options and different Return Code

			if ($http_origin ~* ((https://allowed\.origin\.com))$ ) { set $cors 'true'; }

			if ($request_method = 'OPTIONS') {
				set $cors ${cors}options;
			}

			if ($cors = "true") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
			}

			if ($cors = "trueoptions") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
				more_set_headers 'Content-Type: text/plain charset=UTF-8';
				more_set_headers 'Content-Length: 0';
				return 204;
			}

			add_header Set-Cookie $auth_cookie;

			# Ensure that modsecurity will not run on an internal location as this is not accessible from outside

			return 302 https://api-auth.xyz.com/oauth2/start?rd=$scheme://$best_http_host$request_uri;
		}

		location ~* "^/public-service/(.*)" {

			set $namespace      "default";
			set $ingress_name   "my-ingress";
			set $service_name   "public-service";
			set $service_port   "8080";
			set $location_path  "/public-service/(.*)";
			set $global_rate_limit_exceeding n;

			rewrite_by_lua_block {
				lua_ingress.rewrite({
					force_ssl_redirect = false,
					ssl_redirect = true,
					force_no_ssl_redirect = false,
					preserve_trailing_slash = false,
					use_port_in_redirects = false,
					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
				})
				balancer.rewrite()
				plugins.run()
			}

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
			# other authentication method such as basic auth or external auth useless - all requests will be allowed.
			#access_by_lua_block {
			#}

			header_filter_by_lua_block {
				lua_ingress.header()
				plugins.run()
			}

			body_filter_by_lua_block {
				plugins.run()
			}

			log_by_lua_block {
				balancer.log()

				monitor.call()

				plugins.run()
			}

			port_in_redirect off;

			set $balancer_ewma_score -1;
			set $proxy_upstream_name "default-public-service-8080";
			set $proxy_host          $proxy_upstream_name;
			set $pass_access_scheme  $scheme;

			set $pass_server_port    $server_port;

			set $best_http_host      $http_host;
			set $pass_port           $pass_server_port;

			set $proxy_alternative_upstream_name "";

			# this location requires authentication

			auth_request        /_external-auth-L3B1YmxpYy1zZXJ2aWNlLyguKik-Prefix;
			auth_request_set    $auth_cookie $upstream_http_set_cookie;

			add_header          Set-Cookie $auth_cookie;

			set_escape_uri $escaped_request_uri $request_uri;
			error_page 401 = @e884a17a0753be22d71e9d6bcafec6d2566b9f02;

			# Cors Preflight methods needs additional options and different Return Code

			if ($http_origin ~* ((https://allowed\.origin\.com))$ ) { set $cors 'true'; }

			if ($request_method = 'OPTIONS') {
				set $cors ${cors}options;
			}

			if ($cors = "true") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
			}

			if ($cors = "trueoptions") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
				more_set_headers 'Content-Type: text/plain charset=UTF-8';
				more_set_headers 'Content-Length: 0';
				return 204;
			}

			client_max_body_size                    1m;

			proxy_set_header Host                   $best_http_host;

			# Pass the extracted client certificate to the backend

			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;

			proxy_set_header                        Connection        $connection_upgrade;

			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $remote_addr;

			proxy_set_header X-Forwarded-For        $remote_addr;

			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			proxy_set_header X-Scheme               $pass_access_scheme;

			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";

			# Custom headers to proxied server

			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;

			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;

			proxy_max_temp_file_size                1024m;

			proxy_request_buffering                 on;
			proxy_http_version                      1.1;

			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;

			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_timeout             0;
			proxy_next_upstream_tries               3;

			rewrite "(?i)/public-service/(.*)" /$1 break;
			proxy_pass http://upstream_balancer;

			proxy_redirect                          off;

		}

		location = /_external-auth-Lw-Prefix {
			internal;

			# Ensure that modsecurity will not run on an internal location as this is not accessible from outside

			# ngx_auth_request module overrides variables in the parent request,
			# therefore we have to explicitly set this variable again so that when the parent request
			# resumes it has the correct value set for this variable so that Lua can pick backend correctly
			set $proxy_upstream_name "upstream-default-backend";

			proxy_pass_request_body     off;
			proxy_set_header            Content-Length          "";
			proxy_set_header            X-Forwarded-Proto       "";
			proxy_set_header            X-Request-ID            $req_id;

			proxy_set_header            Host                    auth-service.default.svc.cluster.local;
			proxy_set_header            X-Original-URL          $scheme://$http_host$request_uri;
			proxy_set_header            X-Original-Method       $request_method;
			proxy_set_header            X-Sent-From             "nginx-ingress-controller";
			proxy_set_header            X-Real-IP               $remote_addr;

			proxy_set_header            X-Forwarded-For        $remote_addr;

			proxy_set_header            X-Auth-Request-Redirect $request_uri;

			proxy_buffering                         off;

			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;
			proxy_request_buffering                 on;

			proxy_ssl_server_name       on;
			proxy_pass_request_headers  on;

			client_max_body_size        1m;

			# Pass the extracted client certificate to the auth provider

			proxy_http_version 1.1;
			set $target http://auth-service.default.svc.cluster.local:8080/verify;

			proxy_pass $target;
		}

		location @2bb6bc255b0c4bfd57a8f6f523d918622a087012 {
			internal;

			# Cors Preflight methods needs additional options and different Return Code

			if ($http_origin ~* ((https://allowed\.origin\.com))$ ) { set $cors 'true'; }

			if ($request_method = 'OPTIONS') {
				set $cors ${cors}options;
			}

			if ($cors = "true") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
			}

			if ($cors = "trueoptions") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
				more_set_headers 'Content-Type: text/plain charset=UTF-8';
				more_set_headers 'Content-Length: 0';
				return 204;
			}

			add_header Set-Cookie $auth_cookie;

			# Ensure that modsecurity will not run on an internal location as this is not accessible from outside

			return 302 https://api-auth.xyz.com/oauth2/start?rd=$scheme://$best_http_host$request_uri;
		}

		location ~* "^/" {

			set $namespace      "default";
			set $ingress_name   "my-ingress";
			set $service_name   "";
			set $service_port   "";
			set $location_path  "/";
			set $global_rate_limit_exceeding n;

			rewrite_by_lua_block {
				lua_ingress.rewrite({
					force_ssl_redirect = false,
					ssl_redirect = true,
					force_no_ssl_redirect = false,
					preserve_trailing_slash = false,
					use_port_in_redirects = false,
					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
				})
				balancer.rewrite()
				plugins.run()
			}

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
			# other authentication method such as basic auth or external auth useless - all requests will be allowed.
			#access_by_lua_block {
			#}

			header_filter_by_lua_block {
				lua_ingress.header()
				plugins.run()
			}

			body_filter_by_lua_block {
				plugins.run()
			}

			log_by_lua_block {
				balancer.log()

				monitor.call()

				plugins.run()
			}

			port_in_redirect off;

			set $balancer_ewma_score -1;
			set $proxy_upstream_name "upstream-default-backend";
			set $proxy_host          $proxy_upstream_name;
			set $pass_access_scheme  $scheme;

			set $pass_server_port    $server_port;

			set $best_http_host      $http_host;
			set $pass_port           $pass_server_port;

			set $proxy_alternative_upstream_name "";

			# this location requires authentication

			auth_request        /_external-auth-Lw-Prefix;
			auth_request_set    $auth_cookie $upstream_http_set_cookie;

			add_header          Set-Cookie $auth_cookie;

			set_escape_uri $escaped_request_uri $request_uri;
			error_page 401 = @2bb6bc255b0c4bfd57a8f6f523d918622a087012;

			# Cors Preflight methods needs additional options and different Return Code

			if ($http_origin ~* ((https://allowed\.origin\.com))$ ) { set $cors 'true'; }

			if ($request_method = 'OPTIONS') {
				set $cors ${cors}options;
			}

			if ($cors = "true") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
			}

			if ($cors = "trueoptions") {
				more_set_headers 'Access-Control-Allow-Origin: $http_origin';
				more_set_headers 'Access-Control-Allow-Credentials: true';
				more_set_headers 'Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS';
				more_set_headers 'Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization';

				more_set_headers 'Access-Control-Max-Age: 1728000';
				more_set_headers 'Content-Type: text/plain charset=UTF-8';
				more_set_headers 'Content-Length: 0';
				return 204;
			}

			client_max_body_size                    1m;

			proxy_set_header Host                   $best_http_host;

			# Pass the extracted client certificate to the backend

			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;

			proxy_set_header                        Connection        $connection_upgrade;

			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $remote_addr;

			proxy_set_header X-Forwarded-For        $remote_addr;

			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			proxy_set_header X-Scheme               $pass_access_scheme;

			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";

			# Custom headers to proxied server

			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;

			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;

			proxy_max_temp_file_size                1024m;

			proxy_request_buffering                 on;
			proxy_http_version                      1.1;

			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;

			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_timeout             0;
			proxy_next_upstream_tries               3;

			rewrite "(?i)/" /$1 break;
			proxy_pass http://upstream_balancer;

			proxy_redirect                          off;

		}

	}
	## end server app.test

	# backend for when default-backend-service is not configured or it does not have endpoints
	server {
		listen 8181 default_server reuseport backlog=4096;
		listen [::]:8181 default_server reuseport backlog=4096;
		set $proxy_upstream_name "internal";

		access_log off;

		location / {
			return 404;
		}
	}

	# default server, used for NGINX healthcheck and access to nginx stats
	server {
		listen 127.0.0.1:10246;
		set $proxy_upstream_name "internal";

		keepalive_timeout 0;
		gzip off;

		access_log off;

		location /healthz {
			return 200;
		}

		location /is-dynamic-lb-initialized {
			content_by_lua_block {
				local configuration = require("configuration")
				local backend_data = configuration.get_backends_data()
				if not backend_data then
				ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
				return
				end

				ngx.say("OK")
				ngx.exit(ngx.HTTP_OK)
			}
		}

		location /nginx_status {
			stub_status on;
		}

		location /configuration {
			client_max_body_size                    21M;
			client_body_buffer_size                 21M;
			proxy_buffering                         off;

			content_by_lua_block {
				configuration.call()
			}
		}

		location / {
			content_by_lua_block {
				ngx.exit(ngx.HTTP_NOT_FOUND)
			}
		}
	}
}

stream {
	lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;;";

	lua_shared_dict tcp_udp_configuration_data 5M;

	init_by_lua_block {
		collectgarbage("collect")

		-- init modules
		local ok, res

		ok, res = pcall(require, "configuration")
		if not ok then
		error("require failed: " .. tostring(res))
		else
		configuration = res
		end

		ok, res = pcall(require, "tcp_udp_configuration")
		if not ok then
		error("require failed: " .. tostring(res))
		else
		tcp_udp_configuration = res
		tcp_udp_configuration.prohibited_localhost_port = '10246'

		end

		ok, res = pcall(require, "tcp_udp_balancer")
		if not ok then
		error("require failed: " .. tostring(res))
		else
		tcp_udp_balancer = res
		end
	}

	init_worker_by_lua_block {
		tcp_udp_balancer.init_worker()
	}

	lua_add_variable $proxy_upstream_name;


	access_log /var/log/nginx/access.log log_stream ;

	error_log  /var/log/nginx/error.log notice;

	upstream upstream_balancer {
		server 0.0.0.1:1234; # placeholder

		balancer_by_lua_block {
			tcp_udp_balancer.balance()
		}
	}

	server {
		listen 127.0.0.1:10247;

		access_log off;

		content_by_lua_block {
			tcp_udp_configuration.call()
		}
	}

	# TCP services

	# UDP services

	# Stream Snippets

}

curl -v http://localhost/protected-service/protected -H "Host: app.test" -H "origin: https://allowed.origin.com"

*   Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /protected-service/protected HTTP/1.1
> Host: app.test
> User-Agent: curl/7.79.1
> Accept: */*
> origin: https://allowed.origin.com
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Moved Temporarily
< Date: Tue, 12 Jul 2022 13:05:14 GMT
< Content-Type: text/html
< Content-Length: 138
< Connection: keep-alive
< Location: https://api-auth.xyz.com/oauth2/start?rd=http://app.test/protected-service/protected
< Access-Control-Allow-Origin: https://allowed.origin.com
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
< Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization
< Access-Control-Max-Age: 1728000
<
<html>
<head><title>302 Found</title></head>
<body>
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host localhost left intact

longwuyuan · 2022-07-12T13:29:56Z

Remove the message posted about deployment yaml. It is just clutter and does not help focus on the problem/solution, in my opinion.

Thanks for the other post where the info requested was posted. But that is half clarity. See my thoughts below and correct me or educate me if I am wrong. This is complicated stuff so it will be good if we make all information crystal clear in these conversations.

(1) Your services of --LoadBalancer are in pending state so while I see some of your URL are internal to cluster, I would not trust any of the data presented as it does not simulate a real use case. By real use case I mean if there was a service --type ClusterIP, then it is reasonable that those URLs with internal-to-cluster FQDN/HostName are working.

(2) Where is this https://api-auth.xyz.com

(3) Remove the rewrite annotation

If you can make changes and repost or edit the current post, it will help. That is where the logs become relevant. And although I did not mention it, if you can show that this was done with make dev-env from your branch, for example show kubectl cluster-info and also show git diff in your branch, it will add validity for anyone checking on this in future

longwuyuan · 2022-07-12T13:50:35Z

I edited my previous message so please check. Also suggesting again that you post logs of contrllerpod in addition to kubectl cluster-info and git diff from the branch where you are working

harry1064 · 2022-07-12T13:52:58Z

(1) Your services of --LoadBalancer are in pending state so while I see some of your URL are internal to cluster, I would not trust any of the data presented as it does not simulate a real use case. By real use case I mean if there was a service --type ClusterIP, then it is reasonable that those URLs with internal-to-cluster FQDN/HostName are working.

Do you mean pending state of External IP? As I am running the cluster locally, I don' think we require it.
In this command

curl -v http://localhost/protected-service/protected -H "Host: app.test" -H "origin: https://allowed.origin.com/"

as we are passing Host header which is equal to the fact URL being hit like http://app.test/protected-service/protected, which is similar if this cluster would have been deployed on any cloud platform.

(2) Where is this https://api-auth.xyz.com

This does not matter as the according to my understanding the problem statement is whatever the exernalAuth.SignURL is configured, when we are redirecting the user it should return the CORS origin configured in the ingress resource, as other wise browser will block if the headers are missing.

(3) Remove the rewrite annotation

This case is not even triggered in the use case we are testing here. In our case

curl -v http://localhost/protected-service/protected -H "Host: app.test" -H "origin: https://allowed.origin.com/"

it will go to auth-service, auth-service will return 401 and then based on our nginx config, 401 will get internally redirected to location block which will return the user 302 to redirect him to URL configured in exernalAuth.SignURL. With CORS template now added in this location block, we are able to return the CORS related headers whatever user configured in ingress resource back to user.

harry1064 · 2022-07-12T14:22:44Z

kubectl cluster-info

Kubernetes control plane is running at https://127.0.0.1:60297
CoreDNS is running at https://127.0.0.1:60297/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

kubectl logs ingress-nginx-controller-5689d5bcd6-sq957 -n ingress-nginx

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       1.0.0-dev
  Build:         git-771ce1c6d
  Repository:    [email protected]:harry1064/ingress-nginx.git
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

W0711 13:57:48.236677      10 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0711 13:57:48.238565      10 main.go:230] "Creating API client" host="https://10.96.0.1:443"
I0711 13:57:48.379900      10 main.go:274] "Running in Kubernetes cluster" major="1" minor="21" git="v1.21.12" state="clean" commit="696a9fdd2a58340e61e0d815c5769d266fca0802" platform="linux/amd64"
I0711 13:57:48.651781      10 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0711 13:57:48.708115      10 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0711 13:57:48.743638      10 nginx.go:258] "Starting NGINX Ingress controller"
I0711 13:57:48.763814      10 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"b67da353-4b1f-4111-83ac-cc12b4a847e0", APIVersion:"v1", ResourceVersion:"606", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0711 13:57:49.946600      10 nginx.go:301] "Starting NGINX process"
I0711 13:57:49.947022      10 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0711 13:57:49.947049      10 nginx.go:321] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0711 13:57:49.948218      10 controller.go:167] "Configuration changes detected, backend reload required"
I0711 13:57:49.985818      10 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader
I0711 13:57:49.986141      10 status.go:84] "New leader elected" identity="ingress-nginx-controller-5689d5bcd6-sq957"
I0711 13:57:50.136032      10 controller.go:184] "Backend successfully reloaded"
I0711 13:57:50.136764      10 controller.go:195] "Initial sync, sleeping for 1 second"
I0711 13:57:50.136883      10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5689d5bcd6-sq957", UID:"17c6585f-afef-4b9e-88a3-805a8a7f7bfd", APIVersion:"v1", ResourceVersion:"723", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
E0711 14:49:46.796610      10 leaderelection.go:367] Failed to update lock: Put "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/ingress-nginx/leases/ingress-controller-leader": context deadline exceeded
I0711 14:49:47.876666      10 leaderelection.go:283] failed to renew lease ingress-nginx/ingress-controller-leader: timed out waiting for the condition
I0711 14:49:48.097635      10 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0711 14:50:42.869485      10 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader
W0712 04:32:26.894806      10 controller.go:257] ignoring ingress my-ingress in default based on annotation : ingress does not contain a valid IngressClass
I0712 04:32:26.895637      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 04:32:26.925367      10 store.go:425] "Ignoring ingress because of error while validating ingress class" ingress="default/my-ingress" error="ingress does not contain a valid IngressClass"
W0712 04:35:50.520777      10 controller.go:257] ignoring ingress my-ingress in default based on annotation : ingress does not contain a valid IngressClass
I0712 04:35:50.520827      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 04:40:42.523341      10 store.go:395] "Ignoring ingress because of error while validating ingress class" ingress="default/my-ingress" error="ingress does not contain a valid IngressClass"
W0712 04:41:08.928160      10 controller.go:257] ignoring ingress my-ingress in default based on annotation : ingress does not contain a valid IngressClass
I0712 04:41:08.928430      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 04:41:08.934052      10 store.go:425] "Ignoring ingress because of error while validating ingress class" ingress="default/my-ingress" error="ingress does not contain a valid IngressClass"
I0712 04:48:18.880741      10 store.go:395] "Ignoring ingress because of error while validating ingress class" ingress="default/my-ingress" error="ingress does not contain a valid IngressClass"
W0712 04:48:28.663388      10 controller.go:257] ignoring ingress my-ingress in default based on annotation : ingress does not contain a valid IngressClass
I0712 04:48:28.663443      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 04:48:28.669858      10 store.go:425] "Ignoring ingress because of error while validating ingress class" ingress="default/my-ingress" error="ingress does not contain a valid IngressClass"
I0712 05:04:52.939217      10 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.554s renderingIngressLength:1 renderingIngressTime:0.006s admissionTime:35.8kBs testedConfigurationSize:0.56}
I0712 05:04:52.940122      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 05:04:52.947223      10 store.go:474] "creating ingress" ingress="default/my-ingress" ingressclass="nginx"
I0712 05:04:52.950114      10 controller.go:167] "Configuration changes detected, backend reload required"
I0712 05:04:52.959578      10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"93653ffd-ff34-4c32-9c63-34b96a47f245", APIVersion:"networking.k8s.io/v1", ResourceVersion:"80361", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0712 05:04:53.060842      10 controller.go:184] "Backend successfully reloaded"
I0712 05:04:53.065898      10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5689d5bcd6-sq957", UID:"17c6585f-afef-4b9e-88a3-805a8a7f7bfd", APIVersion:"v1", ResourceVersion:"723", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0712 05:05:43.311293      10 status.go:299] "updating Ingress status" namespace="default" ingress="my-ingress" currentValue=[] newValue=[{IP:10.96.6.142 Hostname: Ports:[]}]
I0712 05:05:43.319314      10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"93653ffd-ff34-4c32-9c63-34b96a47f245", APIVersion:"networking.k8s.io/v1", ResourceVersion:"80439", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
172.18.0.1 - - [12/Jul/2022:05:08:15 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.065 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.004 401 1a1dd8ca331c60e8d6fb768990aa8db3
172.18.0.1 - - [12/Jul/2022:05:08:15 +0000] "GET /protected-service/protected HTTP/1.1" 401 172 "-" "curl/7.79.1" 99 0.065 [default-protected-service-8080] [] - - - - 1a1dd8ca331c60e8d6fb768990aa8db3
172.18.0.1 - - [12/Jul/2022:05:09:55 +0000] "GET /protected-service/protected HTTP/1.1" 204 0 "-" "curl/7.79.1" 0 0.006 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.004 204 f8d1e3d3d58543d84f4bbc55c744eda5
172.18.0.1 - - [12/Jul/2022:05:09:55 +0000] "GET /protected-service/protected HTTP/1.1" 200 26 "-" "curl/7.79.1" 126 0.009 [default-protected-service-8080] [] 10.244.0.10:5000 26 0.003 200 f8d1e3d3d58543d84f4bbc55c744eda5
I0712 05:11:24.021127      10 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.054s renderingIngressLength:1 renderingIngressTime:0s admissionTime:37.2kBs testedConfigurationSize:0.054}
I0712 05:11:24.021188      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 05:11:24.028769      10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"93653ffd-ff34-4c32-9c63-34b96a47f245", APIVersion:"networking.k8s.io/v1", ResourceVersion:"80937", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0712 05:11:24.028823      10 controller.go:167] "Configuration changes detected, backend reload required"
I0712 05:11:24.166045      10 controller.go:184] "Backend successfully reloaded"
I0712 05:11:24.166725      10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5689d5bcd6-sq957", UID:"17c6585f-afef-4b9e-88a3-805a8a7f7bfd", APIVersion:"v1", ResourceVersion:"723", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0712 08:24:43.251226      10 main.go:140] Current config.corsAllowOrigin [https://allowed.origin.com]
I0712 08:24:43.295957      10 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.044s renderingIngressLength:1 renderingIngressTime:0.002s admissionTime:37.3kBs testedConfigurationSize:0.046}
I0712 08:24:43.296008      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 08:24:43.304320      10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"93653ffd-ff34-4c32-9c63-34b96a47f245", APIVersion:"networking.k8s.io/v1", ResourceVersion:"97914", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0712 08:24:43.305137      10 main.go:140] Current config.corsAllowOrigin [https://allowed.origin.com]
I0712 08:24:43.306060      10 controller.go:167] "Configuration changes detected, backend reload required"
I0712 08:24:43.396843      10 controller.go:184] "Backend successfully reloaded"
I0712 08:24:43.397761      10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5689d5bcd6-sq957", UID:"17c6585f-afef-4b9e-88a3-805a8a7f7bfd", APIVersion:"v1", ResourceVersion:"723", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
172.18.0.1 - - [12/Jul/2022:08:26:01 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.008 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.006 401 da23f2fa179a36ddbe4c4a3acb2e3a5a
172.18.0.1 - - [12/Jul/2022:08:26:01 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 99 0.008 [default-protected-service-8080] [] - - - - da23f2fa179a36ddbe4c4a3acb2e3a5a
172.18.0.1 - - [12/Jul/2022:08:26:27 +0000] "GET /protected-service/protected HTTP/1.1" 204 0 "-" "curl/7.79.1" 0 0.003 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.002 204 893e22a572af014c280f630dba0d4dce
172.18.0.1 - - [12/Jul/2022:08:26:27 +0000] "GET /protected-service/protected HTTP/1.1" 200 26 "-" "curl/7.79.1" 126 0.008 [default-protected-service-8080] [] 10.244.0.10:5000 26 0.006 200 893e22a572af014c280f630dba0d4dce
172.18.0.1 - - [12/Jul/2022:08:27:20 +0000] "GET /public-service/public HTTP/1.1" 204 0 "-" "curl/7.79.1" 0 0.003 [default-public-service-8080] [] 10.96.72.241:8080 0 0.003 204 eedc58dbe404e7791fd11739672d250c
172.18.0.1 - - [12/Jul/2022:08:27:20 +0000] "GET /public-service/public HTTP/1.1" 200 23 "-" "curl/7.79.1" 93 0.008 [default-public-service-8080] [] 10.244.0.8:5000 23 0.004 200 eedc58dbe404e7791fd11739672d250c
172.18.0.1 - - [12/Jul/2022:08:33:53 +0000] "GET /public-service/public HTTP/1.1" 204 0 "-" "curl/7.79.1" 0 0.006 [default-public-service-8080] [] 10.96.72.241:8080 0 0.005 204 ce1c079b06f1a1301d6d42490426327c
172.18.0.1 - - [12/Jul/2022:08:33:53 +0000] "GET /public-service/public HTTP/1.1" 200 23 "-" "curl/7.79.1" 129 0.010 [default-public-service-8080] [] 10.244.0.8:5000 23 0.004 200 ce1c079b06f1a1301d6d42490426327c
I0712 10:04:46.841819      10 nginx.go:163] "New NGINX configuration template loaded"
I0712 10:05:38.926813      10 main.go:140] Current config.corsAllowOrigin [https://allowed.orign.com]
I0712 10:05:38.973938      10 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.046s renderingIngressLength:1 renderingIngressTime:0.001s admissionTime:41.3kBs testedConfigurationSize:0.047}
I0712 10:05:38.973987      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 10:05:38.979207      10 main.go:140] Current config.corsAllowOrigin [https://allowed.orign.com]
I0712 10:05:38.979538      10 controller.go:167] "Configuration changes detected, backend reload required"
I0712 10:05:38.984033      10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"93653ffd-ff34-4c32-9c63-34b96a47f245", APIVersion:"networking.k8s.io/v1", ResourceVersion:"106772", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0712 10:05:39.073487      10 controller.go:184] "Backend successfully reloaded"
I0712 10:05:39.074105      10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5689d5bcd6-sq957", UID:"17c6585f-afef-4b9e-88a3-805a8a7f7bfd", APIVersion:"v1", ResourceVersion:"723", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
172.18.0.1 - - [12/Jul/2022:10:07:05 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.004 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.003 401 f0ed015c40e1b08da9e5b404269471ad
172.18.0.1 - - [12/Jul/2022:10:07:05 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 135 0.004 [default-protected-service-8080] [] - - - - f0ed015c40e1b08da9e5b404269471ad
172.18.0.1 - - [12/Jul/2022:10:07:37 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.004 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.004 401 522d13a55412ce434273e34403194874
172.18.0.1 - - [12/Jul/2022:10:07:37 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 134 0.004 [default-protected-service-8080] [] - - - - 522d13a55412ce434273e34403194874
172.18.0.1 - - [12/Jul/2022:11:31:10 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.097 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.094 401 9716ce7365e3b4602e5e8c13c91186d7
172.18.0.1 - - [12/Jul/2022:11:31:10 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 134 0.097 [default-protected-service-8080] [] - - - - 9716ce7365e3b4602e5e8c13c91186d7
172.18.0.1 - - [12/Jul/2022:11:32:02 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.004 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.003 401 5009b6665ba47a4e4726fdbc1b8b5c11
172.18.0.1 - - [12/Jul/2022:11:32:02 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 134 0.004 [default-protected-service-8080] [] - - - - 5009b6665ba47a4e4726fdbc1b8b5c11
172.18.0.1 - - [12/Jul/2022:11:32:47 +0000] "GET /protected-service/protected HTTP/1.1" 204 0 "-" "curl/7.79.1" 0 0.007 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.007 204 0563eb38ca0e6b4d89acafc6a98c15e0
172.18.0.1 - - [12/Jul/2022:11:32:47 +0000] "GET /protected-service/protected HTTP/1.1" 200 26 "-" "curl/7.79.1" 161 0.046 [default-protected-service-8080] [] 10.244.0.10:5000 26 0.039 200 0563eb38ca0e6b4d89acafc6a98c15e0
172.18.0.1 - - [12/Jul/2022:11:34:17 +0000] "GET /protected-service/protected HTTP/1.1" 204 0 "-" "curl/7.79.1" 0 0.011 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.009 204 8db0786e6b9ec3d5f0a641d9c8b24ef1
172.18.0.1 - - [12/Jul/2022:11:34:17 +0000] "GET /protected-service/protected HTTP/1.1" 200 26 "-" "curl/7.79.1" 161 0.038 [default-protected-service-8080] [] 10.244.0.10:5000 26 0.027 200 8db0786e6b9ec3d5f0a641d9c8b24ef1
172.18.0.1 - - [12/Jul/2022:11:34:29 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.002 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.001 401 fafb59f19751d8ea808a7236134b89ad
172.18.0.1 - - [12/Jul/2022:11:34:29 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 134 0.002 [default-protected-service-8080] [] - - - - fafb59f19751d8ea808a7236134b89ad
172.18.0.1 - - [12/Jul/2022:11:36:21 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.006 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.006 401 70f20e0d4fbed1f28d2945d332693ac0
172.18.0.1 - - [12/Jul/2022:11:36:21 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 134 0.006 [default-protected-service-8080] [] - - - - 70f20e0d4fbed1f28d2945d332693ac0
172.18.0.1 - - [12/Jul/2022:11:36:23 +0000] "GET /protected-service/protected HTTP/1.1" 204 0 "-" "curl/7.79.1" 0 0.002 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.003 204 04cc289fbc3bfdc5a8040e57d328c2eb
172.18.0.1 - - [12/Jul/2022:11:36:23 +0000] "GET /protected-service/protected HTTP/1.1" 200 26 "-" "curl/7.79.1" 161 0.006 [default-protected-service-8080] [] 10.244.0.10:5000 26 0.003 200 04cc289fbc3bfdc5a8040e57d328c2eb
I0712 12:55:44.586783      10 main.go:140] Current config.corsAllowOrigin [https://allowed.origin.com]
I0712 12:55:44.633671      10 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.046s renderingIngressLength:1 renderingIngressTime:0.001s admissionTime:41.3kBs testedConfigurationSize:0.047}
I0712 12:55:44.633721      10 main.go:100] "successfully validated configuration, accepting" ingress="default/my-ingress"
I0712 12:55:44.640036      10 main.go:140] Current config.corsAllowOrigin [https://allowed.origin.com]
I0712 12:55:44.640454      10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"my-ingress", UID:"93653ffd-ff34-4c32-9c63-34b96a47f245", APIVersion:"networking.k8s.io/v1", ResourceVersion:"121679", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0712 12:55:44.642413      10 controller.go:167] "Configuration changes detected, backend reload required"
I0712 12:55:44.749662      10 controller.go:184] "Backend successfully reloaded"
I0712 12:55:44.750729      10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5689d5bcd6-sq957", UID:"17c6585f-afef-4b9e-88a3-805a8a7f7bfd", APIVersion:"v1", ResourceVersion:"723", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
172.18.0.1 - - [12/Jul/2022:13:05:04 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.008 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.007 401 d8e4a250d7a99b434cb841259d2988d9
172.18.0.1 - - [12/Jul/2022:13:05:04 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 134 0.008 [default-protected-service-8080] [] - - - - d8e4a250d7a99b434cb841259d2988d9
172.18.0.1 - - [12/Jul/2022:13:05:14 +0000] "GET /protected-service/protected HTTP/1.1" 401 0 "-" "curl/7.79.1" 0 0.004 [default-protected-service-8080] [] 10.96.72.241:8080 0 0.004 401 2475cccd14cec728a385578747819971
172.18.0.1 - - [12/Jul/2022:13:05:14 +0000] "GET /protected-service/protected HTTP/1.1" 302 138 "-" "curl/7.79.1" 135 0.004 [default-protected-service-8080] [] - - - - 2475cccd14cec728a385578747819971

git diff

diff --git a/rootfs/etc/nginx/template/nginx.tmpl b/rootfs/etc/nginx/template/nginx.tmpl
index 561278b6f..28fca9cb7 100755
--- a/rootfs/etc/nginx/template/nginx.tmpl
+++ b/rootfs/etc/nginx/template/nginx.tmpl
@@ -1172,6 +1172,10 @@ stream {
 
             add_header Set-Cookie $auth_cookie;
 
+            {{ if $location.CorsConfig.CorsEnabled }}
+            {{ template "CORS" $location }}
+            {{ end }}
+
             # Ensure that modsecurity will not run on an internal location as this is not accessible from outside
             {{ if $all.Cfg.EnableModsecurity }}
             modsecurity off;

As make dev-env was breaking on main branch, so I change MAC_DOCKER_FLAGS="--load" to MAC_DOCKER_FLAGS= locally to build the image on my machine.

longwuyuan · 2022-07-12T16:41:40Z

@harry1064 this fleshing out of the details, reduces the work for approvers, so thank you very much.
If you can add a e2e-test, we can test this in CI. Let me know your thoughts.

harry1064 · 2022-07-12T17:24:25Z

@longwuyuan Yes sure. I can write the e2-test. I am familiarizing myself with the way the e2e test are structured as I am new to project.
Thank you very much for your inputs on making this PR process easier. :)

longwuyuan · 2022-07-12T17:35:49Z

@harry1064 that is awesome. Please also join the ingress-nginx-dev channel on slack in case you need to talk. This is good stuff so thank you again.

harry1064 · 2022-07-13T11:54:22Z

@longwuyuan Added one e2e test case. Have a look and let me know your thoughts.

longwuyuan · 2022-07-13T11:58:21Z

I saw the test just now. Thank again for this.

/ok-to-test

@tao12345666333 @rikatz @strongjz, it seems like there has been a bug for years, that if auth & cors are used together in one ingress, then cors headers are missing. When you get a chance, please check & label this PR . I am not sure how such a bug can exist for years as auth+cors should be common enterprise use case.

harry1064 · 2022-07-13T12:02:23Z

@longwuyuan Thank you too for guiding through the process.

tao12345666333 · 2022-07-13T12:16:06Z

/assign

I will add it to my list, and check it ASAP. Thanks

longwuyuan · 2022-07-13T12:19:58Z

@tao12345666333 , Also I did not check if this opens a corner-case vulnerability.

therc · 2022-09-16T15:45:44Z

Any progress on this?

0x0dr1y · 2022-10-27T12:07:56Z

Any blocker we can help to resolve here?

sgohlke · 2023-01-17T12:26:27Z

@tao12345666333 @rikatz @strongjz Is there any progress on this PR? We also have this problem and would be happy if this can be released.

rikatz · 2023-01-19T14:55:59Z

/lgtm
/approve
Should be part of v1.6.1

Thanks

k8s-ci-robot · 2023-01-19T14:56:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: harry1064, rikatz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rikatz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ernetes#8814) * Add CORS template check inside location for externalAuth.SignURL * Add testcase for CORS header for auth-signin redirect with CORS enabled.

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 12, 2022

k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 12, 2022

k8s-ci-robot requested review from rikatz and strongjz July 12, 2022 11:56

harry1064 added 2 commits July 13, 2022 19:47

Add CORS template check inside location for externalAuth.SignURL

8799355

Add testcase for CORS header for auth-signin redirect with CORS enabled.

6b97a1c

harry1064 force-pushed the CORS-header-when-authSignUrl-redirect branch from c5abeb9 to 6b97a1c Compare July 13, 2022 11:51

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 13, 2022

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 13, 2022

k8s-ci-robot assigned tao12345666333 Jul 13, 2022

johanneswuerbach mentioned this pull request Nov 3, 2022

CORS Headers missing when using auth-url and auth-signin with Oauth2-Proxy #8786

Closed

k8s-ci-robot assigned rikatz Jan 19, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 19, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 19, 2023

k8s-ci-robot merged commit d1af3b5 into kubernetes:main Jan 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CORS template check inside location for externalAuth.SignURL #8814

Add CORS template check inside location for externalAuth.SignURL #8814

harry1064 commented Jul 12, 2022 •

edited

Loading

k8s-ci-robot commented Jul 12, 2022

k8s-ci-robot commented Jul 12, 2022

k8s-ci-robot commented Jul 12, 2022

longwuyuan commented Jul 12, 2022

harry1064 commented Jul 12, 2022

longwuyuan commented Jul 12, 2022 •

edited

Loading

longwuyuan commented Jul 12, 2022

harry1064 commented Jul 12, 2022 •

edited

Loading

harry1064 commented Jul 12, 2022

longwuyuan commented Jul 12, 2022

harry1064 commented Jul 12, 2022 •

edited

Loading

longwuyuan commented Jul 12, 2022

harry1064 commented Jul 13, 2022

longwuyuan commented Jul 13, 2022

harry1064 commented Jul 13, 2022

tao12345666333 commented Jul 13, 2022

longwuyuan commented Jul 13, 2022

therc commented Sep 16, 2022

0x0dr1y commented Oct 27, 2022

sgohlke commented Jan 17, 2023

rikatz commented Jan 19, 2023

k8s-ci-robot commented Jan 19, 2023

Add CORS template check inside location for externalAuth.SignURL #8814

Add CORS template check inside location for externalAuth.SignURL #8814

Conversation

harry1064 commented Jul 12, 2022 • edited Loading

What this PR does / why we need it:

Types of changes

Which issue/s this PR fixes

How Has This Been Tested?

Checklist:

k8s-ci-robot commented Jul 12, 2022

k8s-ci-robot commented Jul 12, 2022

k8s-ci-robot commented Jul 12, 2022

longwuyuan commented Jul 12, 2022

harry1064 commented Jul 12, 2022

longwuyuan commented Jul 12, 2022 • edited Loading

longwuyuan commented Jul 12, 2022

harry1064 commented Jul 12, 2022 • edited Loading

harry1064 commented Jul 12, 2022

longwuyuan commented Jul 12, 2022

harry1064 commented Jul 12, 2022 • edited Loading

longwuyuan commented Jul 12, 2022

harry1064 commented Jul 13, 2022

longwuyuan commented Jul 13, 2022

harry1064 commented Jul 13, 2022

tao12345666333 commented Jul 13, 2022

longwuyuan commented Jul 13, 2022

therc commented Sep 16, 2022

0x0dr1y commented Oct 27, 2022

sgohlke commented Jan 17, 2023

rikatz commented Jan 19, 2023

k8s-ci-robot commented Jan 19, 2023

harry1064 commented Jul 12, 2022 •

edited

Loading

longwuyuan commented Jul 12, 2022 •

edited

Loading

harry1064 commented Jul 12, 2022 •

edited

Loading

harry1064 commented Jul 12, 2022 •

edited

Loading