Nginx ingress chart is not working in azure (AKS) #10863

bittu664 · 2024-01-16T10:58:20Z

After installing the Nginx ingress controller in AKS, it gives me Loadbalancer ip , but that ip is not working , here is my chart details and steps:-

my AKS is running on v1.28.0 version

here you can see this when i hit this ip its not opening:-

k8s-ci-robot · 2024-01-16T10:58:28Z

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

longwuyuan · 2024-01-17T02:06:57Z

/remove-kind bug

There are many questions asked in the new issue template, when you click the button to create a new bug report. You have not answered any of those questions and those answers are needed for readers to base their comments on. Please check what those questions are and provide the information.
Please read the documentation related to ingress objects and ingress-nginx controller in particualr and help out with a proper ingress resource created, tested and debug info posted here. That is more practical compared to the current information provided
Also the recommended install method for Azure is this https://kubernetes.github.io/ingress-nginx/deploy/#azure

bittu664 · 2024-01-18T07:14:11Z

Thanks , @longwuyuan by refering this docs now i am able to fix this issues. actually the problem is on latest verson:- 1.9.5

so after using this 1.8.2 version its working.

 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml

longwuyuan · 2024-01-18T11:18:58Z

What is the problem on latest version?

…

On Thu, 18 Jan, 2024, 12:44 pm Rock, ***@***.***> wrote: Thanks , @longwuyuan <https://github.com/longwuyuan> by refering this docs now i am able to fix this issues. actually the problem is on latest verson:- 1.9.5 so after using this 1.8.2 version its working. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml — Reply to this email directly, view it on GitHub <#10863 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWTEGZBBXRPZ4R757C3YPDDU5AVCNFSM6AAAAABB4TH5XOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJXHEZDOMBZGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

bittu664 · 2024-01-18T12:25:31Z

i dont know about issue, the thing is in latest version ingress contolerr exposeing loadbalncer but that loadbalncer ip does not work at all. means it does not show 404 nginx page. thats it.

strongjz · 2024-01-18T14:03:45Z

What is the issue? just stops working is not helpful. Could you deploy an ingress object? Is the ingress-nginx service up and running? Is the ingress pod up and running? Are their network policy or cloud firewall policy in place?

bittu664 · 2024-01-18T14:24:29Z

Yes i deployed my ingress object, it didnt work at all in latest version. as i already said that the problem something on latest version, but it is working on 1.8.2

Gacko · 2024-01-21T14:25:37Z

/assign

Gacko · 2024-01-22T16:46:57Z

Hello,

I already wanted to comment on this yesterday but didn't make it. Can you confirm the following:

Installing Ingress NGINX via the following manifest without any changes in your environment works:

kubectl create --filename https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml

Installing Ingress NGINX via the following manifest without any changes in your environment does not work:

kubectl create --filename https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.5/deploy/static/provider/cloud/deploy.yaml

To completely remove all remains of an installation, please run the following command with the respective URL:

kubectl delete --filename URL

If you can confirm the above, I'm concerned about how AKS is handling unprivileged containers recently. The only real difference between the both of them is the PSS stuff, we tightened security settings in the v1.9.5 releases.

Chart: Tighten securityContexts and Pod Security Policies. #10491

Maybe you can also go the extra mile and check which release manifest in particular breaks it on AKS (v1.8.4, v1.9.0, v1.9.1, v1.9.3, v1.9.4, v1.9.5).

In the meantime I will try to reproduce your issue on my own.

Thanks a lot in advance!
Marco

longwuyuan · 2024-01-22T17:01:45Z

I hope this is not about or related to this #9601 (comment)

Can anyone here check or look into Azure/AKS#2907 (comment) which basically boils down to a new requirement to use this annotation

"--set controller.service.annotations."service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"=/healthz"

as described in #9601 . Just so that we can rule it out

Gacko · 2024-01-22T17:04:33Z

I already considered this and it's probably the root cause for a different issue. But right here, the author tells it's working with v1.8.2, so I'd first like to verify this. Maybe it's only occurring because of the update and Azure Cloud Controller Manager reconciling the config.

Gacko · 2024-01-22T19:29:42Z

Ah, damn, I got one thing wrong: Initially you were using the Helm chart in version v4.9.0, right? And this was not working, correct? Then you used the plain deploy.yaml for version v1.8.2 and this was working.

You need to know: The deploy.yaml comes with externalTrafficPolicy being set to Local while the chart defaults to Cluster. For the chart to work on AKS, you wight want to do what @longwuyuan mentioned above or set controller.service.externalTrafficPolicy to Local.

julie-ng · 2024-01-25T16:53:44Z

@longwuyuan I think you are right, it is related to those other networking specific changes :-(

I've set service externalTrafficPolicy=Local
And using this annotation service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz

BUT, it doesn't work. These are what my logs at 16:38 looked like:

│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd W0125 16:36:51.454329       8 controller.go:1457] Using default certificate                                                                                                                                                                                                                                                                                            │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd I0125 16:37:09.693221       8 leaderelection.go:255] successfully acquired lease ingress/ingress-basic-ingress-nginx-leader                                                                                                                                                                                                                                            │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd I0125 16:37:09.693535       8 status.go:84] "New leader elected" identity="ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd"                                                                                                                                                                                                                                    │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-lf2fw I0125 16:37:16.475021       6 status.go:84] "New leader elected" identity="ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd"                                                                                                                                                                                                                                    │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-cnflw I0125 16:37:20.064558       8 status.go:84] "New leader elected" identity="ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd"                                                                                                                                                                                                                                    │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd I0125 16:38:45.035747       8 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.053s renderingIngressLength:1 renderingIngressTime:0s admissionTime:18.3kBs testedConfigurationSize:0.053}                                                                                                                       │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd I0125 16:38:45.035790       8 main.go:107] "successfully validated configuration, accepting" ingress="hello-world/hello-world"                                                                                                                                                                                                                                         │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd I0125 16:38:45.054286       8 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hello-world", Name:"hello-world", UID:"8699411f-ccc8-427a-8214-70a85492bd62", APIVersion:"networking.k8s.io/v1", ResourceVersion:"137592", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                            │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-cnflw I0125 16:38:45.049441       8 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hello-world", Name:"hello-world", UID:"8699411f-ccc8-427a-8214-70a85492bd62", APIVersion:"networking.k8s.io/v1", ResourceVersion:"137592", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                            │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-cnflw I0125 16:38:45.050349       8 controller.go:190] "Configuration changes detected, backend reload required"                                                                                                                                                                                                                                                             │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-lf2fw I0125 16:38:45.049079       6 controller.go:190] "Configuration changes detected, backend reload required"                                                                                                                                                                                                                                                             │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-lf2fw I0125 16:38:45.051853       6 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hello-world", Name:"hello-world", UID:"8699411f-ccc8-427a-8214-70a85492bd62", APIVersion:"networking.k8s.io/v1", ResourceVersion:"137592", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                            │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd I0125 16:38:45.059832       8 controller.go:190] "Configuration changes detected, backend reload required"                                                                                                                                                                                                                                                             │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-lf2fw I0125 16:38:45.229540       6 controller.go:210] "Backend successfully reloaded"                                                                                                                                                                                                                                                                                       │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-lf2fw I0125 16:38:45.231714       6 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"ingress-basic-ingress-nginx-controller-7d4fdcb4cb-lf2fw", UID:"7a263811-6824-4941-a0fa-1e4e79a5e82a", APIVersion:"v1", ResourceVersion:"136869", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration   │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd I0125 16:38:45.230070       8 controller.go:210] "Backend successfully reloaded"                                                                                                                                                                                                                                                                                       │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd I0125 16:38:45.236243       8 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd", UID:"ef41a198-8a1c-4b87-912f-efcff1475c67", APIVersion:"v1", ResourceVersion:"136928", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration   │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-cnflw I0125 16:38:45.291811       8 controller.go:210] "Backend successfully reloaded"                                                                                                                                                                                                                                                                                       │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-cnflw I0125 16:38:45.294060       8 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"ingress-basic-ingress-nginx-controller-7d4fdcb4cb-cnflw", UID:"d63207d2-c29f-42bd-9fb4-5dc6cdd9a839", APIVersion:"v1", ResourceVersion:"136967", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration   │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-cnflw 10.0.2.4 - - [25/Jan/2024:16:42:13 +0000] "GET / HTTP/1.1" 308 164 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" 301 0.000 [hello-world-hello-world-80] [] - - - - 1ce11720043d1c933ddd7174f83e07c3                                                                                                                       │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-lf2fw 10.0.2.4 - - [25/Jan/2024:16:42:14 +0000] "GET / HTTP/2.0" 200 5421 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" 215 0.030 [hello-world-hello-world-80] [] 10.0.2.55:3000 5421 0.030 200 dc0811b348f5a4f7a32137a526958340                                                                                                │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-lf2fw 10.0.2.33 - - [25/Jan/2024:16:42:18 +0000] "GET / HTTP/1.1" 308 164 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" 378 0.000 [hello-world-hello-world-80] [] - - - - 544f98b1d7502d81e36a7d7f55297ea2                                                                                                                      │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd 10.0.2.4 - - [25/Jan/2024:16:42:18 +0000] "GET / HTTP/2.0" 200 5421 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" 286 0.026 [hello-world-hello-world-80] [] 10.0.2.43:3000 5421 0.026 200 84042c2c8f6251a03c6ea0eff1bdb056                                                                                                │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd 10.0.2.4 - - [25/Jan/2024:16:42:18 +0000] "GET /favicon.ico HTTP/2.0" 404 83 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" 68 0.003 [hello-world-hello-world-80] [] 10.0.2.55:3000 83 0.003 404 bcc5d7994711a1045ab66a8389e10a01                                                                                          │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd 10.0.2.4 - - [25/Jan/2024:16:42:29 +0000] "GET / HTTP/2.0" 200 5421 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" 37 0.003 [hello-world-hello-world-80] [] 10.0.2.43:3000 5421 0.003 200 f10b90b4d12456a921384c9516247e1a                                                                                                 │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd 10.0.2.4 - - [25/Jan/2024:16:42:30 +0000] "GET / HTTP/2.0" 200 5421 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" 18 0.002 [hello-world-hello-world-80] [] 10.0.2.55:3000 5421 0.003 200 3fbeefd7db8f2c75de6b461b9f411e11                                                                                                 │
│ ingress-basic-ingress-nginx-controller-7d4fdcb4cb-hkmqd 10.0.2.4 - - [25/Jan/2024:16:42:33 +0000] "GET / HTTP/2.0" 200 5421 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" 18 0.002 [hello-world-hello-world-80] [] 10.0.2.43:3000 5421 0.003 200 c763e11f0048e337f2e05b010b045401

The logs above showed it suddenly changed a few minutes later at 16:42, when I manually inspected the Azure load balancer and noticed the health probe was still pointing to / instead of /healthz.

@longwuyuan @Gacko like the OP, the 1.8.2 version worked for me. What could have changed?

My configuration is open source:

my chart values chart.values.yaml
my helm CLI command Makefile:116

Right now I am using chart version 4.8.0. Going to try 4.7.0 now.

Gacko · 2024-01-25T18:52:13Z

The health check is handled by kube-proxy or its replacement, if you're using Cilium e.g., with externalTrafficPolicy: Local. I was able to get things running with the plain deploy.yaml from our docs (https://kubernetes.github.io/ingress-nginx/deploy/#azure) because this ships with externalTrafficPolicy: Local. I did not need to change the health check path.

julie-ng · 2024-01-25T18:52:19Z

@longwuyuan @Gacko - nevermind. My problem was YAML 🤦‍♀️🤦‍♀️🤦‍♀️🤦‍♀️

Gacko · 2024-01-25T18:53:53Z

julie-ng/cloudkube-aks-clusters@94c2b2e

This one? Can you omit the health check path annotation and check if it's still working?

Gacko · 2024-01-27T16:55:06Z

Closing as there's no feedback and @julie-ng prove it's working on AKS. Please comment on this issue if you have further questions as it's about the same issue.

/close

k8s-ci-robot · 2024-01-27T16:55:12Z

@Gacko: Closing this issue.

In response to this:

Closing as there's no feedback and @julie-ng prove it's working on AKS. Please comment on this issue if you have further questions as it's about the same issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

bittu664 added the kind/bug Categorizes issue or PR as related to a bug. label Jan 16, 2024

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Jan 16, 2024

strongjz added this to [SIG Network] Ingress NGINX Jan 16, 2024

k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jan 17, 2024

617m4rc mentioned this issue Jan 18, 2024

AKS load balancer health check fails with ingress-nginx version v1.9.5 (but works with up to and including v1.8.4) #10869

Closed

k8s-ci-robot assigned Gacko Jan 21, 2024

k8s-ci-robot closed this as completed Jan 27, 2024

github-project-automation bot moved this to Done in [SIG Network] Ingress NGINX Jan 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nginx ingress chart is not working in azure (AKS) #10863

Nginx ingress chart is not working in azure (AKS) #10863

bittu664 commented Jan 16, 2024 •

edited

Loading

k8s-ci-robot commented Jan 16, 2024

longwuyuan commented Jan 17, 2024

bittu664 commented Jan 18, 2024

longwuyuan commented Jan 18, 2024 via email

bittu664 commented Jan 18, 2024 •

edited

Loading

strongjz commented Jan 18, 2024

bittu664 commented Jan 18, 2024 •

edited

Loading

Gacko commented Jan 21, 2024

Gacko commented Jan 22, 2024 •

edited

Loading

longwuyuan commented Jan 22, 2024 •

edited

Loading

Gacko commented Jan 22, 2024

Gacko commented Jan 22, 2024

julie-ng commented Jan 25, 2024 •

edited

Loading

Gacko commented Jan 25, 2024

julie-ng commented Jan 25, 2024

Gacko commented Jan 25, 2024

Gacko commented Jan 27, 2024

k8s-ci-robot commented Jan 27, 2024

Nginx ingress chart is not working in azure (AKS) #10863

Nginx ingress chart is not working in azure (AKS) #10863

Comments

bittu664 commented Jan 16, 2024 • edited Loading

k8s-ci-robot commented Jan 16, 2024

longwuyuan commented Jan 17, 2024

bittu664 commented Jan 18, 2024

longwuyuan commented Jan 18, 2024 via email

bittu664 commented Jan 18, 2024 • edited Loading

strongjz commented Jan 18, 2024

bittu664 commented Jan 18, 2024 • edited Loading

Gacko commented Jan 21, 2024

Gacko commented Jan 22, 2024 • edited Loading

longwuyuan commented Jan 22, 2024 • edited Loading

Gacko commented Jan 22, 2024

Gacko commented Jan 22, 2024

julie-ng commented Jan 25, 2024 • edited Loading

Gacko commented Jan 25, 2024

julie-ng commented Jan 25, 2024

Gacko commented Jan 25, 2024

Gacko commented Jan 27, 2024

k8s-ci-robot commented Jan 27, 2024

bittu664 commented Jan 16, 2024 •

edited

Loading

bittu664 commented Jan 18, 2024 •

edited

Loading

bittu664 commented Jan 18, 2024 •

edited

Loading

Gacko commented Jan 22, 2024 •

edited

Loading

longwuyuan commented Jan 22, 2024 •

edited

Loading

julie-ng commented Jan 25, 2024 •

edited

Loading