Nginx Ingress on Azure Kubernetes does not honor upstream-hash-by

[daninthewoods]

With Nginx Ingress configured with upstream-hash-by: "$request_uri" external requests are not correctly hashed to the same backend pod

Configuration
1 nginx ingress
Version: k8s.gcr.io/ingress-nginx/controller:v0.41.2
Annotation: nginx.ingress.kubernetes.io/upstream-hash-by: "$request_uri"
3 replica backend http-echo test containers, echoing back request headers and podname

Using a simple http echo container and while true; do curl ... | grep ... to check for the pod name returning the response. Each request is a new connection from curl, but is same request_uri.

I expect given the same http request made repeatly to get back responses from the same backend pod.

On AKS I can test at 3 layers

1. External request - fails and returns responses from all backend pods
2. NodePort of nginx-ingress - fails and returns responses from all backend pods
3. IP/port of nginx-ingress - success and returns responses from a specific backend pod

• To prove upstream-hash-by is configured correctly, if I remove the setting, test Move issues and prs over from contrib/ingress #3 behaves same as Basic structure  #1 & Figure out documentation layout #2.
• Using curl -s 127.0.0.1:10246/configuration/backends from within nginx-ingress proves the hash-by configuration has been applied.
• A fresh restart of nginx-ingress also returns the same behavior, versus a dynamic update of the annotation.
• I noticed that on repeat requests using keep-alive (ie. from browser or apache bench) then the same backend is used, but changes periodically. Maybe due to max keep-alive requests per connection.
• Using other values for upstream-hash-by has same result (e.g. $remote_addr)
• I have also tested 3 replicas of nginx-ingress and for test Move issues and prs over from contrib/ingress #3 above, all return responses consistently from the same backend pod - (ie they hash the same)

Azure Kubernetes services (AKS)

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.14", GitCommit:"5de7fd1f9555368a86eb0f8f664dc58055c17269", GitTreeState:"clean", BuildDate:"2021-01-18T09:31:01Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[longwuyuan]

I think the data you posted leads to guessing but not much precise analysis.

Can you post the following information here ;

• kubectl get all,nodes,ing -A -o wide
• kubectl -n describe po
• kubectl -n describe svc
• kubectl -n describe ing
• Your complete curl command and its output
• The logs of the ingressnginx pod related to your curl command
• Any other related info or logs like proof of the header in the request that decides routing like dns etc

[daninthewoods]

Thanks for the follow-up. We finally found a resolution. Someone had declared 80/443 as TCP routes on the AKS load-balancer and as such no http traffic was routing into nginx, it was routing around and into the service behind.

[longwuyuan]

Thnx. Close the issue. Thanks, ; Long

…

On Tue, 13 Apr, 2021, 12:43 AM daninthewoods, ***@***.***> wrote: Thanks for the follow-up. We finally found a resolution. Someone had declared 80/443 as TCP routes on the AKS load-balancer and as such no http traffic was routing into nginx, it was routing around and into the service behind. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7037 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWRFSPHVQNJV2B4UOTTTINA33ANCNFSM42U6KSHQ> .

[longwuyuan]

/remove-kind bug
@daninthewoods Please close the issue. Thanks

[iamNoah1]

/close

[k8s-ci-robot]

@iamNoah1: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[strongjz]

/close

[k8s-ci-robot]

@strongjz: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.