ingress nginx still routes traffic to Unready pod

[vinh-thai]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version:
0.22.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Azure AKS
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

• Created a POC k8s project that has a /ready endpoint for k8s readiness probe,
• deliberately set the /ready having a long waiting response so that Pods are marked it as not ready
• execute kubectl get pods and can see that it is not ready

NAME                                                        READY     STATUS    RESTARTS   AGE
test-aks-healthcheck-7ffd7dffd-tqfzw                        0/1       Running   1          2h

• execute kubectl get ep and sees that:

NAME                                       ENDPOINTS           AGE 
test-aks-healthcheck                                           3h

• Check in ingress controller log

I0221 03:09:30.604376       6 controller.go:460] Replacing location "/" for server "test.aks.ohmdevtest.buddy.io" with upstream "upstream-default-backend" to use upstream "default-test-aks-healthcheck-http" (Ingress "default/test-aks-healthcheck")
I0221 03:09:30.604394       6 controller.go:588] Upstream "default-test-aks-healthcheck-http" has no active Endpoint

• but when I try to access the POC project endpoint , it is still available ( it is expected that it would return a HTTP 503.
  (logs in nginx show

"GET /ping HTTP/2.0" 200 19 "-" "curl/7.54.0" 45 0.002 [default-test-aks-healthcheck-http] 10.244.2.183:8080 30 0.000 200 26ff9c66267807f5e227200fd386b7e3

• a weird thing is that, if I reload the ingress controller by log on the ingress controller pod and do

$ nginx -s reload

then it works, properly receive a 503 Service temporary unavailable.
(nginx log:

 "GET /ping HTTP/2.0" 503 197 "-" "curl/7.54.0" 45 0.000 [default-test-aks-healthcheck-http] - - - - 4a26860e9de3c46e74f3ef265eb96703

What you expected to happen:
503 return when trying to access a service on pod marked as unready

How to reproduce it (as minimally and precisely as possible):

• Create a small nodejs REST API that has long readiness check
• Add in the helm config as below in values.yaml

readinessProbe:
  path: /ready
  port: 8080
  timeoutSeconds: 1
  initialDelaySeconds: 10

• deploy to Azure AKS or other k8s cluster
• try to access the nodejs REST API endpoint after the pod has been marked as not ready

Anything else we need to know:

[aledbf]

Closing. This issue is fixed in master #3628
You can test the fix using the dev image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev

[vinh-thai]

thanks @aledbf , any chance you know when it will be in official release?

[aledbf]

Please check https://github.com/kubernetes/ingress-nginx/projects/30 (I think another week)