liveness probe fails and nginx-ingress-controller restarts on load with unavailable upstreams

[zbitmanis]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT

NGINX Ingress controller version: 0.16., 0.17.,0.18.*,0.19.0, 0.20.0, 0.21.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T09:56:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T09:56:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
  NAME="Ubuntu"
  VERSION="16.04.5 LTS (Xenial Xerus)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 16.04.5 LTS"
  VERSION_ID="16.04"
  HOME_URL="http://www.ubuntu.com/"
  SUPPORT_URL="http://help.ubuntu.com/"
  BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
  VERSION_CODENAME=xenial
  UBUNTU_CODENAME=xenial
• Kernel (e.g. uname -a):
  Linux k8s-master-35100059-0 4.15.0-1021-azure Refactor default backend handling and add better events #21~16.04.1-Ubuntu SMP Fri Aug 10 12:36:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

What happened:
On load on ingress which points to unavailable upstreams leads to regular livenes probe fail and regular nginx ingress pod restart initiated by kubelet

ingress controllers are located on dedicated nodes (taint + tolerations )

What you expected to happen:
ingrress controler should detect that all upstreams are unresolvable

How to reproduce it (as minimally and precisely as possible):

• Deploy Pod with exposed port

kind: Deployment 
metadata:
  name: nginx-unavailable
  namespace: nginx-ingress 
  labels:
    app: nginx-load-generator
spec:
  replicas: 2
  selector:
    matchLabels:
      run: nginx-unresolver
  template: 
      metadata:
        labels:
          run: nginx-unresolver
          app: nginx-load-generator
      spec:
        containers:
          - name: nginx-unavailable
            image: nginx
            ports: 
               - containerPort: 80
                 protocol: TCP

• *Deploy Service which which points one to exposed second unexposed port *

apiVersion: v1
kind: Service 
metadata:
  name: nginx-lg-svc
  namespace: nginx-ingress 
  labels:
    app: nginx-load-generator
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: working
    port: 8081
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx-unresolver
  type: ClusterIP

• *Deploy ingress *

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: nginx-lgunr-foo
    namespace: nginx-ingress 
spec:
    rules:
    - host: nginx-unresolver-foo.example.org
      http:
        paths:
        - backend:
            serviceName: nginx-lg-svc
            servicePort: 8080

• *Generate load from single host using tsenart/vegeta *

• contents of bdload.template
  GET http://nginx-unresolver-foo.example.org

• simulate load
  vegeta attack -insecure -rate=1000 -duration=600s -targets=bdload.templat

• controller deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    k8s-app: nginx-ingress
  name: nginx-ingress-controller
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: nginx-ingress
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: nginx-ingress
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/nginx-default-http-backend
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --publish-service=$(POD_NAMESPACE)/nginx-ingress
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --sort-backends=true
        - --v=3
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: nginx-ingress-controller
        ports:
         /..... skip ..../ 

Anything else we need to know:

• enabled to experimental-allowed-unsafe-sysctls='net.ipv4.,net.core.* - minimal mitigation *
  the same result
  Added to nginx-controller

        security.alpha.kubernetes.io/unsafe-sysctls: net.core.somaxconn=1024,net.ipv4.tcp_max_syn_backlog=1024,net.ipv4.tcp_syn_retries=3,net.ipv4.tcp_synack_retries=2
  

[aledbf]

@zbitmanis please enable the proxy-next-upstream: http_502 http_503 http_504 option https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-next-upstream
in the configuration configmap. By default there is no retries

[aledbf]

Closing. Please reopen if the suggested change does not work.