ingress-nginx > 0.16.x does not support aufs anymore

[ssiergl]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

aufs, overlay2, cap, setcap

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

NGINX Ingress controller version:

0.16.x

Kubernetes version (use kubectl version):

1.10, 1.11

Environment:

debian stretch, bare metal

docker 17.03 with aufs storage driver

What happened:

Pod is crashing

F0713 09:43:31.317783       7 main.go:72] Port 80 is already in use. Please check the flag --http-port

What you expected to happen:

Pod is able to bind port 80 and 443 to start up correctly

How to reproduce it (as minimally and precisely as possible):

Just deploy on a node with aufs storage driver

Anything else we need to know:

With #2619 the aufs storage driver compatibility is broken. Aufs does not support setting caps for nginx-ingress-controller. Hence the pod is not able to get permission to bind privileged ports.

It should be documented that aufs support on privileged ports is dropped and give an example to use ingress-nginx with remapped unprivileged ports (--http-port, --https-port)

See as well: kubernetes-sigs/kubespray#3000

[antoineco]

/kind bug

[aledbf]

@ssiergl please change https://github.com/kubernetes/ingress-nginx/blob/master/deploy/mandatory.yaml#L261 to 0
(basically, this means run as root)

[ProFfeSsoRr]

Containerd (not docker) and overlayfs2 - same problem. And change uid to 0 fix them too.

[aledbf]

@ssiergl please remove the securityContext section in the deployment and use the image quay.io/aledbf/nginx-ingress-controller:0.401

[liqlin2015]

@aledbf We got same issue with docker 18.03.1 and overlay2 storage driver. I was curious why this issue only happens on some of system .

Is there any OS level security setting which restrict the permission?

[BlueBlue-Lee]

have same issue:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.17.1
  Build:      git-12f7966
  Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------

F0809 07:07:36.049759       8 main.go:72] Port 80 is already in use. Please check the flag --http-port

# docker info
Server Version: 18.06.3-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs

# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.9", GitCommit:"16236ce91790d4c75b79f6ce96841db1c843e7d2", GitTreeState:"clean", BuildDate:"2019-03-25T06:40:24Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - ingress-nginx
              topologyKey: kubernetes.io/hostname         
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        node-role.kubernetes.io/node: ""
      containers:
        - name: nginx-ingress-controller
          image: hub.c.163.com/combk8s/nginx-ingress-controller:0.17.1
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          resources:
            limits:
              cpu: 4000m
              memory: 8Gi
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

change runAsUser: 33 to runAsUser: 0, can't resolve this problem, but print another error:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.17.1
  Build:      git-12f7966
  Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------

nginx version: nginx/1.13.12
W0809 07:17:03.518518       8 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0809 07:17:03.518731       8 main.go:191] Creating API client for https://10.178.4.1:443
I0809 07:17:03.531511       8 main.go:235] Running in Kubernetes cluster version v1.11 (v1.11.9) - git (clean) commit 16236ce91790d4c75b79f6ce96841db1c843e7d2 - platform linux/amd64
I0809 07:17:03.534079       8 main.go:100] Validated kube-system/default-http-backend as the default backend.
F0809 07:17:04.015786       8 main.go:113] Error generating self-signed certificate: could not create temp pem file /etc/ingress-controller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem254202187: permission denied

[BlueBlue-Lee]

@aledbf Sorry to bother.

[antoineco]

@shenlanse that issue was fixed in v0.18. You are using v0.17. Please update.

[BlueBlue-Lee]

@antoineco Thanks. Could you please tell me the reason or related git commit?

[antoineco]

Sure, the closing issue is referenced few comments before yours:

k8s-ci-robot closed this issue in #2825 about 1 year ago

If I remember correctly the issue is related to setcap (used to bind to ports below 1024 with non privileged users) not working in all environments.