Unable to run ingress-nginx on readonly fs

[rvadim]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Feature request

NGINX Ingress controller version: 0.24.1

Kubernetes version (use kubectl version): v1.13.4

Environment:

• Cloud provider or hardware configuration: Bare Metal
• OS (e.g. from /etc/os-release): CentOS 7
• Kernel (e.g. uname -a): 3.10

What happened:
I'm need to set readOnlyRootFilesystem: true in PSP for ingress nginx controller for security and performance reasons.
When I start it with this mount points(all volumes is emptyDir):

          volumeMounts:
          - mountPath: /etc/ingress-controller/ssl
            name: ssl
          - mountPath: /tmp/
            name: tmp
          - mountPath: /etc/nginx
            name: nginx

I have 2 errors:

1. main.go:141] Error creating prometheus collector: listen unix /tmp/prometheus-nginx.socket: bind: address already in use
2. nginx.go:170] Invalid NGINX configuration template: unexpected error reading template /etc/nginx/template/nginx.tmpl: open /etc/nginx/template/nginx.tmpl: no such file or directory

What you expected to happen:

1. nginx-ingress-contoller delete or unlink /tmp/prometheus-nginx.socket on starting (https://stackoverflow.com/questions/2208581/socket-listen-doesnt-unbind-in-c-under-linux)
2. Keep nginx templates and configuration at another directory of generated nginx.conf

How to reproduce it (as minimally and precisely as possible):
PodSecurityPolicy

apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
  name: nginx-ingress
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - NET_BIND_SERVICE
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    ranges:
    - max: 65535
      min: 33
    rule: MustRunAs
  hostNetwork: true
  hostPorts:
  - max: 80
    min: 80
  - max: 443
    min: 443
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - ALL
  runAsUser:
    ranges:
    - max: 65535
      min: 33
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 33
    rule: MustRunAs
  volumes:
  - configMap
  - hostPath
  - emptyDir
  - secret
  - downwardAPI
  - persistentVolumeClaim
  - projected

DaemonSet:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: nginx-ingress-controller
  name: nginx-ingress-controller
  namespace: ingress-nginx
spec:
  minReadySeconds: 5
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-ingress-controller
  template:
    metadata:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
      labels:
        app: nginx-ingress-controller
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/$(NGINX_CONFIGMAP_NAME)
        - --tcp-services-configmap=$(POD_NAMESPACE)/$(TCP_CONFIGMAP_NAME)
        - --udp-services-configmap=$(POD_NAMESPACE)/$(UDP_CONFIGMAP_NAME)
        - --publish-service=$(POD_NAMESPACE)/$(SERVICE_NAME)
        - --annotations-prefix=nginx.ingress.kubernetes.io
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        name: nginx-ingress-controller
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          procMount: Default
          runAsUser: 33
        volumeMounts:
        - mountPath: /etc/ingress-controller/ssl
          name: ssl
        - mountPath: /tmp/
          name: tmp
        - mountPath: /etc/nginx
          name: nginx
        - mountPath: /etc/nginx/test
          name: nginx
      securityContext:
        fsGroup: 33
        runAsUser: 33
      serviceAccount: nginx-ingress
      serviceAccountName: nginx-ingress
      volumes:
      - emptyDir: {}
        name: ssl
      - emptyDir: {}
        name: tmp
      - emptyDir: {}
        name: nginx
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

Anything else we need to know:
There are workarounds for both problems:
We can remove /tmp/prometheus-nginx.socket on container start

          lifecycle:
            postStart:
              exec:
                command: ['/bin/sh', '-c', 'rm -f /tmp/prometheus-nginx.socket']

We can copy configuration files to /etc/nginx with initContainer

      initContainers:
        - name: copy-configs
          image: {{ controller_image }}:{{ controller_version }}
          volumeMounts:
          - mountPath: /nginx
            name: nginx
          command: ['sh', '-c', 'cp -r /etc/nginx/* /nginx/']

[aledbf]

I'm need to set readOnlyRootFilesystem: true in PSP for ingress nginx controller for security and performance reasons.

Which performance reasons?

main.go:141] Error creating prometheus collector: listen unix /tmp/prometheus-nginx.socket: bind: address already in use

I don't have a fix for this one. Any idea/proposal is welcome

nginx.go:170] Invalid NGINX configuration template: unexpected error reading template /etc/nginx/template/nginx.tmpl: open /etc/nginx/template/nginx.tmpl: no such file or directory

For this one, you need to copy the content from the image.

I am fixing some issues with PSP and volumes here #4456

[rvadim]

@aledbf thank you for answer and fix for volumes 👍

Which performance reasons?

https://docs.docker.com/storage/storagedriver/select-storage-driver/

devicemapper is supported, but requires direct-lvm for production environments, because loopback-lvm, while zero-configuration, has very poor performance. devicemapper was the recommended storage driver for CentOS and RHEL, as their kernel version did not support overlay2. However, current versions of CentOS and RHEL now have support for overlay2, which is now the recommended driver.

I don't have a fix for this one. Any idea/proposal is welcome

https://stackoverflow.com/questions/34873151/how-can-i-delete-a-unix-domain-socket-file-when-i-exit-my-application

... Put the unlink() right before the bind(). That's how everybody else does it. (Example: BSD syslogd, one of the classic unix-domain-socket-based services)