Using custom nginx.tmpl fails at startup: /etc/nginx/template filesystem is read-only

[bgagnon]

Is this a BUG REPORT or FEATURE REQUEST?: Bug report
NGINX Ingress controller version: (dev)
Kubernetes version: 1.10.3

Environment:

• Cloud provider or hardware configuration: Docker for Mac

What happened:

Configuring a custom nginx.tmpl is no longer working in latest build.

      containers:
      - name: nginx-ingress-controller
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
          runAsUser: 33
        volumeMounts:
        - mountPath: /etc/nginx/template
          name: nginx-template-volume
        - mountPath: /var/lib/nginx/
          name: nginx-var-lib
      volumes:
      - name: nginx-template-volume
        configMap:
          name: nginx-configuration
          items:
          - key: nginx.tmpl
            path: nginx.tmpl
      - name: nginx-var-lib
        emptyDir: {}

The Dockerfile's entrypoint wants to set permissions on the directory, but it's read-only because it is mounted from a ConfigMap:

chown: changing ownership of '/etc/nginx/template/nginx.tmpl': Read-only file system
chown: changing ownership of '/etc/nginx/template/..data': Read-only file system
chown: changing ownership of '/etc/nginx/template/..2018_07_27_19_46_12.086463885/nginx.tmpl': Read-only file system
chown: changing ownership of '/etc/nginx/template/..2018_07_27_19_46_12.086463885': Read-only file system
chown: changing ownership of '/etc/nginx/template': Read-only file system

I tried forcing the ConfigMap volume as readOnly: false but it does not help.

[bgagnon]

Looks like this was added in 8107e0f and further refined in df76d4b.
The chown steps cannot work if the directory is mounted from a ConfigMap.

[aledbf]

@bgagnon this is my fault. As part to run the ingress controller as a user I made changes to the startup of the container.
Tomorrow I will add e2e tests for custom templates to fix and avoid this regression.
Apologies for the troubles.

[bgagnon]

Also note that setting readOnly: false is a no-op in Kubernetes 1.9+ unless a feature gate is toggled: kubernetes/kubernetes#62099

I can think of one workaround involving an emptyDIr volume and an initContainer that moves nginx.tmpl into place.

[bgagnon]

No worries @aledbf, and thank you for the fast response.

[bgagnon]

This workaround is functional:

      initContainers:
      - name: copy-nginx-template
        image: busybox
        args:
        - cp
        - /tmp/nginx/nginx.tmpl
        - /etc/nginx/template/nginx.tmpl
        volumeMounts:
        - name: nginx-template-volume
          mountPath: /tmp/nginx
        - mountPath: /etc/nginx/template
          name: nginx-template-dir
      volumes:
      - name: nginx-template-dir
        emptyDir: {}
      - name: nginx-var-lib
        emptyDir: {}
      - name: nginx-template-volume
        configMap:
          name: nginx-configuration
          items:
          - key: nginx.tmpl
            path: nginx.tmpl

[aledbf]

Closing. Fixed in #2877