rootless HTTPs not working due to missing /data and /config permissions

[codeInTheShell]

A documentation type deployment of caddy which runs as root with docker-compose (https://hub.docker.com/_/caddy) works with auto-ssl flawlessly.
If a docker deployment has the flag user set to 1000:1000 and a named volume is used instead of a local path the user permissions stay root. If a deployment has just user set to 1000:1000 but still uses local paths, the created directories will still have root permissions.

The only solution is to manually chown -R 1000:1000 either the docker volume _data/ directory or the local path.

Sample docker-compose.yaml

version: "3.9"                                                                                                                                                                                                     
services:                                                                                                
  caddy:                                                                                                                                                                               
    image: caddy:2-alpine                                                                               
    restart: unless-stopped                                                                              
    user: 1000:1000     # <--- the culprit                                                                                 
    ports:                                                                                              
      - "80:80"                                                                                         
      - "443:443"                                                                                       
      - "443:443/udp"                                                                                   
    volumes:                                                                                             
      - $PWD/../Caddyfile:/etc/caddy/Caddyfile                                                           
      - $PWD/site:/srv                                                                                   v
      - caddy_data:/data                                                                                 
      - caddy_config:/config            
      - ./caddy_data:/data                                                                                 
      - ./caddy_config:/config                                                                             
                                                                                                         
volumes:                                                                                                 
  caddy_data:                                                                                            
  caddy_config:

Unfortunately the below Dockerfile didn't help either.

ENV UID 1000
ENV GID 1000

RUN addgroup -g $GID -S caddy ; adduser -SDHs /sbin/nologin -u $UID -G caddy caddy  ; chown $UID:$GID /data ; chown $UID:$GID /config
VOLUME /data
VOLUME /config
RUN chown -R $UID:$GID /data /config

The result is the same either way unless i manually chown -R it with the respective user and group id:

{"level":"error","ts":1680045963.1747446,"logger":"tls","msg":"job failed","error":"domain.tld: obtaining certificate: failed storage check: open /data/caddy/rw_test_4836700798867844449: permission denied - storage is probably misconfigured"}
{"level":"error","ts":1680045963.1751785,"logger":"tls","msg":"job failed","error":"domain.tld: obtaining certificate: failed storage check: open /data/caddy/rw_test_148537797091318011: permission denied - storage is probably misconfigured"}
{"level":"error","ts":1680045963.1753407,"msg":"unable to autosave config","file":"/config/caddy/autosave.json","error":"open /config/caddy/autosave.json: permission denied"}
{"level":"info","ts":1680045963.1753864,"msg":"serving initial configuration"}

[francislavoie]

Hey @tianon I hope you don't mind the ping, could you give us some pointers here?

I'm not sure what the correct way to handle volumes that need write access as non-root, for official images. I didn't see any guidance about that in the https://github.com/docker-library/official-images README other than a mention of gosu but I'm not sure how that interacts with volumes or if it helps with that at all 🤔

[paulo-erichsen]

I had the same issue, but found a workaround. Earlier I saw #104 and decided to give it a go at rootless container user.

My understanding is that the Dockerfile calls mkdir -p on some folders, but they seem to be created as root??

caddy-docker/Dockerfile.tmpl

Lines 9 to 13 in bee0fa4

	mkdir -p \
	/config/caddy \
	/data/caddy \
	/etc/caddy \
	/usr/share/caddy \

I use ansible to configure caddy and my workaround was to create all the folders that caddy uses beforehand with the user permissions that I wanted. (I had to rm -r the older caddy folders and rerun my playbook to start from scratch)

roles/caddy/tasks/main.yml

---
- name: Create caddy config directory
  ansible.builtin.file:
    path: "{{ project_directory }}/caddy/{{ item }}"
    state: directory
    mode: "0755"
    owner: "{{ user_id }}"
    group: "{{ group_id }}"
  loop:
    - config
    - data
    - etc
    - etc/caddy
    - logs

- name: Copy caddy config
  ansible.builtin.template:
    src: Caddyfile.j2
    dest: "{{ project_directory }}/caddy/etc/caddy/Caddyfile"
    mode: "0600"
    owner: "{{ user_id }}"
    group: "{{ group_id }}"
  notify: Restart caddy

- name: Copy caddy dockerfile
  ansible.builtin.copy:
    src: Dockerfile
    dest: "{{ project_directory }}/caddy/Dockerfile"
    mode: "0644"
    owner: "{{ user_id }}"
    group: "{{ group_id }}"

- name: Build docker image for caddy with cloudflare plugin
  community.docker.docker_image:
    build:
      path: "{{ project_directory }}/caddy/"
      pull: true
    name: caddy/caddycfdns
    source: build
    force_source: true

- name: Create caddy container
  community.docker.docker_container:
    container_default_behavior: no_defaults
    name: caddy
    image: caddy/caddycfdns:latest
    pull: false
    state: started
    restart_policy: unless-stopped
    network_mode: bridge
    ports:
      - 80:80
      - 443:443
    volumes:
      - "{{ project_directory }}/caddy/etc/caddy:/etc/caddy"
      - "{{ project_directory }}/caddy/config:/config"
      - "{{ project_directory }}/caddy/data:/data"
      - "{{ project_directory }}/caddy/logs:/logs"
    user: "{{ user_id }}:{{ group_id }}"
    env:
      ACME_AGREE: "true"
      CLOUDFLARE_EMAIL: "xxxxxxxxxxxxxxxxxxxxx"
      CLOUDFLARE_API_TOKEN: "{{ cloudflare_dns_token }}"
      TZ: "{{ default_timezone }}"
    labels:
      com.centurylinklabs.watchtower.enable: "false"

NOTE: one interesting thing is that the logs folder isn't created in that mkdir that I've referenced. That must be happening somewhere else maybe after caddy starts?

for reference, this is the Dockerfile that I use

roles/caddy/files/Dockerfile

FROM caddy:builder AS builder
RUN XCADDY_SUDO=0 xcaddy build --with github.com/caddy-dns/cloudflare

FROM caddy:latest
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

[tianon]

Sorry for the delay 🙇

In other images, we've done something like root-owned 1777 (/tmp permissions) on volumes so that "arbitrary user" works, but then we'll usually couple that with an entrypoint that ensures that if we're started as root, we adjust the ownership accordingly before we step down from root so that the runtime security is better (https://github.com/docker-library/rabbitmq/blob/e667276a6f89ce21af3da179fd2178795f73c94e/docker-entrypoint.sh#L4-L11 for example). 😅

In some other images, we've taken the approach that if a user wants to run as a non-default user, then it's their responsibility to ensure that their volume has appropriate permissions (although that's a somewhat less usable solution because it requires something like a k8s "init container" to accomplish in a generic way -- I really wish k8s and Docker would allow specifying ownership/permissions directly when creating/configuring volumes 🙊).

[dimkasta]

A bit late, but I hope this helps.

@tianon Content, ownership and permissions can be defined in volumes using Dockerfile only before they are created. You can use things like COPY and/with chmod/chown and the image that creates the volume will populate it with its definitions.

@codeInTheShell For arbitrary accounts in existing volumes, you can try setting it first to root like 0:0, then exec into the container to do any chown that you need, and then switch to the new IDs. A temp root bastion container could be handy for that. Or an entrypoint script that calls a separate startup script and/or does not start the service if the user is root, but keeps the container running. Just to avoid root creating more files by running a complex service.

In rootless mode, the daemon maps container user IDs to different host user IDs. The container root is the user running the daemon on the host. The user still has CHOWN capabilities though. So you can still fix permissions.

[tcurdt]

This situation is a little unfortunate. I blame docker for not allowing specifying ownership for volumes. 😒

While still not ideal, the most simple workaround seems to be to add one more layer and pass in the UID and GID. At least it seems to work fine for me.

FROM caddy:2.7-alpine

ENV UID 65534
ENV GID 65534
RUN chown -R $UID:$GID /data /config