Run as non-root inside container

[hoshsadiq]

Would be nice to have caddy run as a non-root user inside the container. This is especially useful for rootfull containers (e.g. Docker), where the uid=0 inside the container is the same as uid=0 outside. Additionally, since the /data dir and the /config have been specified as a VOLUME, others are unable to extend the container and change the ownership of those directories. I'm currently using the below Dockerfile to run caddy as a non-root user. I've changed it slightly to suit my needs but much of it should be re-usable for this repo.

FROM caddy:2-builder AS builder

RUN caddy-builder github.com/caddy-dns/cloudflare

FROM caddy:2-alpine AS deps

# We cannot use FROM scratch because, despite adding cap_net_bind_service to the binary
# it still won't run. Presuming because libcap isn't available? Not sure.
FROM alpine:3.12

COPY --from=deps /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=deps /etc/mime.types /etc/nsswitch.conf /etc/

COPY --from=builder /usr/bin/caddy /caddy

RUN set -eux; \

    apk add --no-cache libcap; \
    setcap cap_net_bind_service=ep /caddy; \

    mkdir -p /config/caddy /data/caddy; \

    addgroup -g 101 -S www-data; \
    adduser -u 101 -D -S -G www-data www-data; \

    chown -R www-data:www-data /config /data

USER www-data

ENV XDG_CONFIG_HOME=/config XDG_DATA_HOME=/data

VOLUME /config /data

EXPOSE 80
EXPOSE 443

ENTRYPOINT ["/caddy"]
CMD ["run", "--config", "/Caddyfile", "--adapter", "caddyfile"]

One thing worth considering is that this might be not be an easy upgrade for many, indeed, it may be that we'd need a temporary stop gap that runs the container as root, changes ownership of files/folders, then drops privileges. Then after perhaps a couple of versions, this stop gap can be replaced fully with a non-root user without going through the trouble of dropping privileges.

[hairyhenderson]

Hi @hoshsadiq - my apologies for replying so late (been vacationing and otherwise distracted)!

We actually originally did run as non-root by default, but simplicity we decided to drop that (see #24, and also #103 for some other related discussion).

If your Dockerfile works for you, that's great. In most cases where users want to run as non-root, they also don't need to listen to :80/:443 in the container, so the setpcap magic isn't necessary at all.

It's also worth noting that caddy is an official image, and as such needs to be similarly-shaped to other official images of the same type. At a quick glance, none of nginx, traefik, httpd, or haproxy support running as non-root out of the box either.

Finally, it's worth considering why you want to run as non-root. What attack vectors are you trying to avoid? Container escape vulnerabilities are pretty much the only real risk, but anyone running a modern Docker version is immune to many of them. It's also worth considering user namespace remapping as a mitigation. In my experience the main reason for running as non-root is to pass compliance checks - not a bad reason, but it's also worth recognizing that non-compliance does not automatically equal decreased security (and vice-versa).

I will leave this open to discuss possibilities, but I think the starting point is probably to just provide some better documentation around running as non-root.

[mirekum]

Hi, In the company I work for, I am not allowed to run any docker container as root. It would be nice to use the Caddy in the same way:
es:
container_name: es
image: elasticsearch:6.8.8
read_only: true
user: "elasticsearch:elasticsearch"

[hoshsadiq]

@hairyhenderson sorry, I had forgotten about this issue.

In most cases where users want to run as non-root, they also don't need to listen to :80/:443 in the container, so the setpcap magic isn't necessary at all.

I'm aware of this. I didn't see an option to run caddy on a non-standard port, hence why I used setpcap. Maybe I was blind and/or I misunderstood. If you could let me know what that option is I'll gladly update my image.

It's also worth noting that caddy is an official image, and as such needs to be similarly-shaped to other official images of the same type. At a quick glance, none of nginx, traefik, httpd, or haproxy support running as non-root out of the box either.

While this is true, it's worth noting that nginx and httpd both have the option to delegate the actual workers (I don't know if it's the same with traefik and haproxy). One can indeed run as root, though it is discouraged in multiple places. In addition, for nginx, the configuration ignores the user/group options if the main process runs a non-root user.

Caddy does not have an option to delegate workers (I'm presuming requests run in go routines so that wouldn't even make sense). It is also impossible to run it as a non-root user using docker's (and docker-compose's) user arguments because the volumes /data and /config are owned by root, so when caddy runs as non-root, it does not have the correct permissions to those directories to function correctly.

it's worth considering why you want to run as non-root

Definitely agree with this. I'm using caddy for a personal project, but I think running as non-root is a no-brainer for me considering it's such a trivial task that gives a lot with little effort. Though this is a personal preference, and isn't entirely based on any specific threat model, rather it's more of a precautionary issue.

I think even if Caddy doesn't change the default, that would totally be okay, but making it easier for individual to run caddy in non-root mode, in my opinion is a must. As it stands, that is as simple as chowning the /data and /config directory as a non-root user. This was what initially prompted me to raise this as an issue. Due to docker's design, it is not possible to change the ownership (or the mode) of a VOLUME in a running container. It is only possible by explicitly doing so in another Dockerfile, changing the ownership/mode, and then re-assigning it as a VOLUME (as I've done in the my image).

[hairyhenderson]

@hoshsadiq thanks for following up!

I didn't see an option to run caddy on a non-standard port

That's really just a matter of configuration. In the Caddyfile syntax, there are two global options http_port and https_port to tell Caddy to listen to nonstandard ports. You'll still need to be listening at 80/443 outside the container in order to get automatic TLS from Let's Encrypt though.

So a config like this should work for you (warning, I didn't have time to test this):

{
  http_port 8080
  https_port 8443
}

mysite.com {
  root * /usr/share/caddy
  file_server
}

Then when you run the container, just map the ports appropriately:

$ docker run -d -p 80:8080 -p 443:8443 ...

As it stands, that is as simple as chowning the /data and /config directory as a non-root user

Sure, but that's if you want to use /data and /config - you don't have to. You can set the XDG_DATA_HOME and XDG_CONFIG_HOME environment variables to different locations. (See https://caddyserver.com/docs/conventions#file-locations for details on what these are)

The reason I used VOLUMEs is to prevent these files from being written to the container, and hence lost during a container restart. It's important for the the data directory in particular to persist, because it contains TLS certs from Let's Encrypt, and if those are lost you can very quickly get rate-limited by LE.

So, if you choose to configure Caddy to use different data/config directories (which you'll have to if you're running as non-root), you'll need to make sure you use persisted volumes.

Hope this helps! I'm definitely going to add some of this to the docs for the image on DockerHub, since most of this is definitely non-intuitive for people new to Caddy v2 😅

[westurner]

. It is also impossible to run it as a non-root user using docker's (and docker-compose's) user arguments because the volumes /data and /config are owned by root, so when caddy runs as non-root, it does not have the correct permissions to those directories to function correctly.

The process should not have write access to the config.

More about not running as root here: #24 (comment)

[francislavoie]

/config is actually a config cache directory, for when users are making use of Caddy's config API. If Caddy didn't write its current config to file, then the running config would be lost. In other words, any changes pushed to the config API would be lost. It also allows for the --resume option to work, which will make Caddy start up with the previously run configuration instead on start.

https://caddyserver.com/docs/conventions#configuration-directory

Don't worry, Caddy never writes to your Caddyfile (unless you use the caddy fmt command with the overwrite option, to have it re-indent your config).

[mholt]

If Caddy didn't write its current config to file, then the running config would be lost. In other words, any changes pushed to the config API would be lost.

To clarify, the running config is stored in memory, so it would only be lost lost if the process or container was terminated. (Which, of course, happens all the time.) You're right, but I just wanted to make that distinction.

[hoshsadiq]

Sorry for the late reply - again.

Sure, but that's if you want to use /data and /config - you don't have to. You can set the XDG_DATA_HOME and XDG_CONFIG_HOME environment variables to different locations

Aah, that is fair. I did not think of this, I'll use that. Also thanks for the config options for running on non-standard port, I don't know why, but I just did not find the options prior to you mentioning it.

I'm more than happy for this issue to be closed as my issue is resolved, however, I think it's still worth changing the default container to being back run as a non-root user. Considering root inside a container is root outside a container (for docker anyway), this can be destructive if Caddy is compromised. Security should not be compromised on such a level for a little convenience. This should be okay once rootless becomes the norm, but currently the majority users, and production workloads are still running Docker, where all containers are root-full.

Anyway, if there is no wiggle room, feel free to close this issue. I suppose if that's the case, documentation is key.

[abjugard]

Would it be an acceptable compromise for the official image to additionally install libcap and run setcap cap_net_bind_service=+eip /usr/bin/caddy so that users that want to use a more secure configuration can do so without having to build their own image? As it stands, the image isn't compatible with K8s securityContext runAsNonRoot + runAsUser options, because if I do that then it won't be able to bind port 80, which I personally think it should be able to do, in order to be compatible with e.g. host networking.

I've provided a pull-request which mitigates this issue, by making it possible to at least use the image with a different PID, given the user configures it in a compatible manner.

[francislavoie]

Ah, yeah good idea @abjugard, that seems like a good approach.

[hairyhenderson]

Fixed in #274 - this'll be available in the next Caddy release

[julsemaan]

Sorry for reviving a closed issue here but it seems there are capabilities to run this as non-root but although docs were called out in the issue description, I can't seem to find any docs related to how we can run this image as non-root

Could you point me at the right location for the docs and I think it will help to have them linked here in the OG issue

Appreciate the help!

[jjlin]

@julsemaan In general, to run a container as non-root, you just pass -u/--user to docker run; see https://docs.docker.com/engine/reference/commandline/run/ for details.

Some images (e.g., LinuxServer.io images) use environment variables like PUID/PGID to switch to another user/group during container startup, but there aren't any official standards behind that kind of behavior.

[polarathene]

As it stands, the image isn't compatible with K8s securityContext runAsNonRoot + runAsUser options, because if I do that then it won't be able to bind port 80, which I personally think it should be able to do, in order to be compatible with e.g. host networking.

FWIW, once containerd 2.0 is out k8s and others that leverage that dependency are likely to adopt the same default that Docker has had for a while where the need for the setcap will be unnecessary: kubernetes/kubernetes#102612 (comment)

That said, I do not like the setcap approach as it enforces the capability even if you explicitly do not want it. I go into more detail about that issue here: #299 (comment)

---

This is especially useful for rootfull containers (e.g. Docker), where the uid=0 inside the container is the same as uid=0 outside.

Considering root inside a container is root outside a container (for docker anyway), this can be destructive if Caddy is compromised. Security should not be compromised on such a level for a little convenience.

Just to clear up these two statements. Root in the container is not equivalent to root outside of the container. Yes they both share the same UID/GID, but the root user in the container is run in a namespace, there is isolation going on there, reduced capabilities is a good example of that.

Regarding the statement about compromising security for convenience, that's technically what you get with setcap approach vs having the application request the capability at runtime if it's available.

As the linked comment of mine notes, the setcap approach was being planned at one point to grant much more concerning capability to the binary which would not have been possible to opt-out of as a setcap workaround instead of at runtime.

---

I ended up here looking for why/when the setcap got introduced. Only after I found this issue did I realize it wouldn't have been as tricky to figure out if I had noticed the main Dockerfile.tmpl file 😑

For reference it was part of the 2.6.3 release onwards (would be nicer if the repo actually used git tags for releases since the commit would also be associated to the eventual tagged release).