UID and GID 999 are system IDs and are not to be used by non-default-distro processes such as container services #33
Comments
|
It is fine to use whatever UID/GID in the container. In a rootless container, the container itself can run as UID 0 (root) and be mapped to UID on the host that the host deems appropriate. Then within the container any volume mount has the UID mapped accordingly, which is probably the correct approach. Podman does this well with I haven't looked into what this image is doing, but it's not uncommon for containers that only need to persist data with a specific UID/GID to support making that runtime configurable. You could also use named volumes instead of bind mount volumes if you don't need direct file access in the host (should be fine for a DB). |
|
The problem is (was as I've fixed it) that the container uses UID 999 in the container and you don't document it on https://hub.docker.com/r/valkey/valkey and it, in my case, conflicted with the Normal, I would say sane, defaults for a container is to make it as non-intrusive/secure by default as possible and e.g. use the If it somehow is a requirement for the container to run with UID 999 then please add it to the documentation of the container so that I'm most likely not the only one that tries out the container and gets a UID overlap because of this. Not every test needs to be setup in a full blown lab that thinks of nothing but security. Changing the UID from 999 to 65534 (or there abouts) is the easiest solution to avoid getting a CVE because of overlapping UID's with the host OS system which can grant more access if somehow it was possible to break out of the container and having a valid UID to work with. |
|
Hey, Thank you so much for raising this issue and the information in this issue. To be frank it would be really helpful if you could raise a PR with a fix, that would also help other users to avoid getting a CVE. is this the fix we are looking for? valkey-container/Dockerfile.template Line 12 in ae6141e Also, previously redis container was also setup the same way, were we facing the same issue then? |
|
Might want to give this consideration:
NOTE: I've not investigated that myself, nor am I endorsing their suggestion to default to Quick overview of
I don't see how this is a mixup. It's perfectly normal with containers, I think you're just not familiar with that? If you insist that you know better, please cite images or resources that state this should/is the way it should be within containers (note that I cited links to various official Docker images that say otherwise). I understand your concern from a sysadmin perspective when you don't take into consideration how it is with containers, that threw me off when I first got into working with containers myself. The thing is a container can vary in base image, so the UID/GID assignment by distro is not always in alignment, even with common system users/groups like This UID/GID concern can vary across new releases of the same distro as well, but is more of an evident issue when an image installs packages that create users/groups, as the order can matter. If your base image or your own were to change packages in a future build that would shift that install order, it can affect the UID/GID of the next image release made, such that any existing users persisted storage outside of the image is no longer in alignment with the UID/GID values for your containers users/groups, requiring manual fix by each affected user (or sometimes containers apply this at runtime with an entrypoint script). Trying not to step on UID/GID used on the host system is a bit foolish as you will not know what these are. As the sysadmin for the containers running, any data you persist to the host storage is up to you with ensuring it's not mishandled, should the UID/GID conflict with assignment on the host that it causes a real problem (it rarely does in practice). Keep your volume data persisted to a common location on the host system where this boundary is a non-issue, alternatively deploy with rootless or user namespace remapping (solves your issue properly). There is also ID Mapped mounts which rootful containers can leverage (Podman supports this, while Docker doesn't officially yet, it can be done manually IIRC).
It's a system service, not one dependent upon a user session. Like the references I've provided above, they either explicitly create with
Citation needed for other official database images that are running as
I agree with you here, I wish images would document their UID/GID more visibly, especially when it's not configurable (without extending/customizing the image)
Besides the UID overlap, does it cause any actual problem in practice? I assume your concern is related to a container escape, similar to escape as a root user? If you're properly locking down the containers for security reasons, that really shouldn't be happening, you'll find escapes are reliant upon non-default capabilities being granted to the container The switch to a non-root user already results in all caps being dropped, thus the only time that becomes an issue is when the image modifies binaries with Other than that the most common other way to break out is via access to the Docker API (usually the socket), but that's an explicit mount (and one that is forbidden access to with SELinux enabled for the container IIRC). I have seen some images run as non-root but give that non-root user permission to use the socket within the container 🤦♂ If you're serious about security though, unless you need rootful containers, go for rootless (these have some limitations, but are usually fine). The whole UID/GID concern will be handled for you implicitly (at least it is with Podman IIRC). |
You shouldn't be assigning a new user/group to the existing UID/GID already assigned to Personally I prefer containers to run as root ( The main value of the switch is for the convenience of a non-root user dropping all caps, so that the sysadmin deploying the container doesn't have to explicitly do so (drop caps, or change the container to run as a different UID/GID). Something that shouldn't be an issue if the sysadmin deployed with rootless containers instead, but I understand the precaution is taken given the broad audience that often doesn't know any better. FWIW breakouts happen with non-root users too, and even if you are running in the container as If you want to go the extra mile, go get Valkey packaged as a slice for Canonical's Then if Valkey can run as the The sysadmin should ideally be able to use As I showed in my previous comment, Valkey is presently aligned with the same UID/GID as all other popular DB images. It would seem wise to be consistent there unless a proper discussion with the other projects can all agree on it being pragmatic to change away from that. Personally the reasoning from @bbruun doesn't seem like good enough justification IMO, and I hope I've made that rather evident as to why with this verbose response. |
I understand this and I'm fully aware of this, but the valkey container just hit a ... nerve that overlapped with RHEL's systemd setup which the others don't (or haven't as of yet). But just because "everyone else is doing it" then it does not make it the correct way or safe way or secure way to do it.
I am familiar with it (see my last sentence about my situation). A few sources: From https://linuxhandbook.com/uid-linux/ RHEL that I'm on is in the Fedora family...
I'm fully aware from the first
Yes, hence the best practices to not create users in containers that are in well know UID/GID ranges for system users or accounts to specifically avoid that kind of scenario and
I know - but since the container uses
True yet very avoidable by not using well known UID/GID ranges that are well know for system user and accounts.
A container is not a system service.
I don't have any citations. It is based on old introduction docs from Dockers site back in the day, and almost all other "get started with containers" documentation. The main issue here is that if you install Valkey/Redis/Postgres/Mongo etc. etc. from the package manager then it will create a user (and often group) using adduser/addgroup which will not overlap with any system users or accounts but in the container this is not possible as it is hardcoded.
Agreement is a nice thing :-)
No - which is why I've not made a PR to "fix" the issue "for me" but only asked about it/make the maintainers aware of the problems with using low UID/GIDs, which I hope is OK?
Agree. But for testing out a container and the documentation or Docker Hub "example" does not document this then that is not something to take into account for testing - hence this issue.
Agree. But that is outside the scope of the low hardcoded UID/GID in the container.
I know, but it is often better to be safe than sorry... hence I would guestimate that 99% of all security thinking is about what could happen vs how little the chance of it actually happening. Better safe than sorry.
I agree, but this was found in a test of the container on a test RHEL server. So not in scope to see if using the container is valid choice vs manual installation (I'm thinking ahead in regards to upgrades mostly). Podman is not something my colleagues like - they have trouble enough understanding and accepting the use of containers to begin with as they only see issues with them vs normal package managed applications that they have been using for decades. I'll get them there, but I can't make them do 1000 new things that they don't understand from day 1, I have to do it doucement even if it means using Docker and its root daemon. |
Because it is (was) an inconvenience for me when I had the user overlap but not critical for running the container.
I know how to fix it - that is not the problem. I would add 2 variables and have the docker-entrypoint.sh script do a usermod and chown before running valkey-server to fix it so it is possible to run it with custom a UID/GID.
I didn't use the Redis container - migrating away from AWS version to on-prem and I was looking at alternatives and Valkey seems to be the best choice. And using the container is mostly to test out upgrades later on vs using a package manager or manual install. |
|
TL;DR: Apologies for verbosity, I'm short on time.
Could you please clarify with a reproduction? Are you certain this is what you think it is and not an XY problem? For example these DB images all use That can cause a variety of mishaps if you're not careful. You shouldn't be proposing a fix with a UID/GID change, when that's not reproducible for you on other similar images using that same UID/GID. Instead it's better to understand the difference between the images (or what you did) to cause the underlying failure scenario itself. Adjusting the UID/GID may have "fixed" it, but that's not necessarily the correct solution... you've tried connecting some dots, but from what you've described the same problem could occur if both images changed to the same UID/GID values and you repeated the steps, it may have nothing to do with the existing host assignment. Where I do agree with a change is for consistency. This is not ok: valkey-container/8.1/alpine/Dockerfile Lines 7 to 13 in 796eef7 valkey-container/8.1/debian/Dockerfile Lines 7 to 12 in 796eef7 Image variants should be compatible with their UID/GID for the containerized service. However changing them will also impact all existing users of the images, that is a breaking change. I haven't reviewed the image in full, so the
Regarding OWASP advice, privilege escalation attacks can happen as non-root users but switching away from the root user will drop all capabilities to the container user implicitly for the sysadmin so that they don't have to, and it minimizes some damage in the event a user escapes as that user (there are privilege escalation attacks in which they can become root regardless). I have seen some image authors blindly follow such advice, but do so poorly by implementing workarounds that reduce security via Yes, Non-system users are those with login shells and intended for actual user sessions. The kernel has some features like: # Default is 1024, Docker runs this with it set to 0 implicitly,
# as it's reasonably safe in the container context:
sysctl net.ipv4.ip_unprivileged_port_start=80
# It wasn't the case with other container engines for some time though,
# So some images would instead grant their non-root program this capability
# to ignore the security restriction (as it would apply for root)
#
# NOTE: the `+e` enforces this by the kernel preventing the program from running
# if the capability were denied by the sysadmin, even when the program is configured
# to bind to a port above 1024 which would have otherwise been valid..
setcap 'cap_net_bind_service=+ep' /path/to/programAn unprivileged user can be a system user btw, a UID of 1000+ can also be privileged in the sense of being granted ambient capabilities (Docker does not support this AFAIK, systemd does however), rather than a program/process being granted capabilities to it's permitted set (
valkey-container/docker-entrypoint.sh Lines 10 to 14 in 796eef7 Ah alright, yeah that resolves the ownership concern provided no other separate tooling/images are expecting the However for a common ENV configurable I see across images with Off-topic: That conditional would look better like (works with both bash and ash, which if [[ "$1" == 'valkey-server' && "$(id -u)" == '0' ]]; then
Sorry, doesn't make sense to me here. For clarity so that we're on the same page, when you've explicitly mounted a volume to the host, anything written to that location is where your concern is? Otherwise it's a bit bizarre to expect no conflict in UID/GID with containers and the host, base images differ here by distro, and for each image on those base images, any further system packages installed will shuffle new UID/GID assignment accordingly, you can't do much about that beyond choose fixed UID/GID in advance (I've had to do this for ClamAV with it's DB for example so that it has a stable ownership across image upgrades).
I think you have a misunderstanding with the relation of privilege to capabilities, not UID/GID ownership?
Again... you can't justify that way when it's not consistent across distros 🤷♂ This UID/GID doesn't exist in Fedora by default unless systemd is installed (for desktop/server ISOs that's usually a given). $ docker run --rm -it fedora:42
# Nothing:
$ cat /etc/passwd | grep coredump
# Install systemd:
$ dnf install -y systemd
$ cat /etc/passwd | grep coredump
systemd-coredump:x:998:998:systemd Core Dumper:/:/usr/sbin/nologin
$ cat /etc/group | grep coredump
systemd-coredump:x:998:
# This is what got assigned 999 instead:
$ cat /etc/passwd | grep oom
systemd-oom:x:999:999:systemd Userspace OOM Killer:/:/usr/sbin/nologin
$ cat /etc/group | grep oom
systemd-oom:x:999:Now the UID is 998, while for you it's 999... Does Fedora need to "fix" this now? No, it's no different to using a VM or migrating data from another OS install (even if it's the same one where UID/GID were mismatched due to implicit assignment of UID/GID and package install time).
The container is a sandbox (namespace) that runs one or more processes, those can be services as they would be outside of a container, why you want to make a distinction here I do not know. If you need isolation from host UID/GID values, go with rootless or related rootful features for remapping. Volumes are only allowed to write to disk what it's permitted to on the host, a non-issue when the container is rootful, or when you use rootless containers with user namespace remapping ( So no it's not really the volumes that are the problem... it's the way a user chooses to run the image, and how the image author approaches it in their image. If you just use root in the container, it's not really a problem is it, except for the time before rootless containers were available which is why we have all this "best practice" advice to run containers as non-root users internally (similar to Push for the I'm not sure what you are on about with Podman disadvantage... it's daemonless, unlike Docker. If you want rootful container, then run Podman commands as root?
Perhaps it was removed from the docs for a reason then if it was previously suggested? I don't see how all images using
The system packages avoid the conflict for a good reason, but I really don't see it being an issue with containers. By that logic, containers shouldn't have root
The hard-coding has no relevance...? Without that, it's not going to read the hosts users and groups, that'd be very bad. FWIW, the official Docker docs encourage pinning explicit UID/GID, and they warn about large UID/GID values (for For rootless containers, this also affects the range assignment (you allocate 2^16 sub-UID/GID for a host user).
Yes questions are great, just a reminder that I'm not a maintainer of this image (I recall your first reply to me might have mistaken me for one). BTW, I mean no disrespect in my responses where I'm potentially over-explaining things you already know as a sysadmin, but I have enough experience as a sysadmin and with containers that I feel I can weigh in that this sounds like an XY problem with potential knowledge gaps on your end for the more niche aspects. You do seem rather knowledgeable but something seems off with the reported problem (errors and access) only affecting Valkey.
Locking down capabilities would be more advanced security practice. Using non-root users as default in images or adding the support for such is usually to benefit the majority audience that doesn't have a good grasp on such things and thus rely upon "best practice" advice from resources they trust. As such you won't see that sort of guidance in specific images READMEs. I disagree with the practice of rootful containers using non-root users due to various caveats that can bring, the default capabilities for container root are fine as-is, but I understand the precaution, especially for projects that just want to offer Docker / Containers as a deployment option but otherwise don't have much of a handle on this sort of security that well either (I've seen enterprise grade, well funded projects that are open-source that don't implement things on their end properly too, so you can imagine why such practices are prevalent). Docs wise, the image should at least communicate the UID/GID it uses when deviating from root as default.
Apologies, I'm more focused on the UID/GID concern in general, rather than specific to this image. In that sense I consider it relevant context and in scope, should anyone arrive at this issue to go over the pro/cons of what is being discussed.
Fair point.
Sorry, I don't follow the concern here? If you're evaluating a container vs host install, and your concern is the container writes volume data to the host with a UID/GID already assigned to something else, why wouldn't ID mapped volumes or rootless containers make sense? For context, this specific image may only have the one UID/GID mapping concern so your
FWIW Docker does have rootless too, and if this is mostly a concern on the developers desktop systems, Docker Desktop while rootful is effectively rootless in the sense of it using a VM to manage Docker. You can use that on Linux too, which still provides the docker CLI. I mention this because Docker Desktop also has ECI (enhanced container isolation), should you need even stricter security requirements (this will add friction should a container require access to the docker socket). I understand the choice to go with Docker for users and to an extent with developers. If your colleagues are sysadmins and already comfortable with Systemd however, Podman Quadlets is the Docker Compose equivalent in Podman but with systemd units (they use a generator service to extend config with some Podman/Container specific metadata settings, but the generator outputs a standard systemd service). The difference between rootful and rootless then becomes system or user scope systemd services based on standard systemd config locations 👍 ( |
I was trying out the docker image and setting up docker-compose and kept getting errors about access. That OS distro is RHEL 8.
It appears that "you" use UID 999 (and 1000 for some reason) and GID 999 for the service inside the docker image.
That overlaps with the default <1000 is system user and groups only.
You set the UID/GID's here: https://github.com/search?q=repo%3Avalkey-io%2Fvalkey-container%20999&type=code
On e.g. RHEL 8 then
systemd-coredumpuser managed and installed by systemdinputgroup (I don't know which one uses it on a server).Could you fix this mixup of valkey being a system service maintained by the distros and make the UID and GID larger than 1000 which is also the advised method for non system accounts. E.g. 65535 for both would be ideal as it is the default "nobody" group on systems.
The text was updated successfully, but these errors were encountered: