Run rootless containers while running nomad as root

[optiz0r]

Hi all,

I'd like to be able to run a nomad job using "rootless" style containers while nomad itself is still running as root. That is to say, I'd like to be able to run the container with user namespacing enabled, and uid 0 within the container mapped to a chosen non-root uid on the host. Nomad itself might be called upon to run "rootless" containers on behalf of multiple other uids (which removes running nomad as a non-root uid from consideration).

This Podman driver implements the {{--user}} flag the same way as docker does, all processes within the container are squashed to run as the chosen uid, and the "rootless" functionality is lost (having multiple users within the container, uid 0 having sufficient privilege to switch user, change file owners etc). This is how we currently run docker containers under nomad, but its problematic in that containers that users are developing locally using non-root podman will not behave the same way when launched under nomad, and containers from the docker hub that require processes to run as specific uids (or use multiple uids internally) fail with filesystem permission errors and similar.

Do you think it would be possible to get the "rootless" behaviour using the podman v2 http api, either by having nomad itself drop privileges to the desired user before making the api request, or by passing additional options via the api to setup the user namespacing in a way equivalent to running podman run as the target uid? I see that some user namespacing options exist in the code but are currently commented out (https://github.com/hashicorp/nomad-driver-podman/blob/master/api/structs.go#L297).

Thanks,
Ben

[c-goes]

When nomad runs as root it could setup a systemd socket activation unit for the Podman HTTP API for the other user(s) and enable linger via loginctl. Probably this socket is accessible from root? Running podman cli from another user is a bit harder and requires a machinectl shell every time or you need to start dbus on your own.

[optiz0r]

That would be fine for my use case, even if I have to setup the socket activation and linger options myself. I have a constrained set of users that nomad should be able to invoke jobs as (enforced by hard-mandatory Sentinel policies).

[vlbeaudoin]

This feature would be ideal for our organisation!
If someone has had success implementing something like this i am very interested

[arunkchow]

I wanted to open a fresh issue but I saw this one. As per the Github page for podman-driver, rootless container running is supported. From what I read, Nomad needs to be run as root as it needs some system wide privileges. I tried using a non-root user under the task block with user = "rootless-username" and the jobs fail instantly. I created a non-root user as per Red Hat's documentation. I am able to run/manage the containers as rootless user I created. But no luck with nomad job. Just wondering if there's any way around this to get it working.

[harningt]

I imagine this could work if the nomad driver allowed configuring the podman socket on a per task basis (or did this behind the scenes).
Another option, if we didn't want the nomad users messing around with nomad client low-level podman details - allow for the socket path configuration to be made a map or perhaps a template string. I may try this out as a small experiment to get this going... otherwise my migration to using nomad vs a bunch of systemd config files is dead in the water...

[towe75]

@harningt i'd like to add a map of "logical name" => "/path/to/socket" to the driver options. Each task configuration would refer to such a socket name (think "root", "tenant-a", "tenant-b" etc.).

This would be a huge step for this driver and it's also a quite unique feature. But you can imagine that it's also a lot of work. I can't give you any target date whatsover because it's, after all, a community/volunteer project.