podman run --uidmap: fails on cgroups v1

[edsantiago]

Anything using --uidmap is failing on cgroups v1 with runc:

# podman run --uidmap 0:10001:10002 --rm --hostname BtoukoyxkBlQPjuQDYLbVeZzC quay.io/libpod/testimage:20220615 grep BtoukoyxkBlQPjuQDYLbVeZzC /etc/hosts
Error: runc: runc create failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount /proc/self/fd/11:/sys/fs/cgroup/systemd (via /proc/self/fd/12), flags: 0x20502f: operation not permitted: OCI permission denied
[ rc=126 (** EXPECTED 0 **) ]

[giuseppe]

I've launched manually that command in a Ubuntu 2204 VM configured with cgroupv1 with:

# sed -i -e 's/^GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0"/' /etc/default/grub
# upgrade-grub
# reboot

and I don't see that failure:

# podman --runtime runc run --uidmap 0:10001:10002 --rm --hostname BtoukoyxkBlQPjuQDYLbVeZzC quay.io/libpod/testimage:20220615 grep BtoukoyxkBlQPjuQDYLbVeZzC /etc/hosts
10.88.0.14      BtoukoyxkBlQPjuQDYLbVeZzC nostalgic_euclid

# podman --runtime /root/runc-upstream/runc run --uidmap 0:10001:10002 --rm --hostname BtoukoyxkBlQPjuQDYLbVeZzC quay.io/libpod/testimage:20220615 grep BtoukoyxkBlQPjuQDYLbVeZzC /etc/hosts
10.88.0.15      BtoukoyxkBlQPjuQDYLbVeZzC busy_thompson

# podman --cgroup-manager cgroupfs --runtime /root/runc/runc run --uidmap 0:10001:10002 --rm --hostname BtoukoyxkBlQPjuQDYLbVeZzC quay.io/libpod/testimage:20220615 grep BtoukoyxkBlQPjuQDYLbVeZzC /etc/hosts
10.88.0.16      BtoukoyxkBlQPjuQDYLbVeZzC keen_booth

# podman --cgroup-manager cgroupfs --runtime runc run --uidmap 0:10001:10002 --rm --hostname BtoukoyxkBlQPjuQDYLbVeZzC quay.io/libpod/testimage:20220615 grep BtoukoyxkBlQPjuQDYLbVeZzC /etc/hosts
10.88.0.17      BtoukoyxkBlQPjuQDYLbVeZzC objective_fermat

same results using Podman from the main branch.

Could be something in our images or the runc version used?

@cevich could you point me to how the image was created?

[edsantiago]

There's a script called hack/get_ci_vm that (if it works) can be used to get a CI VM that you can ssh into.

What I normally do these days is, if I have a PR that fails, go to the Cirrus log page, there's a button at top right that says Re-Run, it has an option to Re-run with an in-browser ssh session. It doesn't last long, but it works well enough. You could submit a PR that comments out my Skips, then when it fails, do that magic Re-run and poke in the system.

(Or, you can see if hack/get_ci_vm will work for you.)

[giuseppe]

Thanks! That was helpful.

It is an issue in the runc version installed in the Ubuntu image.

It is already fixed upstream with d370e3c04660201e72ba6968342ce964c31a2d7f

[edsantiago]

Thank you! I guess I know what I'm doing with my week.

[cevich]

Thanks @edsantiago and @giuseppe for tracking this down. Getting this fixed in our CI (finally) will be a significant improvement. There are plenty of people running podman in CGv1 environments. So having things like --uidmap not break for them is important. Knowing about these problems early in developers PR's will goes a long way toward increasing confidence in future released packages.

[edsantiago]

Anyone know where we stand on this? Has the magic version been updated on Ubuntu? Has anyone built VMs recently?

[cevich]

Most recent images I'm aware of are c5473162832904192 (two-days old IIRC). Maybe there are clues in quay.io/libpod/ubuntu_podman:c5473162832904192 (warning: 600+MB image):

||/ Name           Version        Architecture Description
+++-==============-==============-============-=================================
ii  runc           1.1.0-0ubuntu1 amd64        Open Container Project - runtime

So nothing there. The nuclear-option here is to just ask @lsm5 to roll a custom one for us as we've done for many years. He's been avoiding that (IIRC) b/c it's a PITA to maintain.

[edsantiago]

Never mind. No images built since July 27. Opened containers/automation_images#166 to see if Ubuntu is fixed (and to find out what new bugs appear)

[edsantiago]

Hi. Me again. I took a look at Ubuntu image built in containers/automation_images#178 . Nope, still the same broken runc. Carry on.

[cevich]

Nope, still the same broken runc. Carry on.

There should be another Ubuntu release coming up in October (22.10). Worth checking if there's a "beta" available and if it has the "right" runc version?

[edsantiago]

Not really. I care primarily about RHEL, secondarily about Fedora, and pretty much zero about Ubuntu. My experience with Ubuntu is that every time they fix one thing, they break two others, so every VM update is an ordeal. My preference, therefore, is to minimize VM updates.

[cevich]

OMG Ubuntu is a nightmare. I'd prefer to not care about it either, unfortunately, it's quite popular 😞 In any case, given your priorities, I'd suggest not even bothering to check until 22.10 comes out in our CI. I highly doubt we'll see runc change at all/ever in 22.04.

[giuseppe]

would it make sense to test cgroupv1 and runc onto another distro where we have more control (like Fedora or CentOS Stream)? We could restrict Ubuntu to the default configuration with crun and cgroup v2.

[edsantiago]

I would love to get away from Ubuntu; see #15337. And it's not just cgroups v1/v2, see #15360 (comment)

[rhatdan]

Would it make more sense to test on centos/stream 8 for V1 and runc, drop old versions of Ubuntu and only test crun/v2

[lsm5]

ATM, we're only testing the LTS (which is also the current latest) Ubuntu. /cc @cevich

[lsm5]

I guess I can push a new runc to Fedora and use debbuild to generate debian package as well. I don't want to add it to my autobuilder job but I can do runc updates on-demand. Would that work?

[giuseppe]

IMO we are not really testing Ubuntu if we use different packages. We end up doing the job without much advantage out of it.

[cevich]

IMO we are not really testing Ubuntu if we use different packages.

And THAT is the crux of the matter right there.

I also fear we'd end up in that same boat if we tried testing with CentOS, it's just "too old" for the bleeding-edge needs of upstream CI. Heck, we've even turned off F35 in CI despite it still being fully supported due to not wanting to deal with old crap.

So for the "how to test runc" issue which seems central here: I don't see any technical reason we couldn't add a runc item to some parts of our test matrix, it's easy enough to switch it at runtime (the package is typically already installed). Logistically it's a bit of a hassle on humans, simply given the number of tasks we're running in CI.

I can reduce the number of tasks presented, but the trade off is, more are bundled together which has it's own down-sides for flakes and re-running.

[edsantiago]

"Too old" may be exactly what we want: our goal, remember, is to test RHEL — or, since we can't test RHEL, a proxy.

[cevich]

No, assuming we're talking about upstream-CI, that puts us in exactly the same position. We wouldn't be testing an OS that a user would ever actually use - it becomes a franken' OS due to replacing vast numbers of (for example) bleeding-edge podman dependencies.

Big-picture architecture-wise, remember upstream CI is to catch low-hanging fruit problems as quickly as possible, before they make it downstream. That DOES NOT relieve downstream of the responsibility for additional integration testing, in THEIR USERS environment. In other words, we should not try to mix two different testing-contexts all within upstream. That road only leads to pain.

I would even go so far as to suggest, perhaps we shouldn't even be testing Fedora releases upstream, only rawhide. We've not gone there yet but I wouldn't consider it unreasonable.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

Abandoned. Should anyone other than me care about this, please reopen.