Missing FCAP I fail to install httpd into Fedora 33 container in build-using-dockerfile

[praiskup]

With this dockerfile (on Fedora 34):

FROM fedora:33
RUN dnf install -y httpd

I run buildah bud ., and I see:

  Installing       : httpd-2.4.46-9.fc33.x86_64                         187/207
Error unpacking rpm package httpd-2.4.46-9.fc33.x86_64
 
error: unpacking of archive failed on file /usr/sbin/suexec;604723c1: cpio: cap_set_file failed - Inappropriate ioctl for device
error: httpd-2.4.46-9.fc33.x86_64: install failed
...
Failed:
  httpd-2.4.46-9.fc33.x86_64                                                    

Error: Transaction failed

Output of rpm -q buildah or apt list buildah:

buildah-1.20.0-0.12.dev.git7f340f9.fc34.x86_64

Output of buildah version:

Version:         1.20.0-dev
Go Version:      go1.16beta1
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.10.1
Git Commit:      
Built:           Thu Jan  1 01:00:00 1970
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.16
Built:        Mon Feb 22 15:08:57 2021
OS/Arch:      linux/amd64

Output of cat /etc/*release:

Fedora release 34 (Thirty Four)
NAME=Fedora
VERSION="34 (Thirty Four Prerelease)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora 34 (Thirty Four Prerelease)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/34/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
Fedora release 34 (Thirty Four)
Fedora release 34 (Thirty Four)

Output of uname -a:

Linux raiskup 5.11.3-300.fc34.x86_64 #1 SMP Thu Mar 4 19:03:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

$ cat /etc/containers/storage.conf  | grep -v -e ^# -e ^$
[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
[storage.options.overlay]
mountopt = "nodev,metacopy=on"
[storage.options.thinpool]

[praiskup]

Similar older issue.

[praiskup]

FTR, this regressed back at least 3 times during last two years. It would be awesome if I could
help you to install some test-case for this (install the httpd package to the latest stable Fedora
container).

[praiskup]

Also, using the podman run --rm -ti fedora:33 environment - I can install the httpd package,
so that gives me CAP_SETFCAP. Using buidah bud --cap-add=CAP_SETFCAP has no effect.

[praiskup]

I tried to update skopeo and container-common:
sudo dnf update https://kojipkgs.fedoraproject.org//packages/skopeo/1.2.2/22.fc34/x86_64/skopeo-1.2.2-22.fc34.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/containers-common/1/11.fc34/noarch/containers-common-1-11.fc34.noarch.rpm

From the blocked update:
https://bodhi.fedoraproject.org/updates/FEDORA-2021-9e76477662

And neither that helped.

[praiskup]

Fixed by (manually built in mock fedora-34-x86_64) buildah-1.20.0-0.26.dev.gitfd48180.fc34.x86_64 from rawhide.

[praiskup]

Ok, no, I'm lost :-( I built the image only once today so I thought it is fixed - but no. I can't built it again. I can't swear I haven't hit a cached layer before on successful build, but I think it is unlikely because I would notice that (it rather appears to be some randomly appearing problem, as before in the old issue podman#5364).

[jlebon]

We're also hitting this regression in FCOS on f34+: coreos/fedora-coreos-config#883. The test script is here: https://github.com/coreos/fedora-coreos-config/blob/8dbd6fa4878ec0b9a07bc035d05df0ed5bc39997/tests/kola/podman/rootless-systemd.

[rhatdan]

The regression is probably in containers.conf. Can you compare /usr/share/containers/containers.conf on F34 to f33?

[rhatdan]

Specifically on CAP_SETFCAP.

[praiskup]

The option is there, I checked before ... same as I tried --cap-add=CAP_SETFCAP explicitly.
I don't have manual edits in that config file, and the only difference is that in the default F34
config there's a missing [engine.volume_plugins] section (which is anyway empty on F33).

[rhatdan]

@giuseppe Ideas?

[rhatdan]

Anything in /var/log/audit/audit.log for SELinux or SECCOMP?

[giuseppe]

could also be related fuse-overlayfs.

Is this running as root or rootless?

[jlebon]

Is this running as root or rootless?

In the case of FCOS, rootless. I linked to the exact test script used there (you should be able to just run that script directly in an F34 system).

[praiskup]

Anything in /var/log/audit/audit.log for SELinux or SECCOMP?

Nothing obvious.

Is this running as root or rootless?

rootless

[giuseppe]

could also be related fuse-overlayfs.

I could reproduce without fuse-overlayfs, so it should be something related to capabilities

[rhatdan]

Could be a kernel issue?

[giuseppe]

possibly. I am still troubleshooting what is going wrong, podman and buildah behave differently

[giuseppe]

it works if I run a "podman run" and attempt the dnf install from there

[giuseppe]

it looks like a regression in the kernel, simple reproducer:

Fedora 33:

$ uname -a
Linux lithium 5.10.14-200.fc33.x86_64 #1 SMP Sun Feb 7 19:59:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ unshare -r unshare -r sh -c 'touch /tmp/setxattr-test; setcap "cap_setuid=ep" /tmp/setxattr-test' && echo ok
ok

Fedora 34

$ uname -a
Linux localhost.localdomain 5.11.3-300.fc34.x86_64 #1 SMP Thu Mar 4 19:03:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ unshare -r unshare -r sh -c 'touch /tmp/setxattr-test; setcap "cap_setuid=ep" /tmp/setxattr-test' && echo ok
Failed to set capabilities on file `/tmp/setxattr-test' (Invalid argument)
usage: setcap [-h] [-q] [-v] [-n <rootid>] (-r|-|<caps>) <filename> [ ... (-r|-|<capsN>) <filenameN> ]

 Note <filename> must be a regular (non-symlink) file.
 -r          remove capability from file
 -           read capability text from stdin
 <capsN>     cap_from_text(3) formatted file capability

 -h          this message and exit status 0
 -q          quietly
 -v          validate supplied capability matches file
 -n <rootid> write a user namespace limited capability
 --license   display the license info

The same issue happens on Rawhide

[rhatdan]

Can you open a bugzilla on this?

[ebiederm]

it looks like a regression in the kernel, simple reproducer:

Fedora 33:

$ uname -a
Linux lithium 5.10.14-200.fc33.x86_64 #1 SMP Sun Feb 7 19:59:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ unshare -r unshare -r sh -c 'touch /tmp/setxattr-test; setcap "cap_setuid=ep" /tmp/setxattr-test' && echo ok

Fedora 34:

$ uname -a
Linux localhost.localdomain 5.11.3-300.fc34.x86_64 #1 SMP Thu Mar 4 19:03:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ unshare -r unshare -r sh -c 'touch /tmp/setxattr-test; setcap "cap_setuid=ep" /tmp/setxattr-test' && echo ok
Failed to set capabilities on file `/tmp/setxattr-test' (Invalid argument)

I would agree this appears to be a kernel regression. If I am reading the reproducer correctly the change was deliberate
so I want to ask are we doing such silly things deliberately.

It was observed that under certain circumstances it is possible to set a file capability that works as a file capability
in an ancestor user namespace of the user namespace the file capability was set it. This happens if two user namespaces
have the same uid for their root users so the v3 file capabilities can not tell them apart.

Unless I am mistake that is exactly what is happening in the reproducer posted above.

Are you deliberately trying to create a file capability that works outside of the user namespace that created it?

I am trying to figure out if we want to treat this as a security enhancement or a regression. A consequence of a security enhancement is that we would ask you not to nest user namespaces with the same root user if you care about file capabilities.

If it really makes sense to do such nesting then we can treat this as a regression and revert the change.

I believe the commit that caused this in the kernel is:
95ebabde382c ("capabilities: Don't allow writing ambiguous v3 file capabilities")

Can someone revert that commit and verify that it fixes the original issue?

[giuseppe]

thanks for helping us.

I've tried reverting 95ebabde382c ("capabilities: Don't allow writing ambiguous v3 file capabilities") and that solves the issue I've seen.

With rootless containers, we always create an outer user namespace where the unprivileged user is mapped to 0 and any additional ID from /etc/sub?id is mapped starting at 1.
All the containers/user namespaces are created inside the outer namespace. We need this for a couple of reasons: 1) setting up the storage with multiple IDs available before launching the container 2) have a way to share namespaces among containers, e.g. containers in a pod share the network namespace.

From the outer namespace, a pod/container can be confined further with its own user namespace and sometimes it is useful that the root in the inner namespace is mapped back to the initial unprivileged user.

There is also the case when the unprivileged user has no access to additional IDs, so the outer namespace is limited to have just one ID mapped. Even in this configuration it could be useful to create sub-user namespaces. If each container runs in their own user namespace, they won't own or have access to namespaces used by other containers.