BUG: Problem with selinux/runc>=1.0.0-rc7

[h-vetinari]

After building podman from master for #2887, I encountered a bug that is either due to podman, selinux or runc. I'm starting to file here, because I don't know if this is actually a runc-problem, because I'm encountering it when running podman, and because the usual crowd here (esp. @rhatdan) seems to be active in all three projects anyway.

Assume we have a simple file test.dockrf:

FROM docker.io/library/ubuntu:bionic
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
    && apt-get install -y --no-install-recommends curl

CMD ["bash"]

Then, after building runc (and podman) from master as described in the tutorial, I get:

$ sudo podman build -f test.dockrf . --no-cache
STEP 1: FROM docker.io/library/ubuntu:bionic
STEP 2: ARG DEBIAN_FRONTEND=noninteractive
--> Using cache 2e22fdc5b41c5884242479a9aec8d4f3f8f5565ff7a091b39d40760ad34852f7
STEP 3: FROM 2e22fdc5b41c5884242479a9aec8d4f3f8f5565ff7a091b39d40760ad34852f7
STEP 4: RUN apt-get update     && apt-get install -y --no-install-recommends curl
selinux label is specified in config, but selinux is disabled or not supported
error running container: error creating container for [/bin/sh -c apt-get update     && apt-get install -y --no-install-recommends curl]: : exit status 1
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[apt-get update     && apt-get install -y --no-install-recommends curl] Flags:[] Attrs:map[] Message:RUN apt-get update     && apt-get install -y --no-install-recommends curl Original:RUN apt-get update     && apt-get install -y --no-install-recommends curl}: error while running runtime: exit status 1

I know there were some updates of runc regarding selinux recently, but it seems not everything is working yet (but then again, this might just be on the podman side). In any case, the container build succeeds for runc versions 1.0.0-rc4, 1.0.0-rc5, 1.0.0-rc6, but fails for 1.0.0-rc7, 1.0.0-rc8, and master.

The selinux config is from a vanilla azure RHEL machine:

$ cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

[rhatdan]

Did you build your own runc, without the "selinux" buildtag?

[h-vetinari]

@rhatdan: Did you build your own runc, without the "selinux" buildtag?

Yes I did, cf. above:

@h-vetinari: [...] after building runc (and podman) from master as described in the tutorial [...]

The exact build steps from the tutorial are:

git clone https://github.com/opencontainers/runc.git $GOPATH/src/github.com/opencontainers/runc
cd $GOPATH/src/github.com/opencontainers/runc
make BUILDTAGS="seccomp"
sudo cp runc /usr/bin/runc

This works with the runc-tags I indicated:

[...] the container build [i.e. podman build -f test.dockrf .] succeeds for runc versions 1.0.0-rc4, 1.0.0-rc5, 1.0.0-rc6, but fails for 1.0.0-rc7, 1.0.0-rc8, and master.

Each runc-version was built separately of course.

[rhatdan]

Should be

git clone https://github.com/opencontainers/runc.git $GOPATH/src/github.com/opencontainers/runc
cd $GOPATH/src/github.com/opencontainers/runc
make BUILDTAGS="selinux seccomp"
sudo cp runc /usr/bin/runc

[rhatdan]

Since not all platforms support SELinux runc version needs to be modified. Podman is using SELinux so it requires a runc that understands it.

[h-vetinari]

@rhatdan: Since not all platforms support SELinux runc version needs to be modified. Podman is using SELinux so it requires a runc that understands it.

Fair enough, I'll put that additional tag in the PR for updating the installation.

However, the build still does not succeed (now with having built runc from master, using make BUILDTAGS="selinux seccomp"):

$ sudo podman build -f test.dockrf --no-cache .
STEP 1: FROM docker.io/library/ubuntu:bionic
STEP 2: ARG DEBIAN_FRONTEND=noninteractive
--> e4ee292110da7b50baa2c12555987ca18fe65812567fbd62b7d4619a5caaf9bf
STEP 3: RUN apt-get update     && apt-get install -y --no-install-recommends curl
/bin/sh: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
Error: error building at STEP "RUN apt-get update     && apt-get install -y --no-install-recommends curl": error while running runtime: exit status 127

All variants of glib* are installed, and the linker should be able to find it:

$ which podman
/usr/bin/podman
$ ldd /usr/bin/podman
        linux-vdso.so.1 =>  (0x00007ffdfec90000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdfff4aa000)
        libgpgme.so.11 => /lib64/libgpgme.so.11 (0x00007fdfff277000)
        libassuan.so.0 => /lib64/libassuan.so.0 (0x00007fdfff065000)
        libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007fdffee60000)
        libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007fdffec1f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fdffe851000)
        /lib64/ld-linux-x86-64.so.2 (0x0000563afa9f7000)

If I then try to also build podman with make BUILDTAGS="selinux seccomp" (both install tutorials simply say make), I get a bunch of other problems to do with linking libostree-1.so.1. I got past that with some hackery, only to find myself back with the glibc error (even though I unset all environment variables I had used for that, I still cannot reproduce the behaviour anymore; I'm guessing some things are cached...)

$ sudo which podman
/bin/podman
$ sudo ldd /bin/podman
        linux-vdso.so.1 =>  (0x00007ffe827ba000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2a3e1c3000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f2a3df9c000)
        libostree-1.so.1 => /usr/local/lib/libostree-1.so.1 (0x00007f2a3dd12000)
        libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007f2a3d973000)
        libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007f2a3d723000)
        libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f2a3d40c000)
        libgpgme.so.11 => /lib64/libgpgme.so.11 (0x00007f2a3d1d9000)
        libassuan.so.0 => /lib64/libassuan.so.0 (0x00007f2a3cfc8000)
        libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f2a3cdc2000)
        libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007f2a3cb81000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f2a3c979000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f2a3c774000)
        libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x00007f2a3c51d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f2a3c150000)
        /lib64/ld-linux-x86-64.so.2 (0x000055dd367c6000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f2a3beed000)
        libgpgme-pthread.so.11 => /lib64/libgpgme-pthread.so.11 (0x00007f2a3bcba000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2a3ba94000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f2a3b87d000)
        libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f2a3b64c000)
        libffi.so.6 => /lib64/libffi.so.6 (0x00007f2a3b444000)
        libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f2a3b23f000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f2a3b026000)
        libmount.so.1 => /lib64/libmount.so.1 (0x00007f2a3ade3000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f2a3abcc000)
        libsepol.so.1 => /lib64/libsepol.so.1 (0x00007f2a3a92e000)
        libudev.so.1 => /lib64/libudev.so.1 (0x00007f2a3a717000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f2a3a415000)
        libcap.so.2 => /lib64/libcap.so.2 (0x00007f2a3a210000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f2a39ffa000)
        libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x00007f2a39d79000)
        libdw.so.1 => /lib64/libdw.so.1 (0x00007f2a39b2a000)
        libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f2a398e9000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f2a396e4000)
        libattr.so.1 => /lib64/libattr.so.1 (0x00007f2a394de000)
        libelf.so.1 => /lib64/libelf.so.1 (0x00007f2a392c6000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f2a390b6000)

So, after several hours of trying, the simple fact remains that building podman together with the latest runc does not work, but that things do work (albeit potentially with selinux bugs?) before runc-v1.0.0-rc7.

I'd appreciate some help here, because I'm a bit out of my depth here...

[rhatdan]

Is this an SELInux issue?
Does it work with
setenforce 0
If so then execute
restorecon -R -v /var/lib/containers

[h-vetinari]

@rhatdan
It seems to be an selinux issue indeed. After building both runc/podman with the build tags "selinux seccomp", I reproduced the above errors.
After running sudo setenforce 0 and sudo restorecon -R -v /var/lib/containers, the container build (see OP) succeeds.

[rhatdan]

Ok I am going to close this issue then.

[h-vetinari]

Weeeeeell, are you going to open an issue with selinux then? I'm preparing the PR for cleaning up the installation docs, and right now, the (corrected) build instructions (i.e. with the build tag you wanted) yield a broken installation on RHEL with selinux.

(and without the build tag the installation is also broken, unless runc is taken <1.0.0-rc7).

[rhatdan]

Well this is not caused by SELinux, your system got mislabeled somehow. Potentially from running containers with SELinux disabled, but I have no idea how. And there is little SELinux can do about a mislabeled system.

[h-vetinari]

your system got mislabeled somehow.

TBH, I don't see where there could be such a "somehow". I followed the install instructions on a fresh RHEL VM, and I can get a working system without deactivating selinux if building runc<1.0.0-rc7 without the selinux-buildtag.

You said above that runc however must be built with this tag, but then I end up with a broken build, somewhere in the interplay between selinux/runc/podman. Therefore, this issue is transparently not closed IMO, but maybe things will be easier to reason about once I get around to posting the PR for the documentation update.

[rhatdan]

Well see if you can get a repeatable failure.

[h-vetinari]

@rhatdan: Well see if you can get a repeatable failure.

As mentioned in #3068:

@h-vetinari: @rhatdan, following the install instructions for building on RHEL should give you a reproducible example for #3045...