Mounting /sys/kernel

[avikivity]

In a container:

# ls -l /sys/kernel/
total 0

I'd like to run some tracing in a (privileged) container, but I can't mount /sys/kernel (even with a direct -v argument). Is there a way to do it?

[mheon]

Can you try --security-opt unmask=ALL when creating the container?

[rhatdan]

--security-opt unmask=/sys/kernel

Should also work, as I understand it.

[avikivity]

Both work, but fail as soon as I add --network host.

[avikivity]

$ podman run --network host --rm --security-opt unmask=ALL docker.io/fedora:34 ls /sys/kernel
$ podman run  --rm --security-opt unmask=ALL docker.io/fedora:34 ls /sys/kernel
boot_params
btf
cgroup
config
debug
dmabuf
fscaps
iommu_groups
irq
kexec_crash_loaded
kexec_crash_size
kexec_loaded
livepatch
mm
notes
profiling
rcu_expedited
rcu_normal
reboot
security
slab
software_nodes
sunrpc
tracing
uevent_seqnum
vmcoreinfo

[debarshiray]

This works in Toolbx containers:

[rishi@svoboda ~]$ toolbox enter
⬢[rishi@toolbox ~]$ ls /sys/kernel/
boot_params  fscaps              livepatch      reboot          uevent_seqnum
btf          iommu_groups        mm             security        vmcoreinfo
cgroup       irq                 notes          slab
config       kexec_crash_loaded  profiling      software_nodes
debug        kexec_crash_size    rcu_expedited  sunrpc
dmabuf       kexec_loaded        rcu_normal     tracing

So, you might have to turn a few more knobs. Maybe run podman inspect --type container ... against a Toolbx container and look for CreateCommand in the JSON for hints?

[avikivity]

toolbox looks interesting, I think I'm more or less emulating it with my giant script.

However, it doesn't like me:

$ toolbox create --image docker.io/scylladb/scylla-toolchain:fedora-34-20220522
Created container: scylla-toolchain-fedora-34-20220522
Enter with: toolbox enter scylla-toolchain-fedora-34-20220522
$ toolbox enter scylla-toolchain-fedora-34-20220522
Error: invalid entry point PID of container scylla-toolchain-fedora-34-20220522

[debarshiray]

What does this say:

$ podman start --attach scylla-toolchain-fedora-34-20220522

[avikivity]

podman start --attach scylla-toolchain-fedora-34-20220522
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="Migrating to newer Podman"
level=debug msg="Setting up configuration"
level=debug msg="Setting up configuration: file /etc/containers/toolbox.conf not found"
level=debug msg="Setting up configuration: file /root/.config/containers/toolbox.conf not found"
level=debug msg="Resolving image name"
level=debug msg="Distribution (CLI): ''"
level=debug msg="Image (CLI): ''"
level=debug msg="Release (CLI): ''"
level=debug msg="Resolved image name"
level=debug msg="Image: 'fedora-toolbox:34'"
level=debug msg="Release: '34'"
level=debug msg="Resolving container name"
level=debug msg="Container: ''"
level=debug msg="Image: 'fedora-toolbox:34'"
level=debug msg="Release: '34'"
level=debug msg="Resolved container name"
level=debug msg="Container: 'fedora-toolbox-34'"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/Asia/Jerusalem"
level=debug msg="Creating regular file /etc/machine-id"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating directory /run/libvirt"
level=debug msg="Binding /run/libvirt to /run/host/run/libvirt"
level=debug msg="Creating directory /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating directory /run/systemd/resolve"
level=debug msg="Binding /run/systemd/resolve to /run/host/run/systemd/resolve"
level=debug msg="Creating directory /run/udev/data"
level=debug msg="Binding /run/udev/data to /run/host/run/udev/data"
level=debug msg="Creating directory /tmp"
level=debug msg="Binding /tmp to /run/host/tmp"
level=debug msg="Creating directory /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating directory /var/lib/libvirt"
level=debug msg="Binding /var/lib/libvirt to /run/host/var/lib/libvirt"
level=debug msg="Creating directory /var/lib/systemd/coredump"
level=debug msg="Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump"
level=debug msg="Creating directory /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating directory /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Looking up group for sudo"
level=debug msg="Group for sudo is wheel"
level=debug msg="Modifying user avi with UID 1000:"
level=debug msg=usermod
level=debug msg=--append
level=debug msg=--groups
level=debug msg=wheel
level=debug msg=--home
level=debug msg=/home/avi
level=debug msg=--shell
level=debug msg=/bin/bash
level=debug msg=--uid
level=debug msg=1000
level=debug msg=avi
level=debug msg="Removing password for user avi"
Error: failed to remove password for user avi: passwd(1) not found

[debarshiray]

This explains the failure:

Error: failed to remove password for user avi: passwd(1) not found

Toolbx uses passwd(1) to set up sudo(8) inside the container to offer a smoother interactive command-line experience. Would it be possible to add that to the scylla-toolchain-fedora-34 image?

It would be best if the scylla-toolchain-fedora-34 image was derived from the fedora-toolbox:34 image. Failing that, another option is to include at least the toolbox-support RPM in the image. There's also a toolbox-experience RPM that offers a more complete user experience, equivalent to deriving from the fedora-toolbox image.

[avikivity]

sudo is installed in the toolchain image.

[avikivity]

ah, but passwd is not. I typically bind-mount /etc/passwd: https://github.com/scylladb/scylla/blob/adda43edc75b901b2329bca8f3eb74596698d05f/tools/toolchain/dbuild#L147

[debarshiray]

The entry point of a Toolbx container does a passwd --delete root to wipe out the root password so that one can use sudo(8) without any hassle.

Since root inside a rootless container isn't the same as root on the host, and is essentially some random user, this seems to have worked out OK so far.

I can't say off-hand what the trade-offs would be, if we decided to bind mount /etc/passwd and such, instead of using useradd and usermod. I would have to take a closer look at the code, the Git history and scratch my head several times for that. :)

[debarshiray]

@avikivity were you able to do your tracing with /sys/kernel in the end? I am just curious where you ended up.