Nesting, podman inside podman, QEMU VM with cloud image #7
Comments
Cgroups v1 vs v2The best explanation, the one from its creators: Kubernetes Deconstructed: Understanding Kubernetes by Breaking It Down - Carson Anderson, DOMO, start=1473&end=1525 About the freezer cgroup and kvm, Understanding and Working with the Cgroups Interface - Michael Anderson, The PTR Group, LLC, start=1179&end=1191
Containers unplugged: Linux namespaces - Michael Kerrisk Quotes
--cgroup-manager=cgroupfs|systemd --cgroups=enabled|disabled|no-conmon|split Check it: Commands to test v1 vs v2stat -c %T -f /sys/fs/cgroupRefs.:
readlink -f /proc/$$/ns/*Refs.: ls -ahl /sys/fs/cgroup/cgroup.controllersRefs.: cat /sys/fs/cgroup/cgroup.controllers | tr ' ' '\n' | sortRefs.: stat -f -c %t /sys/fs/cgroup 2> /dev/nullls -al /proc/self/nsFrom: https://stackoverflow.com/a/46450691 systemctl --user status dbus.socket | rg activeFrom: containers/podman#5443 (comment) mount | grep '^cgroup' | cut -d ' ' -f 1 | uniqYour kernel may support both cgroup and cgroup2! $ mount | grep '^cgroup' | cut -d ' ' -f 1 | uniq
cgroup2
cgroupFrom: https://unix.stackexchange.com/a/619682 mount -t cgroup2stat -c %T -f $(mount -t cgroup2 | cut -d' ' -f3)In my NixOS: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)In a QEMU + KVM VM running from inside a container running from podman rootless: cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)In a Vagrant Ubuntu 22.04 LTS VM: cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)Does this work? It returned the same output in my tests. grep cgroup /proc/filesystemsFrom: https://unix.stackexchange.com/a/508826 grep ^cgroup /etc/mtabFrom: https://unix.stackexchange.com/a/657535 podman \
run \
--cgroup-manager=cgroupfs \
--log-level=error \
--interactive=true \
--tty=true \
--rm=true \
docker.io/library/alpine:3.14.0 \
sh \
-c \
'echo Hello world!'podman \
run \
--storage-driver=vfs \
--log-level=debug \
--interactive=true \
--tty=true \
--rm=true \
docker.io/library/alpine:3.14.0 \
sh \
-c \
'echo "Hello world!"'From: containers/podman#6667 (comment) TODO: study this and test. --storage-driver overlay \From: containers/podman#3024 (comment), official documentation Configure timezone in a container --cgroups=disabled
--storage-opt "overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs"podman \
run \
--interactive=true \
--memory-reservation=200m \
--memory=300m \
--memory-swap=400m \
--rm=true \
--tty=true \
docker.io/library/alpine:3.14.0 \
echo \
'Hi!'--cgroups=disabledFrom: https://bugzilla.redhat.com/show_bug.cgi?id=1732957#c50 podman \
run \
--security-opt=seccomp=unconfined \
--log-level=error \
--interactive=true \
--tty=true \
--rm=true \
docker.io/library/alpine:3.14.0 \
sh \
-c \
"echo 'Hello world!'"podman \
run \
--device=/dev/fuse:rw \
--log-level=error \
--interactive=true \
--privileged=true \
--tty=true \
--rm=true \
--userns=host \
--user=0 \
docker.io/library/alpine:3.14.0 \
sh \
-c \
'echo "Hello world!" && ls -l /dev/fuse'Checking for kernel parametersFact 1: There is an kernel configuration ( Fact 2: Each distribution puts it in different places, https://superuser.com/a/287372 sudo cat /proc/config.gz | gunzip - | grep CONFIG_USER_NS
sysctl user.max_user_namespaces
unshare -Ur cat /proc/self/uid_map
zgrep CONFIG_USER_NS /proc/config.gzBOOT_CONFIG_PATH="/boot/config-$(uname -r)"
test -f "${BOOT_CONFIG_PATH}" && zgrep USER_ "${BOOT_CONFIG_PATH}"https://unix.stackexchange.com/a/483108 BOOT_CONFIG_PATH="/boot/config-$(uname -r)"
test -f "${BOOT_CONFIG_PATH}" && cat "${BOOT_CONFIG_PATH}"cat /proc/kallsyms | awk '/\[/ {print $NF}' | sort -uRefs: Checking stuffIn Ubuntu 22.04: zgrep CONFIG_USER_NS /boot/config-$(uname -r)cat /etc/sysctl.d/*-default.conf | wc -lsudo sysctl -anix shell nixpkgs#criu --command sh -c 'sudo $(which criu) check'
nix run nixpkgs#k3s check-config
nix shell nixpkgs#libvirt --command virt-host-validateenv | sortmount | sortlocale -a | sort |
|
Example source:
nix flake metadata nixpkgsResolved URL: github:NixOS/nixpkgs
Locked URL: github:NixOS/nixpkgs/56fadd63bdb77e16049d60d83d6b66c36d5ca8b0
Description: A collection of packages for the Nix package manager
Path: /nix/store/1jnpi7c0jkl5zipkx1c5d7rk296lai2m-source
Revision: 56fadd63bdb77e16049d60d83d6b66c36d5ca8b0
Last modified: 2021-08-17 02:47:25
Inputs:nix \
profile \
install \
github:ES-Nix/podman-rootless/from-nixpkgs#podmanpodman infohost:
arch: amd64
buildahVersion: 1.21.0
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: Unknown
path: /nix/store/327d4f50nkajvzg8chq35pkkdk7bz7w4-conmon-2.0.29/bin/conmon
version: 'conmon version 2.0.29, commit: '
cpus: 4
distribution:
distribution: ubuntu
version: "18.04"
eventLogger: journald
hostname: ubuntu
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 165536
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 165536
size: 65536
kernel: 4.15.0-153-generic
linkmode: dynamic
memFree: 2555105280
memTotal: 5193162752
ociRuntime:
name: crun
package: Unknown
path: /nix/store/fzy7y5xiarvxlya8q4cgmjlq9hcp95sy-crun-0.20.1/bin/crun
version: |-
crun version 0.20.1
commit: 0.20.1
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPT
rootless: true
seccompEnabled: true
seccompProfilePath: ""
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /nix/store/d8c1cmcaa3jkf7r7s3sxal5i571n2qrq-slirp4netns-1.1.11/bin/slirp4netns
package: Unknown
version: |-
slirp4netns version 1.1.11
commit: 368e69ccc074628d17a9bb9a35b8f4b9f74db4c6
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.1
swapFree: 0
swapTotal: 0
uptime: 2h 45m 39.02s (Approximately 0.08 days)
registries:
search:
- docker.io
store:
configFile: /home/ubuntu/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /nix/store/xmq9shra3jdv7fyh1zmsx2qpqp1hxinz-fuse-overlayfs-1.6/bin/fuse-overlayfs
Package: Unknown
Version: |-
fuse-overlayfs: version 1.6
FUSE library version 3.10.4
using FUSE kernel interface version 7.31
graphRoot: /home/ubuntu/.local/share/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 5
runRoot: /run/user/1000/containers
volumePath: /home/ubuntu/.local/share/containers/storage/volumes
version:
APIVersion: 3.2.2
Built: 315532800
BuiltTime: Tue Jan 1 00:00:00 1980
GitCommit: ""
GoVersion: go1.16.5
OsArch: linux/amd64
Version: 3.2.2If this podman \
stats \
--cgroup-manager=systemdoutputs this
and this nix shell nixpkgs#bashInteractive nixpkgs#coreutils nixpkgs#mount nixpkgs#gnugrep
mount | grep '^cgroup' | cut -d ' ' -f 1 | uniqshows cat > Containerfile << _EOF
FROM registry.fedoraproject.org/fedora:32
RUN dnf -y install systemd httpd
RUN systemctl enable httpd
EXPOSE 80
CMD [ "/sbin/init" ]
_EOF
podman \
build \
--file=Containerfile \
--tag=httpd . \
&& podman \
run \
--interactive=true \
--tty=true \
--rm=true \
--name=httpd \
httpdfail with the following message: systemd v245.8-2.fc32 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID)
Detected virtualization container-other.
Detected architecture x86-64.
Welcome to Fedora 32 (Container Image)!
Set hostname to <f79b358e1faa>.
Initializing machine ID from random generator.
Failed to create /user.slice/user-1000.slice/session-1.scope/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...In cgroup v2nix flake metadata nixpkgsResolved URL: github:NixOS/nixpkgs
Locked URL: github:NixOS/nixpkgs/70dc433f2eb84cc7ed848a30feaa5aab5890d23c
Description: A collection of packages for the Nix package manager
Path: /nix/store/zfvfgqvcah4fkx2vn47naybjyfhwiabn-source
Revision: 70dc433f2eb84cc7ed848a30feaa5aab5890d23c
Last modified: 2021-08-17 00:10:12
Inputs:ubuntu@ubuntu:~$ podman infopodman \
stats \
--cgroup-manager=systemdshows: ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS
a37d74849551 httpd -- 1.249MB / 8.163GB 0.02% -- / -- -- / -- 1
e18c53edd79c gracious_sanderson 0.38% 1.288GB / 8.163GB 15.78% -- / -- 13.67GB / 22.05GB 9And it works on podman \
run \
--env=PATH=/root/.nix-profile/bin:/usr/bin:/bin \
--device=/dev/kvm \
--device=/dev/fuse \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--log-level=error \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--privileged=true \
--tty=false \
--rm=true \
--user=0 \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:rw \
docker.nix-community.org/nixpkgs/nix-flakes \
<<COMMANDS
mkdir --parent --mode=0755 ~/.config/nix
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf
nix \
profile \
install \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& mkdir --parent --mode=0755 /var/tmp \
&& podman \
run \
--storage-driver="vfs" \
--cgroups=disabled \
--log-level=error \
--interactive=true \
--network=host \
--tty=true \
alpine \
sh \
-c 'apk add --no-cache curl && echo PinP'
COMMANDS |
|
Adapted from: How to use Podman inside of a container: Rootless Podman with containerized rootful Podman result="$({ podman run --privileged quay.io/podman/stable podman run ubi8 echo hello } | rg --count 'hello')"if [[ -n "$result" ]] ; then echo 'Passed!'; fiFrom: https://stackoverflow.com/a/57789801 podman run --privileged --net=host --rm quay.io/podman/stable podman run --net=host --storage-driver=vfs hello-worldpodman \
run \
--privileged \
--net=host \
--rm \
quay.io/podman/stable \
podman \
run \
--net=host \
--storage-driver=vfs \
docker.io/library/ubuntu:22.04 \
bash \
-c \
'apt-get update'xhost +
podman \
run \
--env=PATH=/root/.nix-profile/bin:/usr/bin:/bin \
--device=/dev/kvm \
--device=/dev/fuse \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--log-level=error \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--privileged=true \
--tty=true \
--rm=true \
--user=0 \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
--volume=/etc/localtime:/etc/localtime:ro \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:rw \
docker.nix-community.org/nixpkgs/nix-flakes
xhost -TODO: |
|
It is possible! nix \
profile \
install \
github:ES-Nix/podman-rootless/from-nixpkgs#podmancat > Containerfile << 'EOF'
FROM registry.fedoraproject.org/fedora:36
RUN dnf -y install procps systemd httpd
RUN systemctl enable httpd
EXPOSE 80
CMD [ "/sbin/init" ]
EOF
podman \
build \
--file=Containerfile \
--tag=httpd . \
&& podman rm --force --ignore container-httpd \
&& podman \
run \
--detach=true \
--rm=true \
--name=container-httpd \
httpd \
&& podman \
exec \
--interactive=true \
--tty=false \
container-httpd \
bash <<<'timeout 20 watch -n1 systemctl status httpd'Play around: podman \
exec \
--interactive=true \
--tty=true \
container-httpd \
bashpodman exec -e HOME='/home/abcuser' -e USER='abcuser' --interactive=true --tty=true -u abcuser -w /home/abcuser -v "$(pwd)":/mnt/code container-systemd-ubuntu22 bashRefactorcat > Containerfile << 'EOF'
FROM registry.fedoraproject.org/fedora:36
RUN dnf -y install procps systemd
EXPOSE 80
CMD [ "/sbin/init" ]
EOF
podman \
build \
--file=Containerfile \
--tag=httpd . \
&& podman rm --force --ignore container-httpd \
&& podman \
run \
--detach=true \
--rm=true \
--name=container-httpd \
httpd \
&& podman \
exec \
--interactive=true \
--tty=false \
container-httpd \
bash <<<'systemctl list-unit-files --all'
podman \
exec \
--interactive=true \
--tty=false \
container-httpd \
bash <<<'dnf -y install httpd && systemctl enable httpd && systemctl start httpd'
podman \
exec \
--interactive=true \
--tty=false \
container-httpd \
bash <<<'timeout 20 watch -n1 systemctl status httpd'podman kill container-systemd-ubuntu22Ubuntu 22.04cat > Containerfile << 'EOF'
FROM ubuntu:22.04
RUN apt-get update -y \
&& apt-get install -y systemd
RUN addgroup abcgroup --gid 4455 \
&& adduser -q \
--gecos '"An unpriviliged user with an group"' \
--disabled-password \
--ingroup abcgroup \
--uid 3322 \
abcuser
EXPOSE 80
CMD [ "/lib/systemd/systemd" ]
EOF
podman \
build \
--file=Containerfile \
--tag=systemd-ubuntu22 . \
&& podman rm --force --ignore container-systemd-ubuntu22 \
&& podman \
run \
--detach=true \
--rm=true \
--name=container-systemd-ubuntu22 \
systemd-ubuntu22 \
&& podman \
exec \
--interactive=true \
--tty=false \
container-systemd-ubuntu22 \
bash <<<'systemctl list-unit-files --all'
podman \
exec \
--interactive=true \
--tty=false \
container-systemd-ubuntu22 \
bash <<<'apt-get -y install lighttpd && systemctl enable lighttpd && systemctl start lighttpd'
podman \
exec \
--interactive=true \
--tty=false \
container-systemd-ubuntu22 \
bash <<<'timeout 20 watch -n1 systemctl status lighttpd'TODO: test it! cat << EOF > "$HOME"/hello-world.sh
#!/bin/bash
while true; do echo 'Hello world: '$(date +"%Y/%m/%d %T.%N"); sleep $(shuf -i 1-3 -n 1); done
EOF
chmod a+x "$HOME"/hello-world.sh
cat << EOF > /etc/systemd/system/hello-word.service
[Unit]
Description=Hello World Service Example
After=systend-user-sessions.service
[Service]
Type=simple
ExecStart=$HOME/hello-world.sh
EOF
systemctl start hello-world.servicesystemctl status hello-world.service
journalctl -u -e hello-worldTODO: systemctl --user show-environment | grep DBUS podman kill container-systemd-ubuntu22 |
podman \
run \
--security-opt="label=disable" \
--user=podman \
--device=/dev/fuse \
quay.io/podman/stable \
podman \
run \
--rm=true \
docker.io/library/alpine:3.14.2 \
sh \
-c \
'
echo \
&& cat /etc/os-release \
&& apk update \
&& apk add --no-cache python3 \
&& python3 --version \
&& echo
'podman \
run \
--security-opt="label=disable" \
--user=podman \
--device=/dev/fuse \
quay.io/podman/stable \
podman \
run \
--security-opt="label=disable" \
--user=podman \
--device=/dev/fuse \
quay.io/podman/stable \
podman \
run \
--rm=true \
docker.io/library/alpine:3.14.2 \
sh \
-c \
'
echo \
&& cat /etc/os-release \
&& apk update \
&& apk add --no-cache python3 \
&& python3 --version \
&& echo
' |
Ubuntu 22.04cat > Containerfile << 'EOF'
FROM ubuntu:22.04
RUN apt-get update -y \
&& apt-get install -y systemd \
&& apt-get -y autoremove \
&& apt-get -y clean \
&& rm -rf /var/lib/apt/lists/*
RUN addgroup abcgroup --gid 4455 \
&& adduser -q \
--gecos '"An unpriviliged user with an group"' \
--disabled-password \
--ingroup abcgroup \
--uid 3322 \
abcuser
RUN apt-get update -y \
&& apt-get install --no-install-recommends --no-install-suggests -y \
ca-certificates \
curl \
tar \
xz-utils \
nix-bin \
&& apt-get -y autoremove \
&& apt-get -y clean \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir -v /nix && chown -v abcuser:abcgroup /nix
EXPOSE 80
CMD [ "/lib/systemd/systemd" ]
EOF
podman \
build \
--file=Containerfile \
--tag=systemd-ubuntu22 . \
&& podman rm --force --ignore container-systemd-ubuntu22 \
&& podman \
run \
--detach=true \
--rm=true \
--name=container-systemd-ubuntu22 \
systemd-ubuntu22 \
&& podman \
exec \
--interactive=true \
--tty=false \
container-systemd-ubuntu22 \
bash <<<'systemctl list-unit-files --all'
podman \
exec \
--interactive=true \
--tty=false \
container-systemd-ubuntu22 \
bash <<<'stat /nix'
podman exec --interactive=true --tty=true -u abcuser -w /home/abcuser container-systemd-ubuntu22 bashmkdir -v -p -m 0755 ~/.config/nix && echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.confpodman \
exec \
-e USER=root \
-e PATH=/root/.nix-profile:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
--interactive=true \
--privileged=true \
--tty=true \
-u 0 \
-w /root \
container-systemd-ubuntu22 \
bash \
<<'COMMANDS'
apt-get -qq update \
&& apt-get install -y nix-bin \
&& chown abcuser: -R /nix
COMMANDSnix --extra-experimental-features 'nix-command flakes' profile install nixpkgs#hello |
nix run nixpkgs#udocker -- run quay.io/podman/stable podman imagesnix run nixpkgs#udocker -- run docker.io/nixos/nix:latest sh -c 'nix flake --version'nix \
run \
nixpkgs#udocker \
-- \
run \
alpine \
sh \
-c \
'
echo \
&& cat /etc/os-release \
&& apk update \
&& apk add --no-cache python3 \
&& python3 --version \
&& echo
'nix \
run \
nixpkgs#udocker \
-- \
run \
--security-opt="label=disable" \
--user=podman \
--device=/dev/fuse \
quay.io/podman/stable \
podman \
run \
--rm=true \
docker.io/library/alpine:3.14.2 \
sh \
-c \
'
echo \
&& cat /etc/os-release \
&& apk update \
&& apk add --no-cache python3 \
&& python3 --version \
&& echo
'nix run nixpkgs#udocker -- run localhost/unprivileged-ubuntu22:latest |
Abstract
List:
https://www.devopschina.org/wp-content/uploads/2023/08/Podman-in-Action-ebook-FINAL-Red-Hat-Developer.pdf
TODO: reproduce it
containers/podman#10282 (comment)
TODO: reproduce it
https://stackoverflow.com/a/56856410
The text was updated successfully, but these errors were encountered: