podman run fails on tinycore

[imperialguy]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman run fails on tinycore 11.1 VM because of some issue related to slirp4netns. I built podman and related deps manually. Are there any plans to release podman distros for tinycore? That would be very helpful.

Steps to reproduce the issue:

1. Install podman on tinycore using manual install

2. Run podman run hello-world

Describe the results you received:

tc@box:~/tczs$ podman run  hello-world
✔ docker.io/library/hello-world:latest
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 0e03bdcc26d7 done
Copying config bf756fb1ae done
Writing manifest to image destination
Storing signatures
Error: /home/tc/.nix-profile/bin/slirp4netns failed: "sent tapfd=7 for tap0\nWARNING: Support for seccomp is experimental\nreceived tapfd=7\ncannot pivot_root to /tmp\ncreate_sandbox failed\ndo_slirp is exiting\ndo_slirp failed\nparent failed\nWARNING: Support for seccomp is experimental\nStarting slirp\n* MTU:             65520\n* Network:         10.0.2.0\n* Netmask:         255.255.255.0\n* Gateway:         10.0.2.2\n* DNS:             10.0.2.3\n* Recommended IP:  10.0.2.100\n"

Describe the results you expected:
Expected it to work

Additional information you deem important (e.g. issue happens only occasionally):
Tried this workaround, but it produces a different error

tc@box:~/tczs$ podman run  --network=host --cgroups disabled hello-world
Error: OCI runtime error: invalid file system type on '/sys/fs/cgroup'

Output of podman version:

Version:      3.0.0-dev
API Version:  3.0.0
Go Version:   go1.15.6
Git Commit:   9c9f02aad773051fa742a874844f08f2fb567d3b-dirty
Built:        Tue Jan  1 00:00:00 1980
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.0-dev
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/bin/conmon
    version: 'conmon version 2.0.23-dev, commit: 05b804604d6da1b3faaee7defb65d3b9b3771888-dirty'
  cpus: 1
  distribution:
    distribution: tinycore
    version: "11.1"
  eventLogger: file
  hostname: box
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 50
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 5.4.3-tinycore64
  linkmode: static
  memFree: 187273216
  memTotal: 3123281920
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/local/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /home/tc/.nix-profile/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 759169024
  swapTotal: 769392640
  uptime: 4h 33m 43.52s (Approximately 0.17 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /home/tc/.config/containers/storage.conf
  containerStore:
    number: 12
    paused: 0
    running: 0
    stopped: 12
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/tc/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 2
  runRoot: /run/user/1001/containers
  volumePath: /home/tc/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 315532800
  BuiltTime: Tue Jan  1 00:00:00 1980
  GitCommit: 9c9f02aad773051fa742a874844f08f2fb567d3b-dirty
  GoVersion: go1.15.6
  OsArch: linux/amd64
  Version: 3.0.0-dev

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Tinycore runs on virtual box.

[mheon]

This does not appear to be a Podman issue. Please file an issue against the slirp4netns Github repo at https://github.com/rootless-containers/slirp4netns

[imperialguy]

What about this?

tc@box:~/tczs$ podman run  --network=host --cgroups disabled hello-world
Error: OCI runtime error: invalid file system type on '/sys/fs/cgroup'

[mheon]

That's an error from either crun or runc, whichever you are using, from what I can see.

[imperialguy]

Using crun version 0.16

[afbjorklund]

I had podman running on boot2podman (based on tinycore), you might want to look there for clues:

https://github.com/boot2podman/boot2podman/blob/master/building_rootless.md

I used https://github.com/tianon/cgroupfs-mount in order to do the required cgroupfs mounting there

But I don't really plan on continuing it, so the last release was for podman 1.9.3 and tinycore 10.0

[afbjorklund]

I think the problem is trying to use pivot_root from a rootfs (the default tinycore installation runs off the initrd directly):

sent tapfd=7 for tap0
WARNING: Support for seccomp is experimental
received tapfd=7
cannot pivot_root to /tmp
create_sandbox failed
do_slirp is exiting
do_slirp failed
parent failed
WARNING: Support for seccomp is experimental
Starting slirp

The workaround that was done for boot2docker was to copy everything to tmpfs using the "noembed" bootcode

10.32. noembed - use a separate tmpfs
This is an advanced option that changes where in RAM Core is run
from. By default, Core uses the tmpfs setup by the kernel; with this
bootcode, Core will setup a new tmpfs file system, and use that
instead.
Using this bootcode temporarily doubles the RAM use, as both
copies are kept in RAM at once during boot. As an extra copy is
made, it also slows the boot time. It allows GNU df to detect the
free space in /, used by some proprietary software installers.
Example:
• noembed

From http://tinycorelinux.net/book.html

The alternative is running podman with "no_pivot_root", but this is not really secure and sort of deprecated.

It allows running from a rootfs though, so it was still used by boot2podman. And worked with old slirp4netns

[afbjorklund]

@imperialguy : you will need some kind of podman switch, to avoid adding the --enable-sandbox feature

Alternatively add some kind of --no-pivot to slirp4netns, to avoid trying call pivot_root (but use chroot)

It was added here: rootless-containers/slirp4netns@6f46a05

See 7c3428d (for slirp4netns 0.4.1)

[imperialguy]

@afbjorklund I'll give the noembed thing a try. The podman switch option sounds good as well. @giuseppe Can such a switch be added to podman?

[giuseppe]

@giuseppe Can such a switch be added to podman?

I think it should be ok to add a new option like slirp_disable__features=... to specify some options to slirp4netns.

Alternatively, could slirp4netns be compiled without the sandboxing feature? We could make it configurable at configure time.

[imperialguy]

@giuseppe Can such a switch be added to podman?

I think it should be ok to add a new option like slirp_disable__features=... to specify some options to slirp4netns.

Alternatively, could slirp4netns be compiled without the sandboxing feature? We could make it configurable at configure time.

Either way works for me. Can this be made configurable via nix as well? Although, podman switch is the most easiest for the end user.

[giuseppe]

yes it can be configurable via nix as well.

[imperialguy]

It would be great if you can make the podman switch the primary resolution option. If that's not possible, then slirp configuration it is.

[afbjorklund]

The best would be to make it configurable, based on whether using a rootfs or not. (i.e. no_pivot)

        // NetworkCmdPath is the path to the slirp4netns binary.
        NetworkCmdPath string `toml:"network_cmd_path,omitempty"`

        // NetworkCmdOptions is the default options to pass to the slirp4netns binary.
        // For example "allow_host_loopback=true"
        NetworkCmdOptions []string `toml:"network_cmd_options,omitempty"`

        // NoPivotRoot sets whether to set no-pivot-root in the OCI runtime.
        NoPivotRoot bool `toml:"no_pivot_root,omitempty"`

Converting to a tmpfs, or compiling out or disabling the sandbox feature sounds like workarounds ?

It is podman that decides to "call" runc or buildah or slirp4netns, and needs to use the proper flags.

        noPivot := r.config.Engine.NoPivotRoot

        if !noPivot && slirpFeatures.HasEnableSandbox {
                cmdArgs = append(cmdArgs, "--enable-sandbox")
        }

[imperialguy]

The best would be to make it configurable, based on whether using a rootfs or not. (i.e. no_pivot)

        // NetworkCmdPath is the path to the slirp4netns binary.
        NetworkCmdPath string `toml:"network_cmd_path,omitempty"`

        // NetworkCmdOptions is the default options to pass to the slirp4netns binary.
        // For example "allow_host_loopback=true"
        NetworkCmdOptions []string `toml:"network_cmd_options,omitempty"`

        // NoPivotRoot sets whether to set no-pivot-root in the OCI runtime.
        NoPivotRoot bool `toml:"no_pivot_root,omitempty"`

Converting to a tmpfs, or compiling out or disabling the sandbox feature sounds like workarounds ?

It is podman that decides to "call" runc or buildah or slirp4netns, and needs to use the proper flags.

        noPivot := r.config.Engine.NoPivotRoot

        if !noPivot && slirpFeatures.HasEnableSandbox {
                cmdArgs = append(cmdArgs, "--enable-sandbox")
        }

The above fixes the slirp4netns related issue. For the problem below, is this the recommended solution?

tc@box:~/tczs$ podman run  --network=host --cgroups disabled hello-world
Error: OCI runtime error: invalid file system type on '/sys/fs/cgroup'

[afbjorklund]

For the problem below, is this the recommended solution?

tc@box:~/tczs$ podman run  --network=host --cgroups disabled hello-world
Error: OCI runtime error: invalid file system type on '/sys/fs/cgroup'

Yes, I would recommend using that script when not using systemd.

Tiny Core Linux uses init (from busybox) as PID 1 and not systemd

You will need to compile a new kernel with cgroups support, though.

The default Tiny Core Linux kernel does not support cgroupfs/overlayfs.

[afbjorklund]

@imperialguy

Are there any plans to release podman distros for tinycore? That would be very helpful.

We deprecated and buried both the boot2docker and boot2podman distributions in 2020...
They were container-enabled downstream distributions of tinycore (new kernel + packages)

• https://github.com/boot2docker/boot2docker (docker 19.03.12)
• https://github.com/boot2podman/boot2podman (podman 1.9.3)

There is much more info here: https://podman.io/community/meeting/notes/2020-11-03/
Feel free to contact me directly if you want more details about these projects (or "machine").

Presentation: The boot2podman project - "rise and fall of", 2013-2020

The main focus for a container distribution is Fedora CoreOS, featuring Podman:
https://docs.fedoraproject.org/en-US/fedora-coreos/running-containers/

For the casual user, we just used Vagrant (with regular Ubuntu or Fedora) instead:
https://boot2podman.github.io/2020/07/22/machine-replacement.html

---

There has been some talk about doing a community distribution, based on Tiny Core Linux...
I would recommend coordinating such efforts with the upstream project: http://tinycorelinux.net/

The best would be if the default kernel could be container-enabled, and the packages included.
The old/dead projects left off at TCL 10.1 (with 11.x on "master"), but maybe for e.g. TCL 12.x ?

• 9.0: Linux 4.14 based (2018)
• 10.1: Linux 4.19 based (2019)
• 11.1: Linux 5.4 based (2020)
• 12.0: Linux 5.10 based (2021)

The package manager is called TCE and uses TCZ packages, there's more info here: packages.md
Some of the system components don't use packages at all, but just regular .tgz tarballs to the initrd.

It would be possible to support both Docker 20.10+ and Podman 2.1+ from the same "base" TCL OS.
You would just install different .tcz packages on top, docker/containerd/runc or podman/conmon/crun.

[imperialguy]

For me, there's broadly two use cases - lightweight and not-so-lightweight (or let's just call it heavy weight:) ) For light weight, I am using tinycore. For heavy-weight, I am thinking vagrant+libvirt+archlinux. But, now that you mentioned, I'll definitely look at your vagrant+libvirt+fedora32 option as well. I usually try to stay away from large disk size VMs as much as possible.

As far as the future options go, hopefully some day you will release a "Podman for Desktop" that runs on fedora CoreOS VM. So, I'll wait for that. And look forward to using it if and when it's available.

In the meantime my use case is mostly light-weight. Hence tinycore. Haven't planned on coordinating with upstream, but I usually custom build the tczs as I go based on my use cases (podman and related deps in this case).

[afbjorklund]

For me, there's broadly two use cases - lightweight and not-so-lightweight (or let's just call it heavy weight:) ) For light weight, I am using tinycore. For heavy-weight, I am thinking vagrant+libvirt+archlinux. But, now that you mentioned, I'll definitely look at your vagrant+libvirt+fedora32 option as well. I usually try to stay away from large disk size VMs as much as possible.

Assuming podman package support in the repositories, it would probably be trivial to adopt it to other distributions.

Vagrant.configure("2") do |config|
  config.vm.box = "generic/fedora32"

  config.vm.provider "libvirt" do |lv|
    lv.memory = "1024"
  end

  config.vm.provision "shell", privileged: false, inline: <<-SHELL
    sudo yum install -y podman

    systemctl enable --user podman.socket
    systemctl start --user podman.socket
  SHELL
end

Like using "generic/arch" instead of fedora and "pacman -Syu --noconfirm podman" instead of yum, or something ?

https://www.redhat.com/sysadmin/podman-clients-macos-windows

As far as the future options go, hopefully some day you will release a "Podman for Desktop" that runs on fedora CoreOS VM. So, I'll wait for that. And look forward to using it if and when it's available.

If you want to use Fedora CoreOS, there is already CodeReady Containers. But I can't speak for Red Hat plans...

As far as "weight" goes, it's like 10x vagrant (which is 10x tinycore). Like 10G, when compared with 1G and 100M ?

In the meantime my use case is mostly light-weight. Hence tinycore. Haven't planned on coordinating with upstream, but I usually custom build the tczs as I go based on my use cases (podman and related deps in this case).

Sounds like the same story then: https://podman.io/blogs/2019/01/14/podman-machine-and-boot2podman.html

But it quickly grows into a full "distribution" and takes up a lot of maintenance time, which is why it was deprecated...