kubic xUbuntu_18.04 broken default OCI runtime config? dockerfile RUN lines fail with default installation

[buck2202]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Simple dockerfile builds fail on a default configuration install of podman 3 from the kubic xUbuntu_18.04 repository. The default oci runtime configuration seems broken

Steps to reproduce the issue:
I created a very dumb dockerfile which fails during its build

$ cat test_dockerfile 
FROM ubuntu
RUN set -x && apt-get -q update
$ podman build -t test -f test_dockerfile .
STEP 1: FROM ubuntu
STEP 2: RUN set -x && apt-get -q update
error running container: error creating container for [/bin/sh -c set -x && apt-get -q update]: : exec: "runc": executable file not found in $PATH
Error: error building at STEP "RUN set -x && apt-get -q update": error while running runtime: exit status 1

runc is not present, but the default runtime seems like it should be crun. The crun package is pulled in by podman:

$ which runc
$ which crun
/usr/bin/crun
$ podman info
*snip*
host:
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17.6-58ef-dirty
      commit: fd582c529489c0738e7039cbc036781d1d039014
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
$ dpkg -S /usr/bin/crun
crun: /usr/bin/crun
$
$ apt show crun
Package: crun
Version: 100:0.17-4
Priority: optional
Section: devel
Maintainer: Lokesh Mandvekar <[email protected]>
Installed-Size: 1364 kB
Depends: libyajl2
Homepage: https://github.com/containers/crun.git
Download-Size: 231 kB
APT-Manual-Installed: no
APT-Sources: http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  Packages
Description: OCI runtime written in C
$
$ apt show podman
Package: podman
Version: 100:3.0.0-1
Priority: optional
Section: devel
Maintainer: Lokesh Mandvekar <[email protected]>
Installed-Size: 85.2 MB
Depends: libseccomp2 (>= 2.4.3-1), libdevmapper1.02.1, libgpgme11, catatonit, conmon (>= 100:2.0.25~2), containers-common (>= 100:1-7), containernetworking-plugins (>= 100:0.9.1-1), dbus-user-session, iptables, podman-plugins (>= 100:1.1.1-3), crun (>= 100:0.17-4)
Recommends: slirp4netns (>= 100:1.1.8-3), uidmap, fuse-overlayfs
Conflicts: podman-rootless
Homepage: https://github.com/containers/podman.git
Download-Size: 18.5 MB
APT-Manual-Installed: yes
APT-Sources: http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  Packages
Description: Manage pods, containers and container images.

The default /etc/containers/containers.conf has all relevant sections commented out

$ cat /etc/containers/containers.conf
*snip*
# Default OCI runtime
#
# runtime = "crun"
*snip*
# Paths to look for a valid OCI runtime (crun, runc, kata, etc)
# [engine.runtimes]
# crun = [
#            "/usr/bin/crun",
#            "/usr/sbin/crun",
#            "/usr/local/bin/crun",
#            "/usr/local/sbin/crun",
#            "/sbin/crun",
#            "/bin/crun",
#            "/run/current-system/sw/bin/crun",
# ]

# runc = [
#        "/usr/bin/runc",
#        "/usr/sbin/runc",
#        "/usr/local/bin/runc",
#        "/usr/local/sbin/runc",
#        "/sbin/runc",
#        "/bin/runc",
#        "/usr/lib/cri-o-runc/sbin/runc",
# ]

# kata = [
#            "/usr/bin/kata-runtime",
#            "/usr/sbin/kata-runtime",
#            "/usr/local/bin/kata-runtime",
#            "/usr/local/sbin/kata-runtime",
#            "/sbin/kata-runtime",
#            "/bin/kata-runtime",
#            "/usr/bin/kata-qemu",
#            "/usr/bin/kata-fc",
# ]

Even though podman info seems to indicate that crun is the default and podman knows where to find it, manually forcing it avoids the issue

$ podman --runtime /usr/bin/crun build -t test -f test_dockerfile .
STEP 1: FROM ubuntu
STEP 2: RUN set -x && apt-get -q update
+ apt-get -q update
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [165 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [13.3 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [612 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [664 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1016 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [21.1 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [198 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4301 B]
Fetched 17.0 MB in 3s (5169 kB/s)
Reading package lists...
STEP 3: COMMIT test
--> 02e2f54e157
02e2f54e157d7778c6e197abeb8a53c56fa357d3f274e4d41d6e182291bc0247

It's possible I'm missing something in other configuration files somewhere, but podman seems to look for runc as the default runtime despite indications that crun is properly installed and set as the default

Output of podman version:

$ podman version
Version:      3.0.0
API Version:  3.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.26, commit: '
  cpus: 1
  distribution:
    distribution: ubuntu
    version: "18.04"
  eventLogger: journald
  hostname: mw-podman-base-bionic
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1003
      size: 1
    - container_id: 1
      host_id: 296608
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 296608
      size: 65536
  kernel: 4.15.0-1092-gcp
  linkmode: dynamic
  memFree: 75235328
  memTotal: 608616448
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17.6-58ef-dirty
      commit: fd582c529489c0738e7039cbc036781d1d039014
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1002/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.3.1
  swapFree: 0
  swapTotal: 0
  uptime: 2h 3m 59.2s (Approximately 0.08 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 5
  runRoot: /run/user/1002/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown,now 100:3.0.0-1 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
google cloud, ubuntu bionic image

[buck2202]

explicitly uncommenting the crun lines in /etc/containers/containers.conf still isn't enough to get past this without manually specifying the runtime per-command. There is no user-level override present

$ grep -n crun /etc/containers/containers.conf 
394:runtime = "crun"
399:# runtime_supports_json = ["crun", "runc", "kata"]
403:# runtime_supports_nocgroups = ["crun"]
428:# Paths to look for a valid OCI runtime (crun, runc, kata, etc)
430:crun = [
431:           "/usr/bin/crun",
432:           "/usr/sbin/crun",
433:           "/usr/local/bin/crun",
434:           "/usr/local/sbin/crun",
435:           "/sbin/crun",
436:           "/bin/crun",
437:           "/run/current-system/sw/bin/crun",

$ grep -n runc /etc/containers/containers.conf |grep -v truncate
399:# runtime_supports_json = ["crun", "runc", "kata"]
428:# Paths to look for a valid OCI runtime (crun, runc, kata, etc)
440:# runc = [
441:#        "/usr/bin/runc",
442:#        "/usr/sbin/runc",
443:#        "/usr/local/bin/runc",
444:#        "/usr/local/sbin/runc",
445:#        "/sbin/runc",
446:#        "/bin/runc",
447:#        "/usr/lib/cri-o-runc/sbin/runc",

$ diff /usr/share/containers/containers.conf /etc/containers/containers.conf 
394c394
< # runtime = "crun"
---
> runtime = "crun"
430,438c430,438
< # crun = [
< #            "/usr/bin/crun",
< #            "/usr/sbin/crun",
< #            "/usr/local/bin/crun",
< #            "/usr/local/sbin/crun",
< #            "/sbin/crun",
< #            "/bin/crun",
< #            "/run/current-system/sw/bin/crun",
< # ]
---
> crun = [
>            "/usr/bin/crun",
>            "/usr/sbin/crun",
>            "/usr/local/bin/crun",
>            "/usr/local/sbin/crun",
>            "/sbin/crun",
>            "/bin/crun",
>            "/run/current-system/sw/bin/crun",
> ]

$ cat $HOME/.config/containers/containers.conf
cat: /home/user/.config/containers/containers.conf: No such file or directory

$ podman build -t test -f test_dockerfile .
STEP 1: FROM ubuntu
STEP 2: RUN set -x && apt-get -q update
error running container: error creating container for [/bin/sh -c set -x && apt-get -q update]: : exec: "runc": executable file not found in $PATH
Error: error building at STEP "RUN set -x && apt-get -q update": error while running runtime: exit status 1

$ podman --runtime=/usr/bin/crun build -t test -f test_dockerfile .
STEP 1: FROM ubuntu
STEP 2: RUN set -x && apt-get -q update
+ apt-get -q update
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [612 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [664 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [165 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [13.3 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:11 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1016 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [198 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [21.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4301 B]
Fetched 17.0 MB in 3s (5270 kB/s)
Reading package lists...
STEP 3: COMMIT test
--> 75d03142de9
75d03142de9d788b887d1e49fb789c279ae63c3558805eb05dbd8a0202a4acf6

edit: fixed ugly linebreak in paste

[Luap99]

@rhatdan PTAL

[fugkco]

Got the same issue. Happy to provide output of podman info and the lot if needed.

One additional thing I'd like to add is I attempted to explicitly add the runtime to the rootless config (since I was running rootless) in addition to the global config, but still not working:

cat ~/.config/containers/containers.conf
[engine]
runtime="crun"

As OP mentioned, running manually by adding --runtime flag works. I did notice that running podman --runtime crun ps throws an error, but then subsequently does show all the containers:

$ podman --runtime $(which crun) ps
ERRO[0000] OCI Runtime runc is in use by a container, but is not available (not in configuration file or not installed)
CONTAINER ID  IMAGE                            COMMAND  CREATED  STATUS  PORTS   NAMES
eb938a7a413a  docker.io/library/alpine:latest                        About an hour ago                                 Up About an hour ago          trusting_chaplygin

Note, I'm running Pop!_OS 20.10.

[vrothberg]

Thanks for reaching out! I'll take it and prepare a fix.

[vrothberg]

Opened #9368 to fix the issue.

[fugkco]

@vrothberg just upgraded to v3.0.1, maybe I'm missing something but seems to still happen for rootless. Error doesn't show up when running rootful:

$ podman --version
podman version 3.0.1

$ podman --runtime=crun ps
ERRO[0000] OCI Runtime runc is in use by a container, but is not available (not in configuration file or not installed)
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

$ podman --runtime=crun ps
ERRO[0000] OCI Runtime runc is in use by a container, but is not available (not in configuration file or not installed)
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

$ sudo podman --version
podman version 3.0.1

$ sudo podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

$ sudo podman --runtime=crun ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

[vrothberg]

@fugkco this issue relates to which runtime is being used when building.

I do not know your system and the configuration but I assume that the update to 3.0.1 taught Podman the path to runc. I assume that containers.conf was updated.

[buck2202]

@vrothberg is it possible that the full path is still not being passed to buildah?

I'm using runc because of its CRIU support, and have found that the system-packaged runc on ubuntu bionic throws seccomp errors (I think it was fine for 2.x, but not the point here)

Error: OCI runtime error: time="2021-02-20T22:15:19Z" level=error msg="container_linux.go:349: starting container process caused \"error adding seccomp rule for syscall socket: requested action matches default action of filter\""

I removed the runc package and installed cri-o-runc instead, which installs a runc executable which is not added to the system path by default. podman info knows where it is

  ociRuntime:
    name: runc
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.2-dev'

but it seems that buildah does not

$ cat test_dockerfile 
FROM ubuntu
RUN set -x && apt-get -q update
$ podman build -t test -f test_dockerfile .
STEP 1: FROM ubuntu
✔ docker.io/library/ubuntu:latest
Getting image source signatures
Copying blob 83ee3a23efb7 [======================================] 27.2MiB / 27.2MiB
Copying blob f611acd52c6c done  
Copying blob db98fc6f11f0 done  
Copying config f63181f19b done  
Writing manifest to image destination
Storing signatures
STEP 2: RUN set -x && apt-get -q update
error running container: error creating container for [/bin/sh -c set -x && apt-get -q update]: : exec: "runc": executable file not found in $PATH
Error: error building at STEP "RUN set -x && apt-get -q update": error while running runtime: exit status 1

This is easily fixable by adding runc from cri-o-runc into the system path, but it was still unexpected since containers.conf suggests that podman manages its own search paths. Maybe this is intended behavior (for buildah to rely on the system path), but I wanted to flag it just in case.

[rhatdan]

podman build should get the same defaults as podman run as of podman 3.0.1 release, I believe. This was recently fixed.

[buck2202]

Sorry, I didn't mention in my most recent comment that I was seeing this happen on updated v3.0.1 from kubic.

From the outside, it seems like the name of the runtime is now matching between run and build, but build is relying on the system $PATH to find it, while run is using the search paths from containers.conf.

[mheon]

@buck2202 Can you open a new issue for that bug?

[fugkco]

@fugkco this issue relates to which runtime is being used when building.

I do not know your system and the configuration but I assume that the update to 3.0.1 taught Podman the path to runc. I assume that containers.conf was updated.

@vrothberg no worries. For some reason I thought build/run was had the same issue in 3.0.1, but I just confirmed it did not. I will raise a separate ticket.