Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: open executable: Operation not permitted: OCI permission denied #9377

Closed
ebousse opened this issue Feb 15, 2021 · 19 comments · Fixed by #9404
Closed

Error: open executable: Operation not permitted: OCI permission denied #9377

ebousse opened this issue Feb 15, 2021 · 19 comments · Fixed by #9404
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@ebousse
Copy link

ebousse commented Feb 15, 2021

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When running a simple podman run command, I get an error: Error: open executable: Operation not permitted: OCI permission denied.

Steps to reproduce the issue:

  1. Run podman --log-level=debug run --rm docker.io/klakegg/hugo:0.80.0-ext-alpine-ci hugo version

Describe the results you received:

Without debug log level:

Error: open executable: Operation not permitted: OCI permission denied

With debug log level:

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm docker.io/klakegg/hugo:0.80.0-ext-alpine-ci hugo version) 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.33.1 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/1000/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/bousse-e/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/1000/libpod/tmp VolumePath:/home/bousse-e/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/bousse-e/.config/cni/net.d}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/bousse-e/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/bousse-e/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/bousse-e/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/bousse-e/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
INFO[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
INFO[0000] Setting parallel job count to 13             
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm docker.io/klakegg/hugo:0.80.0-ext-alpine-ci hugo version) 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.33.1 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/1000/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/bousse-e/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/1000/libpod/tmp VolumePath:/home/bousse-e/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/bousse-e/.config/cni/net.d}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/bousse-e/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/bousse-e/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/bousse-e/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/bousse-e/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
INFO[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
INFO[0000] Setting parallel job count to 13             
DEBU[0000] parsed reference into "[overlay@/home/bousse-e/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] exporting opaque data as blob "sha256:a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] parsed reference into "[overlay@/home/bousse-e/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] exporting opaque data as blob "sha256:a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] using systemd mode: false                    
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] Allocated lock 0 for container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 
DEBU[0000] parsed reference into "[overlay@/home/bousse-e/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] exporting opaque data as blob "sha256:a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] created container "a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423" 
DEBU[0000] container "a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423" has work directory "/home/bousse-e/.local/share/containers/storage/overlay-containers/a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423/userdata" 
DEBU[0000] container "a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423" has run directory "/run/user/1000/containers/overlay-containers/a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423/userdata" 
DEBU[0000] Not attaching to stdin                       
DEBU[0000] overlay: mount_data=lowerdir=/home/bousse-e/.local/share/containers/storage/overlay/l/Q2KX5JJVYJUFZXIKECE73QBBIF:/home/bousse-e/.local/share/containers/storage/overlay/l/XYV7JDDTNRWCTKXBCK6LTGBPWY:/home/bousse-e/.local/share/containers/storage/overlay/l/LKMJCJILATTJUXWNXVEG6JGMW7:/home/bousse-e/.local/share/containers/storage/overlay/l/UJWDGGYPQTNZSOYTINJLRGAB7K,upperdir=/home/bousse-e/.local/share/containers/storage/overlay/96675f36b591b17513189b8e96294f1738368408caa96dad524bd809ed7557dd/diff,workdir=/home/bousse-e/.local/share/containers/storage/overlay/96675f36b591b17513189b8e96294f1738368408caa96dad524bd809ed7557dd/work,context="system_u:object_r:container_file_t:s0:c872,c971" 
DEBU[0000] Made network namespace at /run/user/1000/netns/cni-7ab36677-878e-3eba-e79d-161e3ba97a8f for container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 
DEBU[0000] mounted container "a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423" at "/home/bousse-e/.local/share/containers/storage/overlay/96675f36b591b17513189b8e96294f1738368408caa96dad524bd809ed7557dd/merged" 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-7ab36677-878e-3eba-e79d-161e3ba97a8f tap0 
DEBU[0000] Created root filesystem for container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 at /home/bousse-e/.local/share/containers/storage/overlay/96675f36b591b17513189b8e96294f1738368408caa96dad524bd809ed7557dd/merged 
DEBU[0000] Workdir "/src" resolved to host path "/home/bousse-e/.local/share/containers/storage/overlay/96675f36b591b17513189b8e96294f1738368408caa96dad524bd809ed7557dd/merged/src" 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] Setting CGroups for container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 to user.slice:libpod:a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 at /home/bousse-e/.local/share/containers/storage/overlay-containers/a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 -u a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 -r /usr/bin/crun -b /home/bousse-e/.local/share/containers/storage/overlay-containers/a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423/userdata -p /run/user/1000/containers/overlay-containers/a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423/userdata/pidfile -n gallant_boyd --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -s -l k8s-file:/home/bousse-e/.local/share/containers/storage/overlay-containers/a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/bousse-e/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/cni-7ab36677-878e-3eba-e79d-161e3ba97a8f for container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 
DEBU[0000] unmounted container "a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423" 
DEBU[0000] Removing container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 
DEBU[0000] Removing all exec sessions for container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 
DEBU[0000] Cleaning up container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 storage is already unmounted, skipping... 
DEBU[0000] Container a185829c14d461c3c812d1e0ba34b6d281ac58f7c1fc28bf762de7773a807423 storage is already unmounted, skipping... 
DEBU[0000] ExitCode msg: "open executable: operation not permitted: oci permission denied" 
Error: open executable: Operation not permitted: OCI permission denied

Describe the results you expected:

The command should run.

Note that another command such as podman run --rm mariadb ls works without problem.

Additional information you deem important (e.g. issue happens only occasionally): N/A

Output of podman version:

Version:      3.0.0-dev
API Version:  3.0.0
Go Version:   go1.16rc1
Built:        Wed Feb  3 13:07:25 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.3
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: knodel
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.11-200.fc33.x86_64
  linkmode: dynamic
  memFree: 20326567936
  memTotal: 33541758976
  ociRuntime:
    name: crun
    package: crun-0.17-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 7m 14.32s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/bousse-e/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/bousse-e/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/bousse-e/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1612354045
  BuiltTime: Wed Feb  3 13:07:25 2021
  GitCommit: ""
  GoVersion: go1.16rc1
  OsArch: linux/amd64
  Version: 3.0.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.0-0.204.dev.gita086f60.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): N/A
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Feb 15, 2021
@mheon
Copy link
Member

mheon commented Feb 15, 2021

Can you upgrade to 3.0 final and see if the issue still occurs?

@ebousse
Copy link
Author

ebousse commented Feb 15, 2021

Sure! I've installed the build from: https://koji.fedoraproject.org/koji/buildinfo?buildID=1708795

Now I have:

$ podman version
Version:      3.0.0
API Version:  3.0.0
Go Version:   go1.16rc1
Built:        Fri Feb 12 15:17:06 2021
OS/Arch:      linux/amd64

But the exact same error remains:

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm docker.io/klakegg/hugo:0.80.0-ext-alpine-ci hugo version) 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.33.4 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/1000/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/bousse-e/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/1000/libpod/tmp VolumePath:/home/bousse-e/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/bousse-e/.config/cni/net.d}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/bousse-e/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/bousse-e/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/bousse-e/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/bousse-e/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
INFO[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
INFO[0000] Setting parallel job count to 13             
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm docker.io/klakegg/hugo:0.80.0-ext-alpine-ci hugo version) 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.33.4 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{ImageBuildFormat:oci CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/1000/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand: InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/bousse-e/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/1000/libpod/tmp VolumePath:/home/bousse-e/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/bousse-e/.config/cni/net.d}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/bousse-e/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/bousse-e/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/bousse-e/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/bousse-e/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
INFO[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
INFO[0000] Setting parallel job count to 13             
DEBU[0000] parsed reference into "[overlay@/home/bousse-e/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] exporting opaque data as blob "sha256:a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] parsed reference into "[overlay@/home/bousse-e/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] exporting opaque data as blob "sha256:a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] using systemd mode: false                    
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] Allocated lock 0 for container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 
DEBU[0000] parsed reference into "[overlay@/home/bousse-e/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] exporting opaque data as blob "sha256:a93ad6a0848d92142b50f94f10bf6c1a774e7ebc73df82a83c015b0052a12303" 
DEBU[0000] created container "8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30" 
DEBU[0000] container "8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30" has work directory "/home/bousse-e/.local/share/containers/storage/overlay-containers/8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30/userdata" 
DEBU[0000] container "8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30" has run directory "/run/user/1000/containers/overlay-containers/8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30/userdata" 
DEBU[0000] Not attaching to stdin                       
DEBU[0000] Made network namespace at /run/user/1000/netns/cni-7c5f91e2-a54b-1171-5a83-b745b11369ed for container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 
DEBU[0000] overlay: mount_data=lowerdir=/home/bousse-e/.local/share/containers/storage/overlay/l/Q2KX5JJVYJUFZXIKECE73QBBIF:/home/bousse-e/.local/share/containers/storage/overlay/l/XYV7JDDTNRWCTKXBCK6LTGBPWY:/home/bousse-e/.local/share/containers/storage/overlay/l/LKMJCJILATTJUXWNXVEG6JGMW7:/home/bousse-e/.local/share/containers/storage/overlay/l/UJWDGGYPQTNZSOYTINJLRGAB7K,upperdir=/home/bousse-e/.local/share/containers/storage/overlay/d21e5218c6795865b682691efcb7db489bfc442cc4d46fb4a669245b6962af46/diff,workdir=/home/bousse-e/.local/share/containers/storage/overlay/d21e5218c6795865b682691efcb7db489bfc442cc4d46fb4a669245b6962af46/work,context="system_u:object_r:container_file_t:s0:c76,c643" 
DEBU[0000] mounted container "8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30" at "/home/bousse-e/.local/share/containers/storage/overlay/d21e5218c6795865b682691efcb7db489bfc442cc4d46fb4a669245b6962af46/merged" 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-7c5f91e2-a54b-1171-5a83-b745b11369ed tap0 
DEBU[0000] Created root filesystem for container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 at /home/bousse-e/.local/share/containers/storage/overlay/d21e5218c6795865b682691efcb7db489bfc442cc4d46fb4a669245b6962af46/merged 
DEBU[0000] Workdir "/src" resolved to host path "/home/bousse-e/.local/share/containers/storage/overlay/d21e5218c6795865b682691efcb7db489bfc442cc4d46fb4a669245b6962af46/merged/src" 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] Setting CGroups for container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 to user.slice:libpod:8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 at /home/bousse-e/.local/share/containers/storage/overlay-containers/8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 -u 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 -r /usr/bin/crun -b /home/bousse-e/.local/share/containers/storage/overlay-containers/8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30/userdata -p /run/user/1000/containers/overlay-containers/8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30/userdata/pidfile -n upbeat_clarke --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -s -l k8s-file:/home/bousse-e/.local/share/containers/storage/overlay-containers/8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/bousse-e/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/cni-7c5f91e2-a54b-1171-5a83-b745b11369ed for container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 
DEBU[0000] unmounted container "8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30" 
DEBU[0000] Removing container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 
DEBU[0000] Removing all exec sessions for container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 
DEBU[0000] Cleaning up container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 storage is already unmounted, skipping... 
DEBU[0000] Container 8aa726d53c3dddc73b0b95e647e22f500b5f3f9aceedfa7de1a41bcc7f64cc30 storage is already unmounted, skipping... 
DEBU[0000] ExitCode msg: "open executable: operation not permitted: oci permission denied" 
Error: open executable: Operation not permitted: OCI permission denied

@rhatdan
Copy link
Member

rhatdan commented Feb 15, 2021 •
edited
Loading

Permission denied usually means Security.
First thing I try is to run --privileged

$ podman run --privileged --rm docker.io/klakegg/hugo:0.80.0-ext-alpine-ci hugo version
Error: open executable: Operation not permitted: OCI permission denied

Since this fails in rootless --privileged, I will try rootfull.

# podman run --privileged --rm docker.io/klakegg/hugo:0.80.0-ext-alpine-ci hugo version
Trying to pull docker.io/klakegg/hugo:0.80.0-ext-alpine-ci...
Getting image source signatures
Copying blob 801bfaa63ef2 done  
Copying blob 50675e4fd1b6 done  
Copying blob d35553704998 done  
Copying blob 67e45b4e49c7 done  
Copying config a93ad6a084 done  
Writing manifest to image destination
Storing signatures
Error: open executable: Operation not permitted: OCI permission denied

Blows up there as well.

At this point it looks like something strange is going on. Does this image work within Docker?

I think your entrypoint is screwed up.

# podman run --rm -ti --entrypoint hugo docker.io/klakegg/hugo:0.80.0-ext-alpine-ci version
Hugo Static Site Generator v0.80.0-792EF0F4/extended linux/amd64 BuildDate: 2020-12-31T13:46:18Z

This does not look like a podman bug.

@rhatdan rhatdan closed this as completed Feb 15, 2021
@ebousse
Copy link
Author

ebousse commented Feb 16, 2021

At this point it looks like something strange is going on. Does this image work within Docker?

Yes it does. I tried it with docker before trying it with podman, and it works without problem. That's part of why I expected it to work with podman. Also note that this image is quite popular (5M+ pulls) and is even the one recommended by the hugo development team for using hugo through docker, as shown here: https://gohugo.io/getting-started/installing/

Thanks for the workaround though, this greatly helps!

@mheon
Copy link
Member

mheon commented Feb 16, 2021

If this works on Docker and --privileged does not solve it this seems like a Podman bug. I'm going to reopen.

@mheon mheon reopened this Feb 16, 2021
@rhatdan
Copy link
Member

rhatdan commented Feb 16, 2021

Ok then it is something wrong with the way we are handling entrypoint.

@rhatdan
Copy link
Member

rhatdan commented Feb 16, 2021

The image has this:

            "Entrypoint": [
                ""
            ],

Which might be confusing us, and not confusing Docker.

@rhatdan rhatdan mentioned this issue Feb 16, 2021
Merged
rhatdan added a commit to rhatdan/podman that referenced this issue Feb 17, 2021
@rhatdan
Ignore entrypoint=[\"\"]
12a577a 
We recieved an issue with an image that was built with
entrypoint=[""]
This blows up on Podman, but works on Docker.

When we setup the OCI Runtime, we should drop
entrypoint if it is == [""]

containers#9377

Signed-off-by: Daniel J Walsh <[email protected]>
mheon pushed a commit to mheon/libpod that referenced this issue Feb 18, 2021
@rhatdan @mheon
Ignore entrypoint=[\"\"]
865769d 
We recieved an issue with an image that was built with
entrypoint=[""]
This blows up on Podman, but works on Docker.

When we setup the OCI Runtime, we should drop
entrypoint if it is == [""]

containers#9377

Signed-off-by: Daniel J Walsh <[email protected]>
@kofemann
Copy link

Had a similar issue. Turned out that the entrypoint script was missing execution bit. To simple

RUN chmod +x /entrypoint.sh

have fixed that. Unfortunately the error messages was not clear about it.

@PavelSosin-320
Copy link

I got it in 3.4.4. when using rootless user. But I have the specific suspect: conmon. It looks strange:
From rootless: ls -alZ /usr/bin/conmon
-rwxr-xr-x. 1 nobody nobody system_u:object_r:bin_t:s0 139448 Sep 24 19:09 /usr/bin/conmon
and as root ls -alZ /usr/bin/conmon
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0 139448 Sep 24 19:09 /usr/bin/conmon.
Attempt to write /proc/self/oom_score_adj by the rootless user fails and causes conmon exit with code -1.
Everything in /proc/self of rootless user is owned by root and labeled as system_u:system_r:kernel_t:s0 . Is nobody's binary able to write into the Kernel? From th "Root" point of view everything is owned by root but labeled as root root unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 0. Maybe, root can adjust Kernel OOM behavior but not rootless? The scope of rootless user is User.service and Session.scope.

@mheon
Copy link
Member

mheon commented Feb 7, 2022

That error message re: /proc/self/oom_score_adj is expected, and will occur even on successful runs of rootless Podman. Conmon unconditionally makes that call, whether as root or rootless, but it is not allowed for rootless users; since it's non-fatal, we've never found a need to fix this. If you are having issues with rootless Podman I assure you this error is unrelated; also, you should open a fresh bug report, as this one is over a year old.

@PavelSosin-320
Copy link

PavelSosin-320 commented Feb 8, 2022 •
edited
Loading

@mheon I really trying to help to make Podman usable for "Normal GNOME-based workstations users". Unfortunately, regardless how image and container has been created I get OCI permission denied. If log-level-debug option is provided the log always contains the similar sequence of lines:
Running ["crun" "create" "--bundle" "/var/tmp/buildah075687185" "--pid-file" "/var/tmp/buildah075687185/pid" "--no-new-keyring" "--console-socket" "/var/tmp/buildah075687185/console.sock" "buildah-buildah075687185"]
2022-02-07T06:45:35.000883799Z: open executable: Permission denied and then Podman and buildah fall to cleanup.
It happens both for Podman and Buildah, start and init.
Unfortunately logs don't contain who is sender of -1. and which OCI permission is required. Also replacement CRun with Runc changes nothing. CRun itself can be invoked by a rootless user with --help option, its permissions and labelling look OK. Some parameters passed to CRun look suspectable, like combination of 2 logging mechanisms: journald and syslog, console socket that doesn't exists for a GNOME terminal user which uses GUI terminal emulator application running bash. Since most of recent Linux distros, including WSL use GNOME the problem looks as very common. 😒.
Possibly, it is like #10337 ? - related to close_range syscall. The problem is that Podman 3.4.4 doesn't accept --security-opt at all. Just did CRun upgrade to 1.4.2 spec 1.0.0 - the same result. There is no new containers-common for Fedora 35.

@PavelSosin-320
Copy link

Possible hint:
systemd[2777]: Started libcrun container !!!!!!!!!!
........
conmon[7746]: conmon f762d93c06e0d37e4f0d : runtime stderr: open executable: Permission denied
conmon[7746]: conmon f762d93c06e0d37e4f0d : Failed to create container: exit status 1
podman[7750]: 2022-02-08 11:22:12.036817177 +0200 IST m=+0.126761968 container cleanup ....
Container is created but can't bind in/out/err to the tty? GNOME provides pty1,2,3. Session is started by systemd and output goes to systemd log.

@mheon
Copy link
Member

mheon commented Feb 8, 2022

I believe that runtime stderr indicates that this is an error message from the OCI runtime, not that we're having issues working with that file descriptor (though I may be wrong, I've never seen this error before). It seems like it may be failing when attempting to exec the OCI runtime binary. Is there any difference in error message between runc and crun?

Also, I again suggest you open a fresh issue, this is unrelated to the bug we're posting in.

@rhatdan
Copy link
Member

rhatdan commented Feb 8, 2022

This looks like SELinux causing the issue, if you are on F36/Rawhide, there was an issue with the container-selinux package which could cause this porblem.

@PavelSosin-320
Copy link

@mheon, As I mentioned above Crun and Runc work similarly. If the SELinux labeling is suspected, please, explain why I see different executable files attributes of CRun, RunC (root system_u:object_r:container_runtime_exec_t:s0) and Conmon (oot system_u:object_r:bin_t:s0) and different owners from root and rootless ( root / nobody ). Is it installation glitch?
bus.socket in Fedora 35 and before are also created and labeled differently. But, since switching. SELinux mode back and forward doesn't solve the issue, probability that SELinux is the root cause is law. The important fact that when Podman is run ui GNOME it is actually run indirectly as gnome-terminal -x podman. GNOME doesn't expose the console to users. It is always spawned child process related to pts/2 pseudo X-Terminal. The echo "tty" > /dev/tty works OK and "tty" is displayed on gnome-terminal. Tty exists but volatile because can be destroyed when Terminal's tab is closed. Terminal app warns about it !!!

@rhatdan
Copy link
Member

rhatdan commented Feb 9, 2022

If SELinux is causing issues, please attach the AVCs.

@PavelSosin-320
Copy link

Absolutely nothing is shown in the running SEAlert UI. But inside Container's runtime I found:
cat /run/user/1000/containers/storage/btrfs-containers/f762d93c06e0d37e4f0df9dec63388f967ee3bbb1e3259b9dfc7a86abaa5d431/userdata/oci-log
{"msg":"open executable: Permission denied","level":"error","time":"2022-02-13T14:05:25.000427144Z"}
{"msg":"open executable: Permission denied","level":"error","time":"2022-02-13T14:09:53.000118101Z"}
[root@Dell ~]#
What is written to oci-log inside the starting container? It looks like OCI permission denied inside oci-log is the same as in the podman debug log. Image's CMD is bash and its owner is root as expected. But execution is allowed to everybody. Image creator was some root from Canonical but runner is me, rootless.
Is ls -alZ /run/user/1000/containers/storage/btrfs-containers/f762d93c06e0d37e4f0df9dec63388f967ee3bbb1e3259b9dfc7a86abaa5d431/
total 0
drwx------. 3 root root unconfined_u:object_r:user_tmp_t:s0 60 Feb 13 12:19 .
correct?

@rhatdan
Copy link
Member

rhatdan commented Feb 14, 2022

Run semodule -DB before the failure, and see if additional AVC messages show up.

ausearch -m avc -ts recent

@PavelSosin-320
Copy link

Actually, I expect to see that container in the runtime references the "writable snapshot" of the container in the user's container storage as btrfs assumes but don't see it. Only container's pre-declared mount points have to be not a snapshot

@skyblaster skyblaster mentioned this issue Aug 7, 2022
Closed
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ignore entrypoint=[""] rhatdan/podman
6 participants
@kofemann @rhatdan @ebousse @mheon @openshift-ci-robot @PavelSosin-320