Podman failed to mount runtime directory for rootless netns: no such file or directory #13703

fraschm1998 · 2022-03-29T20:15:48Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

podman build .
podman-compose up

Describe the results you received:
Podman failed to mount runtime directory for rootless netns: no such file or directory. I'm not sure if it's an issue with SELinux: https://dpaste.com/ANRM893KY

I found where the error message is printed (lines 128-138): https://fossies.org/linux/podman/libpod/networking_linux.go

echo $XDG_RUNTIME_DIR prints: /run/user/1000

Edit according to lines 114-119 it seems as tho I need to have the following binds mounted:

// The following bind mounts are needed
// 1. XDG_RUNTIME_DIR -> XDG_RUNTIME_DIR/rootless-netns/XDG_RUNTIME_DIR
// 2. /run/systemd -> XDG_RUNTIME_DIR/rootless-netns/run/systemd (only if it exists)
// 3. XDG_RUNTIME_DIR/rootless-netns/resolv.conf -> /etc/resolv.conf or XDG_RUNTIME_DIR/rootless-netns/run/symlink/target
// 4. XDG_RUNTIME_DIR/rootless-netns/var/lib/cni -> /var/lib/cni (if /var/lib/cni does not exists use the parent dir)
// 5. XDG_RUNTIME_DIR/rootless-netns/run -> /run

However in /run/user/1000 the only file related to netns is /run/user/1000/netns/rootless-netns-*

fd rootless in /

root@asus-g14 / # fd rootless
home/massimo/podman/cni/rootless-cni-infra
run/user/1000/libpod/tmp/rootless-netns.lock
run/user/1000/libpod/tmp/rootless-netns
run/user/1000/netns/rootless-netns-987c9a1aa493ae43558a
dev/shm/libpod_rootless_lock_1000
run/user/1000/libpod/tmp/rootless-netns/rootless-netns-slirp4netns.pid
var/db/repos/gentoo/metadata/md5-cache/sys-apps/rootlesskit-0.14.2
var/db/repos/gentoo/sys-apps/rootlesskit
var/db/repos/gentoo/sys-apps/rootlesskit/rootlesskit-0.14.2.ebuild
usr/share/doc/containerd-1.5.11/rootless.md.bz2
usr/share/selinux/targeted/include/services/rootlesskit.if
usr/share/selinux/strict/include/services/rootlesskit.if
usr/libexec/podman/rootlessport

root@asus-g14 / # fd rootless-netns
run/user/1000/libpod/tmp/rootless-netns.lock
run/user/1000/libpod/tmp/rootless-netns
run/user/1000/libpod/tmp/rootless-netns/rootless-netns-slirp4netns.pid
run/user/1000/netns/rootless-netns-987c9a1aa493ae43558a
root@asus-g14 / # v /run/user/1000/libpod/tmp/rootless-netns/resolv.conf

Edit I tried starting up the container again after doing podman system reset and get this:

ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: The name org.freedesktop.systemd1 was not provided by any .service files
Error: unable to start container 159d3d6111ff17c9c33108d37a547fb382e4909300ee07873f842ae2ac505fb7: could not create relabel rootless-netns run directory: setxattr /run/user/1000/libpod/tmp/
rootless-netns/run: invalid argument

Output of podman version:

podman version 4.0.2

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: app-containers/conmon-2.1.0
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.0, commit: v2.1.0'
  cpus: 16
  distribution:
    distribution: gentoo
    version: "2.8"
  eventLogger: file
  hostname: asus-g14
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.16-gentoo-x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 34120912896
  memTotal: 41527205888
  networkBackend: cni
  ociRuntime:
    name: crun
    package: app-containers/crun-1.4.4
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.1.12
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 27m 10.13s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
  docker.io/library:
    Blocked: false
    Insecure: false
    Location: docker.io/library
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io/library
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  configFile: /home/massimo/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/massimo/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 32
  runRoot: /run/user/1000/containers
  volumePath: /home/massimo/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646619707
  BuiltTime: Sun Mar  6 21:21:47 2022
  GitCommit: 342c8259381b63296e96ad29519bd4b9c7afbf97
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

app-containers/podman-4.0.2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Gentoo SeLinux Openrc

The text was updated successfully, but these errors were encountered:

Luap99 · 2022-03-30T09:57:26Z

The error ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: The name org.freedesktop.systemd1 was not provided by any .service files should be ignored since you do not run systemd. I will fix this so that we do not log the error when you do not use systemd.

Error: unable to start container 159d3d6111ff17c9c33108d37a547fb382e4909300ee07873f842ae2ac505fb7: could not create relabel rootless-netns run directory: setxattr /run/user/1000/libpod/tmp/ rootless-netns/run: invalid argument sound like selinux problem. Is selinux enabled on your system?

When you run podman on a non systemd system we should not try to move the process under a new systemd scope. [NO NEW TESTS NEEDED] Ref containers#13703 Signed-off-by: Paul Holzinger <[email protected]>

fraschm1998 · 2022-03-30T14:27:28Z

Error: unable to start container 159d3d6111ff17c9c33108d37a547fb382e4909300ee07873f842ae2ac505fb7: could not create relabel rootless-netns run directory: setxattr /run/user/1000/libpod/tmp/ rootless-netns/run: invalid argument sound like selinux problem. Is selinux enabled on your system?

I do have selinux installed but getenforce reports that it's in permissive mode. These are the logs I could find related to podman in /var/log/messages: http://dpaste.com/FRZSBJKZL

Edit and podman related entried in audit.log: http://dpaste.com/4BEM47HHL

Luap99 · 2022-03-30T14:39:33Z

We are tying to label it with the iptables_var_run_t type, is that available on your system?
@rhatdan Any idea how to check what package provides this type?

fraschm1998 · 2022-03-30T14:42:27Z

We are tying to label it with the iptables_var_run_t type, is that available on your system? @rhatdan Any idea how to check what package provides this type?

These are the packages I have related to iptables:

 ~  ip                                                                                                                                                                      ok | 10:11:41
ip                        ip6tables-legacy-restore  ipcmk                     ippeveprinter             iptables-legacy           iptables-save
ip6tables                 ip6tables-legacy-save     ipcrm                     ipptool                   iptables-legacy-restore   iptables-xml
ip6tables-apply           ip6tables-restore         ipcs                      iptables                  iptables-legacy-save      iptunnel
ip6tables-legacy          ip6tables-save            ipmaddr                   iptables-apply            iptables-restore

Luap99 · 2022-03-30T15:23:07Z

This is not about missing binaries, you are missing a selinux policy which provides this selinux type.
@rhatdan ideas how to debug this?

rhatdan · 2022-03-30T18:14:54Z

This looks like you have a custom policy and not the default policy. Your policy does not support MLS labels (MCS) which Podman relies on for container separation.

Someone wrote podman_t types, and the person writing and shipping this policy needs to diagnose and figure out what is going on. This is not something the upstream podman team can fix.

rhatdan · 2022-03-30T18:15:30Z

I would be willing to work with whomever wrote this policy, but this is not a Podman issue.

When you run podman on a non systemd system we should not try to move the process under a new systemd scope. [NO NEW TESTS NEEDED] Ref containers#13703 Signed-off-by: Paul Holzinger <[email protected]>

fraschm1998 · 2022-03-30T20:38:34Z

I would be willing to work with whomever wrote this policy, but this is not a Podman issue.

Could it be related to this? opencontainers/selinux#171

rhatdan · 2022-03-30T20:42:41Z

It could be, but I thought some fixes had gone in for this.

devinrsmith · 2022-05-17T04:07:23Z

I ran into this, found the ultimate issue was

exec: "slirp4netns": executable file not found in $PATH

sudo dnf install slirp4netns seemed to fix the issue.

0xC0ncord · 2022-06-21T13:30:43Z

I would be willing to work with whomever wrote this policy, but this is not a Podman issue.

@rhatdan For the record, I'm certain this is the policy I originally wrote and upstreamed to refpolicy which is now part of Gentoo's policy. I agree I don't think it's a bug in podman but unfortunately I can't provide more details without seeing any AVCs, and I never ran into this issue during the policy's development.

opencontainers/selinux#171 is a separate issue.

fraschm1998 · 2022-06-23T13:41:37Z

I can't provide more details without seeing any AVCs

This is the most recent audit log: http://sprunge.us/2a3zQC

rhatdan · 2022-06-23T14:38:08Z

@fraschm1998 Looks like your system is badly mislabeled. touch /.autorelabel; reboot
Will cleanup labeling.

0xC0ncord · 2022-06-23T14:41:12Z

This is the most recent audit log: http://sprunge.us/2a3zQC

The behavior for running podman as an unconfined user in refpolicy has changed recently, but that change is not part of Gentoo's policy yet.

If you aren't hitting this issue while SELinux is in permissive mode, I suggest filing a bug in the Gentoo bug tracker since that would indicate the problem is outside the scope of Podman.

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 29, 2022

Luap99 added the network Networking related issue or feature label Mar 30, 2022

Luap99 mentioned this issue Mar 30, 2022

rootless netns: move process to scope only with systemd #13712

Merged

rhatdan closed this as completed Mar 30, 2022

fraschm1998 mentioned this issue Mar 31, 2022

Podman failed to mount runtime directory for rootless netns: no such file or directory opencontainers/selinux#175

Closed

edsantiago mentioned this issue Jul 7, 2022

port forward range test: fix an oops #14855

Merged

Douile mentioned this issue Oct 1, 2022

Systemd check wrong on systems with hidepid=invisible #16022

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman failed to mount runtime directory for rootless netns: no such file or directory #13703

Podman failed to mount runtime directory for rootless netns: no such file or directory #13703

fraschm1998 commented Mar 29, 2022

Luap99 commented Mar 30, 2022

fraschm1998 commented Mar 30, 2022 •

edited

Loading

Luap99 commented Mar 30, 2022

fraschm1998 commented Mar 30, 2022

Luap99 commented Mar 30, 2022

rhatdan commented Mar 30, 2022 •

edited

Loading

rhatdan commented Mar 30, 2022

fraschm1998 commented Mar 30, 2022

rhatdan commented Mar 30, 2022

devinrsmith commented May 17, 2022

0xC0ncord commented Jun 21, 2022

fraschm1998 commented Jun 23, 2022

rhatdan commented Jun 23, 2022

0xC0ncord commented Jun 23, 2022

Podman failed to mount runtime directory for rootless netns: no such file or directory #13703

Podman failed to mount runtime directory for rootless netns: no such file or directory #13703

Comments

fraschm1998 commented Mar 29, 2022

Luap99 commented Mar 30, 2022

fraschm1998 commented Mar 30, 2022 • edited Loading

Luap99 commented Mar 30, 2022

fraschm1998 commented Mar 30, 2022

Luap99 commented Mar 30, 2022

rhatdan commented Mar 30, 2022 • edited Loading

rhatdan commented Mar 30, 2022

fraschm1998 commented Mar 30, 2022

rhatdan commented Mar 30, 2022

devinrsmith commented May 17, 2022

0xC0ncord commented Jun 21, 2022

fraschm1998 commented Jun 23, 2022

rhatdan commented Jun 23, 2022

0xC0ncord commented Jun 23, 2022

fraschm1998 commented Mar 30, 2022 •

edited

Loading

rhatdan commented Mar 30, 2022 •

edited

Loading