Podman hanging #5350

AstraLuma · 2020-02-27T23:43:28Z

/kind bug

Description

Might be #5291, but I'm not using systemd-spawned containers.

I'm consistently having problems with podman hanging

Steps to reproduce the issue:
Ok, this is a multi-terminal thing, but I'll try my best.

I'm starting with a fresh podman instance with nothing else on the system.

🐚 podman run --rm --publish 5984:80 nginx
<pulls image>
<waits>

$ podman ps
CONTAINER ID  IMAGE                           COMMAND               CREATED         STATUS             PORTS                 NAMES
f2ee8b412cc8  docker.io/library/nginx:latest  nginx -g daemon o...  12 seconds ago  Up 12 seconds ago  0.0.0.0:5984->80/tcp  boring_chaum
$ podman rm -f --latest
f2ee8b412cc8cb0a63652018c813d21f978cd3819bc6ef0891fc2dd97c52fa3f

(The command in 🐚 exits)

🐚 podman run --rm --publish 5984:80 nginx
<waits>

$ podman ps
<waits>
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

$ podman rm -f --all
<waits>                                                                                                                     
Error: error removing container d498db43fe12dea382155bcb3c35d12c92c5328f7ea1bafccda94724f2d69d10 root filesystem: unlinkat /home/astraluma/.local/share/containers/storage/overlay-containers/d498db43fe12dea382155bcb3c35d12c92c5328f7ea1bafccda94724f2d69d10/userdata/shm: device or resource busy

Ctrl-C in 🐚 does work, and the other two commands run (producing the output above).

Describe the results you expected:

No hanging and no weird errors.

Additional information you deem important (e.g. issue happens only occasionally):

I'm running sudo rm -r ~/.local/share/containers/ between runs to consistently get the above behavior.

Output of podman version:

Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.8
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.8
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.9, commit: unknown'
  Distribution:
    distribution: debian
    version: unknown
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 380334080
  MemTotal: 8229736448
  OCIRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.0~rc10+dfsg1
      commit: 1.0.0~rc10+dfsg1-1
      spec: 1.0.1-dev
  SwapFree: 8390701056
  SwapTotal: 8467247104
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: cayde7
  kernel: 5.4.0-4-amd64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: 'slirp4netns: /usr/bin/slirp4netns'
    Version: |-
      slirp4netns version 0.4.3
      commit: 2244b9b6461afeccad1678fac3d6e478c28b4ad6
  uptime: 1h 7m 33.24s (Approximately 0.04 days)
registries:
  search:
  - docker.io
store:
  ConfigFile: /home/astraluma/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  GraphRoot: /home/astraluma/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /run/user/1000
  VolumePath: /home/astraluma/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 1.8.0~7 amd64 [installed]
podman/unknown 1.8.0~7 arm64
podman/unknown 1.8.0~7 armhf
podman/unknown 1.8.0~7 ppc64el

(Project Kubic builds)

Additional environment details (AWS, VirtualBox, physical, etc.):

Real metal laptop, running debian, same for all my other bugs.

The text was updated successfully, but these errors were encountered:

AstraLuma · 2020-02-27T23:44:54Z

Oh, and even though podman rm throws an error, the container is no longer listed in podman ps.

mheon · 2020-02-28T00:34:04Z

Can you reproduce with a podman run command with --log-level=debug added and provide the output?

AstraLuma · 2020-02-28T02:50:48Z

DEBU[0000] Reading configuration file "/home/astraluma/.config/containers/libpod.conf" 
DEBU[0000] Rewriting /home/astraluma/.config/containers/libpod.conf for CGroup v2 upgrade 
DEBU[0000] Merged system config "/home/astraluma/.config/containers/libpod.conf": &{{false false false true true true} 0 {   [] [] []} /home/astraluma/.local/share/containers/storage/volumes docker://  runc map[runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [] [] [] [/usr/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] cgroupfs /usr/libexec/podman/catatonit /home/astraluma/.local/share/containers/storage/libpod /run/user/1000/libpod/tmp -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]  []   k8s.gcr.io/pause:3.1 /pause true true  2048 shm journald  ctrl-p,ctrl-q false false} 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/astraluma/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/astraluma/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /home/astraluma/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/astraluma/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] No store required. Not opening container store. 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/sbin/runc"               
INFO[0000] running as rootless                          
DEBU[0000] Reading configuration file "/home/astraluma/.config/containers/libpod.conf" 
DEBU[0000] Rewriting /home/astraluma/.config/containers/libpod.conf for CGroup v2 upgrade 
DEBU[0000] Merged system config "/home/astraluma/.config/containers/libpod.conf": &{{false false false true true true} 0 {   [] [] []} /home/astraluma/.local/share/containers/storage/volumes docker://  runc map[runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [] [] [] [/usr/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] cgroupfs /usr/libexec/podman/catatonit /home/astraluma/.local/share/containers/storage/libpod /run/user/1000/libpod/tmp -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]  []   k8s.gcr.io/pause:3.1 /pause true true  2048 shm journald  ctrl-p,ctrl-q false false} 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/astraluma/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/astraluma/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /home/astraluma/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/astraluma/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/sbin/runc"               
DEBU[0000] parsed reference into "[overlay@/home/astraluma/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/nginx:latest" 
DEBU[0000] parsed reference into "[overlay@/home/astraluma/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@a1523e859360df9ffe2b31a8270f5e16422609fe138c1636383efdc34b9ea2d6" 
DEBU[0000] exporting opaque data as blob "sha256:a1523e859360df9ffe2b31a8270f5e16422609fe138c1636383efdc34b9ea2d6" 
DEBU[0000] Using slirp4netns netmode                    
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 1 for container a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893 
DEBU[0000] parsed reference into "[overlay@/home/astraluma/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@a1523e859360df9ffe2b31a8270f5e16422609fe138c1636383efdc34b9ea2d6" 
DEBU[0000] exporting opaque data as blob "sha256:a1523e859360df9ffe2b31a8270f5e16422609fe138c1636383efdc34b9ea2d6" 
DEBU[0000] created container "a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893" 
DEBU[0000] container "a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893" has work directory "/home/astraluma/.local/share/containers/storage/overlay-containers/a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893/userdata" 
DEBU[0000] container "a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893" has run directory "/run/user/1000/overlay-containers/a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893/userdata" 
DEBU[0000] New container created "a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893" 
DEBU[0000] container "a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893" has CgroupParent "/libpod_parent/libpod-a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893" 
DEBU[0000] Not attaching to stdin                       
DEBU[0000] Made network namespace at /run/user/1000/netns/cni-65e97db9-1983-9223-0a4a-93fcf2a163bc for container a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893 
DEBU[0000] overlay: mount_data=lowerdir=/home/astraluma/.local/share/containers/storage/overlay/l/CFOR4VZ7YWRQ5UGZZFZF2E4NQD:/home/astraluma/.local/share/containers/storage/overlay/l/OPG5M5WPVRVMNBSOG3YDATTADM:/home/astraluma/.local/share/containers/storage/overlay/l/ZF43HFGOXAJGUCMV5SNGP4BCQL,upperdir=/home/astraluma/.local/share/containers/storage/overlay/238fa502d54e01999a8fd88d04dc199042075a835406e6560e16c9b22bfe270b/diff,workdir=/home/astraluma/.local/share/containers/storage/overlay/238fa502d54e01999a8fd88d04dc199042075a835406e6560e16c9b22bfe270b/work 
DEBU[0000] mounted container "a79c03ee14728bfa3648e2077442eb940fccb187cb66167a1cefccc5c6a0d893" at "/home/astraluma/.local/share/containers/storage/overlay/238fa502d54e01999a8fd88d04dc199042075a835406e6560e16c9b22bfe270b/merged" 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-65e97db9-1983-9223-0a4a-93fcf2a163bc tap0 
DEBU[0000] rootlessport: time="2020-02-27T21:49:59-05:00" level=info msg="starting parent driver" 
DEBU[0000] rootlessport: time="2020-02-27T21:49:59-05:00" level=info msg="opaque=map[builtin.readypipepath:/run/user/1000/libpod/tmp/rootlessport203537987/.bp-ready.pipe builtin.socketpath:/run/user/1000/libpod/tmp/rootlessport203537987/.bp.sock]" 
DEBU[0000] rootlessport: time="2020-02-27T21:49:59-05:00" level=info msg="starting child driver in child netns (\"/proc/46082/exe\" [containers-rootlessport-child])" 
DEBU[0000] rootlessport: time="2020-02-27T21:49:59-05:00" level=info msg="waiting for initComplete"

mheon · 2020-02-28T05:49:22Z

Seems like slirp. @giuseppe @AkihiroSuda PTAL

github-actions · 2020-03-30T00:04:15Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2020-03-30T10:38:51Z

@giuseppe Did you ever get a chance to look at this?

giuseppe · 2020-03-30T11:09:01Z

I think this can be fixed by: 647dc33

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Feb 27, 2020

AstraLuma mentioned this issue Feb 27, 2020

Podman hanging with couchdb #5351

Closed

mheon added the rootless label Feb 28, 2020

TomSweeneyRedHat mentioned this issue Feb 28, 2020

build: specify input fd to buildah #5354

Merged

github-actions bot added the stale-issue label Mar 30, 2020

rhatdan closed this as completed Jun 9, 2020

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman hanging #5350

Podman hanging #5350

AstraLuma commented Feb 27, 2020

AstraLuma commented Feb 27, 2020

mheon commented Feb 28, 2020

AstraLuma commented Feb 28, 2020

mheon commented Feb 28, 2020

github-actions bot commented Mar 30, 2020

rhatdan commented Mar 30, 2020

giuseppe commented Mar 30, 2020

Podman hanging #5350

Podman hanging #5350

Comments

AstraLuma commented Feb 27, 2020

AstraLuma commented Feb 27, 2020

mheon commented Feb 28, 2020

AstraLuma commented Feb 28, 2020

mheon commented Feb 28, 2020

github-actions bot commented Mar 30, 2020

rhatdan commented Mar 30, 2020

giuseppe commented Mar 30, 2020