sometimes, /usr/bin/fuse-overlayfs remains after container exist.

[coldbloodx]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1.start a container:
[xianwu@rcn07 ~]$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
34335b25d1e4 localhost/centos:latest /home/xianwu/.lsb... About a minute ago Exited (0) 26 seconds ago ulaworks008.job.4039

2. after the container exit, check container related processes, the fuse-overlayfs remains
   [xianwu@rcn07 ~]$ ps -ewf |grep 205298
   xianwu 205298 1 0 08:21 ? 00:00:00 /usr/bin/fuse-overlayfs -o lowerdir=/podmanfs/overlay/l/D6ZILYLI2BLBDF65YSKZRRBKOB:/podmanfs/overlay/l/UNZMQEUSSGU7ZCGQZ6ICUEFNHX,upperdir=/podmanfs/overlay/6ea6e0c4da2c6f350a07279c6d71525be9f8c9cb66cc53272b6d42a8a60b26f7/diff,workdir=/podmanfs/overlay/6ea6e0c4da2c6f350a07279c6d71525be9f8c9cb66cc53272b6d42a8a60b26f7/work,uidmapping=0:1:33857:33857:0:1:33858:33858:31679,gidmapping=0:1:10007:10007:0:1:10008:10008:55529 /podmanfs/overlay/6ea6e0c4da2c6f350a07279c6d71525be9f8c9cb66cc53272b6d42a8a60b26f7/merged

Describe the results you received:
the /usr/bin/fuse-overlayfs remains.

Describe the results you expected:
the /usr/bin/fuse-overlayfs exists automatically.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

[xianwu@rcn07 ~]$ podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64```

**Output of `podman info --debug`:**

[xianwu@rcn07 ~]$ podman info --debug
debug:
compiler: gc
git commit: ""
go version: go1.13.4
podman version: 1.6.4
host:
BuildahVersion: 1.12.0-dev
CgroupVersion: v1
Conmon:
package: conmon-2.0.6-1.module_el8.2.0+305+5e198a41.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.6, commit: a2b11288060ebd7abd20e0b4eb1a834bbf0aec3e'
Distribution:
distribution: '"centos"'
version: "8"
IDMappings:
gidmap:
- container_id: 0
host_id: 10007
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 33857
size: 1
- container_id: 1
host_id: 100000
size: 65536
MemFree: 5225304064
MemTotal: 6244052992
OCIRuntime:
name: runc
package: runc-1.0.0-65.rc10.module_el8.2.0+305+5e198a41.x86_64
path: /usr/bin/runc
version: 'runc version spec: 1.0.1-dev'
SwapFree: 3951292416
SwapTotal: 3999264768
arch: amd64
cpus: 2
eventlogger: journald
hostname: rcn07
kernel: 4.18.0-193.el8.x86_64
os: linux
rootless: true
slirp4netns:
Executable: /bin/slirp4netns
Package: slirp4netns-0.4.2-3.git21fdece.module_el8.2.0+305+5e198a41.x86_64
Version: |-
slirp4netns version 0.4.2+dev
commit: 21fdece2737dc24ffa3f01a341b8a6854f8b13b4
uptime: 19h 48m 3.57s (Approximately 0.79 days)
registries:
blocked: null
insecure: null
search:

• registry.access.redhat.com
• registry.redhat.io
• docker.io
  store:
  ConfigFile: /home/xianwu/.config/containers/storage.conf
  ContainerStore:
  number: 40
  GraphDriverName: overlay
  GraphOptions:
  overlay.mount_program:
  Executable: /usr/bin/fuse-overlayfs
  Package: fuse-overlayfs-0.7.2-5.module_el8.2.0+305+5e198a41.x86_64
  Version: |-
  fuse-overlayfs: version 0.7.2
  FUSE library version 3.2.1
  using FUSE kernel interface version 7.26
  GraphRoot: /podmanfs
  GraphStatus:
  Backing Filesystem: extfs
  Native Overlay Diff: "false"
  Supports d_type: "true"
  Using metacopy: "false"
  ImageStore:
  number: 1
  RunRoot: /tmp/run-33857
  VolumePath: /podmanfs/storage/volumes

**Package info (e.g. output of `rpm -q podman` or `apt list podman`):**

(paste your output here)

**Additional environment details (AWS, VirtualBox, physical, etc.):**
centos 8.2 virtual machine on vmware.

[rhatdan]

Could you check if this still happens on podman 1.9.3 version which should be on RHEL8.2.1 now.

[coldbloodx]

I'll try it out later.

[giuseppe]

that is the equivalent of leaking a mount point. When that happens, can you do: podman unshare cat /proc/self/mountinfo ?

[coldbloodx]

just hit this this issue, see my below test.

no podman process here:
[xianwu@rcn07 ~]$ ps -ewf |grep podman
xianwu 689115 687539 0 08:52 pts/0 00:00:00 grep --color=auto podman

[xianwu@rcn07 ~]$ podman run -it localhost/centos hostname
d8be9f57555e
[xianwu@rcn07 ~]$ ps -ewf |grep podman
xianwu 689293 1 0 08:54 ? 00:00:00 podman
xianwu 689312 1 0 08:54 ? 00:00:00 /usr/bin/fuse-overlayfs -o lowerdir=/podmanfs/overlay/l/D6ZILYLI2BLBDF65YSKZRRBKOB:/podmanfs/overlay/l/UNZMQEUSSGU7ZCGQZ6ICUEFNHX,upperdir=/podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/diff,workdir=/podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/work /podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/merged. --> this one remains!!!
xianwu 689356 687539 0 08:54 pts/0 00:00:00 grep --color=auto podman

[xianwu@rcn07 ~]$ ps -ewf |grep podman
xianwu 689293 1 0 08:54 ? 00:00:00 podman
xianwu 689312 1 0 08:54 ? 00:00:00 /usr/bin/fuse-overlayfs -o lowerdir=/podmanfs/overlay/l/D6ZILYLI2BLBDF65YSKZRRBKOB:/podmanfs/overlay/l/UNZMQEUSSGU7ZCGQZ6ICUEFNHX,upperdir=/podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/diff,workdir=/podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/work /podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/merged
xianwu 689423 687539 0 08:55 pts/0 00:00:00 grep --color=auto podman

Here is output of: podman unshare cat /proc/self/mountinfo
[xianwu@rcn07 ~]$ podman unshare cat /proc/self/mountinfo
178 177 8:2 / / rw,relatime master:1 - ext4 /dev/sda2 rw
179 178 0:20 / /sys rw,nosuid,nodev,noexec,relatime master:2 - sysfs sysfs rw
180 179 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:3 - securityfs securityfs rw
181 179 0:24 / /sys/fs/cgroup ro,nosuid,nodev,noexec master:4 - tmpfs tmpfs ro,mode=755
182 181 0:25 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime master:5 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
183 181 0:28 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime master:6 - cgroup cgroup rw,net_cls,net_prio
184 181 0:29 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime master:7 - cgroup cgroup rw,cpu,cpuacct
185 181 0:30 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime master:8 - cgroup cgroup rw,freezer
186 181 0:31 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime master:9 - cgroup cgroup rw,pids
187 181 0:32 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime master:10 - cgroup cgroup rw,hugetlb
188 181 0:33 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,devices
189 181 0:34 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime master:12 - cgroup cgroup rw,perf_event
190 181 0:35 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime master:13 - cgroup cgroup rw,memory
191 181 0:36 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,cpuset
192 181 0:37 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,blkio
193 181 0:38 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,rdma
194 179 0:26 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:17 - pstore pstore rw
195 179 0:27 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime master:18 - bpf bpf rw,mode=700
196 179 0:39 / /sys/kernel/config rw,relatime master:19 - configfs configfs rw
197 179 0:8 / /sys/kernel/debug rw,relatime master:27 - debugfs debugfs rw
198 179 0:54 / /sys/fs/fuse/connections rw,relatime master:96 - fusectl fusectl rw
199 178 0:6 / /dev rw,nosuid master:20 - devtmpfs devtmpfs rw,size=3038356k,nr_inodes=759589,mode=755
200 199 0:21 / /dev/shm rw,nosuid,nodev master:21 - tmpfs tmpfs rw
201 199 0:22 / /dev/pts rw,nosuid,noexec,relatime master:22 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
202 199 0:40 / /dev/hugepages rw,relatime master:26 - hugetlbfs hugetlbfs rw,pagesize=2M
203 199 0:18 / /dev/mqueue rw,relatime master:28 - mqueue mqueue rw
204 178 0:23 / /run rw,nosuid,nodev master:23 - tmpfs tmpfs rw,mode=755
205 204 0:47 / /run/user/33857 rw,nosuid,nodev,relatime master:155 - tmpfs tmpfs rw,size=609768k,mode=700,uid=33857,gid=10007
206 204 0:49 / /run/user/0 rw,nosuid,nodev,relatime master:165 - tmpfs tmpfs rw,size=609768k,mode=700
207 178 0:4 / /proc rw,nosuid,nodev,noexec,relatime master:24 - proc proc rw
208 207 0:19 / /proc/sys/fs/binfmt_misc rw,relatime master:25 - autofs systemd-1 rw,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13750
209 178 0:41 / /var/lib/nfs/rpc_pipefs rw,relatime master:57 - rpc_pipefs sunrpc rw
210 178 0:44 / /home rw,relatime master:142 - autofs /etc/auto.homexian rw,fd=5,pgrp=651,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=17258
211 210 0:48 / /home/xianwu rw,relatime master:160 - nfs 9.111.248.224:/data/export/home/xianwu rw,vers=3,rsize=8192,wsize=8192,namlen=255,hard,noacl,proto=tcp,timeo=6000,retrans=2,sec=sys,mountaddr=9.111.248.224,mountvers=3,mountport=63759,mountproto=tcp,local_lock=none,addr=9.111.248.224
212 178 0:45 / /pcc rw,relatime master:147 - autofs /etc/auto.pccxian rw,fd=11,pgrp=651,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=17267
213 212 0:43 / /pcc/dev rw,relatime master:93 - nfs 9.111.248.224:/data/export/pcc_dev rw,vers=3,rsize=8192,wsize=8192,namlen=255,hard,noacl,proto=tcp,timeo=6000,retrans=2,sec=sys,mountaddr=9.111.248.224,mountvers=3,mountport=63759,mountproto=tcp,local_lock=none,addr=9.111.248.224
214 178 0:46 / /scratch rw,relatime master:151 - autofs /etc/auto.scratchxian rw,fd=17,pgrp=651,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=17271
215 178 0:52 / /lsf rw,relatime master:99 - nfs 9.111.251.179:/lsf rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=9.111.251.179,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=9.111.251.179
216 178 8:2 /podmanfs/overlay /podmanfs/overlay rw,relatime - ext4 /dev/sda2 rw
217 178 0:50 / /podmanfs/overlay-containers/d8be9f57555e519adaf46de02472bd45e1545c6647f91f5fb4471ccf3d908894/userdata/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=64000k,uid=33857,gid=10007
220 205 0:47 /netns /run/user/33857/netns rw,nosuid,nodev,relatime shared:102 master:155 - tmpfs tmpfs rw,size=609768k,mode=700,uid=33857,gid=10007
229 220 0:3 net:[4026532606] /run/user/33857/netns/cni-08c1003b-1edb-e284-8551-a39abf0172c4 rw shared:103 - nsfs nsfs rw
230 216 0:51 / /podmanfs/overlay/ef6d5c3b189742dc20ffe523f4152866d661273e5e8dedf70f4bdef5208ec8df/merged rw,nodev,noatime - fuse.fuse-overlayfs fuse-overlayfs rw,user_id=0,group_id=0,default_permissions,allow_other

[wwilson]

I have run into this issue pretty often, and it seems to exist on a wide selection of podman versions.

[wwilson]

You can see if you’re affected by running these commands:

ps aux | grep fuse-overlayfs | wc -l
podman ps -q | grep wc -l

The defunct fuse-overlayfs processes are owned by init and are sleeping (not zombies).

[rhatdan]

@giuseppe Thoughts on what is causing this? Do we need something in containers storage to remove these on shutdown?

[giuseppe]

@giuseppe Thoughts on what is causing this? Do we need something in containers storage to remove these on shutdown?

I think the cleanup process fails. I see sometimes leaked mounts from root as well. The difference with fuse-overlayfs is that when we leak the mount we also leak the process that is managing that mount

[rhatdan]

I see 4 fuse-overlayfs running in my homedir right now.

ps -eZ | grep fuse-
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 280864 ? 00:00:00 fuse-overlayfs
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 280932 ? 00:00:00 fuse-overlayfs
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 282384 ? 00:00:00 fuse-overlayfs
ps -eZ | grep podman
unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 110035 ? 00:00:00 podman pause

[rhatdan]

$ mount | grep fuse
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
gvfsd-fuse on /run/user/3267/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=3267,group_id=3267)
$ ps aux | grep fuse-overlayfs 
dwalsh    280864  0.0  0.0   7056  4692 ?        Ss   Jul28   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/D6MYXF5LJWYT63GXZH6ZQTKCIQ,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/work,context="system_u:object_r:container_file_t:s0:c15,c290" /home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/merged
dwalsh    280932  0.0  0.0   8076  5480 ?        Ss   Jul28   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/diff:/home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/empty,context="system_u:object_r:container_file_t:s0:c15,c290" /home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/merged
dwalsh    282384  0.0  0.0   5124  1732 ?        Ss   Jul28   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/D6MYXF5LJWYT63GXZH6ZQTKCIQ,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/work,context="system_u:object_r:container_file_t:s0:c119,c292" /home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/merged
dwalsh    562517  0.0  0.0 216220  2272 pts/2    S+   17:51   0:00 grep --color=auto fuse-overlayfs

So these are fuse-overlay that are not mounting anything?

[wwilson]

Could this possible dirty shutdown be the same root cause that sometimes corrupts ~/.local/share/containers/storage/overlay-containers/containers.json ?

[giuseppe]

So these are fuse-overlay that are not mounting anything?

so that seems like a fuse-overlayfs issue. Can you please strace these processes? In what syscall are they blocked?

[giuseppe]

$ mount | grep fuse
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
gvfsd-fuse on /run/user/3267/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=3267,group_id=3267)
$ ps aux | grep fuse-overlayfs
dwalsh 280864 0.0 0.0 7056 4692 ? Ss Jul28 0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/D6MYXF5LJWYT63GXZH6ZQTKCIQ,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/work,context="system_u:object_r:container_file_t:s0:c15,c290" /home/dwalsh/.local/share/containers/storage/overlay/9c3847385c56fda91796ec164a50066d744706ae0a8dbd6e22ef7745b2edf67d/merged
dwalsh 280932 0.0 0.0 8076 5480 ? Ss Jul28 0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/diff:/home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/empty,context="system_u:object_r:container_file_t:s0:c15,c290" /home/dwalsh/.local/share/containers/storage/overlay/5a0b3dffd3f565da9d8d0a9a16149614220fd142e88804ba019ffb8eb641ad45/merged
dwalsh 282384 0.0 0.0 5124 1732 ? Ss Jul28 0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/D6MYXF5LJWYT63GXZH6ZQTKCIQ,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/work,context="system_u:object_r:container_file_t:s0:c119,c292" /home/dwalsh/.local/share/containers/storage/overlay/fdb93042a85659ada459b3169f767bc03741d3e63a5a8e5409ff7f8ad8e907bc/merged
dwalsh 562517 0.0 0.0 216220 2272 pts/2 S+ 17:51 0:00 grep --color=auto fuse-overlayfs

can you try podman unshare mount | grep fuse

The mounts created by fuse-overlayfs are in the new user+mount namespace, not visible from the host

[rhatdan]

Yes the mount points were in the unshare mount namespace.

[wwilson]

I tried stracing a few of these like giuseppe suggested (not all of them because I umm... apparently have hundreds of these). They all appear to be hung on either read or splice on a file descriptor that does not appear to exist in my system.

[giuseppe]

They all appear to be hung on either read or splice on a file descriptor that does not appear to exist in my system.

so it is waiting for events from FUSE. I think that is expected. I still think the issue is that we fail the cleanup and don't even get to do the Unmount.

You can check if it is fuse-overlays to hang by doing:

podman unshare umount /path/to/the/merged/path/specified/in/the/fuse-overlayfs/cmdline`

Does fuse-overlayfs exit after that?

[wwilson]

It does indeed look like that causes the defunct fuse-overlays to exit.

So this is probably a podman shutdown bug?

[mheon]

If you are on Podman v2.0.3 or later, we should be printing full logs from the cleanup process (podman container cleanup) to syslog if the container is started with --log-level=debug. The failure should be in there.

[wwilson]

I am currently unable to change my podman version. @rhatdan it looked like you had also run into this? Can you try to get the logs that @mheon needs?

[rhatdan]

I have two running on my system right now.

dwalsh    869895  0.0  0.0   4520  1776 ?        Ss   Aug05   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/UONY5FHLNK2NTAIM3A5XG4R43O,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/243777689c53a0240a0579ee04d026bb9d7e0b9e3e6a31aa93478fc728b46689/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/243777689c53a0240a0579ee04d026bb9d7e0b9e3e6a31aa93478fc728b46689/work,context="system_u:object_r:container_file_t:s0:c615,c823" /home/dwalsh/.local/share/containers/storage/overlay/243777689c53a0240a0579ee04d026bb9d7e0b9e3e6a31aa93478fc728b46689/merged
dwalsh   1022603  0.0  0.0   4240  2592 ?        Ss   13:44   0:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/dwalsh/.local/share/containers/storage/overlay/l/4455UXPC5G6NMXLCSLAS45VFSF:/home/dwalsh/.local/share/containers/storage/overlay/l/OS5U2G5S46Y4BSPLFCX2JNIZ5T:/home/dwalsh/.local/share/containers/storage/overlay/l/V4NL7AHHESQZHGDHTW6CC7KRZZ:/home/dwalsh/.local/share/containers/storage/overlay/l/4AUPK5LUJQDWR7OQP3K4KYHKO2:/home/dwalsh/.local/share/containers/storage/overlay/l/DPGJTLOFCHWWZL6RIRQBSPZC5N:/home/dwalsh/.local/share/containers/storage/overlay/l/3HO3BBVOF7QZSWEZ7ZAPR2Q2DM:/home/dwalsh/.local/share/containers/storage/overlay/l/XU5LJTGPV7H7TALNYVWOCLK5LS:/home/dwalsh/.local/share/containers/storage/overlay/l/ELEYLJDEOFQPELSQNVDTSLMM2E:/home/dwalsh/.local/share/containers/storage/overlay/l/SLVLTGJIP2T5TT3GVGS2WQRXTS:/home/dwalsh/.local/share/containers/storage/overlay/l/MTF5NX5ITW7EVOU2VNTJ4MZW55:/home/dwalsh/.local/share/containers/storage/overlay/l/TIVZ2PBB2GCNTXEREVDBVY34O3:/home/dwalsh/.local/share/containers/storage/overlay/l/EP2XCXA4MHLGEXTM56SES6LSC3:/home/dwalsh/.local/share/containers/storage/overlay/l/VHT7FEVQODCAVUEYAIFJIIYFJ4:/home/dwalsh/.local/share/containers/storage/overlay/l/TYDNRNZUFA6PM37LEMI5IPDLSF:/home/dwalsh/.local/share/containers/storage/overlay/l/ZSKIIYOH63XKFUXDJ7KZ7YT77D,upperdir=/home/dwalsh/.local/share/containers/storage/overlay/054ae403adbe18c034e809a0a6fdcff121a0e289f4d9064c80dd72be7fbfbb9a/diff,workdir=/home/dwalsh/.local/share/containers/storage/overlay/054ae403adbe18c034e809a0a6fdcff121a0e289f4d9064c80dd72be7fbfbb9a/work,context="system_u:object_r:container_file_t:s0:c388,c958" /home/dwalsh/.local/share/containers/storage/overlay/054ae403adbe18c034e809a0a6fdcff121a0e289f4d9064c80dd72be7fbfbb9a/merged
$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

But I don't see any mounts in /proc/mounts even if I do a podman unshare.
I don't know how to get these to happen.

[giuseppe]

have you used pods? I think #7283 could help to clean up leaked mounts

[mheon]

Has anyone seem this since the release of Podman 2.0.5? I suspect it should be fixed with the pod cleanup code having merged

[rhatdan]

Closing, reopen if you have seen it.

[wwilson]

I've upgraded to Podman 2.0.6, and this problem looks like it is still very much a problem.

[wwilson]

What's the right way to reopen this? I'll try to collect the debugging info @mheon was looking for.

[wwilson]

Pretty weird -- I ran with --log-level=debug as suggested, and there is nothing from the shutdown process getting printed at all. Last things I see are:

DEBU[0000] Started container 6823e20fe3f4f7bd212d37f7bb8b5d7a1c9ade4b50761ac506060e7e8c9f179e 
DEBU[0000] Enabling signal proxying                     
DEBU[0000] Called run.PersistentPostRunE([a bunch of stuff])

[TomSweeneyRedHat]

@wwilson thanks for the follow-up. To reopen, you need to click on the "Reopen and comment" button. However, I think that only shows up for members of the "Containers" organization and the reporter of the issue. I'll go ahead and reopen the issue for you assuming that the button is not being displayed.

[wwilson]

Update: the fuse-overlayfs process is killed if the halted container is destroyed. Dunno if that helps narrow down on where the issue could be.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I don't believe we have this issue in latest podman in github or fuse-overlayfs in github.