Can't mount or unmount filesystem with legacy mountpoint

[gg7]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	14.04.5 LTS
Linux Kernel	4.4.0-62-generic #83~14.04.1-Ubuntu
Architecture	x86_64
ZFS Version	v0.6.5.9-1~trusty, ZFS pool version 5000, ZFS filesystem version 5
SPL Version	v0.6.5.9-1~trusty

Problem

After unmounting a ZFS filesystem sometimes I can't re-mount it. Example:

# From history, I did this:
george@george:~$ sudo umount /var/lib/docker
george@george:~$ sudo mount --bind /old-docker /var/lib/docker
george@george:~$ sudo umount /var/lib/docker

# now:
george@george:~$ grep docker /etc/mtab /proc/mounts
george@george:~$ mount | grep docker

george@george:~$ sudo mount -t zfs zp0/docker /var/lib/docker
filesystem 'zp0/docker' is already mounted
george@george:~$ sudo umount /var/lib/docker
umount: /var/lib/docker: not mounted

george@george:~$ sudo umount -t zfs zp0/docker
umount: zp0/docker: not found
george@george:~$ sudo zfs umount zp0/docker
cannot unmount 'zp0/docker': legacy mountpoint
use umount(1M) to unmount this filesystem

Workaround

george@george:~$ sudo rmdir /var/lib/docker
george@george:~$ sudo mkdir /var/lib/docker
george@george:~$ sudo mount /var/lib/docker

I have experienced this several times and it is somewhat annoying.

[tuxoko]

Did you have any mount namespaces with zp0/docker mounted?

[gg7]

Did you have any mount namespaces with zp0/docker mounted?

Not intentionally at least. The Docker daemon was stopped but maybe it didn't clean up after itself?

george@george:~$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 807
Server Version: 1.11.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: zfs
 Dirs: 1152
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-62-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.28 GiB
Name: george
ID: <whatever>
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

How can I list all mount namespaces on my system?

[tuxoko]

I don't know if there's any tool that does that. But you can scan through all process through procfs.
sudo sh -c 'for i in /proc/*/ns/mnt; do ls -l $i; done' | uniq -f 10
(you might need to change 10 to other number)
And then do findmnt -N <pid>

[pacocc]

Also unable to unmount zfs volumes with mountpoint=legacy under Ubuntu 16.04.2. Certain (rsync vs other filesystem) issues cause the process to hang, and it can't be killed because of the mountpoint=legacy setting, which causes "umount -f" to also fail. Toggling mountpoint=/some/path allows the rsync process to be killed after it hangs and so the fs can be unmounted. havent tried "umount -f" with the zf process running.

[snhrdt]

Same problem here: I ran a keycloak docker container, stopped it, wanted to restart, and now docker tells me:

# docker start keycloak
Error response from daemon: error creating zfs mount of z/docker/6f5c6faba0a81ec262381bdc65a978cda6426170ad384fe8721babe026ea2b49 to /var/lib/docker/zfs/graph/6f5c6faba0a81ec262381bdc65a978cda6426170ad384fe8721babe026ea2b49: device or resource busy Error: failed to start containers: keycloak

Any idea how to solve this?