Unable to cleanly exit from containers using NFS volumes created by podman volume create

[toddhpoole]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Based on our reading of the 1.6.0 changelog, podman volume create can now "create and mount volumes with options, allowing volumes backed by NFS." When we try to exercise this feature and create an NFS-backed volume and attach it to a container, we're then unable cleanly exit from that container.

Documentation covering NFS volumes is non-existent, so if this is user error, please advise. Either way, expanding the Examples section of podman-volume-create.1.md and podman-run.1.md with more examples, including several NFS ones, would be helpful.

Steps to reproduce the issue:

1. Create a volume backed by an NFS filesystem (guessing at the invocation here... again, there are no NFS examples in the documentation to reference).

$ podman volume create --opt type=nfs --opt o=addr=192.168.2.126,rw --opt device=:/exports/test test_nfs_vol
test_nfs_vol

2. Confirm that podman is aware of the volume.

$ podman volume inspect --all
[
     {
          "Name": "test_nfs_vol",
          "Driver": "local",
          "Mountpoint": "/home/testuser/.local/share/containers/storage/volumes/test_nfs_vol/_data",
          "CreatedAt": "2019-10-12T20:25:25.000893895-07:00",
          "Labels": {
               
          },
          "Scope": "local",
          "Options": {
               
          }
     }
]

3. Try to run a container with the volume attached:

$ podman run --rm --interactive --tty --volume test_nfs_vol:/mnt/test test_container

4. Exit the container:

[root@0452086601a2 /]# exit
exit
ERRO[1085] Error removing container 0452086601a208847ae52d4a224d4779e435249159118f6da06b1fa5ae11d3dd: error cleaning up container 0452086601a208847ae52d4a224d4779e435249159118f6da06b1fa5ae11d3dd: error unmounting container 0452086601a208847ae52d4a224d4779e435249159118f6da06b1fa5ae11d3dd storage: error unmounting volume test_nfs_vol for container 0452086601a208847ae52d4a224d4779e435249159118f6da06b1fa5ae11d3dd: error unmounting volume test_nfs_vol: invalid argument

Describe the results you received:
The container exits with ERRO[1085] and a long run-on error message stating "Error removing container [...]: error cleaning up container [...]: error unmounting container [...] storage: error unmounting volume test_nfs_vol for container [...]: error unmounting volume test_nfs_vol: invalid argument"

Describe the results you expected:
The container exits cleanly; no error message printed.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:            1.6.1
RemoteAPI Version:  1
Go Version:         go1.12.9
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.9
  podman version: 1.6.1
host:
  BuildahVersion: 1.11.2
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.1-1.fc30.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.1, commit: 4346fbe0b2634b05857973bdf663598081240374'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 31369199616
  MemTotal: 33539690496
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 16840126464
  SwapTotal: 16840126464
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: host0
  kernel: 5.2.18-200.fc30.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-4.git19d199a.fc30.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.2
      commit: 19d199a6ca424fcf9516320a327cedad85cf4dfb
  uptime: 3h 28m 18.42s (Approximately 0.12 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/testuser/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.6.3-2.0.dev.git46c0f8e.fc30.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.6.3
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/testuser/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /run/user/1000
  VolumePath: /home/testuser/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-1.6.1-2.fc30.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Fresh minimal install of Fedora 30 with yum -y install vim nfs-utils podman buildah.

Exports are visible to host:

$ showmount -e 192.168.2.126
Export list for 192.168.2.126:
/exports/test  192.168.2.0/24

Exports can be mounted outside of podman using mount 192.168.2.126:/exports/test /mnt/test.

[mheon]

This is completely expected when the volume fails to unmount. The container will be successfully removed, but we will report errors that occurred along the way.

[mheon]

The error that causes the failure to unmount will be handled in #4248