Podman does not detect volume from the volume plugin, unlike docker

[mangeshpanche]

/kind bug

Description
If volume is created out of band in the volume plugin, the volume is not detected by podman, unlike docker. This affects deployment of podman in clustered environment where the volume is created on shared storage from one node, is not detected by podman running on the other node.

Steps to reproduce the issue:

1. Create volume using custom volume driver on shared storage.

2. The podman running on the other node, does not detect the volume, using same custom volume driver.

Describe the results you received:
Podman can detect only volumes created locally using podman create.

Describe the results you expected:
Podman should detect all the volumes seen by the volume plugin driver.

Additional information you deem important (e.g. issue happens only occasionally):
In case of docker, all the volumes seen by the volume plugin driver reflected inside "docker volume ls".

Output of podman version:

# podman version
Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.17.7

Built:      Tue Mar 15 12:15:06 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

# podman info --debug
host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: ae467a0c8001179d4d0adf4ada381108a893d7ec'
  cpus: 8
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: journald
  hostname: eagappflx038
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-305.40.2.el8_4.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 49152798720
  memTotal: 66800738304
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.0.0-74.rc95.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.13
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/opt/veritas/flex/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 68708425728
  swapTotal: 68719472640
  uptime: 140h 12m 38.68s (Approximately 5.83 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
  - filevol
  - veritas
registries:
  docker.io:
    Blocked: true
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: docker.io
  registry.access.redhat.com:
    Blocked: true
    Insecure: false
    Location: registry.access.redhat.com
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: registry.access.redhat.com
  search:
  - console:8443
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 8
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1647371706
  BuiltTime: Tue Mar 15 12:15:06 2022
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q podman
podman-4.0.2-1.module_el8.7.0+1106+45480ee0.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical

[Luap99]

@mheon PTAL

[mheon]

This was a deliberate choice on our part. We use the Podman database as our single source of truth on what volumes exist on the system at present, and where. The main reason for this is a simple question: what happens if I have two volume plugins, and in each of them I out-of-band create a volume named "testvol". We can only have one volume with the same name, so what do we do? Displaying only the first one we encounter means that a podman volume rm testvol doesn't actually remove it - there's another. Do we try and prefix the name with the volume driver - if so, what happens to existing containers that want to mount "testvol" which technically no longer exists, replaced with "plugin1/testvol" or similar.

There are a hundred small questions like this. All of them make sourcing volume definitions from outside of Podman complicated - not impossible, but a very significant amount of work. We can potentially revisit this decision in the future but it will require a significant time and code investment to adequately handle the edge cases it introduces.

[mangeshpanche]

Thanks @mheon for the reply.

Docker supports this configuration, where in volume available in the plugin are detected and listed. This is a must have requirement for deployment in the clustered environment. Not supporting this use case would make it difficult for migration from docker to podman in those deployments.

Presence of same volume name in two different plugin can be an invalid configuration or a corner case and the behavior can be defined and handled accordingly. Even in case of docker the behavior is indeterministic, if two drivers report the same volume.

The following options can be considered:

1. If the same volume is reported by two plugins, the remove of the volume has to be done by specifying the driver.
2. A mechanism to discover the volumes from the plugin via podman volume refresh/reload
3. A option to bypass the database for a plugin. The volume create/delete for this plugin will be sent to the driver directly.

[mangeshpanche]

Is there any update on this?

[mheon]

No. If you'd like this escalated and have a RHEL support contract, I suggest opening an RFE bugzilla to request that this work be prioritized.