containers.podman.podman connection plugin does not work with podman 2 anymore #70

tyll · 2020-07-03T20:10:42Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

The podman connection plugin does not work anymore in non-root mode with Podam 2

Steps to reproduce the issue:

podman run --name c8 --rm --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro docker.io/nmstate/centos8-nmstate-dev
ansible -i c8, -c containers.podman.podman -m setup all

Describe the results you received:

[WARNING]: Unhandled error in Python interpreter discovery for host c8: Expecting value: line 1 column 1 (char 0)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: FileNotFoundError: [Errno 2] No such file or directory: b'/home/till/.local/share/containers/storage/overlay/ce57833ce38f71131609ff6bd6419280bb2da69dc6026ae29a2701562b149d51/merged/root/.ansible/tmp/ansible-tmp-1593806813.5026007-1934528-92979037090521/AnsiballZ_setup.py'
c8 | FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}

Describe the results you expected:

Successful Ansible command

Additional information you deem important (e.g. issue happens only occasionally):

Output of ansible --version:

ansible 2.9.10
  config file = /home/till/.ansible.cfg
  configured module search path = ['/home/till/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.3 (default, May 29 2020, 00:00:00) [GCC 10.1.1 20200507 (Red Hat 10.1.1-1)]

Output of podman version:

podman version 2.0.1

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.1-1.fc32.x86_64

Command line and output of ansible run with high verbosity:

ansible -vvvvi c8, -c containers.podman.podman -m setup all
ansible 2.9.10
  config file = /home/till/.ansible.cfg
  configured module search path = ['/home/till/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.3 (default, May 29 2020, 00:00:00) [GCC 10.1.1 20200507 (Red Hat 10.1.1-1)]
Using /home/till/.ansible.cfg as config file
setting up inventory plugins
Parsed c8, inventory source with host_list plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python3.8/site-packages/ansible/plugins/callback/minimal.py
META: ran handlers
Using podman connection from collection
<c8> RUN [b'/usr/bin/podman', b'mount', b'c8']
<c8> RUN [b'/usr/bin/podman', b'exec', b'c8', b'/bin/sh', b'-c', b'echo ~ && sleep 0']
<c8> RUN [b'/usr/bin/podman', b'exec', b'c8', b'/bin/sh', b'-c', b'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir /root/.ansible/tmp/ansible-tmp-1593806986.5295613-1935085-94392905227443 && echo ansible-tmp-1593806986.5295613-1935085-94392905227443="` echo /root/.ansible/tmp/ansible-tmp-1593806986.5295613-1935085-94392905227443 `" ) && sleep 0']
<c8> Attempting python interpreter discovery
<c8> RUN [b'/usr/bin/podman', b'exec', b'c8', b'/bin/sh', b'-c', b"echo PLATFORM; uname; echo FOUND; command -v '/usr/bin/python'; command -v 'python3.7'; command -v 'python3.6'; command -v 'python3.5'; command -v 'python2.7'; command -v 'python2.6'; command -v '/usr/libexec/platform-python'; command -v '/usr/bin/python3'; command -v 'python'; echo ENDFOUND && sleep 0"]
<c8> RUN [b'/usr/bin/podman', b'exec', b'c8', b'/bin/sh', b'-c', b'/usr/bin/python && sleep 0']
[WARNING]: Unhandled error in Python interpreter discovery for host c8: Expecting value: line 1 column 1 (char 0)
Using module file /usr/lib/python3.8/site-packages/ansible/modules/system/setup.py
<c8> PUT /home/till/.ansible/tmp/ansible-local-1935082f_sfc05r/tmpzicpeliy TO /root/.ansible/tmp/ansible-tmp-1593806986.5295613-1935085-94392905227443/AnsiballZ_setup.py
<c8> RUN [b'/usr/bin/podman', b'exec', b'c8', b'/bin/sh', b'-c', b'rm -f -r /root/.ansible/tmp/ansible-tmp-1593806986.5295613-1935085-94392905227443/ > /dev/null 2>&1 && sleep 0']
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 146, in run
    res = self._execute()
  File "/usr/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 654, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python3.8/site-packages/ansible/plugins/action/normal.py", line 46, in run
    result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/usr/lib/python3.8/site-packages/ansible/plugins/action/__init__.py", line 858, in _execute_module
    self._transfer_data(remote_module_path, module_data)
  File "/usr/lib/python3.8/site-packages/ansible/plugins/action/__init__.py", line 469, in _transfer_data
    self._transfer_file(afile, remote_path)
  File "/usr/lib/python3.8/site-packages/ansible/plugins/action/__init__.py", line 446, in _transfer_file
    self._connection.put_file(local_path, remote_path)
  File "/home/till/.ansible/collections/ansible_collections/containers/podman/plugins/connection/podman.py", line 184, in put_file
    shutil.copyfile(
  File "/usr/lib64/python3.8/shutil.py", line 261, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: b'/home/till/.local/share/containers/storage/overlay/ce57833ce38f71131609ff6bd6419280bb2da69dc6026ae29a2701562b149d51/merged/root/.ansible/tmp/ansible-tmp-1593806986.5295613-1935085-94392905227443/AnsiballZ_setup.py'
c8 | FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}

Additional environment details (AWS, VirtualBox, physical, etc.):

Downgrading to podman-2:1.8.2-2.fc32 fixes the problem.

The text was updated successfully, but these errors were encountered:

manics · 2020-07-04T18:59:50Z

I think this is related to a change in the behaviour of podman mount on rootless podman.

Podman 2.0.1: podman mount succeeds but the merged directory is empty

$ podman run -d nginx
c2b13744d8a0d9063592cc215854a54e68e616ea71fda5c47eb70f54eee79111

$ podman ps
CONTAINER ID  IMAGE                           COMMAND               CREATED     
   STATUS            PORTS   NAMES
c2b13744d8a0  docker.io/library/nginx:latest  nginx -g daemon off;  8 seconds ago  Up 8 seconds ago          eager_fermi

$ podman mount c2b13744d8a0d9063592cc215854a54e68e616ea71fda5c47eb70f54eee79111
/home/vagrant/.local/share/containers/storage/overlay/c2e2b9e2ec04b1935658d99173bdbff9ba8ef796f520d8cf83360c718e61cafa/merged

$ ls -la /home/vagrant/.local/share/containers/storage/overlay/c2e2b9e2ec04b1935658d99173bdbff9ba8ef796f520d8cf83360c718e61cafa/merged
total 0
drwx------. 2 vagrant vagrant  6 Jul  4 18:49 .
drwx------. 5 vagrant vagrant 69 Jul  4 18:49 ..

Podman 1.9.3:

$ podman mount 3e74a16fddc0
Error: cannot mount using driver overlay in rootless mode

$ echo $?
125

This means this should fail on 1.9.3:

ansible-podman-collections/plugins/connection/podman.py

Lines 141 to 146 in d53539c

    
           rc, self._mount_point, stderr = self._podman("mount") 
        
           if rc != 0: 
        
               display.v("Failed to mount container %s: %s" % (self._container_id, stderr.strip())) 
        
           else: 
        
               self._mount_point = self._mount_point.strip() 
        
               display.vvvvv("MOUNTPOINT %s RC %s STDERR %r" % (self._mount_point, rc, stderr))

so the plugin falls back to using podman cp:

ansible-podman-collections/plugins/connection/podman.py

Lines 169 to 181 in d53539c

    
           if not self._mount_point or self.user: 
        
               rc, stdout, stderr = self._podman( 
        
                   "cp", [in_path, self._container_id + ":" + out_path], use_container_id=False 
        
               ) 
        
               if rc != 0: 
        
                   rc, stdout, stderr = self._podman( 
        
                       "cp", ["--pause=false", in_path, self._container_id + ":" + out_path], use_container_id=False 
        
                   ) 
        
                   if rc != 0: 
        
                       raise AnsibleError( 
        
                           "Failed to copy file from %s to %s in container %s\n%s" % ( 
        
                               in_path, out_path, self._container_id, stderr) 
        
                       )

whereas on 2.0.1 it tries to use shutil.copyfile which fails because merged is empty:

ansible-podman-collections/plugins/connection/podman.py

Lines 183 to 187 in d53539c

    
           real_out_path = self._mount_point + to_bytes(out_path, errors='surrogate_or_strict') 
        
           shutil.copyfile( 
        
               to_bytes(in_path, errors='surrogate_or_strict'), 
        
               to_bytes(real_out_path, errors='surrogate_or_strict') 
        
           )

Note I can get podman mount to work on 1.9.3 using podman unshare podman mount ..., but the merged directory is also empty.

In case it's useful this is what's under the other container directories on 2.0.1:

$ ls -la /home/vagrant/.local/share/containers/storage/overlay/c2e2b9e2ec04b1935658d99173bdbff9ba8ef796f520d8cf83360c718e61cafa/*
-rw-r--r--. 1 vagrant vagrant  26 Jul  4 18:49 /home/vagrant/.local/share/containers/storage/overlay/c2e2b9e2ec04b1935658d99173bdbff9ba8ef796f520d8cf83360c718e61cafa/link
-rw-rw-r--. 1 vagrant vagrant 144 Jul  4 18:49 /home/vagrant/.local/share/containers/storage/overlay/c2e2b9e2ec04b1935658d99173bdbff9ba8ef796f520d8cf83360c718e61cafa/lower

/home/vagrant/.local/share/containers/storage/overlay/c2e2b9e2ec04b1935658d99173bdbff9ba8ef796f520d8cf83360c718e61cafa/diff:
total 0
drwxr-xr-x. 5 vagrant vagrant 39 Jul  4 18:49 .
drwx------. 5 vagrant vagrant 69 Jul  4 18:49 ..
drwxr-xr-x. 3 vagrant vagrant 32 Jul  4 18:49 etc
drwxr-xr-x. 3 vagrant vagrant 59 Jul  4 18:49 run
drwxr-xr-x. 3 vagrant vagrant 19 Jul  4 18:49 var

/home/vagrant/.local/share/containers/storage/overlay/c2e2b9e2ec04b1935658d99173bdbff9ba8ef796f520d8cf83360c718e61cafa/merged:
total 0
drwx------. 2 vagrant vagrant  6 Jul  4 18:49 .
drwx------. 5 vagrant vagrant 69 Jul  4 18:49 ..

/home/vagrant/.local/share/containers/storage/overlay/c2e2b9e2ec04b1935658d99173bdbff9ba8ef796f520d8cf83360c718e61cafa/work:
total 0
drwx------. 3 vagrant vagrant 18 Jul  4 18:49 .
drwx------. 5 vagrant vagrant 69 Jul  4 18:49 ..
drwx------. 2 vagrant vagrant  6 Jul  4 18:49 work

sshnaidm · 2020-07-05T21:40:46Z

@tyll please paste podman info --debug output.
@manics on which system do you run test? Please paste podman info --debug output.

I suspect it depends on cgroups version on host. On ubuntu 20 it's not reproducible with podman 2.0.1

tyll · 2020-07-06T09:53:07Z

For future reference, here is the debug output:

working:

debug:
  compiler: gc
  git commit: ""
  go version: go1.14
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  Distribution:
    distribution: fedora
    version: "32"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 3922255872
  MemTotal: 16652611584
  OCIRuntime:
    name: crun
    package: crun-0.13-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 15671275520
  SwapTotal: 17175670784
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: caledvwlch
  kernel: 5.6.18-300.fc32.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-1.1.1-1.fc32.x86_64
    Version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  uptime: 456h 7m 58.91s (Approximately 19.00 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /home/till/.config/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.1-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  GraphRoot: /home/till/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 7
  RunRoot: /run/user/1000/containers
  VolumePath: /home/till/.local/share/containers/storage/volumes

Previously not working but now it does: Thank you!

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: caledvwlch
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.18-300.fc32.x86_64
  linkmode: dynamic
  memFree: 4036943872
  memTotal: 16652611584
  ociRuntime:
    name: crun
    package: crun-0.14-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14
      commit: ebc56fc9bcce4b3208bb0079636c80545122bf58
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.1-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 15703257088
  swapTotal: 17175670784
  uptime: 456h 14m 0.94s (Approximately 19.00 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/till/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 1
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.1-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/till/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 7
  runRoot: /run/user/1000/containers
  volumePath: /home/till/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.1

sshnaidm added the bug Something isn't working label Jul 4, 2020

sshnaidm added this to the 1.1.0 milestone Jul 4, 2020

sshnaidm self-assigned this Jul 4, 2020

manics mentioned this issue Jul 5, 2020

Patch podman connection plugin for 2.0.1 manics/jupyterhub-ansiblespawner#7

Merged

This was referenced Jul 5, 2020

Podman doesn't return error when mounting rootless container with cgroups2, but mounted directory is empty. containers/podman#6856

Closed

Add check for empty dir for podman connection mount #72

Merged

sshnaidm closed this as completed in #72 Jul 5, 2020

manics mentioned this issue Jul 25, 2020

Update podman dependencies manics/jupyterhub-ansiblespawner#9

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

containers.podman.podman connection plugin does not work with podman 2 anymore #70

containers.podman.podman connection plugin does not work with podman 2 anymore #70

tyll commented Jul 3, 2020

manics commented Jul 4, 2020

sshnaidm commented Jul 5, 2020 •

edited

Loading

tyll commented Jul 6, 2020 •

edited

Loading

containers.podman.podman connection plugin does not work with podman 2 anymore #70

containers.podman.podman connection plugin does not work with podman 2 anymore #70

Comments

tyll commented Jul 3, 2020

manics commented Jul 4, 2020

sshnaidm commented Jul 5, 2020 • edited Loading

tyll commented Jul 6, 2020 • edited Loading

sshnaidm commented Jul 5, 2020 •

edited

Loading

tyll commented Jul 6, 2020 •

edited

Loading