Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

connection plugin does not work with rootless containers #29

Closed
tyll opened this issue May 3, 2020 · 3 comments · Fixed by #30
Closed

connection plugin does not work with rootless containers #29

tyll opened this issue May 3, 2020 · 3 comments · Fixed by #30
Assignees
@sshnaidm
Labels
bug Something isn't working

Comments

@tyll
Copy link

tyll commented May 3, 2020

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. podman run --rm -it centos:8

  2. ansible -vvi a935777de543, -c podman -m setup all

Describe the results you received:
Failed to mount container a935777de543: b'Error: cannot mount using driver overlay in rootless mode'
[WARNING]: Unhandled error in Python interpreter discovery for host a935777de543: Expecting value: line 1 column 1
(char 0)
a935777de543 | FAILED! => {
"msg": "Failed to set execute bit on remote files (rc: 1, err: chmod: cannot access '/root/.ansible/tmp/ansible-tmp-1588529245.4183488-80743-226755747598850/AnsiballZ_setup.py': No such file or directory\nError: non zero exit code: 1: OCI runtime error\n)"
}

Describe the results you expected:

Ansible should connect to the host.

If this is not supposed to work, the error message should be improved. Without extra verbosity it fails with the message about the temp file in /root. I guess the failure to mount the container should be a hard error instead of a warning.

Additional information you deem important (e.g. issue happens only occasionally):

Output of ansible --version:

ansible 2.9.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/till/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Feb 28 2020, 00:00:00) [GCC 10.0.1 20200216 (Red Hat 10.0.1-0.8)]

Output of podman version:

1.8.2

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.14
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.15-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 33da5ef83bf2abc7965fc37980a49d02fdb71826'
  Distribution:
    distribution: fedora
    version: "32"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 2131616
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 2131616
      size: 65536
  MemFree: 1978478592
  MemTotal: 8217530368
  OCIRuntime:
    name: crun
    package: crun-0.13-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8321495040
  SwapTotal: 8321495040
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: excalibur
  kernel: 5.6.7-300.fc32.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-1.0.0-1.fc32.x86_64
    Version: |-
      slirp4netns version 1.0.0
      commit: a3be729152a33e692cd28b52f664defbf2e7810a
      libslirp: 4.2.0
  uptime: 28h 23m 11.46s (Approximately 1.17 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /home/till/.config/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.0.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  GraphRoot: /home/till/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 3
  RunRoot: /run/user/1000
  VolumePath: /home/till/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.2-2.fc32.x86_64

Playbok you run with ansible (e.g. content of playbook.yaml):

(paste your output here)

Command line and output of ansible run with high verbosity:

ansible 2.9.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/till/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Feb 28 2020, 00:00:00) [GCC 10.0.1 20200216 (Red Hat 10.0.1-0.8)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed a935777de543, inventory source with host_list plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python3.8/site-packages/ansible/plugins/callback/minimal.py
META: ran handlers
<a935777de543> RUN [b'podman', b'mount', b'a935777de543']
STDOUT b''
STDERR b'Error: cannot mount using driver overlay in rootless mode\n'
RC CODE 125
Failed to mount container a935777de543: b'Error: cannot mount using driver overlay in rootless mode'
<a935777de543> RUN [b'podman', b'exec', b'a935777de543', b'/bin/sh', b'-c', b'echo ~ && sleep 0']
STDOUT b'/root\n'
STDERR b''
RC CODE 0
STDOUT b'' STDERR b''
<a935777de543> RUN [b'podman', b'exec', b'a935777de543', b'/bin/sh', b'-c', b'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir /root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188 && echo ansible-tmp-1588529460.721404-81242-28143645726188="` echo /root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188 `" ) && sleep 0']
STDOUT b'ansible-tmp-1588529460.721404-81242-28143645726188=/root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188\n'
STDERR b''
RC CODE 0
STDOUT b'' STDERR b''
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/namespace.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/ansible_collector.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/basic.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/default_collectors.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/timeout.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/collector.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/parsing/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/_utils.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/validation.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/process.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/parsing/convert_bool.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/parameters.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/sys_info.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/text/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/file.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/_text.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/six/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/pycompat24.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/_collections_compat.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/_json_compat.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/text/converters.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/text/formatters.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/common/collections.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/distro/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/distro/_distro.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/compat.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/python.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/other/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/freebsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/platform.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/other/ohai.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/hurd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/freebsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/caps.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/other/facter.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/date_time.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/aix.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/sunos.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/base.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/cmdline.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/pkg_mgr.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/base.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/openbsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/nvme.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/service_mgr.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/hpux.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/fc_wwn.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/chroot.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/darwin.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/dragonfly.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/ssh_pub_keys.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/netbsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/dns.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/local.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/linux.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/selinux.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/iscsi.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/freebsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/aix.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/fips.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/hurd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/sunos.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/sunos.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/env.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/lsb.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/distribution.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/__init__.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/hpux.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/openbsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/base.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/openbsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/netbsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/hpux.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/dragonfly.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/netbsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/apparmor.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/dragonfly.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/linux.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/system/user.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/linux.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/hardware/darwin.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/network/generic_bsd.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/utils.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/virtual/sysctl.py
Using module_utils file /usr/lib/python3.8/site-packages/ansible/module_utils/facts/sysctl.py
<a935777de543> Attempting python interpreter discovery
<a935777de543> RUN [b'podman', b'exec', b'a935777de543', b'/bin/sh', b'-c', b"echo PLATFORM; uname; echo FOUND; command -v '/usr/bin/python'; command -v 'python3.7'; command -v 'python3.6'; command -v 'python3.5'; command -v 'python2.7'; command -v 'python2.6'; command -v '/usr/libexec/platform-python'; command -v '/usr/bin/python3'; command -v 'python'; echo ENDFOUND && sleep 0"]
STDOUT b'PLATFORM\nLinux\nFOUND\n/usr/libexec/platform-python\nENDFOUND\n'
STDERR b''
RC CODE 0
STDOUT b'' STDERR b''
<a935777de543> RUN [b'podman', b'exec', b'a935777de543', b'/bin/sh', b'-c', b'/usr/libexec/platform-python && sleep 0']
STDOUT b''
STDERR b''
RC CODE 0
STDOUT b'' STDERR b''
[WARNING]: Unhandled error in Python interpreter discovery for host a935777de543: Expecting value: line 1 column 1
(char 0)
Using module file /usr/lib/python3.8/site-packages/ansible/modules/system/setup.py
<a935777de543> PUT /home/till/.ansible/tmp/ansible-local-81239icyqidip/tmp_54n02qf TO /root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/AnsiballZ_setup.py
<a935777de543> RUN [b'podman', b'cp', b'/home/till/.ansible/tmp/ansible-local-81239icyqidip/tmp_54n02qf', b'a935777de543:/root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/AnsiballZ_setup.py']
STDOUT b''
STDERR b'writing file `/sys/fs/cgroup//cgroup.freeze`: Permission denied\nError: `/usr/bin/crun pause a935777de543ade936bfcd412343f3038e538513ce397e5349b3c7eaa17f508e` failed: exit status 1\n'
RC CODE 125
<a935777de543> RUN [b'podman', b'exec', b'a935777de543', b'/bin/sh', b'-c', b'chmod u+x /root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/ /root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/AnsiballZ_setup.py && sleep 0']
STDOUT b''
STDERR b"chmod: cannot access '/root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/AnsiballZ_setup.py': No such file or directory\nError: non zero exit code: 1: OCI runtime error\n"
RC CODE 1
STDOUT b"chmod: cannot access '/root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/AnsiballZ_setup.py': No such file or directory\nError: non zero exit code: 1: OCI runtime error\n" STDERR b"chmod: cannot access '/root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/AnsiballZ_setup.py': No such file or directory\nError: non zero exit code: 1: OCI runtime error\n"
<a935777de543> RUN [b'podman', b'exec', b'a935777de543', b'/bin/sh', b'-c', b'rm -f -r /root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/ > /dev/null 2>&1 && sleep 0']
STDOUT b''
STDERR b''
RC CODE 0
STDOUT b'' STDERR b''
a935777de543 | FAILED! => {
    "msg": "Failed to set execute bit on remote files (rc: 1, err: chmod: cannot access '/root/.ansible/tmp/ansible-tmp-1588529460.721404-81242-28143645726188/AnsiballZ_setup.py': No such file or directory\nError: non zero exit code: 1: OCI runtime error\n)"
}

Additional environment details (AWS, VirtualBox, physical, etc.):
@sshnaidm
Copy link
Member

sshnaidm commented May 4, 2020

@tyll it should work. The problem is here:

STDERR b'writing file `/sys/fs/cgroup//cgroup.freeze`: Permission denied\nError: `/usr/bin/crun pause a935777de543ade936bfcd412343f3038e538513ce397e5349b3c7eaa17f508e` failed: exit status 1\n'
RC CODE 125"

For some reason you can't pause container. It reminds me problem containers/podman#4813 which was solved in ansible/ansible#66583. But seems like didn't work in your case, error message is different. Can you try please such a file? http://paste.openstack.org/show/793035/ Copy it to ~/.ansible/collections/ansible_collections/containers/podman/plugins/connection/podman.py and make sure you use collection plugin, not the built one. Run it with verbosity 5:
ansible -vvvvvi 4fa82881440d, -c containers.podman.podman -m setup all

@sshnaidm sshnaidm added bug Something isn't working needs_info More information about case is required labels May 4, 2020
@sshnaidm sshnaidm self-assigned this May 4, 2020
@tyll
Copy link
Author

tyll commented May 4, 2020

@tyll it should work. The problem is here:

For some reason you can't pause container. It reminds me problem containers/libpod#4813 which was solved in ansible/ansible#66583. But seems like didn't work in your case, error message is different. Can you try please such a file? http://paste.openstack.org/show/793035/ Copy it to ~/.ansible/collections/ansible_collections/containers/podman/plugins/connection/podman.py and make sure you use collection plugin, not the built one. Run it with verbosity 5:
ansible -vvvvvi 4fa82881440d, -c containers.podman.podman -m setup all

Great, thank you. This fixes it for me. There are now only warnings related to the python interpreter:

<2b0085e9ba9c> Attempting python interpreter discovery
<2b0085e9ba9c> RUN [b'/usr/bin/podman', b'exec', b'2b0085e9ba9c', b'/bin/sh', b'-c', b"echo PLATFORM; uname; echo FOUND; command -v '/usr/bin/python'; command -v 'python3.7'; command -v 'python3.6'; command -v 'python3.5'; command -v 'python2.7'; command -v 'python2.6'; command -v '/usr/libexec/platform-python'; command -v '/usr/bin/python3'; command -v 'python'; echo ENDFOUND && sleep 0"]
STDOUT b'PLATFORM\nLinux\nFOUND\n/usr/libexec/platform-python\nENDFOUND\n'
STDERR b''
RC CODE 0
STDOUT b'' STDERR b''
<2b0085e9ba9c> RUN [b'/usr/bin/podman', b'exec', b'2b0085e9ba9c', b'/bin/sh', b'-c', b'/usr/libexec/platform-python && sleep 0']
STDOUT b''
STDERR b''
RC CODE 0
STDOUT b'' STDERR b''
[WARNING]: Unhandled error in Python interpreter discovery for host 2b0085e9ba9c: Expecting value: line 1 column 1 (char 0)

Let me know if you need more info.

@sshnaidm
Copy link
Member

sshnaidm commented May 4, 2020

@tyll warning is fine. Glad it helped, will prepare a patch then.

@tyll tyll mentioned this issue May 4, 2020
Closed
@sshnaidm sshnaidm removed the needs_info More information about case is required label May 4, 2020
@sshnaidm sshnaidm mentioned this issue May 4, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sshnaidm sshnaidm

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Run pause=false w/o message condition sshnaidm/ansible-podman-collections
2 participants
@tyll @sshnaidm