Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ansible + podman not working #4813

Closed
dschier-wtd opened this issue Jan 8, 2020 · 26 comments · Fixed by #4828
Closed

ansible + podman not working #4813

dschier-wtd opened this issue Jan 8, 2020 · 26 comments · Fixed by #4828
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@dschier-wtd
Copy link

dschier-wtd commented Jan 8, 2020

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I am using ansible to interact with podman via ansible podman connector. when using a rootless container, I am not able to execute any command via ansible and will get some errors. The command that should be executed is a simple cat /etc/hostname

This may not be 100% related to podman, but maybe to the ansible code (podman connector), since podman cp works in general.

Errors:

Error: cannot copy into running rootless container with pause set - pass --pause=false to force copying\\n

Steps to reproduce the issue:

  1. Start a container as user
podman run -d --rm --name instance fedora:31 sleep 300
  1. Execute some ansible
ansible instance -c podman -m setup -i inventory

instance = name of the container
inventory file = a plain file with the word instance in it

Describe the results you received:

Error: cannot copy into running rootless container with pause set - pass --pause=false to force copying\\n

Describe the results you expected:

either a working execution or some hints, how to manipulate the "pause" behaviour.

Output of podman version:

Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.1
  podman version: 1.6.2
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.2-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.2, commit: 186a550ba0866ce799d74006dab97969a2107979'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 9151815680
  MemTotal: 16602112000
  OCIRuntime:
    name: crun
    package: crun-0.10.6-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.10.6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8371826688
  SwapTotal: 8371826688
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: nb01
  kernel: 5.4.7-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 21h 3m 30.47s (Approximately 0.88 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /var/home/dschier/.config/containers/storage.conf
  ContainerStore:
    number: 4
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.2-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.2
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /var/home/dschier/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 10
  RunRoot: /run/user/1000
  VolumePath: /var/home/dschier/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-manpages-1.6.2-2.fc31.noarch
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jan 8, 2020
@dschier-wtd
Copy link
Author

I have also opened a bug for the ansible devs

ansible/ansible#66263

@sshnaidm
Copy link
Member

sshnaidm commented Jan 8, 2020

@daniel-wtd I think it should have been fixed in d40b450
I can't currently reproduce it, for me it works with same podman and ansible versions. We have a little bit different builds https://linediff.com/?id=5e15a11d687f4b2e6e8b4567 but it doesn't seems as critical.
Will look again what can be wrong

@sshnaidm
Copy link
Member

sshnaidm commented Jan 8, 2020

@daniel-wtd please paste your "ansible --version"

@dschier-wtd
Copy link
Author

dschier-wtd commented Jan 8, 2020

Thanks for having a look. Below you can find my ansible version

ansible 2.9.2
  config file = None
  configured module search path = ['/var/home/dschier/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /var/home/dschier/.venv-python3/lib64/python3.7/site-packages/ansible
  executable location = /var/home/dschier/.venv-python3/bin/ansible
  python version = 3.7.5 (default, Dec 15 2019, 17:54:26) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]

@sshnaidm
Copy link
Member

sshnaidm commented Jan 8, 2020

@daniel-wtd thanks, also the same. Do you have a possibility to try it on a fresh system? I'll try to do the same today on f31

@dschier-wtd
Copy link
Author

yep, will do this today in the evening (CET) and report back to you.

@dschier-wtd
Copy link
Author

I have tested it a on 2 fresh installations minute ago:

  • fedora 31 (silverblue / iot)
  • podman 1.6.2
  • ansible 2.9.2 (via pip / overlay package)

and got the same result.

@dschier-wtd
Copy link
Author

minor addition: on the same host running the exact same stuff as root is working perfectly fine.

@sshnaidm
Copy link
Member

sshnaidm commented Jan 9, 2020

OK, I reproduced it on the fresh fedora31 system:

fedora@localhost ~]$ podman run -d --rm --name instance fedora:31 sleep 1h
9f894952cb740f2d7aae0ccb5e4e688dfe256f04835693acbbf86e103c3bc716
[fedora@localhost ~]$ podman cp /home/fedora/myfile instance:/root/
Error: cannot copy into running rootless container with pause set - pass --pause=false to force copying
[fedora@localhost ~]$ podman cp --pause=false /home/fedora/myfile instance:/root/
[fedora@localhost ~]$ podman version
Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64

@mheon @rhatdan any ideas why it works on fedora30 with d40b450 but doesn't work on fedora31? With same podman version 1.6.2

@mheon
Copy link
Member

mheon commented Jan 9, 2020 via email

@sshnaidm
Copy link
Member

sshnaidm commented Jan 9, 2020

@mheon yeah, seems like it's cgroups. It works fine with cgroups v1, but doesn't work with v2. Still not sure why it ignores default setting from d40b450
Should we set "--pause false" explicitly in ansible plugin? https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/podman.py#L165-L166

CGroups v2, maybe? Pause requires CGroups which for rootless containers are only available on v2.

@mheon
Copy link
Member

mheon commented Jan 9, 2020

Might be a good idea.

@mheon
Copy link
Member

mheon commented Jan 9, 2020

@giuseppe We expect pause to work rootless on v2, correct?

@giuseppe
Copy link
Member

giuseppe commented Jan 9, 2020

@giuseppe We expect pause to work rootless on v2, correct?

yes, pause should work on cgroup v2 as rootless as well

@sshnaidm
Copy link
Member

sshnaidm commented Jan 9, 2020

As I see in code pause=false is default only with cgroups v1 and it's what causes problems for fedora31. Is that a specific reason for it? https://github.com/containers/libpod/blob/f85b3a01f050e4f8af8471f396013f18647d9241/cmd/podman/cp.go#L496
@giuseppe @mheon ^^

@mheon
Copy link
Member

mheon commented Jan 9, 2020

It sounds like we expect pause to work for CGroups v2, so the fact that it's not is probably a bug

@sshnaidm
Copy link
Member

sshnaidm commented Jan 9, 2020

@mheon so should we stop to require pause=False on cgroups v2 for rootless containers?
If I set now explicitly --pause false in ansible module - will it break some higher/future versions..? Would be better to have it solved in podman maybe.

@giuseppe
Copy link
Member

If pause can be performed in rootless depends whether we are using cgroups v2, the systemd manager and the freezer controller is available (it wasn't on older kernels for cgroup v2). The last condition can fail for root as well.

I think we should just drop the check and let the OCI runtime complain if pause cannot be performed.

giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 10, 2020
rootless containers can use pause on cgroups v2.

Whether it is possible or not to use pause depends from multiple
conditions, such as:

- be on a cgroup v2 unified hierarchy,
- using systemd cgroup manager,
- the kernel has the freezer controller,

The last one may fail for root as well.

Instead of trying to catch all the possible conditions in Podman, let
the OCI runtime complain if pause cannot be performed.

Closes: containers#4813

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe
Copy link
Member

PR here: #4828

@sshnaidm
Copy link
Member

@giuseppe thanks! Will it be ported to 1.6.2 or later? Because if I change ansible plugin to use "--pause false" now, it might be broken for later podman? Or I miss something?

@giuseppe
Copy link
Member

@giuseppe thanks! Will it be ported to 1.6.2 or later? Because if I change ansible plugin to use "--pause false" now, it might be broken for later podman? Or I miss something?

I don't think this will be back ported as it is not a security problem. @mheon or we could backport it?

@mheon
Copy link
Member

mheon commented Jan 13, 2020 via email

@rhatdan
Copy link
Member

rhatdan commented Jan 13, 2020

Yes no back porting, unless absolutely necessary.

@samdoran
Copy link

samdoran commented Jan 17, 2020

I merged the fix for this in Ansible (ansible/ansible#66583) and opened a backport to stable-2.9. It should be in the next 2.9 release.

@dschier-wtd
Copy link
Author

with 1.8.0 on fedora, everything is working fine :)

@rhatdan
Copy link
Member

rhatdan commented Feb 10, 2020

Awesome.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants