podman_containers re-creating the container for each run #125

dschier-wtd · 2020-10-19T12:11:24Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I am running the below playbook and everytime it is running, the container will be re-created.
This also applies when recreate is set to false.

Steps to reproduce the issue:

Run the below playbook
Run it again and watch podman ps -a

Describe the results you received:

The described container will be recreated, even if recreate: false

Describe the results you expected:

The container should only be recreated when recreate: true

Additional information you deem important (e.g. issue happens only occasionally):

Output of ansible --version:

ansible 2.10.2

Output of podman version:

podman version 2.1.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: nb01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.15-201.fc32.x86_64
  linkmode: dynamic
  memFree: 12203294720
  memTotal: 16600129536
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 8371826688
  swapTotal: 8371826688
  uptime: 9m 46.43s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/dschier/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/dschier/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 11
  runRoot: /run/user/1000
  volumePath: /var/home/dschier/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601494271
  BuiltTime: Wed Sep 30 21:31:11 2020
  GitCommit: ""
  GoVersion: go1.14.9
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-7.fc32.x86_64

Playbok you run with ansible (e.g. content of playbook.yaml):

    - name: "Run httpd Container"
      podman_container:
        name: "web-test"
        image: "docker.io/library/httpd:2"
        state: "started"
        force_restart: false
      become: true
      tags:
        - "container"
        - "podman"
        - "httpd"

The text was updated successfully, but these errors were encountered:

sshnaidm · 2020-10-20T08:08:27Z

@daniel-wtd can you please paste a log run with "-vvvvv" and "--diff"?
So far I detected a different bug with this task that SIGWINCH is not in signals table and I'm fixing it now.
Do you use podman_container from collections or from Ansible distribution?

In containers#113 we made a change to lowercase only keys, so signals should be now upper case. Related to containers#125.

dschier-wtd · 2020-10-20T09:21:18Z

Hi @sshnaidm , thank a lot for having a look. I am also facing the SIGWINCH bug now oO. Maybe there is a difference between localhost execution and remote hosts? I will have a look today.

Run on localhost:

TASK [Run httpd Container] *****************************************************************************************************************
task path: /var/home/dschier/Projects/homelab/container.yml:106
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: dschier
<localhost> EXEC /bin/sh -c 'echo ~dschier && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/home/dschier/.ansible/tmp `"&& mkdir "` echo /var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459 `" && echo ansible-tmp-1603185517.6571913-19491-66016853244459="` echo /var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459 `" ) && sleep 0'
Using module file /var/home/dschier/.ansible/collections/ansible_collections/containers/podman/plugins/modules/podman_container.py
<localhost> PUT /var/home/dschier/.ansible/tmp/ansible-local-19286icbbtgte/tmp9nol00nk TO /var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py
<localhost> EXEC /bin/sh -c 'chmod u+x /var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/ /var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py && sleep 0'
<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-pfgczrwonypzemdsavjkqpxkjunnygug ; /usr/bin/python /var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py'"'"' && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py", line 102, in <module>
    _ansiballz_main()
  File "/var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.containers.podman.plugins.modules.podman_container', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib64/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py", line 2237, in <module>
  File "/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py", line 2233, in main
  File "/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py", line 2104, in execute
  File "/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py", line 2041, in make_started
  File "/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py", line 1886, in different
  File "/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py", line 1814, in is_different
  File "/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py", line 1721, in diffparam_stop_signal
KeyError: 'SIGWINCH'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/var/home/dschier/.ansible/tmp/ansible-tmp-1603185517.6571913-19491-66016853244459/AnsiballZ_podman_container.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.containers.podman.plugins.modules.podman_container', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py\", line 2237, in <module>\n  File \"/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py\", line 2233, in main\n  File \"/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py\", line 2104, in execute\n  File \"/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py\", line 2041, in make_started\n  File \"/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py\", line 1886, in different\n  File \"/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py\", line 1814, in is_different\n  File \"/tmp/ansible_podman_container_payload_p2kczj0n/ansible_podman_container_payload.zip/ansible_collections/containers/podman/plugins/modules/podman_container.py\", line 1721, in diffparam_stop_signal\nKeyError: 'SIGWINCH'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

In containers#113 we made a change to lowercase only keys, so signals should be now upper case. Fix tests - stop signal should be word, not number. Related to containers#125.

In containers#113 we made a change to lowercase only keys, but signals in image info are upper case. Fix tests - stop signal should be word, not number. Related to containers#125.

In #113 we made a change to lowercase only keys, but signals in image info are upper case. Fix tests - stop signal should be word, not number. Related to #125.

sshnaidm · 2020-10-20T15:06:57Z

@daniel-wtd I fixed signals issue and released a new version with hotfix, please try to update the collection from Galaxy and try if it's OK.
Then let's see how to reproduce the issue with recreation, currently I can't reproduce it, logs with a "--diff" run would be useful.

dschier-wtd · 2020-10-20T17:57:52Z

Hi, the recreation isue seems to be related to the tags:

version 1 (with tag)

    - name: "Run httpd Container"
      podman_container:
        name: "web-test"
        image: "docker.io/library/httpd:2"
        state: "started"
        force_restart: false
      become: true
      tags:
        - "container"
        - "podman"
        - "httpd"

TASK [Run httpd Container] *****************************************************************************************************************
--- before
+++ after
@@ -1 +1 @@
-image - httpd
+image - httpd:2

changed: [hodgins01]

version 2 (without tag)

    - name: "Run httpd Container"
      podman_container:
        name: "web-test"
        image: "docker.io/library/httpd"
        state: "started"
        force_restart: false
      become: true
      tags:
        - "container"
        - "podman"
        - "httpd"

TASK [Run httpd Container] *****************************************************************************************************************
ok: [hodgins01]

sshnaidm · 2020-10-20T18:20:25Z

@daniel-wtd It's still not reproducing for me. Is it a whole playbook? Can you please try on a fresh system?
Also you can run with -vvvv and add --diff 2>&1 | grep Using to see what exactly module is used here.
Another helpful thing might be running podman inspect web-test | grep -i image to see what is the image in the container.
Thanks

dschier-wtd · 2020-10-20T18:28:34Z

Ha, the problem seems to be, that httpd:2 and httpd:latest are pointing to the same image.

After removing httpd:latest, the play runs fine.

Nevertheless, I have uploaded a paste of --diff -vvv.

with httpd:latest AND httpd:2 present: https://paste.centos.org/view/06f9c48f
only httpd:2 present: https://paste.centos.org/view/34e55edb

sshnaidm · 2020-10-20T19:26:37Z

OK, so for reproducing it was required to pull docker.io/library/httpd:latest as well, when both images present on the system the problem starts to appear.
This is bug in podman. I'll open the issue there.
For a workaround you can remove image "httpd:latest": podman rmi httpd:latest, it will remove the tag only.

dschier-wtd · 2020-10-21T06:06:58Z

I have already done so. Thank you for investigating :)

sshnaidm · 2020-10-21T06:11:35Z

I think it may be good idea to check if we have the same image hash anyway, and if it's same - not to recreate. Will work on it.

dschier-wtd · 2020-10-26T17:34:04Z

Thanks a lot for the investigation and effort. Really appreciate it.

sshnaidm self-assigned this Oct 20, 2020

sshnaidm added bug Something isn't working bug/idempotency Bug related to idempotency of modules labels Oct 20, 2020

sshnaidm added a commit to sshnaidm/ansible-podman-collections that referenced this issue Oct 20, 2020

Fix signals case for podman_container

33339ad

In containers#113 we made a change to lowercase only keys, so signals should be now upper case. Related to containers#125.

sshnaidm mentioned this issue Oct 20, 2020

Fix signals case for podman_container #126

Merged

sshnaidm added a commit that referenced this issue Oct 20, 2020

Fix signals case for podman_container (#126)

2007d84

In #113 we made a change to lowercase only keys, but signals in image info are upper case. Fix tests - stop signal should be word, not number. Related to #125.

sshnaidm mentioned this issue Oct 20, 2020

Podman prints wrong image name in inspection when image has two tags containers/podman#8082

Closed

sshnaidm mentioned this issue Oct 26, 2020

Add strict image compare with hashes #136

Merged

sshnaidm closed this as completed in #136 Oct 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman_containers re-creating the container for each run #125

podman_containers re-creating the container for each run #125

dschier-wtd commented Oct 19, 2020

sshnaidm commented Oct 20, 2020

dschier-wtd commented Oct 20, 2020

sshnaidm commented Oct 20, 2020

dschier-wtd commented Oct 20, 2020

sshnaidm commented Oct 20, 2020 •

edited

Loading

dschier-wtd commented Oct 20, 2020

sshnaidm commented Oct 20, 2020

dschier-wtd commented Oct 21, 2020

sshnaidm commented Oct 21, 2020

dschier-wtd commented Oct 26, 2020

podman_containers re-creating the container for each run #125

podman_containers re-creating the container for each run #125

Comments

dschier-wtd commented Oct 19, 2020

sshnaidm commented Oct 20, 2020

dschier-wtd commented Oct 20, 2020

sshnaidm commented Oct 20, 2020

dschier-wtd commented Oct 20, 2020

version 1 (with tag)

version 2 (without tag)

sshnaidm commented Oct 20, 2020 • edited Loading

dschier-wtd commented Oct 20, 2020

sshnaidm commented Oct 20, 2020

dschier-wtd commented Oct 21, 2020

sshnaidm commented Oct 21, 2020

dschier-wtd commented Oct 26, 2020

sshnaidm commented Oct 20, 2020 •

edited

Loading