Feature request podman support

[aboe76]

Feature request salstack state/module support for podman

Saltstack has already support for docker please add podman support

Development Information to get started:
https://medium.com/cri-o/python3-support-for-podman-a0a2395c3b4

podman is already build for suse and kubic-project so it should fit well with suse manager 3

[Ch3LL]

will approve as a feature request

[deuscapturus]

I just starting looking into creating some podman modules.

I think this is a good starting point: https://github.com/containers/libpod/tree/master/contrib/python/podman

[fignew]

Now that both Fedora and RHEL/CentOS 8 no longer have Docker packages in their native repos I think that podman support is very much needed.

[aboe76]

The podman python has it's own module:
https://github.com/containers/python-podman

[defensivedepth]

I agree with @fignew, this becomes a more pressing issue with the lack of Docker packages with Fedora and RHEL/CentOS 8.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[aboe76]

Please keep this open, podman support would be beneficial to saltstack.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[ankitola]

Yes, It will be great if SaltStack provide Podman support. Please keep open this.

[max-arnold]

I did some initial research and found this:

containers/python-podman#70 (comment)
https://podman.io/blogs/2020/01/17/podman-new-api.html

A couple of important highlights:

python-podman implementation is not fully compatible with libpod varlink defined interface

The bottom line we are moving away from varlink to a similar Protocol to the Docker.API

Thanks for the heads up, I will take a look to the APIv2 to avoid to spend time on something who will be deprecated in the next releases.

But we heard from users that varlink was a hurdle for libpod adoption especially for those who were using the Docker API and its bindings. They simply could not or did not want to rewrite their custom applications for libpod’s new, varlink-based API.

The new API is a simpler implementation based on HTTP/REST. We provide two basic groups of endpoints. The first one is for libpod; the second is for Docker compatibility, to ease adoption. The two endpoints are namespaced to keep them separate. Our goal with implementing a portion of the Docker API, is to be as compatible as possible; while similar calls in the libpod API might bring back additional libpod specific information.

In our proof of concepts, we have tested our endpoint with the docker-py project. There are of course subtle differences which we are still working on. And there are compatibility endpoints that we can not support like swarm which Podman does not support.

As for the existing varlink code, it has been in maintenance mode already. We will continue to address bugs but no new functionality will be developed. Once the new API is fully implemented, we plan to make a deprecation announcement.

This potentially means that some of the existing Salt docker modules might be somewhat compatible with the upcoming Podman APIv2 endpoint.

However, I'm not really familiar with Podman and not sure if APIv2 could provide a viable shortcut for Salt to support Podman containers eventually.

[doesitblend]

ZD-4685

[max-arnold]

I made some progress trying to run podman containers using its new Docker-compatible API and the existing dockermod execution module.

Installation instructions (Ubuntu 20.04, Salt 3001):

Pip pkg:
  pkg.installed:
    - name: python3-pip

Podman repo:
  pkgrepo.managed:
    - name: deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ /
    - key_url: https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_20.04/Release.key
    - file: /etc/apt/sources.list.d/podman.list

Podman pkg:
  pkg.installed:
    - name: podman

Podman service:
  file.managed:
    - name: /etc/systemd/system/podman.service
    - source: salt://podman/podman.service

Podman socket:
  file.managed:
    - name: /etc/systemd/system/podman.socket
    - source: salt://podman/podman.socket
  service.running:
    - name: podman.socket
    - enable: true

Docker socket:
  file.symlink:
    - name: /var/run/docker.sock
    - target: /var/run/podman/podman.sock

Docker python:
  pip.installed:
    - bin_env: /usr/bin/pip3
    - reload_modules: true
    - pkgs:
        - certifi==2019.11.28
        - chardet==3.0.4
        - docker==4.2.1
        - idna==2.9
        # - requests==2.23.0
        - six==1.14.0
        - urllib3==1.25.8
        - websocket-client==0.57.0

restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - pip: Docker python

`podman.service`

[Unit]
Description=Podman API Service
Requires=podman.socket
After=podman.socket
Documentation=man:podman-api(1)
StartLimitIntervalSec=0

[Service]
Type=oneshot
Environment=REGISTRIES_CONFIG_PATH=/etc/containers/registries.conf
ExecStart=/usr/bin/podman system service
TimeoutStopSec=30
KillMode=process

[Install]
WantedBy=multi-user.target
Also=podman.socket

`podman.socket`

[Unit]
Description=Podman API Socket
Documentation=man:podman-api(1)

[Socket]
ListenStream=%t/podman/podman.sock
SocketMode=0660

[Install]
WantedBy=sockets.target

By default any docker.* functions will fail due to the hardcoded version check:

docker.errors.InvalidVersion: API versions below 1.21 are no longer supported by this library

To fix that, run the following command and restart salt-minion:

echo "docker.version: '1.35'" > /etc/salt/minion.d/docker.conf

Here is an incomplete list of docker functions that work (i.e., do not crash and output something meaningful):

• salt-call docker.version
• salt-call docker.info
• salt-call docker.images (only if there is no images)
• salt-call docker.list_containers
• salt-call docker.ps
• salt-call docker.pull ubuntu:18.04

There is one minor blocker that prevents docker.images from succeeding: containers/podman#6796 (comment) (it should be fixed, but for some strange reason still exists in podman 2.0.4). UPD: it will be released in 2.0.5

• salt-call docker.run_container docker.io/library/ubuntu:18.04 name=ubuntu replace=true bg=true will fail due to incompatible network settings format (this is the issue I'm currently testing)
• docker.{login,logout,copy_from,load,layers,save} will fail because they depend on docker cli command (some of them will work if you run ln -s /usr/bin/podman /usr/bin/docker, but I didn't test this thoroughly)

If you want to help, testing other docker functions/states and reporting any issues is appreciated. Podman developers tend to ask for shell-based reproducers, you can find a couple of examples here: containers/podman#5553. Please do not ask them to run any Salt commands to reproduce an issue :)

To trace the HTTP requests that are made by Salt docker module, patch the /usr/lib/python3/dist-packages/requests/sessions.py file (the debug output will be visible when you use salt-call):

--- sessions.py.orig	2020-08-04 00:28:13.977552983 -0700
+++ sessions.py	2020-08-04 00:20:30.776382799 -0700
@@ -530,7 +530,9 @@
             'allow_redirects': allow_redirects,
         }
         send_kwargs.update(settings)
+        print('REQ', prep.method, prep.url, prep.body)
         resp = self.send(prep, **send_kwargs)
+        print('RES', resp.text)
         return resp

     def get(self, url, **kwargs):

The Docker API reference can be useful in comparing and reporting any inconsistencies: https://docs.docker.com/engine/api/v1.35/

[mchugh19]

Is the goal to make the current docker modules work with both docker and podman, or to introduce a new podman module with a docker virtualname?

[max-arnold]

The current goal is to make the existing modules work with minimal or no changes (it was stated that docker-py should just work with Podman APIv2).

If it turns out to be unachievable, then the only option left is to write a new set of Podman-specific state/execution modules using the new podman-py module that supports APIv2 (not to be confused with the obsolete python-podman that uses Podman Varlink protocol).

Right now I'm only considering the easy path :)

[max-arnold]

Another progress update: first successful launch of a Podman container using Salt docker module containers/podman#5580 (comment)

[aboe76]

@max-arnold that's sounds great!!!

[paususe]

Hey @max-arnold, what's the status?

[max-arnold]

@aboe76 @paususe I think I got a single podman container running with some issues, I hope I can find some time this August to document and upload my WIP code here

[aboe76]

@max-arnold nice work, thanks for the status update.

[mostlymaxi]

@max-arnold any updates? Been looking for this as well and interested in contributing.

[hcdeeken]

I had to fiddle a bit with the environment (podman adds environment variables to running containers) to get docker_container.running to behave, but anything else I tried with podman 4.2 worked nicely.
(after adding the symlink and the python modules from #50624 (comment))

[max-arnold]

Here you go! The states were written in Feb 2021 and are outdated, but if anyone wants to continue the work, feel free to create a repo:

podman.zip

These are not meant to be used in production, because Podman and its components are compiled from source (to be able to test the latest fixes).

Kudos to @defensivedepth for making this happen!

[max-arnold]

Spent some time updating the states with the latest and greatest versions:

• Ubuntu 22.04 (removed the Centos support because I do not have time to figure out all this Centos/Stream/Rocky mess)
• Salt 3005.1 onedir
• Podman 4.3.0, CNI plugins 1.1.1, Crun 1.6 (instead of Runc), Conmon 2.1.4
• Docker Python library 5.0.3 (>=6.0.0 is not supported by Salt, see [BUG] docker_image.present erroring out: docker_image' __virtual__ returned False: 'docker.version' is not available. #62602)

For some reason the requests patch doesn't emit any output with salt-call, so another way to debug Docker API requests needs to be implemented.

podman-2022-11-03.zip

The primary issue that needs to be fixed is non-idempotent docker_container.running state. You'll get a new container id each time you apply the state. There are three possible projects that can affect this:

1. Podman and its components. Most likely its API is not 100% compatible, esp. if it is related to networking
2. Docker Python library
3. Salt Docker modules and states (they aren't getting enough attention from VMware), and how they interact with docker module

P.S. Sorry for making you wait for so long. I procrastinated on publishing this WIP for a while :)

[andrelop]

Is there any update on this issue ? I tried to use podman and it mostly works, but docker_container.running generating a new container on each run is not helping.

It seems to assume there's a new imagem even though the image is exactly the same.

[DaAwesomeP]

Hey @max-arnold any updates? It seems that we are close but just barely not quite there yet!

[max-arnold]

@DaAwesomeP Unfortunately, I no longer work on this. Feel free to pick it up and continue the work.