[Feature Request] Support podman for running containers

[hvindin]

Description

In RHEL 8 redhat have largely replaced their native support of docker with podman/buildah. Architecturally and from a security perspective, this seems like a good thing, the separation of concerns as well as the ability to run containers without root access are both ideas that give me warm fuzzy feelings that this is a good idea.

It would be nice if nomad were able to follow this switch along, and while I haven't looked too closely to see if there's a socket I can just point the docker driver at and see how far that gets me, it seems like it might be worth at least a nomad 0.9.0+ plugin for the functionality and behavioral differences.

Use Case

When RHEL 8 goes GA I'm sure that docker (as in the company) will release some RPM's to install docker on RHEL 8 so everything can keep humming along as per the status quo. However, one of the nice-to-have items on our roadmap, whenever RHEL 8 becomes available internally in our org, is to get off docker and start making our internal security team happy by not requiring processes run as root which will execute tasks arbitrarily such as scheduling docker containers.

[schmichael]

Love the idea and just as a note to any brave readers: Nomad 0.9 (currently in beta) supports external task driver plugins. No dev docs yet unfortunately, but using the lxc driver as a template should work for anyone comfortable with Go: https://github.com/hashicorp/nomad-driver-lxc

Podman: https://podman.io/
Buildah: https://github.com/containers/buildah

[fatherlinux]

So, sadly, the docker plugin likely wouldn't work here 1. I "think" the Nomad task plugins use the API, right? Not the CLI? If it used the CLI, Podman would likely be a drop-in replacement...

[tcassaert]

@fatherlinux I haven't looked in detail, but I'm afraid that's correct:

nomad/drivers/docker/driver.go

Line 15 in 55ee03a

docker "github.com/fsouza/go-dockerclient"

It's including https://github.com/fsouza/go-dockerclient, which is a client for the Docker API.

[hvindin]

It does appear to be one of the shortcomings of podman as it stands, also something which sort of indicates that Redhat really would like everything to be kubernetes based... The API, to me, seems like it's basically just a shim tacked on as an afterthought.

Realisticallg it might be worth looking at just running raw containers using the containerd/runc/etc bits that seem to be common for anything which supports OCI containers.

I assume that that would be a ton of work because it would basically mean writing a container engine basically starting with the shared parts of the model... but honestly might the only really sustainable way to support generic containers as a first class citizen.

I think I saw another feature request about that... which would probably be better as it would rely on someone else's specific implementation and would essentially make containers native to nomad....

[towe75]

What about building the driver directly on top of libpod instead of thinking in remote-API terms. After all that's what podman wants to do: no big fat daemons. Of course this restricts us to either run the nomad agent as root or to use rootless podman containers because it would manipulate the filesystem directly.

It should be possible to build something along the lines of the existing external lxc plugin. This one does more or less the same but links to liblxc instead of libpod.

What do you think about this approach? I might find the time to do a proof of concept.

[corbinlc]

@towe75 did you get this working? I can see from some other issues that you have been working towards this. Nomad + podman would be very useful to me too.

[towe75]

@corbinlc i am working on it in my spare time. For now i am using the podman varlink api over a socket connection instead of linking libpod directly (because of dependency issues, see my references).

The results so far are very promissing, i am able to start and stop tasks, ip number is reported back so that a consul service is registered properly. Currently i am working on stats reporting, then i'd like to add volume mounts and maybe exec support. Also port mapping is not handlet yet.

After a bit polishing i might be able to publish a initial draft to github in one or two weeks and add some CI integration to get binaries. I let you know in this issue and also in the related one in the podman community. Code base is very related to the old lxc plugin because i used it as a template. I am also working on enhancing this one, btw.

[towe75]

@corbinlc i managed to upload my project to https://github.com/pascomnet/nomad-driver-podman. There is of course no release yet but you can download the binary from the linked circleci build or just compile it yourself. Featureset is very limited but it's some start, also it lacks tests so far.

[corbinlc]

@towe75 this is great work! Thanks for sharing.

[mildred]

Now that Fedora 31 ships by default with cgroups v2 enabled, docker/moby will no longer work on it. having podman available in Nomad allows for a fallback there.

[towe75]

@mildred maybe you want to try out my driver. It's still in an early phase but i'll provide a first binary release very soon.

[mildred]

@towe75 Thank you for your work, I'll test it soon.

[towe75]

You can now find a first binary release at https://github.com/pascomnet/nomad-driver-podman/releases . I hope a binary will attract more users, some feeback would be nice.

@schmichael what to do with this ticket in general?

[schmichael]

This is very exciting @towe75! Thanks for the update.

Next steps

If it's ok with you let's let this issue linger open until one of us on the Nomad team can test the release.

At that point we'll update https://www.nomadproject.io/docs/drivers/external/index.html and close this issue.

Announcements

Feel free to send an [ANN] Nomad Podman Task Driver v0.0.1 Released announcement to the mailing list.

We should probably update our docs to encourage community plugin drivers to send announcements there. Really any community project should feel free to do that. If the list gets too noisy we can always create a new one or find another communication channel.

Improving ecosystem

We're very eager to improve the ecosystem around Nomad in general, so plugin developers should feel free to email the list or open Github issues if you have any ideas/requests/pain points regarding how we handle the Nomad community!

[onlyjob]

@schmichael, I recommend to add nomad-driver-podman to list of external drivers ASAP.
Currently less mature and less usable (and less actively developed) Singularity driver is already listed there. It would only be fair to mention Podman driver among other external drivers. Thanks.

[schmichael]

I whipped together a Vagrantfile to hopefully make testing Nomad+podman easy: https://github.com/schmichael/nomad-podman-vagrant

After some light testing I submitted #6899 to officially add https://github.com/pascomnet/nomad-driver-podman to the community task driver plugin list on https://www.nomadproject.io/ !

Reviews/comments/suggestions welcome on that PR. Also feel free to create your own fork to make changes and send your own PR against Nomad master.

Thanks @towe75 for your contribution and thanks to everyone else who engaged on this subject! I'm excited to play with podman further in the future.

[towe75]

@onlyjob, @schmichael this is good news, thank you!
I will send the suggested [ANN] post to the mailing list to get more attention.

I'll try your vagrantfile ASAP but also i am playing with a more lite weight approach: the plugin does not need direct control (/proc, ...) over the host since a few commits. So i have a setup where nomad itself is running in a systemd controlled podman container. My plan is to commit this into a extra git repo and have a pullable container image with nomad and the plugin.

Later, a advanced version could setup a actual podman pod with nomad, consul and the plugin to turn a vanilla (debian, ubuntu, fedora) podman host into a cluster node.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.