Podman inside of podman, understanding the pros/cons

[KCSesh]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

This is more of a question to the podman team - apologies if this is the wrong forum.

I have a use-case where I run a managed service on a managed device. This service is a systemd process that runs customer supplied containers.

I am looking to containerize my service, which has introduced me to the interesting world of Docker in Docker (DinD) or Docker outside of Docker (DooD) and the associated pros/cons.

Today I am using rootful podman, but launching my customer containers as unprivileged users and manually creating the user namespaces for them.
i.e.:

sudo podman --root /path/to/podman-root/ --runroot /path/to/podman-run-root/ run --name test -it --rm --uidmap 0:1000:1 --uidmap 1:100000:65536 --gidmap 0:1001:1 --gidmap 1:100000:65536 --read-only --rootfs /path/to/rootfs /bin/bash

My device is using apparmor (not SELinux).

I have read these RedHat links on the matter (additional below):
https://www.redhat.com/sysadmin/podman-inside-container
https://www.redhat.com/sysadmin/podman-inside-kubernetes

In the past it felt that running containers inside of containers was something that should not be done i.e popular reference, but then podman has the guide on exactly how to do it.

I have a few questions...

1. In this guide it is said that for "Podman-remote in rootful Podman with a leaked Podman socket from the host" has the warning "Note, however, this is extremely insecure. The processes within the container can totally take over the host machine." I am curious, why this is so much more insecure then, Rootful Podman in rootful Podman. Are they not both having access to root?

2. My understanding of the biggest downsides of podman inside of podman is that there are 2 versions of podman that exist, and that the inner container can't use SELinux. Am I missing any other large issues here?

3. Long term support for podman in podman should be expected?

4. Another big difference is that if you are using 1 single instance of podman you can share all of the layers with the same podman version, this can be a pro or a con depending on your use case. But from the readings it seems Podman inside of Podman also supports sharing image layers by doing the volume mount: /var/lib/containers. Is this safe to do this across versions, or should the innie/outtie podman always be running the same version? Is there any guidance here? i.e should the outtie be always the "latest" or is it fine if the "innie" is more up to date.

5. In Docker the more common approach seems to be DooD. Docker now of course supports DinD. It seems Podman is taking a stance at primarily recommending Podman inside of Podman as the more common approach. Is this accurate?

Additional readings I have done:
https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
https://blog.container-solutions.com/running-docker-in-jenkins-in-docker
https://applatix.com/case-docker-docker-kubernetes-part/

I know this is opinion based questions, but I am not finding much discussion on this topic specific to Podman.

The podman docs are very very useful, but don't necessarily go into the pros/cons of each or why to chose one over the other, it more just shows you how to do it for all the scenarios.

So these are some big questions I have and would really appreciate some insight.

[rhatdan]

1. In this guide it is said that for "Podman-remote in rootful Podman with a leaked Podman socket from the host" has the warning "Note, however, this is extremely insecure. The processes within the container can totally take over the host machine." I am curious, why this is so much more insecure then, Rootful Podman in rootful Podman. Are they not both having access to root?

Dan> Because it will allow the process within the container to escape to the host. One of the goals of Podman is to isolate workloads from containerized workloads. In Both rootful and rootless modes. Rootfull podman based containers do not have full root access to the host, we block via seccomp, apparmor (SELinux is better), dropped capabilities namespaces ...

I have been writing about this for years.
https://projectatomic.io/blog/2015/08/why-we-dont-let-non-root-users-run-docker-in-centos-fedora-or-rhel/

2. My understanding of the biggest downsides of podman inside of podman is that there are 2 versions of podman that exist, and that the inner container can't use SELinux. Am I missing any other large issues here?

Dan> There can be other things blocked, but overall I am fine with Podman within Podman or within Docker for that matter.

3. Long term support for podman in podman should be expected?

Dan> Yes. depending on your definition of support. We will help you. You can probably buy RHEL and get real support. :^)

4. Another big difference is that if you are using 1 single instance of podman you can share all of the layers with the same podman version, this can be a pro or a con depending on your use case. But from the readings it seems Podman inside of Podman also supports sharing image layers by doing the volume mount: /var/lib/containers. Is this safe to do this across versions, or should the innie/outtie podman always be running the same version? Is there any guidance here? i.e should the outtie be always the "latest" or is it fine if the "innie" is more up to date.

Dan> It should be safe across versions, definitely if the major version matches. We have never broken backwards compatibility that I recall. We want to allow users to upgrade versions so newer versions of Podman with older storage will always work. The reverse could break, but the bits stored on disk for images seldom change.

5. In Docker the more common approach seems to be DooD. Docker now of course supports DinD. It seems Podman is taking a stance at primarily recommending Podman inside of Podman as the more common approach. Is this accurate?

Yes for the security reasons stated above as well as the default way Podman works is not Client-Server but Fork/Exec. But if the users understand the risk, then we are not opposed to them leaking the Podman.socket.