Make podman first class citizens for quarkus usecases

[maxandersen]

Description

Recent podman desktop (0.12) and podman (4.4) by default now activates docker compatibility making it simple to setupand start across all three major os's.

This issue is to capture outstanding issues in quarkus, testcontainers, podman,devsevices containers etc that prevents a smooth podman experience.

Overall setup

Issues that makes installation and setup problematic/challenging/more difficult than we would prefer

• have docker socket enabled by default on windows and OSX (Fixed since podman 4.4.1+ and desktop 0.12.0+)
• test containers requiring enabling privileged ryuk to be enabled

Privileged vs non-priviliged by default

In general docker runs in a setup that defaults to risky/unsafe; where as podman does the opposite. runs in non-priviliged mode. We can probably explicitly make our usage enable unsafe running for both docker and podman.

Downside is that security concerned podman users would not like to do that by default; but on the other hand users running on docker already run it this way.

A classic security first vs usability first issue.

What approach we take here is yet unlcear

Affected Devservices

• Elasticsearch (chroot: cannot change root directory to '/': Operation not permitted)
• DB2 (multitude of root / syscall issues)

Affected container centric usecases

• quarkus docker build extension fails on podman
• docker compose with volumes behaves differently

[holly-cummins]

Here are some sources of friction. Many of these aren't 'defects', exactly, because they can be resolved with appropriate config, but they require debugging and config work when people start using podman - so a definitely "not-first-class" experience:

• Testcontainers ryuk configuration. This is needed for test containers. With no config, users will get an error similar to panic: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/_ping: dial unix /var/run/docker.sock: connect: permission denied. Many people set export TESTCONTAINERS_RYUK_DISABLED=true, but this prevents container cleanup. Editing ~/.testcontainers.properties and adding ryuk.container.privileged=true is a better solution. Podman support testcontainers/testcontainers-java#2088 has some of this discussion (but the issue is closed, and there's a lot of different errors discussed in the one issue). Permission denied when trying to use the `/var/run/docker.sock` file generated by the mac-helper containers/podman#14238 also tracks this issue (among others). We should perhaps raise an issue against testcontainers asking them to detect podman and suggest changing the setting.

• Short names of containers. Testcontainers and Quarkus Dev Services also expect the container service they make requests against to be non-interactive. If someone has multiple registries configured, and when using short image names, Podman responds with a prompt asking which registry should be used to pull images. Testcontainers defaults to using short names internally, so users need to change this default this prompt by setting the short-name-mode="disabled" configuration property of Podman in /etc/containers/registries.conf. Error, when pulling short-named image via docker api containers/podman#16400 may be tracking this, or it may be a related testcontainers + podman issue. I can't quite tell.

• Missing qemu on Arm podman machines. Users need to hand-tune the machine image created by podman to install qemu, for AMD64 images to work. This step isn't needed in Docker, so it can surprise some users. It needs to be redone every time a new podman image is set up. It looks like this is podman default machine on macOS M1/M2 could ship qemu-user-static out of the box to use non-ARM64 containers containers/podman#16148 and it was done in November, so we should test the update and update our instructions if it works.

• It's often necessary to specify the architecture explicitly with podman, in a way it isn't for Docker. See, for example, Use health check to detect mariadb startup and skip volume modifier on M1 #25805, Disable Oracle db tests on Mac M1 because of segfault (and updated readiness check) #25832, MySQL on M1: use architecture-specific image, and health check for readiness detection #25878. These are a mix of M1 issues and Podman issues, but what we've seen is that Podman behaves worse on M1 than Docker does. In some cases we were able to update the dev service configuration to work with podman+M1, but in other cases we had to disable the test (ie we now pass CI, but don't actually work), or do configuration in the test launching that users would have to replicate.

• Test coverage. Although this isn't the same sort of issue, we have a hard time running our CI on podman because of intermittent podman launch errors. We've raised Podman machine start fails with 'no such file or directory', reboot resolves issue on Mac M1 containers/podman#16163 to track this, but it's hard to provide a good reproducer. We're also seeing purgeable space leaks on the machine, which may or may not be related to podman.

[edeandrea]

podman version 4.4.1 on Darwin edeandrea-m1pro 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64

Here is another one - try to run the Quarkus Superheroes via Docker Compose - https://github.com/quarkusio/quarkus-super-heroes#running-locally-via-docker-compose

This does not work. I don't think it is a Quarkus thing either - more that podman (at least on macOS M1) doesn't seem to like volumes.

╰─ docker compose -f deploy/docker-compose/java17.yml -f deploy/docker-compose/monitoring.yml up  
[+] Running 6/10
 ⠿ Network docker-compose_default     Created                                                                                0.0s
 ⠿ Container ui-super-heroes          Created                                                                                0.1s
 ⠹ Container villains-db              Creating                                                                               0.2s
 ⠹ Container fights-db                Creating                                                                               0.2s
 ⠿ Container jaeger                   Created                                                                                0.1s
 ⠿ Container apicurio                 Created                                                                                0.1s
 ⠹ Container prometheus               Creating                                                                               0.2s
 ⠹ Container heroes-db                Creating                                                                               0.2s
 ⠋ Container otel-collector           Creating                                                                               0.0s
 ⠿ Container fights-kafka             Created                                                                                0.0s
 ⠿ Container event-statistics-java17  Created                                                                                0.0s
Error response from daemon: make cli opts(): making volume mountpoint for volume /Users/edeandre/workspaces/quarkus/quarkus-super-heroes-main/rest-fights/deploy/db-init/initialize-database.js: mkdir /Users: operation not permitted

[n1hility]

• Testcontainers ryuk configuration. This is needed for test containers.

-snip-

Improving this experience has been on my TODO for awhile, the current solution is as you mention to use the ryuku privileged flag and to run as rootful on Mac and Windows. On Linux, you need a podman service and some file permission changes. The underlying problem is some assumptions in the ryuk containers that do not interplay well with rootless. A solution with improved usability likely involves patches to testcontainers, and perhaps some on the Quarkus side.

• Short names of containers. Testcontainers and Quarkus Dev Services also expect the container service they make requests against to be non-interactive.

I need to double check this one, I think there were improvements to this with recent images of podman machine

• Missing qemu on Arm podman machines.

This should be resolved now on Mac. I still need to bring this change for windows.

• It's often necessary to specify the architecture explicitly with podman, in a way it isn't for Docker. See, for example, Use health check to detect mariadb startup and skip volume modifier on M1 #25805, Disable Oracle db tests on Mac M1 because of segfault (and updated readiness check) #25832, MySQL on M1: use architecture-specific image, and health check for readiness detection #25878.

IIUC this is about selinux label enforcement and 9p. I wonder if the recent change to switch the qemu 9p security model away from mapped-xattr helps here. A possible improvement could be to ignore the Z/z flags when the FS does not support them

" but what we've seen is that Podman behaves worse on M1 than Docker does."

Do you mean from these issues you linked or something else?

• Test coverage. Although this isn't the same sort of issue, we have a hard time running our CI on podman because of intermittent podman launch errors. We've raised Podman machine start fails with 'no such file or directory', reboot resolves issue on Mac M1 containers/podman#16163 to track this, but it's hard to provide a good reproducer.

We really need to get a reproducer and some logs on this one. I haven't been luck (unlucky?) enough to see this one locally.

We're also seeing purgeable space leaks on the machine, which may or may not be related to podman.

I think this is unlikely to be related to podman since mac has no insight into the linux filesystem podman is using inside a single contiguous loopback file. It's more likely this is coming from files directly on the APFS. One way to test could be to do a podman machine rm and see if purgeable space recovers.

[n1hility]

@edeandrea hey i have super heroes working here. Are you running in rootful mode? I recommend doing the following:

podman machine rm -f
podman machine init
podman machine set -m 8192
podman machine set --rootful
podman machine start

This would blow away any stale image you might have from an old podman release, and set things as rootful. One issue I did notice is that the flights-kaka container will fail on subsequent starts because /tmp is not mounted to tmpfs so is preserved across restarts. If you add the following to your compose under fights-kaka it resolves it:

  tmpfs:
      - /tmp

I need to look into why this isn't done by default.

[edeandrea]

@n1hility I did exactly this today:

podman machine ls

# Then removed all the ones that showed up

brew update && brew upgrade

podman machine init --cpus 4 --memory 6144 --rootful

PODMAN_VERSION=`podman -v | sed 's/[a-zA-Z ]*//'`

sudo /opt/homebrew/Cellar/podman/$PODMAN_VERSION/bin/podman-mac-helper install

# I already had ryuk.container.privileged=true in ~/.testcontainers.properties

podman machine start

podman machine ssh
sudo -i
rpm-ostree install qemu-user-static

# add short-name-mode="disabled" configuration property in /etc/containers/registries.conf

systemctl reboot

podman machine stop

podman machine start

# git clone superheroes
cd quarkus-super-heroes

docker compose -f deploy/docker-compose/java17.yml -f deploy/docker-compose/monitoring.yml pull

docker compose -f deploy/docker-compose/java17.yml -f deploy/docker-compose/monitoring.yml up

# Boom
[+] Running 6/10
 ⠿ Network docker-compose_default     Created                                                                                0.0s
 ⠿ Container ui-super-heroes          Created                                                                                0.1s
 ⠹ Container villains-db              Creating                                                                               0.2s
 ⠹ Container fights-db                Creating                                                                               0.2s
 ⠿ Container jaeger                   Created                                                                                0.1s
 ⠿ Container apicurio                 Created                                                                                0.1s
 ⠹ Container prometheus               Creating                                                                               0.2s
 ⠹ Container heroes-db                Creating                                                                               0.2s
 ⠋ Container otel-collector           Creating                                                                               0.0s
 ⠿ Container fights-kafka             Created                                                                                0.0s
 ⠿ Container event-statistics-java17  Created                                                                                0.0s
Error response from daemon: make cli opts(): making volume mountpoint for volume /Users/edeandre/workspaces/quarkus/quarkus-super-heroes-main/rest-fights/deploy/db-init/initialize-database.js: mkdir /Users: operation not permitted

If I then podman machine stop then start docker desktop & re-run docker compose -f deploy/docker-compose/java17.yml -f deploy/docker-compose/monitoring.yml up everything works fine

[n1hility]

@edeandrea hmm interesting what version of Docker cli are you using?

[edeandrea]

╰─ docker version                                                                                                  
Client:
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Wed Jun  1 21:26:39 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.16.2 (95914)
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:43 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[edeandrea]

[n1hility]

@edeandrea Interesting I’m on a slightly newer version I’ll try to mirror your setup and see what happens

[edeandrea]

Newer version? Docker Desktop says I'm on the latest....

[n1hility]

@edeandrea Yeah I'm on 20.10.21. Although I am on arm so maybe different versions released?

I can't seem to reproduce for some reason with 20.10.16 either. Could you paste a podman info and then a docker info with podman running? I'll keep trying other things to see if I can figure out how you are getting that.

I am on an older mac os release, and I have been looking for an excuse to upgrade, so might try that this weekend.

[n1hility]

oh you are arm too? not sure how we ended up with different versions

[edeandrea]

I'm on arm as well (latest macos version). I'm away from the computer now but will try to get to it over the weekend at some point. If not, definitely on Monday.

[edeandrea]

Did you brew install the docker cli? I have the full docker Desktop installed, so maybe that is where the libs are coming from?

[quarkus-bot]

/cc @geoand (devservices), @stuartwdouglas (devservices)

[fedinskiy]

Hi, we have run Quarkus on podman some time ago, and found several issues. Hope, they can be of use:

1. There are problems(Db2 container started via podman is not acessible for connection containers/podman#16432) when accessing DB2 on rootless podman. If DB2 is going to be supported, then this can cause problems for devmode.
2. Issue with short names, mentioned earlier(Error, when pulling short-named image via docker api containers/podman#16400) also affects ryuk containers, since ryuk images are pulled by their short name. The known workaround is to pull them by full name before running any tests.
3. Quarkus fails to build image, using podman and quarkus-container-image-docker together: Automatic build of Docker image fails when running with podman #28721
4. I do not know, if Quarkus uses testcontainers classes, other than GenericContainer, but beware, that for fully-qualified names you need to explicitly set asCompatibleSubstituteFor. So, eg new MySQLContainer<>("mysql:8.0.16") will work, but new MySQLContainer<>("docker.io/library/mysql:8.0.16") probably would not not (see [Bug]: library/mysql is not a valid substitute for mysql testcontainers/testcontainers-java#5612 and comments. Original issue with "library/mysql:8.0.16" is fixed here: Accept image names with library/ prefix as a valid substitute testcontainers/testcontainers-java#6174, but it omitted the case with fully-qualified names )

[Sanne]

The devservices container we use as default image for Elasticsearch will trigger this problem on podman:

chroot: cannot change root directory to '/': Operation not permitted

This gets fixed by adding the --cap-add sys_chroot parameters to podman.

Wondering if we could automatically set this, but wondering about implications.

[maxandersen]

related #31490

[dmarrazzo]

Another problem specific of a podman 4.4.2
#31652

[Karm]

related #31857

[maxandersen]

I opened containers/podman#18479 to make it easier to get info about rootful status on podman machine.

[maxandersen]

I opened containers/podman#18480 to capture the issue found with docker.sock / podman.sock misalignment when running podman machine (i.e. affects windows/mac users but not linux)

[fedinskiy]

Two issues, related to dev mode and tests on podman 4.5.1:
#33985
#33896

[janderssonse]

Uploaded a simple devservice project which one can show that podman in podman with quarkus devservices enabled fails.
However it works if the container is started with podman run and connected - so something, as other also reported, is up with devservices. https://github.com/janderssonse/quarkus-podman-in-podman-db-devserv-fail-example. (The use case also with fine with docker in docker and Quarkus Devservices so yet again points to podman , quarkus and devservices combination).

I hope to see Podman as the expected default container environment in the future for Quarkus and not docker as now :). That would also align with the expectations (they should work smooth as butter together) of both Podman and Quarkus having heavy Red Hat contributions. Thanks!

[Inithron]

Hi,
I stumbled on this issue and I only wanted to ask if this issue is still relevant, or if it can be closed. I use Podman in Version 5.1.1 and Podman Desktop in version 1.11.1 without any issues.

[maxandersen]

@Inithron there are still devservices that doesn't work great on podman. situation is much better so we aren't there yet.