Podman v3 cannot load container images built with BuildKit

[apyrgio]

We got bit by a Podman v3 issue, which we found out when we were about to publish our Debian packages to apt-tools-prod (see the CI failure):

Writing manifest to image destination
Storing signatures
Loaded image(s): localhost/latest:latest
Traceback (most recent call last):
  File "/usr/bin/dangerzone-cli", line 33, in <module>
    sys.exit(load_entry_point('dangerzone==0.8.0', 'console_scripts', 'dangerzone-cli')())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/dangerzone/errors.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/dangerzone/cli.py", line 87, in cli_main
    dangerzone.isolation_provider.install()
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/container.py", line 194, in install
    if not Container.is_container_installed(raise_on_error=True):
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/container.py", line 245, in is_container_installed
    raise ImageNotPresentException(
dangerzone.isolation_provider.container.ImageNotPresentException: Image is not listed after installation. Bailing out.

Background

Recent Docker Desktop versions ship with the containerd image store (BuildKit) by default, which is a different way to build and store images. This change caused a mixup with image IDs, and was the reason behind the 0.7.1 release (#933). Our way to fix it was to build the container image, and register two image IDs; one for the containerd image store (Docker Desktop only), and one for the legacy image store (Podman and older Docker users).

Turns out that Podman versions prior to 4.0 cannot load images built by the containerd image store. The result is to tag them in a weird way (localhost/latest:latest): containers/podman#11619

Why now?

How come it bit us now? Well, it's not possible to catch this error during QA. The only way to catch it is to:

• Build the container image with the containerd image store (this happened by sheer chance, we could have built it with another image store).
• Test the Dangerzone Debian package in one of our older distros (Ubuntu 20.04 or 22.04)

Next steps

We should add a line in our release notes to always build container images with the legacy image store.

[apyrgio]

As a quick fix for the 0.8.0 release, we have rebuilt our container image with Podman, and have shipped that with our Debian packages. This means the image ID and hashes will be different from the one published in our 0.8.0 release assets.

[almet]

Thanks Alex for the detailed notes on the matter.

I believe two things are worth noting:

• Even when building the images with Docker Desktop and the containerd store, we couldn't produce a container image that would work on Ubuntu 20.04 and 22.04. As a result, and after some tinkering, we decided to move to Podman Desktop for building the image, which produces an image that works for all our supported distributions.
• We are not sure why we are bitten by this, because the Docker Desktop we were using for building the images is the same as it was before. One of the reason could be that we did a reset of the Docker VM at some point, but we are not sure.

The win for us, is that this issue seem to point to the direction of using Podman Desktop as a replacement for Docker Desktop when building the images, which is something we are currently considering in #925