[Auraboot] Can't find local Podman container

[GrabbenD]

Auraboot can't find local images when using Podman (4.5.1).
I've verified that this works work Docker.

Podman installation

$ sudo pacman -S crun podman
$ sudo systemctl enable --now podman.socket
$ echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf
$ sudo -i

Auraboot issue

$ podman build -t local/system:latest .
< ... >
--> f3bec925af71
Successfully tagged localhost/test-byoi-fips:latest
f3bec925af71515054267677ebe7a694b9c78f04c5a980ff7116ab53264df4d8

$ podman run \
  -v "$PWD"/build:/tmp/auroraboot \
  -v /var/run/podman/podman.sock:/var/run/docker.sock \
  --rm -ti quay.io/kairos/auroraboot \
  --set container_image=docker://test-byoi-fips \
  --set "disable_http_server=true" \
  --set "disable_netboot=true" \
  --set "state_dir=/tmp/auroraboot"
4:20PM INF Pulling container image 'test-byoi-fips' to '/tmp/auroraboot/temp-rootfs' (local: true)
4:20PM ERR Failed pulling container image 'test-byoi-fips' to '/tmp/auroraboot/temp-rootfs' (local: true): exit status 1
2 errors occurred:
        * exit status 1
        * 'gen-iso' deps container-pull failed

$ podman image ls
REPOSITORY                         TAG                  IMAGE ID      CREATED         SIZE
localhost/test-byoi-fips           latest               f3bec925af71  23 minutes ago  846 MB

I've tried the following combinations

container_image=test-byoi-fips
container_image=localhost/test-byoi-fips
container_image=docker://test-byoi-fips
container_image=docker://test-byoi-fips:latest
container_image=docker://localhost/test-byoi-fips
container_image=docker://localhost/test-byoi-fips:latest

For reference, using -v /var/run/podman/podman.sock:/var/run/docker.sock works in Elemental-Toolkit when pulling local images with docker://

[jimmykarily]

I'm trying to see if the approach of mounting the podman socket as the docker socket inside an image works at all. This is what I see:

# List local podman images (works)
~/tmp $ podman images
REPOSITORY                 TAG         IMAGE ID      CREATED      SIZE
quay.io/kairos/auroraboot  latest      d651acee6d20  2 weeks ago  1.17 GB

# List local podman images by mounting the podman socket as docker socket inside the auroraboot image (doesn't work)
~/tmp $ podman  run --entrypoint  "" -it -v /var/run/podman/podman.sock:/var/run/docker.sock:ro quay.io/kairos/auroraboot docker images
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/json": dial unix /var/run/docker.sock: connect: permission denied

# List podman local image as root (works)
~/tmp $ sudo podman images
REPOSITORY                 TAG         IMAGE ID      CREATED      SIZE
docker.io/library/busybox  latest      a416a98b71e2  6 days ago   4.5 MB
quay.io/kairos/auroraboot  latest      d651acee6d20  2 weeks ago  1.17 GB
docker.io/library/ubuntu   latest      5a81c4b8502e  3 weeks ago  80.3 MB

# List local podman images as root by mounting the podman socket as docker socket inside the auroraboot image (works!)
~/tmp $ sudo podman  run --entrypoint  "" -it -v /var/run/podman/podman.sock:/var/run/docker.sock:ro quay.io/kairos/auroraboot docker images
REPOSITORY                  TAG       IMAGE ID       CREATED       SIZE
ubuntu                      latest    5a81c4b8502e   3 weeks ago   80.3MB
busybox                     latest    a416a98b71e2   6 days ago    4.5MB
quay.io/kairos/auroraboot   latest    d651acee6d20   2 weeks ago   1.17GB

This issue also seems a bit relevant: containers/podman#6015 (that's where I got the idea of running as root in the first place). I'm not sure if there is a simple way to fix it. If the issue I linked to is actually relevant, all the workarounds seems to involve becoming root in one way or another.

[GrabbenD]

For reference, this happens with a rootful container too. It doesn't have to be rootless

[jimmykarily]

In the description above, I can't tell if you are root or not when running those commands. Specifically, in my case, I can't run podman run -v /var/run/podman/podman.sock:/var/run/docker.sock --entrypoint /bin/bash --rm -ti quay.io/kairos/auroraboot without sudo because I get:

Error: statfs /var/run/podman/podman.sock: permission denied

Now, assuming you run this as root, the question is whether you run podman build -t local/system:latest . as root as well. If not, then the image you built, is not available to the root user. You can try sudo podman images to verify.

In other words, if you use the same user to:

• build the image (e.g. use sudo in podman build)
• create the auroraboot container (e.g. use sudo in podman run)
• run commands inside the container (e.g. be root inside the container)

you should be able to see your image inside the container. If you get to that point (with no auroraboot involved so far), I think auroraboot should also see the image.

[GrabbenD]

Sorry for the confusion @jimmykarily
Yes, I'm running the container as root and I'm building it as root. I forgot to include that I'm actually in the root account while testing all of this (sudo -i).

[jimmykarily]

I created a script that reproduces the problem (runs fine on Arch):

#!/bin/bash

# Reproduce: https://github.com/kairos-io/kairos/issues/1631

set -xe

export IMG=quay.io/kairos/core-ubuntu-22-lts:v2.3.3-rc1

function createworkdir() {
  # the directory of the script
  export DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
  # the temp directory used, within $DIR
  # omit the -p parameter to create a temporal directory in the default location
  export WORK_DIR=`mktemp -d -p "$DIR"`

  # check if tmp dir was created
  if [[ ! "$WORK_DIR" || ! -d "$WORK_DIR" ]]; then
    echo "Could not create temp dir"
    exit 1
  fi
}

# deletes the temp directory
function cleanup {      
  sudo rm -rf "$WORK_DIR"
  echo "Deleted temp working directory $WORK_DIR"
}

sudo systemctl enable --now podman.socket

createworkdir
# register the cleanup function to be called on the EXIT signal
trap cleanup EXIT

cd $WORK_DIR
mkdir -p build

echo "Pulling $IMG outside the auroraboot container"
sudo podman pull $IMG
export LOCALIMG="myimage:latest"
sudo podman tag $IMG $LOCALIMG


echo "These are the images docker sees inside the auroraboot container:"
sudo podman run --rm --entrypoint  "" -it -v /var/run/podman/podman.sock:/var/run/docker.sock:ro quay.io/kairos/auroraboot docker images

echo "Running auroraboot pointing to the local image"
sudo podman run \
  --rm -it \
  -v "$PWD"/build:/tmp/auroraboot \
  -v /var/run/podman/podman.sock:/var/run/docker.sock:ro \
  quay.io/kairos/auroraboot \
  --set container_image=docker://$LOCALIMG \
  --set "disable_http_server=true" \
  --set "disable_netboot=true" \
  --set "state_dir=/tmp/auroraboot"

The relevant output is this:

These are the images docker sees inside the auroraboot container:
+ sudo podman run --rm --entrypoint '' -it -v /var/run/podman/podman.sock:/var/run/docker.sock:ro quay.io/kairos/auroraboot docker images
REPOSITORY                          TAG          IMAGE ID       CREATED        SIZE
ubuntu                              latest       5a81c4b8502e   6 weeks ago    80.3MB
busybox                             latest       a416a98b71e2   3 weeks ago    4.5MB
quay.io/kairos/auroraboot           latest       d651acee6d20   5 weeks ago    1.17GB
localhost/myimage-root              latest       67ae8d21498b   10 days ago    4.5MB
quay.io/kairos/core-ubuntu-22-lts   v2.3.3-rc1   2aea9453135b   23 hours ago   2.62GB
localhost/local/system              latest       2aea9453135b   23 hours ago   2.62GB
localhost/myimage                   latest       2aea9453135b   23 hours ago   2.62GB
+ echo 'Running auroraboot pointing to the local image'
Running auroraboot pointing to the local image
+ sudo podman run --rm -it -v /home/dimitris/tmp2/tmp.4jDs9e3arB/build:/tmp/auroraboot -v /var/run/podman/podman.sock:/var/run/docker.sock:ro quay.io/kairos/auroraboot --set container_image=docker://myimage:latest --set disable_http_server=true --set disable_netboot=true --set state_dir=/tmp/auroraboot
12:18PM INF Pulling container image 'myimage:latest' to '/tmp/auroraboot/temp-rootfs' (local: true)
12:18PM ERR Failed pulling container image 'myimage:latest' to '/tmp/auroraboot/temp-rootfs' (local: true): exit status 1
2 errors occurred:
	* exit status 1
	* 'gen-iso' deps container-pull failed

so docker sees the image as being there but auroraboot still tries to pull it. Maybe auroraboot is set to always-pull mode somehow?

[jimmykarily]

Running these inside the auroraboot container:

sh-4.4# docker images
...
localhost/myimage                   latest       2aea9453135b   23 hours ago   2.62GB

sh-4.4# docker run -it myimage ls
IMAGE	 bin   dev  home  lib32  libx32  mnt  proc  run   snap	sys	tmp  var
VERSION  boot  etc  lib   lib64  media	 opt  root  sbin  srv	system	usr

sh-4.4# luet util unpack --local myimage build
 INFO   Downloading myimage to /build
  ERROR    Error response from daemon: docker.io/library/myimage:latest: image not known

sh-4.4# luet util unpack --local localhost/myimage build
 INFO   Downloading localhost/myimage to /build
  ERROR    Error response from daemon: docker.io/localhost/myimage:latest: image not known

so it's luet that doesn't see the image

[jimmykarily]

luet is using this project to manipulate images: https://github.com/google/go-containerregistry

but the "image not known" error seems to be coming from podman itself. I guess the way the library queries podman for an image is incompatible with podman (and would probably work with docker). Simply "masquerading" the podman socket as docker socket is probably not enough to fake it.

If there is no specific reason to mix tools like this, I would say we can close this issue. There is not much we can do on this repo anyway.

[jimmykarily]

Thinking again, I guess the goal was never to mix tools but use podman instead of docker right? The mounting of podman socket inside auroraboot is just a "hack" to fool auroraboot.

Rephrasing then: This is a feature request for auroraboot to support podman (with no hacks). I'll leave it open for discussion in the next planning.

[jimmykarily]

It turns out, the library we use wants to always have a registry. With some insight from @mudler we figured out a workaround:

#!/bin/bash

# Reproduce: https://github.com/kairos-io/kairos/issues/1631

set -xe

export IMG=quay.io/kairos/core-ubuntu-22-lts:v2.3.3-rc1

function createworkdir() {
  # the directory of the script
  export DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
  # the temp directory used, within $DIR
  # omit the -p parameter to create a temporal directory in the default location
  export WORK_DIR=`mktemp -d -p "$DIR"`

  # check if tmp dir was created
  if [[ ! "$WORK_DIR" || ! -d "$WORK_DIR" ]]; then
    echo "Could not create temp dir"
    exit 1
  fi
}

# deletes the temp directory
function cleanup {      
  sudo rm -rf "$WORK_DIR"
  echo "Deleted temp working directory $WORK_DIR"
}

sudo systemctl enable --now podman.socket

createworkdir
# register the cleanup function to be called on the EXIT signal
trap cleanup EXIT

cd $WORK_DIR
mkdir -p build

echo "Pulling $IMG outside the auroraboot container"
sudo podman pull $IMG
export LOCALIMG="myimage:latest"
sudo podman tag $IMG $LOCALIMG


echo "This are the image docker sees inside the auroraboot container:"
sudo podman run --rm --entrypoint  "" -it -v /var/run/podman/podman.sock:/var/run/docker.sock:ro quay.io/kairos/auroraboot docker images

echo "Running auroraboot pointing to the local image"
sudo podman run \
  --rm -it \
  -v "$PWD"/build:/tmp/auroraboot \
  -v /var/run/podman/podman.sock:/var/run/docker.sock:ro \
  quay.io/kairos/auroraboot \
  --set container_image=docker://docker.io/$LOCALIMG \
  --set "disable_http_server=true" \
  --set "disable_netboot=true" \
  --set "state_dir=/tmp/auroraboot"

You can prefix the image with docker.io/ and it will be able to use it. Either you add it to the container_image like in the script above, or you can just tag your image with docker.io/ in front.

[jimmykarily]

Let's keep this open to document it.