podman run seems to be trying to pull local images as if they don't exist and or are hosted at localhost.

[bayeslearner]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

bug

Description
podman can't pull image from docker.io and built one locally, and preappends it with a "localhost" as repository. But later when podman run is executed, it tries to connect to "container registry localhost. "

[fedora@fedora34 HomelabOS-v0.8.6]$ sudo docker image ls
REPOSITORY                     TAG         IMAGE ID      CREATED             SIZE
localhost/nickbusey/homelabos  v0.8.6      a57d91c52b78  About a minute ago  1.33 GB
docker.io/library/python       3.8-alpine  d314e28e240c  5 days ago          45.3 MB
docker.io/library/alpine       latest      c059bfaa849c  10 days ago         5.87 MB
docker.io/library/wordpress    latest      e3a452c0a154  2 weeks ago         629 MB
docker.io/library/mariadb      latest      e2278f24ac88  3 weeks ago         416 MB

Trying to pull localhost/nickbusey/homelabos:v0.8.6...
WARN[0000] failed, retrying in 1s ... (1/3). Error: initializing source docker://localhost/nickbusey/homelabos:v0.8.6: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
WARN[0001] failed, retrying in 1s ... (2/3). Error: initializing source docker://localhost/nickbusey/homelabos:v0.8.6: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
WARN[0002] failed, retrying in 1s ... (3/3). Error: initializing source docker://localhost/nickbusey/homelabos:v0.8.6: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
Error: initializing source docker://localhost/nickbusey/homelabos:v0.8.6: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

Steps to reproduce the issue:

1. I used this git repository https://gitlab.com/NickBusey/HomelabOS

2. ran "make config" and it somehow decides to build (successfully) a local image.

[fedora@fedora34 HomelabOS-v0.8.6]$ sudo docker image ls
REPOSITORY                     TAG         IMAGE ID      CREATED             SIZE
localhost/nickbusey/homelabos  v0.8.6      a57d91c52b78  About a minute ago  1.33 GB

3. later when I try to run it. podman tries to find this image by "pinging container registry localhost". What is this???

Describe the results you received:
see above

Describe the results you expected:
podman should reuse local images and not try to ping a container registry called localhost

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:
3.4.2

(paste your output here)

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

3.4.2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
yes

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[vrothberg]

Thanks for reaching out. Any chance you can share a sequence of podman commands that lead to the issue? Cloning a git repo and analyzing everything is time costly.

[bayeslearner]

By the way, I am able to successfully setup everything using docker (not podman) on a different machine. So it is definitely a parity issue between them.

Here are some relevant code snippets of how docker is used.
docker asks to tag image as nickbusey/homelabos:$(VERSION),
podman would tag it with localhost prefix.

Later docker run will work fine, but it doesn't with podman.

# First build the docker images needed to deploy
        @sudo docker pull nickbusey/homelabos:$(VERSION) || true
        @sudo docker inspect --type=image nickbusey/homelabos:$(VERSIO
N) > /dev/null && printf "\033[92m========== Docker image already buil
t ==========\033[0m\n" || sudo docker build . -t nickbusey/homelabos:$(VERSION)

if [ -f "${SSH_KEY}" -a -f "${SSH_KEY}.pub" -a -f "$HOME/.homelabos_vault_pass" ]; then
    docker run --rm -it \
      -v ${SSH_KEY}:/root/.ssh/id_rsa \
      -v ${SSH_KEY}.pub:/root/.ssh/id_rsa.pub \
      -v $(pwd):/data \
      -v $HOME/.homelabos_vault_pass:/ansible_vault_pass \
      nickbusey/homelabos:${VERSION} "$@"

[vrothberg]

@bayeslearner, how are you running Podman? By aliasing docker=podman or by pointing the Docker client to the Podman socket?

[bayeslearner]

Not a problem with how podman is run. I'm using it for other docker-compose/docker related stuff.

dnf install docker-compose podman-docker
systemctl enable --now --user podman.socket
export DOCKER_HOST="unix:$XDG_RUNTIME_DIR/podman/podman.sock" in bashrc.

[bayeslearner]

Is this related to this error: #9111 #9127 #11181

[vrothberg]

Thanks, @bayeslearner. This issue should already be fixed in the main branch by commit 5bdd571.

Short-name normalization works very differently in Podman when compared to Docker. However, we recently made Podman behave like Docker on the Docker REST API.

What happens is that the image is being tagged as localhost/$tag and it seems there's a follow-up attempt to pull the image which ultimately fails. After commit 5bdd571, the image will be tagged with docker.io/$tag and the pull will succeed.

The fix will be shipped with the next major release of Podman (i.e., 4.0 early next year).

[bayeslearner]

Note that in docker hub the tag 0.86 for that container image doesn't exist, being the dev branch. In the mean time, is there a work-around, given the snippet below?

if [ -f "${SSH_KEY}" -a -f "${SSH_KEY}.pub" -a -f "$HOME/.homelabos_vault_pass" ]; then
    docker run --rm -it \
      -v ${SSH_KEY}:/root/.ssh/id_rsa \
      -v ${SSH_KEY}.pub:/root/.ssh/id_rsa.pub \
      -v $(pwd):/data \
      -v $HOME/.homelabos_vault_pass:/ansible_vault_pass \
      nickbusey/homelabos:${VERSION} "$@"

[vrothberg]

@bayeslearner, a workaround would be to replace all image references on nickbusey/homelabos with docker.io/nickbusey/homelabos.

[bayeslearner]

I did prefix everything with "docker.io" and the errors are still there. Upon further trial and error, I think the errors are due to various issues besides the tag's naming issue.

Issue 1. rootful podman build and rootless podman run clash. podman won't find the image built for root.
Solution: removed sudo for the build or pull everywhere.

# First build the docker images needed to deploy
        @sudo docker pull nickbusey/homelabos:$(VERSION) || true
        @sudo docker inspect --type=image nickbusey/homelabos:$(VERSIO
N) > /dev/null && printf "\033[92m========== Docker image already buil
t ==========\033[0m\n" || sudo docker build . -t nickbusey/homelabos:$(VERSION)

# First build the docker images needed to deploy
        @sudo docker pull nickbusey/homelabos:$(VERSION) || true
        @sudo docker inspect --type=image nickbusey/homelabos:$(VERSIO
N) > /dev/null && printf "\033[92m========== Docker image already buil
t ==========\033[0m\n" || sudo docker build . -t nickbusey/homelabos:$(VERSION)

Issue2. selinux volume mounting.
Solution: I added :z for all the mount.

After fixing Issue 1 and 2, I was able to run ansible inside the built container image.

---
- hosts: 127.0.0.1
  connection: local
....

However, I ran into the third issue:

Issue3. ansible template can't move temp files into mount directory.
See error below. It seems to be reported here: ansible/ansible#67857

Traceback (most recent call last):
  File "/tmp/ansible_copy_payload_p594yybw/ansible_copy_payload.zip/ansible/module_utils/basic.py", line 2299, in atomic_move
    shutil.copy2(b_src, b_tmp_dest_name)
  File "/usr/lib/python3.9/shutil.py", line 444, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.9/shutil.py", line 387, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/usr/lib/python3.9/shutil.py", line 337, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: b'/data/settings/.ansible_tmpw5pt1oioconfig.yml'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "checksum": "b8660ef41e657777b56f109886d710549b9e883a",
    "diff": [],
    "invocation": {
        "module_args": {
            "_original_basename": "config.yml",
            "attributes": null,
            "backup": false,
            "checksum": "b8660ef41e657777b56f109886d710549b9e883a",
            "content": null,
            "delimiter": null,
            "dest": "/data/settings/config.yml",
            "directory_mode": null,
            "follow": false,
            "force": true,
            "group": null,
            "local_follow": null,
            "mode": "0777",
            "owner": null,
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": "/root/.ansible/tmp/ansible-tmp-1638914630.0026531-17671289020759/source",
            "unsafe_writes": null,
            "validate": null
        }
    },
    "msg": "Failed to replace file: b'/root/.ansible/tmp/ansible-tmp-1638914630.0026531-17671289020759/source' to /data/settings/config.yml: [Errno 13] Permission denied: b'/data/settings/.ansible_tmpw5pt1oioconfig.yml'"
}

Any suggestions? From the perspective of container namespace, the folder inside the podman container is owned by root and ansible is run as root too. So why can't it write to that folder /data/settings?

[vrothberg]

rootful podman build and rootless podman run clash. podman won't find the image built for root.
Solution: removed sudo for the build or pull everywhere.

Yes, it is not possible to access the storage of other users. So root and rootless cannot be mixed. That may explain the original error with trying to pull from localhost/....

After fixing Issue 1 and 2, I was able to run ansible inside the built container image.

Both solutions look good to me.

Any suggestions? From the perspective of container namespace, the folder inside the podman container is owned by root and ansible is run as root too. So why can't it write to that folder /data/settings?

I have two guesses: 1) try running without selinux or disable it temporarily just to be sure. 2) is there a symlink somewhere pointing to another destination as the reported one?

[bayeslearner]

Yeah, it seems to be selinux again. Thanks for the suggestion. Not sure how to fix it, except disable it.

[vrothberg]

Did you try using :Z on all volumes?

[rhatdan]

NO, do not relabel content that is not private to the container. Relabeling ssh keys is a bad idea, since other confined objects on the host could break.

Disable SELinux separation for the container if you are going to leak data like this inside.

--security-opt label:disable

You are basically giving the container the ability to attack every host with keys in ~/.ssh. Confining the container with SELinux is the least of your problems, if this is a hostile container.

If a container broke out, you would definitely want SELinux to block access to these files.

  -v ${SSH_KEY}:/root/.ssh/id_rsa \
  -v ${SSH_KEY}.pub:/root/.ssh/id_rsa.pub 

[vrothberg]

That makes perfect sense, thanks @rhatdan !