podman --remote run pulls image on every invocation

[afbjorklund]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman4 client pulls image every single run, where podman3 client only pulls if missing.

Steps to reproduce the issue:

1. podman --remote run busybox true

2. podman --remote run busybox true

Describe the results you received:

$ podman --version
podman version 4.0.0-dev
$ podman --remote run busybox true
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob sha256:01c2cdc137396a9b78be86c47e5773388e6eed953dec79e257777518134b45f1
Copying config sha256:cabb9f684f8ba3edb303d578bfd7d709d853539ea1b420a3f6c81a08e85bb3d7
Writing manifest to image destination
Storing signatures
$ podman --remote run busybox true
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob sha256:01c2cdc137396a9b78be86c47e5773388e6eed953dec79e257777518134b45f1
Copying config sha256:cabb9f684f8ba3edb303d578bfd7d709d853539ea1b420a3f6c81a08e85bb3d7
Writing manifest to image destination
Storing signatures
$

Describe the results you expected:

$ podman --version
podman version 3.2.1
$ podman --remote run busybox true
Error: 1 error occurred:
	* short-name "busybox" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"


$ podman --remote run docker.io/busybox true
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob sha256:01c2cdc137396a9b78be86c47e5773388e6eed953dec79e257777518134b45f1
Copying config sha256:cabb9f684f8ba3edb303d578bfd7d709d853539ea1b420a3f6c81a08e85bb3d7
Writing manifest to image destination
Storing signatures
$ podman --remote run docker.io/busybox true
$

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.17.2
Git Commit:   abbd6c167e8163a711680db80137a0731e06e564
Built:        Sun Nov  7 18:40:23 2021
OS/Arch:      linux/amd64

Server:
Version:      3.2.1
API Version:  3.2.1
Go Version:   go1.16.7
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Client: Ubuntu 20.04
Server: Ubuntu 21.10

[afbjorklund]

This seems related to the new policy:

"Pulling image docker.io/busybox (policy: always)"

• Commit 5fc622f

Previously it would only pull if missing

---

DEBU[0000] DoRequest Method: GET URI: http://d/v4.0.0/libpod/_ping 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.0.0/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] DoRequest Method: POST URI: http://d/v4.0.0/libpod/images/pull 
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob sha256:01c2cdc137396a9b78be86c47e5773388e6eed953dec79e257777518134b45f1
Copying config sha256:cabb9f684f8ba3edb303d578bfd7d709d853539ea1b420a3f6c81a08e85bb3d7
Writing manifest to image destination
Storing signatures
DEBU[0004] DoRequest Method: POST URI: http://d/v4.0.0/libpod/containers/create 

DEBU[0000] DoRequest Method: GET URI: http://d/v3.2.1/libpod/_ping 
DEBU[0000] DoRequest Method: GET URI: http://d/v3.2.1/libpod/_ping 
DEBU[0000] DoRequest Method: GET URI: http://d/v3.2.1/libpod/images/docker.io%2Fbusybox/exists 
DEBU[0000] DoRequest Method: POST URI: http://d/v3.2.1/libpod/containers/create 

[afbjorklund]

This scenario happens with:

1. Server is running the latest version available for ubuntu: 3.2.1
   Installing a newer version is problematic (requires kubic packages)

https://packages.ubuntu.com/impish/podman

limactl start examples/podman.yaml

2. Client is running the latest version available for brew: 3.4.1
   Installing an older version is problematic (requires binary download)

https://formulae.brew.sh/formula/podman#default

brew install podman

So instead, it pulls every time.

[afbjorklund]

The workaround is to run podman on the server instead, over ssh.

LIMA_INSTANCE=podman lima podman

(lima uses limactl shell to run the ssh command on the VM)

Instead of trying to use the podman-remote binary locally, over ssh.

export CONTAINER_HOST=unix://$HOME/podman.sock

(lima sets up a ssh tunnel from the host home, to the VM socket)

[rhatdan]

This looks like it works correctly in podman 3.4.

$ podman --remote run docker.io/busybox true
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob sha256:01c2cdc137396a9b78be86c47e5773388e6eed953dec79e257777518134b45f1
Copying blob sha256:01c2cdc137396a9b78be86c47e5773388e6eed953dec79e257777518134b45f1
Copying config sha256:cabb9f684f8ba3edb303d578bfd7d709d853539ea1b420a3f6c81a08e85bb3d7
Writing manifest to image destination
Storing signatures
$ podman --remote run docker.io/busybox true
$

[afbjorklund]

It looks like there are no kubic packages available for Ubuntu 21.10.

https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman

So that is not really an option (yet), unless using the debs from 21.04 ?

. /etc/os-release
VERSION_ID=$(echo $VERSION_ID | sed -e 's/21.10/21.04/g')

---

EDIT:

That installed 3.3.1, but it does understand the policy value - so it works.

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Apparently the new OS distribution and podman 3.4 release was still in "testing":

https://build.opensuse.org/package/show/devel:kubic:libcontainers:testing/podman