placeholder: reason for emergency podman-build --arch exception

[edsantiago]

PR #9821 vendored in buildah v1.20.0. That brought in a new test of the --arch flag:
https://github.com/containers/buildah/blob/b405ca6551576a98fbc79ae8fe701246d1c7a4f0/tests/bud.bats#L2596

This test failed under podman CI, requiring an emergency skip to get CI passing:

# $ podman build --force-rm=false --layers=false --arch=arm64 -t arch-test --signature-policy /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.20.0/tests/policy.json /var/tmp/tmp61d96368bc5feadfcbeaa482/my-dir
# STEP 1: FROM alpine
# time="2021-03-25T17:38:35-05:00" level=warning msg="failed, retrying in 2s ... (1/3). Error: Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get \"https://localhost/v2/\": dial tcp [::1]:443: connect: connection refused"
# time="2021-03-25T17:38:37-05:00" level=warning msg="failed, retrying in 2s ... (2/3). Error: Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get \"https://localhost/v2/\": dial tcp [::1]:443: connect: connection refused"
# time="2021-03-25T17:38:39-05:00" level=warning msg="failed, retrying in 2s ... (3/3). Error: Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get \"https://localhost/v2/\": dial tcp [::1]:443: connect: connection refused"
# Error: error creating build container: Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

The reason for this has not been understood. It now seems like it's related to the prefetch helper. In particular, here is a reproducer:

# podman pull quay.io/libpod/alpine
# podman images
REPOSITORY             TAG     IMAGE ID      CREATED        SIZE
quay.io/libpod/alpine  latest  961769676411  19 months ago  5.85 MB
# podman save --format oci-archive --output=/tmp/alpine.tar alpine
# podman rmi -a
# podman load -i /tmp/alpine.tar     ! this changes the image name and ID
# podman images
REPOSITORY        TAG     IMAGE ID      CREATED        SIZE
localhost/alpine  latest  1f97f0559cbd  19 months ago  5.85 MB
# echo "FROM alpine" | podman build --arch=arm64 -t foo -
STEP 1: FROM alpine
WARN[0000] failed, retrying in 2s ... (1/3). Error: Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
WARN[0002] failed, retrying in 2s ... (2/3). Error: Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
WARN[0004] failed, retrying in 2s ... (3/3). Error: Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
Error: error creating build container: Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

The image does include arm64 and other arches:

# podman manifest inspect alpine|gron|grep -i arch
json.manifests[0].platform.architecture = "amd64";
json.manifests[1].platform.architecture = "arm";
json.manifests[2].platform.architecture = "arm";
json.manifests[3].platform.architecture = "arm64";
json.manifests[4].platform.architecture = "386";
json.manifests[5].platform.architecture = "ppc64le";
json.manifests[6].platform.architecture = "s390x";

I wonder if this is related to #3761, in which podman save/load changes digests?

Anyhow, sorry for the non-helpful issue; I'm out of time for today. Hope someone can run with this.

[rhatdan]

@vrothberg This looks like containers/image is not catching the localhost case?

[vrothberg]

@vrothberg This looks like containers/image is not catching the localhost case?

Not quite. docker://localhost/alpine:latest instructs c/image to copy that image from a registry (i.e., "docker" transport).

There are a couple of things to unpack:

1. "localhost" prefixes are a constant pain
2. podman save/load may use these prefixes
3. podman build and buildah bud have a bug in the image pull/look up logic. While there is a local localhost/alpine image in @edsantiago's reproducer, the local image has the wrong architecture so we need to pull one in the desired architecture. This image reference (localhost/alpine) then seems to be used for pulling which is arguably wrong.

I think it's a Buildah bug. Added to my todo.

containers/image is entirely unaware of the "localhost" prefixing that Podman and Buildah do.

[vrothberg]

Update. It's a podman-side bug. Cannot reproduce with buildh bud.

[vrothberg]

I opened containers/buildah#3129 to fix it. It worked with Buildah since it's using a different default pull policy.

[vrothberg]

containers/buildah#3129 merged. @rhatdan, do you want to backport or shall we wait for the next release?

[rhatdan]

Wait for next release. which might be coming up quickly.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

This looks like it is fixed in the main branch now.