podman build fails with local image as FROM argument
#10914
Labels
kind/bug
Categorizes issue or PR as related to a bug.
locked - please file new issue/PR
Assist humans wanting to comment on an old issue or PR with locked comments.
Comments
|
@vrothberg PTAL |
|
I cannot reproduce: @srcshelton, if possible, could share the output of |
|
To confirm, I can do:
… on every one of a variety of hosts, some Working case (with Non-working case (with |
|
Literal |
|
( diff'ing the two outputs above, I notice that we have - Working: Broken: … which is slightly surprising, but perhaps indicates some change in this specific area between |
|
@srcshelton, there was a number of regressions in v3.2 with respect to multi-arch images. Many fixes haven been backported to the v3.2 branch (https://github.com/containers/podman/tree/v3.2). Any chance you could compile the v3.2 branch and test? I am unable to extract a reproducer but maybe the regression has been fixed in the meantime. We plan to release v3.2.3 soon. |
|
I'll give that branch a try today! Please note, however, that this issue also occurs on |
I'm now on: ... and get slightly different (but still failing) output. Firstly, without ... and with Immediately afterwards: The image in question is based on ... so running a container seems to find the image in question, so it definitely appears to be a change around the interaction with |
|
|
@srcshelton, can you try with Thanks for your cooperation, which is super helpful to isolate the source of the issue. |
|
Looks as if it's a |
|
@srcshelton, thanks for checking. Any chance you can share the reproducer? Then you're off the hook and I can start running :) |
|
I've not tested this on a host not running Gentoo to begin with (... although the intent is that it should run from anywhere!), but hopefully if you clone https://github.com/srcshelton/docker-gentoo-build then it should be as simple as running (I have this in Let me try this on a non-Gentoo host system and see if any changes are needed... |
|
Yup, so there are a few uses of the Gentoo-specific The error occurs during the 'stage3' section (the second part, immediately after 'env'). |
|
Thanks, @srcshelton. FWIW, I can reproduce but it does not work with Podman v3.2.1 or any other version of Podman/Buildah I've tried. Whatever version I use, it always attempts to pull |
|
Could you post the full output? I did a test install on Ubuntu 21.04 using their standard packages ( Note that the process as a whole will fail if not run on Gentoo right now, but the |
|
What I did was: diff --git a/gentoo-init.docker b/gentoo-init.docker
index d5d84f4ccbf1..535ecd4a159e 100755
--- a/gentoo-init.docker
+++ b/gentoo-init.docker
@@ -245,8 +245,8 @@ if [ -z "${*:-}" ] || echo " ${*} " | grep -Fq -- ' --env '; then
# USED_BY: stage3 build
# shellcheck disable=SC2086
- $docker image build \
- --platform "linux/${docker_arch}" \
+ $PODMAN build \
+ --platform "linux/${docker_arch}" \
--build-arg ARCH="${arch}" \
--build-arg PROFILE_PATH="${profile:-}" \
--build-arg CHOST="${chost:-}" \Just tried the v3.0.1-rhel branch but it also fails. |
|
So on an Ubuntu laptop (e.g. bare-metal, not virtualised or within a chroot() environment, etc.) which is rarely used but was upgraded to Ubuntu 21.04, I literally did: ... which succeeded past ... so I guess the question has to be, what are the possible differences between your environment and that of a stock Ubuntu installation (or, indeed, the original Gentoo installations) which cause this to always fail for you, but only fail for Gentoo after |
|
Just noticed that I had to change another |
vrothberg
added a commit
to vrothberg/common
that referenced
this issue
Jul 16, 2021
Commit bee8cae enforced the pull policy to "always" when a custom platform was specified. The reason for always pulling is that many multi-arch images are broken; wrong configs, wrong platforms, etc. We cannot perform reliable platform checks. While we may to have to revisit this strategy in the future, it is more important to keep existing workloads running; a bit between a rock and hard place. This change complements commit bee8cae: if attempt to pull an image that resolves to "localhost/", set the pull policy "newer" instead of "always" such that the image may be used instead of erroring out. Ultimately to preserve previous behavior. Context: containers/podman/issues/10914 Signed-off-by: Valentin Rothberg <[email protected]>
|
@srcshelton thank you again for your great help in debugging the issue. I opened containers/common#679 to fix the issue. I hope to get that backported today to Podman v3.2 (Cc @mheon @rhatdan) |
|
Phew - I was still kinda worried it was just me 😅 Glad to help! |
vrothberg
added a commit
to vrothberg/common
that referenced
this issue
Jul 16, 2021
Commit bee8cae enforced the pull policy to "always" when a custom platform was specified. The reason for always pulling is that many multi-arch images are broken; wrong configs, wrong platforms, etc. We cannot perform reliable platform checks. While we may to have to revisit this strategy in the future, it is more important to keep existing workloads running; a bit between a rock and hard place. This change complements commit bee8cae: if attempt to pull an image that resolves to "localhost/", set the pull policy "newer" instead of "always" such that the image may be used instead of erroring out. Ultimately to preserve previous behavior. Context: containers/podman/issues/10914 Signed-off-by: Valentin Rothberg <[email protected]>
vrothberg
added a commit
to vrothberg/common
that referenced
this issue
Jul 16, 2021
Commit bee8cae enforced the pull policy to "always" when a custom platform was specified. The reason for always pulling is that many multi-arch images are broken; wrong configs, wrong platforms, etc. We cannot perform reliable platform checks. While we may to have to revisit this strategy in the future, it is more important to keep existing workloads running; a bit between a rock and hard place. This change complements commit bee8cae: if attempt to pull an image that resolves to "localhost/", set the pull policy "newer" instead of "always" such that the image may be used instead of erroring out. Ultimately to preserve previous behavior. Context: containers/podman/issues/10914 Signed-off-by: Valentin Rothberg <[email protected]>
vrothberg
added a commit
to vrothberg/common
that referenced
this issue
Jul 16, 2021
Commit bee8cae enforced the pull policy to "always" when a custom platform was specified. The reason for always pulling is that many multi-arch images are broken; wrong configs, wrong platforms, etc. We cannot perform reliable platform checks. While we may to have to revisit this strategy in the future, it is more important to keep existing workloads running; a bit between a rock and hard place. This change complements commit bee8cae: if attempt to pull an image that resolves to "localhost/", set the pull policy "newer" instead of "always" such that the image may be used instead of erroring out. Ultimately to preserve previous behavior. Context: containers/podman/issues/10914 Signed-off-by: Valentin Rothberg <[email protected]>
vrothberg
added a commit
to vrothberg/common
that referenced
this issue
Jul 16, 2021
* pull with custom platform: handle "localhost/" (containers/podman/issues/10914) Signed-off-by: Valentin Rothberg <[email protected]>
|
This is now fixed in the main branch and was backported to v3.2.3. |
github-actions
bot
added
the
locked - please file new issue/PR
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
podman-3.2.2(buildah-1.21.0):podman-3.2.1(alsobuildah-1.21.0) and prior:It appears that
podman-3.2.2is trying to make a registry request for anyFROMimage, even if prefixed withlocalhost/.This may be a
buildahissue - containers/buildah#2904 appears similar, but was resolved by addinglocalhost/so I suspect this is something different.I've raised containers/buildah#3375 but since the first affected podman version (
3.2.2) and the last unaffected podman version (3.2.1) both incorporate the same release of buildah, this might actually be apodmanissue.This is reproducible on multiple architectures (output below from
arm64as I'd already downgraded theamd64host topodman-3.2.1).Output of
podman version:Output of
podman info --debug:Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes
The text was updated successfully, but these errors were encountered: