Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman-remote build: can fail but exit 0 (success) #10029

Closed
edsantiago opened this issue Apr 14, 2021 · 5 comments · Fixed by #10030
Closed

podman-remote build: can fail but exit 0 (success) #10029

edsantiago opened this issue Apr 14, 2021 · 5 comments · Fixed by #10030
Labels
flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. remote Problem is in podman-remote

Comments

@edsantiago
Copy link
Member

[ Forked from #9751 ]

podman-remote build sometimes fails but returns exit status 0. Trivial reproducer (assumes podman system service running elsewhere):

$ cat Containerfile
FROM quay.io/libpod/nosuchimage:nosuchtag
RUN echo hi
$ while :;do ../bin/podman-remote build -t build_test --pull-never . && break;done
...runs for a minute or two, then

STEP 1: FROM quay.io/libpod/nosuchimage:nosuchtag
Error: error creating build container: pull policy is "never" but "quay.io/libpod/nosuchimage:nosuchtag" could not be found locally

STEP 1: FROM quay.io/libpod/nosuchimage:nosuchtag
$ <--- back to prompt, with exit status 0. This should never happen.

master @ 14ec3a6, also in podman-3.1.0-3.fc33
@edsantiago edsantiago added flakes Flakes from Continuous Integration remote Problem is in podman-remote labels Apr 14, 2021
@mheon
Copy link
Member

mheon commented Apr 14, 2021

Is this a flake? If it's not, and it's consistent, this is probably a mistake in how remote Build is written (either not returning or correctly handling an error)

@edsantiago
Copy link
Member Author

Flake. The above reproducer works fine for the first 50-100 iterations, it just fails randomly. (Same in CI).

@mheon
Copy link
Member

mheon commented Apr 14, 2021

Alright, it's a race then. Damn, I was hoping this one was going to be simple.

@rhatdan
Copy link
Member

rhatdan commented Apr 14, 2021

I might have a fix.

@rhatdan rhatdan mentioned this issue Apr 14, 2021
Merged
rhatdan added a commit to rhatdan/podman that referenced this issue Apr 14, 2021
@rhatdan
Fix flake on failed podman-remote build
4347a62 
We have a race condition where podman build can fail
but still return an exit code of 0.  This PR ensures
that as soon as the build fails, the failed flag is set
eliminating the race.

Fixes: containers#10029

[NO TESTS NEEDED] Tests of failed builds are already in place, and
the elimination of the race should be enough.

Signed-off-by: Daniel J Walsh <[email protected]>
@edsantiago
Copy link
Member Author

@rhatdan I'm sorry... no-go. It takes a lot longer than before (30 seconds instead of about 10), but the race is still there.

mheon pushed a commit to mheon/libpod that referenced this issue Apr 16, 2021
@rhatdan @mheon
Fix flake on failed podman-remote build
5c21861 
We have a race condition where podman build can fail
but still return an exit code of 0.  This PR ensures
that as soon as the build fails, the failed flag is set
eliminating the race.

Fixes: containers#10029

[NO TESTS NEEDED] Tests of failed builds are already in place, and
the elimination of the race should be enough.

Signed-off-by: Daniel J Walsh <[email protected]>
jmguzik pushed a commit to jmguzik/podman that referenced this issue Apr 26, 2021
@rhatdan @jmguzik
Fix flake on failed podman-remote build
8e33559 
We have a race condition where podman build can fail
but still return an exit code of 0.  This PR ensures
that as soon as the build fails, the failed flag is set
eliminating the race.

Fixes: containers#10029

[NO TESTS NEEDED] Tests of failed builds are already in place, and
the elimination of the race should be enough.

Signed-off-by: Daniel J Walsh <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. remote Problem is in podman-remote
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix flake on failed podman-remote build rhatdan/podman
3 participants
@edsantiago @rhatdan @mheon