Ensure that container still exists when removing #10476
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mheon The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
Is this to fix THE race? |
|
Nope. Separate BZ, unfortunately. Would not be surprised if it fixes some races in CI, though. |
After containers#8906, there is a potential race condition in container removal of running containers with `--rm`. Running containers must first be stopped, which was changed to unlock the container to allow commands like `podman ps` to continue to run while stopping; however, this also means that the cleanup process can potentially run before we re-lock, and remove the container from under us, resulting in error messages from `podman rm`. The end result is unchanged, the container is still cleanly removed, but the `podman rm` command will seem to have failed. Work around this by pinging the database after we stop the container to make sure it still exists. If it doesn't, our job is done and we can exit cleanly. Signed-off-by: Matthew Heon <[email protected]>
mheon
force-pushed
the
ensure_exists_on_remove
branch
from
9a8d9ca to
fad6e1d
Compare
|
FWIW, addresses https://bugzilla.redhat.com/show_bug.cgi?id=1964852 |
|
/lgtm |
openshift-ci
bot
added
the
do-not-merge/hold
openshift-ci
bot
removed
the
do-not-merge/hold
github-actions
bot
added
the
locked - please file new issue/PR
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After #8906, there is a potential race condition in container removal of running containers with
--rm. Running containers must first be stopped, which was changed to unlock the container to allow commands likepodman psto continue to run while stopping; however, this also means that the cleanup process can potentially run before we re-lock, and remove the container from under us, resulting in error messages frompodman rm. The end result is unchanged, the container is still cleanly removed, but thepodman rmcommand will seem to have failed.Work around this by pinging the database after we stop the container to make sure it still exists. If it doesn't, our job is done and we can exit cleanly.
[NO TESTS NEEDED] because the race doesn't trigger easily and I can't make a test to reproduce it on demand anywhere.