Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

container wait: indicate timeout in error #18869

Merged
merged 1 commit into from
Jun 13, 2023

Conversation

vrothberg
Copy link
Member

When waiting for a container, there may be a time window where conmon has already exited but the container hasn't been fully cleaned up. In that case, we give the container at most 20 seconds to be fully cleaned up. We cannot wait forever since conmon may have been killed or something else went wrong.

After the timeout, we optimistically assume the container to be cleaned up and its exit code to present. If no exit code can be found, we return an error.

Indicate in the error whether the timeout kicked in to help debug (transient) errors and flakes (e.g., #18860).

[NO NEW TESTS NEEDED]

Does this PR introduce a user-facing change?

None

@vrothberg vrothberg marked this pull request as ready for review June 13, 2023 09:07
@openshift-ci openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 13, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 13, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 13, 2023
@Luap99
Copy link
Member

Luap99 commented Jun 13, 2023

Not sure how this helps to debug #18860? It is obvious to me that we hit the timeout in the flake.

Regardless this change LGTM, should give us some clarity without looking at the timings, especially if we start seeing these in the wild.

@vrothberg
Copy link
Member Author

Not sure how this helps to debug #18860? It is obvious to me that we hit the timeout in the flake.

I strongly suspect so as well but without the log I cannot be 100 percent sure.

@vrothberg
Copy link
Member Author

@edsantiago PTAL

libpod/container_api.go Outdated Show resolved Hide resolved
When waiting for a container, there may be a time window where conmon
has already exited but the container hasn't been fully cleaned up.
In that case, we give the container at most 20 seconds to be fully
cleaned up.  We cannot wait forever since conmon may have been killed or
something else went wrong.

After the timeout, we optimistically assume the container to be cleaned
up and its exit code to present.  If no exit code can be found, we
return an error.

Indicate in the error whether the timeout kicked in to help debug
(transient) errors and flakes (e.g., containers#18860).

[NO NEW TESTS NEEDED]

Signed-off-by: Valentin Rothberg <[email protected]>
Copy link
Member

@Luap99 Luap99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@edsantiago
Copy link
Member

LGTM2

@vrothberg
Copy link
Member Author

ready to merge

@Luap99
Copy link
Member

Luap99 commented Jun 13, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 13, 2023
@openshift-merge-robot openshift-merge-robot merged commit 2a947c2 into containers:main Jun 13, 2023
@vrothberg vrothberg deleted the debug-18860 branch August 24, 2023 07:19
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Nov 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. release-note-none
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants