[4.6] kludges for the freeze race #40
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Before this commit, Set() used GetState() to check the freezer state and retry the operation if the actual state still differs from requested. This should help with the situation when a new process (such as one added by runc exec) is added to the container's cgroup while it's being freezed by the kernel, but it's not working as it should. The problem is, GetState() never returns FREEZING state, looping until the state is either FROZEN or THAWED, so Set() does not have a chance to repeate the freeze attempt. As a result, the container might end up stuck in a FREEZING state, with GetState() never returning (which in turn blocks some other operations). One way to fix this would be to have GetState returning FREEZING state instead of retrying ad infinitum. It would result in changing the public API, and no callers of GetState expects it to return this. To fix, let's not use GetState() from Set(). Instead, read the freezer.state file directly and act accordingly -- return success on FROZEN, retry on FREEZING, and error out on any other (unexpected) value. While at it, further improve the code: - limit the number of retries; - if retries are exceeded, thaw and return an error; - don't retry (or read the state back) on THAW. I played a lot with various reproducers for this bug, including - parallel runc execs and runc pause/resumes - parallel runc execs and runc --systemd-cgroup update (the latter performs freeze/unfreeze); - continuously running /bin/printf inside container in parallel with runc pause/resume; - running pthread bomb (from criu test suite) in parallel with runc pause/resume; and I was not able to make freeze work 100%, meaning sometimes runc pause fails, or runc --systemd-cgroup update produces a warning. With that said, it's still a big improvement over the previous state of affairs where container is stuck in FREEZING state, and GetState() (and all its users) are also stuck. Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 76ae1f5) Signed-off-by: Kir Kolyshkin <[email protected]>
It appears that briefly thawing the cgroup while freezing greatly increases its chances to freeze successfully. The test case I used is doing runc exec in a look parallel with runc pause/resume in another loop, and the failure to freeze rate reduced from 40 to 0 per minute (tested inside a VM using a busybox container running sleep 1h, doing about 1500 pause/resumes and 650 execs per minute), with max retries being 150 (of 1000). This is still a game of chances, so failures are possible. Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit d1007b0) Signed-off-by: Kir Kolyshkin <[email protected]>
mrunalp
approved these changes
Mar 1, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This backports upstream PRs opencontainers/runc#2774 and opencontainers/runc#2791 to rhaos-4.6 branch, which should fix https://bugzilla.redhat.com/show_bug.cgi?id=1903228