Harden the reconciliation loop to work best with the APIserver #29
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
metral
changed the title
The use of GET and UPDATE API calls to the apiserver operate on optimistic locking, which can lead to invalid operations if an object becomes stale. This a feature in k8s, not a bug. We harden status updates and finalizers using a best-effort approach of getting the latest resource first before attempting to update it. However, loops can still fail on outdated objects albeit less frequently, and practically all events on a Stack can invoke yet another reconciliation loop. We defensively avoid loops where possible, and lean on the resourceGeneration predicate to help lower eventing from invoking more loops than necessary. See for more details: https://git.io/JJlcx
metral
force-pushed
the
metral/rework-reconcile-loop
branch
2 times, most recently
from
e4afee8 to
271c098
Compare
lblackstone
approved these changes
Jul 24, 2020
metral
force-pushed
the
metral/rework-reconcile-loop
branch
from
271c098 to
532533f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the build out of the integration testing suite (#28), we uncovered issues in how the finalizer and status updates operated, mainly related to loop failures and as a result, extra loops spawning and causing indeterministic outcomes.
This PR hardens the reconciliation loop based on the findings from running the tests, borrowing approaches from the Helm operator in the operator-sdk examples.
(stack-cntlr): do not err on CreateStack if stack already exists
fix(stack-cntlr): harden status and finalizers based on Helm operator
The use of GET and UPDATE API calls to the apiserver operate on optimistic
locking, which can lead to invalid operations if an object becomes stale. This a feature in k8s, not a bug.
We harden status updates and finalizers using a best-effort approach of getting
the latest resource first before attempting to update it.
However, loops can still fail on outdated objects albeit less frequently, and practically
all events on a Stack can invoke yet another reconciliation loop. We defensively avoid loops
where possible, and lean on the resourceGeneration predicate to help lower eventing from invoking more loops than necessary.
See for more details: https://git.io/JJlcx
fix(stack-cntlr): retry HTTP 409s optionally, and requeue HTTP 404s