PipelineRun Cancellation is not working #2369
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
requested review from
tekton-robot
added
the
do-not-merge/work-in-progress
googlebot
added
the
cla: yes
tekton-robot
added
the
size/S
tekton-robot
added
size/XXL
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
googlebot
added
cla: no
and removed
cla: yes
tekton-robot
added
size/M
ghost
changed the title
tekton-robot
removed
the
do-not-merge/work-in-progress
|
CLAs look good, thanks! ℹ️ Googlers: Go here for more info. |
googlebot
added
cla: yes
Cancelling a PipelineRun is supposed to also cancel the TaskRuns spawned by that PipelineRun. The way that we do this is to issue `Update` and `UpdateStatus` calls on each TaskRun. Unfortunately this can (and does) fail often because modifications to the TaskRun race each other. This has resulted in many many failed integration tests, giving the PipelineRun cancellation e2e tests the appearance of being flakey. In fact they were actually catching real problems! This commit updates the PipelineRun reconciler's behaviour to PATCH TaskRuns associated with a PipelineRun. This updates the TaskRun's `spec.status` regardless of its current resourceVersion / generation. A similar related issue was happening in the PipelineRun cancel test itself. When submitting the cancellation status to the test's PipelineRun the API server was sometimes rejecting that update and failing the test. This has also been replaced with a PATCH. The PipelineRun cancellation e2e tests now appear to be passing consistently. coauthored with @bobcatfish
thanks. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: vdemeester The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
tekton-robot
added
the
approved
ghost
mentioned this pull request
bobcatfish
added a commit
to bobcatfish/pipeline
that referenced
this pull request
Apr 17, 2020
While working on tektoncd#2369 (flakey tests around cancellation that actually revealed underlying bugs) we were running into a case where trying to cancel a PipelineRun's TaskRuns was failing (due to a race condition). In that case, the PipelineRun would be marked as cancelled and done, even though the PipelineRun was actually still executing in that one or more TaskRuns were actually still running. Now when that happens we will indicate that the PipelineRun is still running and return an error, which will mean that on the next reconcile, the Reconciler will try to reconcile again, and the PipelineRun conditions will reflect what is actually happening. Co-authored-by: Sharon Jerop Kipruto <[email protected]> Fixes tektoncd#2381
2 tasks
bobcatfish
added a commit
to bobcatfish/pipeline
that referenced
this pull request
Apr 27, 2020
While working on tektoncd#2369 (flakey tests around cancellation that actually revealed underlying bugs) we were running into a case where trying to cancel a PipelineRun's TaskRuns was failing (due to a race condition). In that case, the PipelineRun would be marked as cancelled and done, even though the PipelineRun was actually still executing in that one or more TaskRuns were actually still running. Now when that happens we will indicate that the PipelineRun is still running and return an error, which will mean that on the next reconcile, the Reconciler will try to reconcile again, and the PipelineRun conditions will reflect what is actually happening. Co-authored-by: Sharon Jerop Kipruto <[email protected]> Fixes tektoncd#2381
bobcatfish
added a commit
to bobcatfish/pipeline
that referenced
this pull request
May 11, 2020
While working on tektoncd#2369 (flakey tests around cancellation that actually revealed underlying bugs) we were running into a case where trying to cancel a PipelineRun's TaskRuns was failing (due to a race condition). In that case, the PipelineRun would be marked as cancelled and done, even though the PipelineRun was actually still executing in that one or more TaskRuns were actually still running. Now when that happens we will indicate that the PipelineRun is still running and return an error, which will mean that on the next reconcile, the Reconciler will try to reconcile again, and the PipelineRun conditions will reflect what is actually happening. Co-authored-by: Sharon Jerop Kipruto <[email protected]> Fixes tektoncd#2381
tekton-robot
pushed a commit
that referenced
this pull request
May 11, 2020
While working on #2369 (flakey tests around cancellation that actually revealed underlying bugs) we were running into a case where trying to cancel a PipelineRun's TaskRuns was failing (due to a race condition). In that case, the PipelineRun would be marked as cancelled and done, even though the PipelineRun was actually still executing in that one or more TaskRuns were actually still running. Now when that happens we will indicate that the PipelineRun is still running and return an error, which will mean that on the next reconcile, the Reconciler will try to reconcile again, and the PipelineRun conditions will reflect what is actually happening. Co-authored-by: Sharon Jerop Kipruto <[email protected]> Fixes #2381
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Cancelling a PipelineRun is supposed to also cancel the TaskRuns spawned by that PipelineRun. The way that we do this is to issue
UpdateandUpdateStatuscalls on each TaskRun. Unfortunately this can (and does) fail often because modifications to the TaskRun race each other. This has resulted in many many failed integration tests, giving the PipelineRun cancellation e2e tests the appearance of being flakey. In fact they were actually catching real problems!This commit updates the PipelineRun reconciler's behaviour to PATCH TaskRuns associated with a PipelineRun. This updates the TaskRun's
spec.statusregardless of its current resourceVersion / generation.A similar related issue was happening in the PipelineRun cancel test itself. When submitting the cancellation status to the test's PipelineRun the API server was sometimes rejecting that update and failing the test. This has also been replaced with a PATCH.
The PipelineRun cancellation e2e tests now appear to be passing consistently.
Co-authored by @bobcatfish
Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them: