Design: "Finally" Steps in Tasks

[chhsia0]

Related issue: #1559

Motivating Use Cases

• A unit test step fails but a subsequent step in the same Task should upload the results to a bucket regardless of that failure.
• A Pull Request pipeline resource should update a PR with a comment reflecting the status of a failed build. The build happens in one step and then the PR resource could inject a step at the end which runs regardless of the error and knows how to update github/gitlab/etc with the status comment.

Proposed User-Facing Changes

Add a new Finally field to specify a list of sequential steps that run regardless of whether previous steps have succeeded or not. Example:

kind: Task
metadata:
  name: hello-world-finally-steps
spec:
  steps:
  - name: hello
    image: busybox
    command: ["echo", "hello"]
  - name: fail
    image: busybox
    command: ["false"]
  finally:
  - name: world
    image: busybox
    command: ["echo", "world"]

Design Doc

https://docs.google.com/document/d/1e1fagYDErliLnIwU1g_N3YKVyAProhs1l14UyPIoZuM/edit?usp=sharing

Alternative Design

Error Strategy in Task Steps (#1573)

[dibyom]

/kind feature
/area api

Also related to #1684

[NikeNano]

The design doc. Is behind restricted access :( Should I apply for access through the google docs link or should the document be made public @dibyom?

[bobcatfish]

Hi @NikeNano ! Joining our mailing list should give you access: https://github.com/tektoncd/community/blob/master/contact.md#mailing-list

[NikeNano]

Thanks @bobcatfish :)

[afrittoli]

@chhsia0 Are you still looking into this?

This may be rendered unnecessary by the pipeline resource redesign. If you feel this is high prio still feel free to bring it back at the API WG

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vdemeester]

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

[tekton-robot]

@vdemeester: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[vdemeester]

/remove-lifecycle stale

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[jerop]

The use cases described here could be better solved using Pipelines in Pipelines because it:

• allows us to reuse Finally Tasks instead of introducing Finally Steps
• encourages writing Tasks with singular concerns then allowing them to be composed together as a unit of execution -- and this further supports reusing of tasks, such as slack and github catalog tasks to do the notifications when the notification is at task level instead of step level

A unit test step fails but a subsequent step in the same Task should upload the results to a bucket regardless of that failure.

This use case can be solved using a unit-test-and-upload Pipeline which has a unit-test Task and a upload Finally Task, and the unit-test-and-upload Pipeline is passed in as a Task in the main Pipeline

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: unit-test-and-upload
spec:
  tasks:
    - name: unit-test
      taskRef:
        name: unit-test
  finally:  
    - name: upload
      taskRef:
        name: upload

---

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline
spec:
  tasks:
    - name: unit-test-and-upload
      taskRef:
        apiVersion: tekton.dev/v1beta1
        kind: Pipeline
        name: unit-test-and-upload
... 

A Pull Request pipeline resource should update a PR with a comment reflecting the status of a failed build. The build happens in one step and then the PR resource could inject a step at the end which runs regardless of the error and knows how to update github/gitlab/etc with the status comment.

This use case can be solved using a build-and-notify Pipeline which has a build Task and a guarded notify Finally Task, and the build-and-notify Pipeline is passed in as a Task in the main Pipeline

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: build-and-notify
spec:
  tasks:
    - name: build
      taskRef:
        name: build
  finally:  
    - name: notify
      when:
        - input: $(tasks.build.status)
          operator: in
          values: [ "Failed" ] 
      taskRef:
        name: notify

---

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline
spec:
  tasks:
    - name: build-and-notify
      taskRef:
        apiVersion: tekton.dev/v1beta1
        kind: Pipeline
        name: build-and-notify
... 

So I propose going forward with Pipelines in Pipelines to solve these use cases, and this still leaves Finally Steps as an option we can explore in the future if needed, please let me know what you think @chhsia0

/cc @imjasonh @pritidesai @bobcatfish @afrittoli

[bobcatfish]

Thanks for summarizing this and adding examples @jerop !

One detail that I would add which makes using Pipelines as the solution difficult is that sharing data between Tasks in a Pipeline requires a workspace (and probably a backing PVC) in order to make the data available to all the Tasks (pods) (e.g. the unit test upload example above would probably need a workspace too).

If it was possible to colocate Tasks and Workspaces (TEP-0046) or if we had a way of composing Tasks with Tasks (executing in the same pod) (TEP-0044) that would be a different story!

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.