Update TaskRun Step States After Cancellation or Timeout

[danielhelfand]

Closes tektoncd/cli#1095

Changes

In the event a TaskRun is timed out or cancelled, pipelines currently leaves the state of steps as Running or PodInitializing depending on how far the TaskRun has proceeded. This pull request adds a feature to update the status of steps to TaskRunTimeout or TaskRunCancelled. This will help with properly reporting step statuses through tkn to end users to avoid confusion.

The current approach only updates a step status if the step state is Running or Waiting. If a step has already completed, the status will be preserved.

I am open to suggestions on a different status that can be returned for steps other than the TaskRun reason.

Submitter Checklist

• Includes tests (if functionality changed/added)
• Includes docs (if user facing)
• Commit messages follow commit message best practices
• Release notes block has been filled in or deleted (only if no user facing changes)

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

Update step statuses on TaskRun in event of cancellation or timeout

[danielhelfand]

/kind feature

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[vdemeester]

/lgtm
/cc @bobcatfish @pritidesai @afrittoli

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[pritidesai]

@danielhelfand the changes looks good and fixes the step status for a TaskRun but we have similar issue for a PipelineRun. We can merge this and open an issue or another PR for PipelineRun. WDYT?

When a task times out as part of a Pipeline, these changes does update the pipeline run status with an appropriate step status but tkn CLI still shows the taskrun and step both Running. Is this something we have to fix in CLI Repo?

            "status": {
                "taskRuns": {
                    "pipelinerun-with-timeout-echo-good-morning-758nt": {
                        "status": {
                            "conditions": [
                                {
                                    "lastTransitionTime": "2020-08-21T23:35:20Z",
                                    "message": "TaskRun \"pipelinerun-with-timeout-echo-good-morning-758nt\" failed to finish within \"30s\"",
                                    "reason": "TaskRunTimeout",
                                    "status": "False",
                                    "type": "Succeeded"
                                }
                            ],
                            "steps": [
                                {
                                    "terminated": {
                                        "exitCode": 1,
                                        "finishedAt": "2020-08-21T23:35:20Z",
                                        "reason": "TaskRunTimeout",
                                        "startedAt": "2020-08-21T23:34:58Z"
                                    }
                                }
                            ],

I tried on my local cluster for this sample pipeline run:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pipelinerun-with-timeout
spec:
  timeout: "0h0m3s"
  pipelineSpec:
    tasks:
      - name: echo-good-morning
        timeout: "0h0m30s"
        taskSpec:
          steps:
            - name: echo
              image: ubuntu
              script: |
                #!/usr/bin/env bash
                echo "Good Morning!"
                sleep 60
---

And here is the tkn describe pr and tr:

 [66] ->  tkn pr list
NAME                       STARTED         DURATION    STATUS
pipelinerun-with-timeout   5 seconds ago   3 seconds   Failed(PipelineRunTimeout)

 [67] -> tkn pr describe pipelinerun-with-timeout
Name:        pipelinerun-with-timeout
Namespace:   default
Timeout:     3s
Labels:
 tekton.dev/pipeline=pipelinerun-with-timeout

🌡️  Status

STARTED          DURATION    STATUS
14 seconds ago   3 seconds   Failed(PipelineRunTimeout)

💌 Message

PipelineRun "pipelinerun-with-timeout" failed to finish within "3s"

📦 Resources

 No resources

⚓ Params

 No params

🗂  Taskruns

 NAME                                                 TASK NAME           STARTED          DURATION   STATUS
 ∙ pipelinerun-with-timeout-echo-good-morning-758nt   echo-good-morning   14 seconds ago   ---        Running

 [68] -> tkn tr describe pipelinerun-with-timeout-echo-good-morning-758nt
Name:        pipelinerun-with-timeout-echo-good-morning-758nt
Namespace:   default
Timeout:     30s
Labels:
 app.kubernetes.io/managed-by=tekton-pipelines
 tekton.dev/pipeline=pipelinerun-with-timeout
 tekton.dev/pipelineRun=pipelinerun-with-timeout
 tekton.dev/pipelineTask=echo-good-morning

🌡️  Status

STARTED          DURATION    STATUS
26 seconds ago   ---         Running

📨 Input Resources

 No input resources

📡 Output Resources

 No output resources

⚓ Params

 No params

🦶 Steps

 NAME     STATUS
 ∙ echo   Running

🚗 Sidecars

No sidecars

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[danielhelfand]

@pritidesai Looking into this a bit further, I think the status needs some time before it updates. I noticed the same issue you pointed out, but it eventually does update the statuses correctly:

$ tkn pr desc --last

Name:        pipelinerun-with-timeout
Namespace:   default
Timeout:     3s
Labels:
 tekton.dev/pipeline=pipelinerun-with-timeout

🌡️  Status

STARTED         DURATION    STATUS
9 minutes ago   3 seconds   Failed(PipelineRunTimeout)

💌 Message

PipelineRun "pipelinerun-with-timeout" failed to finish within "3s" (TaskRun "pipelinerun-with-timeout-echo-good-morning-hkvjs" failed to finish within "30s")

📦 Resources

 No resources

⚓ Params

 No params

🗂  Taskruns

 NAME                                                 TASK NAME           STARTED         DURATION     STATUS
 ∙ pipelinerun-with-timeout-echo-good-morning-hkvjs   echo-good-morning   9 minutes ago   30 seconds   Failed(TaskRunTimeout)

$ tkn tr desc --last

Name:        pipelinerun-with-timeout-echo-good-morning-hkvjs
Namespace:   default
Timeout:     30s
Labels:
 app.kubernetes.io/managed-by=tekton-pipelines
 tekton.dev/pipeline=pipelinerun-with-timeout
 tekton.dev/pipelineRun=pipelinerun-with-timeout
 tekton.dev/pipelineTask=echo-good-morning

🌡️  Status

STARTED          DURATION     STATUS
10 minutes ago   30 seconds   Failed(TaskRunTimeout)

Message

TaskRun "pipelinerun-with-timeout-echo-good-morning-hkvjs" failed to finish within "30s"

📨 Input Resources

 No input resources

📡 Output Resources

 No output resources

⚓ Params

 No params

🦶 Steps

 NAME     STATUS
 ∙ echo   TaskRunTimeout

🚗 Sidecars

No sidecars

[danielhelfand]

Looking further into the issue, this looks like a case where the TaskRun is actually still Running and is not timed out despite the PipelineRun timeout. This is occurring in v0.15.2 with this same PipelineRun, so it might be related to issues around timeouts with pipelines. But I do think this is not related to the changes here.

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[pritidesai]

Sounds good thanks @danielhelfand.

@bobcatfish was looking into timeout and we have an issue for it #3038.

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pritidesai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [pritidesai]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/taskrun/taskrun.go	78.3%	78.9%	0.6

[danielhelfand]

/cc @vdemeester @bobcatfish @afrittoli

[vdemeester]

/lgtm