Running PipelineRun fails to update when embedded-status is modified

[JeromeJu]

Expected Behavior

The process of the v1 migration should cause no existent PipelineRun to fail. And PipelineRunStatus can be used in both Full or Minimal EmbeddedStatus and also in the process of switching the default value at #5833 .

Actual Behavior

It will fail validateChildObjectsInPipelineRunStatus as there are status.runs and status.taskruns left along with ChildReferences while the EmbeddedStatus has been changed from Full to Minimal. And thus the PipelineRun will not succeed and runs indefinitely while the taskRuns Succeeded.

→ tkn pr list
NAME                  STARTED         DURATION   STATUS
test                  5 minutes ago   ---        Running

→ tkn tr list
NAME                                      STARTED         DURATION   STATUS
test-fetch-secure-data                    9 minutes ago   55s        Succeeded

Steps to Reproduce the Problem

1. Have a PipelineRun that would populate status.runs or status.taskruns started with Full EmbeddedStatus
2. Switch the EmbeddedStatus from Full to Minimal while it's running
3. Similar failure could happen if we switch from Minimal to Full

The example pipelineRun is pasted below, but it could be any PipelineRun that stays longer than the time the EmbeddedStatus is switched.

Additional Info

• Tekton Pipeline version:
  >= v0.43.2

PipelineRun yaml

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: test
spec:
  serviceAccountName: default
  timeout: 600s
  pipelineSpec:
    tasks:
    - name: fetch-secure-data
      taskSpec:
        steps:
        - name: fetch-and-write-secure
          image: ubuntu
          script: sleep 50s

PipelineRun described:

→ k describe pipelineruns.v1beta1.tekton.dev/test
  Type     Reason         Age                   From         Message
  ----     ------         ----                  ----         -------
  Normal   Started        7m15s                 PipelineRun  
  Normal   Running        7m15s                 PipelineRun  Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0
  Warning  InternalError  52s (x17 over 6m20s)  PipelineRun  1 error occurred:
           * expected no TaskRun statuses with embedded-status=minimal, but found 1

Expanded

→ k describe pipelineruns.v1beta1.tekton.dev/test
Name:         test
Namespace:    default
API Version:  tekton.dev/v1beta1
Kind:         PipelineRun
Metadata:
Spec:
  Pipeline Spec:
    Tasks:
      Name:  fetch-secure-data
      Task Spec:
        Metadata:
        Spec:  <nil>
        Steps:
          Image:  ubuntu
          Name:   fetch-and-write-secure
          Resources:
          Script:        sleep 50s
  Service Account Name:  default
  Timeout:               1m40s
Status:
  Child References:
    API Version:         tekton.dev/v1beta1
    Kind:                TaskRun
    Name:                test-fetch-secure-data
    Pipeline Task Name:  fetch-secure-data
  Conditions:
    Last Transition Time:  2023-01-13T16:10:48Z
    Message:               Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0
    Reason:                Running
    Status:                Unknown
    Type:                  Succeeded
  Pipeline Spec:
    Tasks:
      Name:  fetch-secure-data
      Task Spec:
        Metadata:
        Spec:  <nil>
        Steps:
          Image:  ubuntu
          Name:   fetch-and-write-secure
          Resources:
          Script:  sleep 50s
  Task Runs:
    Test - Fetch - Secure - Data:
      Pipeline Task Name:  fetch-secure-data
      Status:
        Conditions:
          Last Transition Time:  2023-01-13T16:10:52Z
          Message:               Not all Steps in the Task have finished executing
          Reason:                Running
          Status:                Unknown
          Type:                  Succeeded
        Pod Name:                test-fetch-secure-data-pod
        Start Time:              2023-01-13T16:10:48Z
        Steps:
          Container:  step-fetch-and-write-secure
          Image ID:   docker.io/library/ubuntu@sha256:27cb6e6ccef575a4698b66f5de06c7ecd61589132d5a91d098f7f3f9285415a9
          Name:       fetch-and-write-secure
          Running:
            Started At:  2023-01-13T16:10:51Z
        Task Spec:
          Steps:
            Image:  ubuntu
            Name:   fetch-and-write-secure
            Resources:
            Script:  sleep 50s
Events:
  Type     Reason         Age                   From         Message
  ----     ------         ----                  ----         -------
  Normal   Started        7m15s                 PipelineRun  
  Normal   Running        7m15s                 PipelineRun  Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0
  Warning  InternalError  52s (x17 over 6m20s)  PipelineRun  1 error occurred:
           * expected no TaskRun statuses with embedded-status=minimal, but found 1

[JeromeJu]

Related issues:

• Switch the default for "embedded-status" feature flag to "minimal" on or after Jan 25, 2023 #5833
• TEP-0100: PipelineRun Status Migration #4954

Related PRs:

• Fix PipelineRunStatus Reconciler for EmbeddedStatus Switch #5989
• [TEP0100] Default to minimal embedded-status #5934