Skipping Strategies

This change implements skipping strategies to give users the flexibility to skip a single guarded Task only and unblock execution of its dependent Tasks. Today, WhenExpressions are specified within Tasks but they guard the Task and its dependent Tasks. To provide flexible skipping strategies, we want to change the scope of WhenExpressions from guarding a Task and its dependent Tasks to guarding the Task only. If a user wants to guard a Task and its dependent Tasks, they can: - cascade the WhenExpressions to the dependent Tasks - compose the Task and its dependent Tasks as a sub-Pipeline that's guarded and executed together using Pipelines in Pipelines (but this is still an experimental feature) Changing the scope of WhenExpressions to guard the Task only is backwards-incompatible, so to make the transition smooth: - we'll provide a feature flag, scope-when-expressions-to-task, which: - will default to false to guard a Task and its dependent Tasks - can be set to true to guard a Task only - after migration, we'll change the global default for the feature flag to true to guard a Task only by default - eventually, we'll remove the feature flag and guard a Task only going forward Implements [TEP-0059: Skipping Strategies](https://github.com/tektoncd/community/blob/main/teps/0059-skipping-strategies.md) Closes tektoncd#2127
jerop · Aug 11, 2021 · e8299ca · e8299ca
1 parent 9d74f3f
commit e8299ca
Show file tree

Hide file tree

Showing 15 changed files with 1,522 additions and 162 deletions.
diff --git a/config/config-feature-flags.yaml b/config/config-feature-flags.yaml
@@ -82,3 +82,6 @@ data:
   # Setting this flag will determine which gated features are enabled.
   # Acceptable values are "stable" or "alpha".
   enable-api-fields: "stable"
+  # Setting this flag to "true" scopes when expressions to guard a Task only
+  # instead of a Task and its dependent Tasks.
+  scope-when-expressions-to-task: "false"
diff --git a/docs/deprecations.md b/docs/deprecations.md
@@ -25,3 +25,5 @@ being deprecated.
 | [`Conditions` CRD is deprecated and will be removed. Use `when` expressions instead.](https://github.com/tektoncd/community/blob/main/teps/0007-conditions-beta.md)       | [v0.16.0](https://github.com/tektoncd/pipeline/releases/tag/v0.16.0)         | Alpha                                                                                                     | Nov 02 2020                         |
 | [The `disable-home-env-overwrite` flag will be removed](https://github.com/tektoncd/pipeline/issues/2013)                                     | [v0.24.0](https://github.com/tektoncd/pipeline/releases/tag/v0.24.0)         | Beta                                                                                                     | February 10 2022                    |
 | [The `disable-working-dir-overwrite` flag will be removed](https://github.com/tektoncd/pipeline/issues/1836)                                  | [v0.24.0](https://github.com/tektoncd/pipeline/releases/tag/v0.24.0)         | Beta                                                                                                     | February 10 2022                    |
+| [The `scope-when-expressions-to-task` flag will be flipped from "false" to "true"](https://github.com/tektoncd/pipeline/issues/1836)          | [v0.27.0](https://github.com/tektoncd/pipeline/releases/tag/v0.27.0)         | Beta                                                                                                     | February 10 2022                    |
+| [The `scope-when-expressions-to-task` flag will be removed](https://github.com/tektoncd/pipeline/issues/1836)                                 | [v0.27.0](https://github.com/tektoncd/pipeline/releases/tag/v0.27.0)         | Beta                                                                                                     | March 10 2022                       |
diff --git a/docs/install.md b/docs/install.md
@@ -366,6 +366,10 @@ use of custom tasks in pipelines.
 most stable features to be used. Set it to "alpha" to allow alpha
 features to be used.
 
+- `scope-when-expressions-to-task`: set this flag to "true" to scope `when` expressions to guard a `Task` only. Set it
+  to "false" to guard a `Task` and its dependent `Tasks`. It defaults to "false". For more information, see [guarding
+  `Task` execution using `when` expressions](pipelines.md#guard-task-execution-using-whenexpressions).
+
 For example:
 
 ```yaml

diff --git a/docs/pipelines.md b/docs/pipelines.md
@@ -474,6 +474,226 @@ There are a lot of scenarios where `when` expressions can be really useful. Some
 - Checking if the name of a CI job matches
 - Checking if an optional Workspace has been provided
 
+#### Guarding a `Task` and its dependent `Tasks`
+
+When  `when` expressions evaluate to `False`, the `Task` and its dependent `Tasks` will be skipped by default while the
+rest of the `Pipeline` will execute. Dependencies between `Tasks` can be either ordering ([`runAfter`](https://github.com/tektoncd/pipeline/blob/main/docs/pipelines.md#using-the-runafter-parameter))
+or resource (e.g. [`Results`](https://github.com/tektoncd/pipeline/blob/main/docs/pipelines.md#using-results))
+dependencies, as further described in [configuring execution order](#configuring-the-task-execution-order). The global
+default scope of `when` expressions is set to a `Task` and its dependent`Tasks`; `scope-when-expressions-to-task` field
+in [`config/config-feature-flags.yaml`](install.md#customizing-the-pipelines-controller-behavior) defaults to "false".
+
+**Note:** Scoping `when` expressions to a `Task` and its dependent `Tasks` is deprecated
+
+To guard a `Task` and its dependent Tasks:
+- cascade the `when` expressions to the specific dependent `Tasks` to be guarded as well
+- compose the `Task` and its dependent `Tasks` as a unit to be guarded and executed together using `Pipelines` in `Pipelines`
+
+##### Cascade `when` expressions to the specific dependent `Tasks`
+
+Pick and choose which specific dependent `Tasks` to guard as well, and cascade the `when` expressions to those `Tasks`. 
+
+Taking the use case below, a user who wants to guard `manual-approval` and its dependent `Tasks`:
+
+```
+                                     tests
+                                       |
+                                       v
+                                 manual-approval
+                                 |            |
+                                 v        (approver)
+                            build-image       |
+                                |             v
+                                v          slack-msg
+                            deploy-image
+```
+
+The user can design the `Pipeline` to solve their use case as such:
+
+```yaml
+tasks:
+...
+- name: manual-approval
+  runAfter:
+    - tests
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  taskRef:
+    name: manual-approval
+
+- name: build-image
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  runAfter:
+    - manual-approval
+  taskRef:
+    name: build-image
+
+- name: deploy-image
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  runAfter:
+    - build-image
+  taskRef:
+    name: deploy-image
+
+- name: slack-msg
+  params:
+    - name: approver
+      value: $(tasks.manual-approval.results.approver)
+  taskRef:
+    name: slack-msg
+```  
+
+##### Compose using Pipelines in Pipelines
+
+Compose a set of `Tasks` as a unit of execution using `Pipelines` in `Pipelines`, which allows for guarding a `Task` and 
+its dependent `Tasks` (as a sub-`Pipeline`) using `when` expressions. 
+
+**Note:** `Pipelines` in `Pipelines` is an [experimental feature](https://github.com/tektoncd/experimental/tree/main/pipelines-in-pipelines)
+
+Taking the use case below, a user who wants to guard `manual-approval` and its dependent `Tasks`:
+
+```
+                                     tests
+                                       |
+                                       v
+                                 manual-approval
+                                 |            |
+                                 v        (approver)
+                            build-image       |
+                                |             v
+                                v          slack-msg
+                            deploy-image
+```
+
+The user can design the `Pipelines` to solve their use case as such:
+
+```yaml
+## sub pipeline (approve-build-deploy-slack)
+tasks:
+  - name: manual-approval
+    runAfter:
+      - integration-tests
+    taskRef:
+      name: manual-approval
+
+  - name: build-image
+    runAfter:
+      - manual-approval
+    taskRef:
+      name: build-image
+
+  - name: deploy-image
+    runAfter:
+      - build-image
+    taskRef:
+      name: deploy-image
+
+  - name: slack-msg
+    params:
+      - name: approver
+        value: $(tasks.manual-approval.results.approver)
+    taskRef:
+      name: slack-msg
+      
+---
+## main pipeline
+tasks:
+...
+- name: approve-build-deploy-slack
+  runAfter:
+    - tests
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  taskRef:
+    apiVersion: tekton.dev/v1beta1
+    kind: Pipeline
+    name: approve-build-deploy-slack
+```
+
+#### Guarding a `Task` only
+
+To guard a `Task` only and unblock execution of its dependent `Tasks`, set the global default scope of `when` expressions
+to `Task` using the `scope-when-expressions-to-task` field in [`config/config-feature-flags.yaml`](install.md#customizing-the-pipelines-controller-behavior)
+by changing it to "true"
+- The ordering-dependent `Tasks` will be executed
+- The resource-dependent `Tasks` (and their dependencies) will be skipped because of missing `Results` from the skipped 
+  parent `Task`. When we add support for [default `Results`](https://github.com/tektoncd/community/pull/240), then the 
+  resource-dependent `Tasks` may be executed if the default `Results` from the skipped parent `Task` are specified. In 
+  addition, if a resource-dependent `Task` needs a file from a guarded parent `Task` in a shared `Workspace`, make sure
+  to handle the execution of the child `Task` in case the expected file is missing from the `Workspace` because the 
+  guarded parent `Task` is skipped. 
+
+```
+                                     tests
+                                       |
+                                       v
+                                 manual-approval
+                                 |            |
+                                 v        (approver)
+                            build-image       |
+                                |             v
+                                v          slack-msg
+                            deploy-image
+```
+
+Taking the use case above, a user who wants to guard `manual-approval` only can design the `Pipeline` as such:
+
+```yaml
+tasks:
+...
+- name: manual-approval
+  runAfter:
+    - tests
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  taskRef:
+    name: manual-approval
+
+- name: build-image
+  runAfter:
+    - manual-approval
+  taskRef:
+    name: build-image
+
+- name: deploy-image
+  runAfter:
+    - build-image
+  taskRef:
+    name: deploy-image
+
+- name: slack-msg
+  params:
+    - name: approver
+      value: $(tasks.manual-approval.results.approver)
+  taskRef:
+    name: slack-msg
+```
+
+With `when` expressions scoped to `Task`, if `manual-approval` is skipped, execution of it's dependent `Tasks` 
+(`slack-msg`, `build-image` and `deploy-image`) would be unblocked regardless:
+- `build-image` and `deploy-image` should be executed successfully
+- `slack-msg` will be skipped because it is missing the `approver` `Result` from `manual-approval`
+  - dependents of `slack-msg` would have been skipped too if it had any of them
+  - if `manual-approval` specifies a default `approver` `Result`, such as "None", then `slack-msg` would be executed 
+    ([supporting default `Results` is in progress](https://github.com/tektoncd/community/pull/240))
+
 ### Guard `Task` execution using `Conditions`
 
 **Note:** `Conditions` are [deprecated](./deprecations.md), use [`when` expressions](#guard-task-execution-using-when-expressions) instead.
@@ -700,10 +920,13 @@ so that one will run before another and the execution of the `Pipeline` progress
 without getting stuck in an infinite loop.
 
 This is done using:
-
-- [`from`](#using-the-from-parameter) clauses on the [`PipelineResources`](resources.md) used by each `Task`
-- [`runAfter`](#using-the-runafter-parameter) clauses on the corresponding `Tasks`
-- By linking the [`results`](#configuring-execution-results-at-the-pipeline-level) of one `Task` to the params of another
+- _resource dependencies_:
+  - [`from`](#using-the-from-parameter) clauses on the [`PipelineResources`](resources.md) used by each `Task`
+  - [`results`](#configuring-execution-results-at-the-pipeline-level) of one `Task` being pa `params` or
+    `when` expressions of another
+
+- _ordering dependencies_:
+  - [`runAfter`](#using-the-runafter-parameter) clauses on the corresponding `Tasks`
 
 For example, the `Pipeline` defined as follows
 

diff --git a/pkg/apis/config/feature_flags.go b/pkg/apis/config/feature_flags.go
@@ -37,6 +37,7 @@ const (
 	enableTektonOCIBundles                  = "enable-tekton-oci-bundles"
 	enableCustomTasks                       = "enable-custom-tasks"
 	enableAPIFields                         = "enable-api-fields"
+	scopeWhenExpressionsToTask              = "scope-when-expressions-to-task"
 	DefaultDisableHomeEnvOverwrite          = true
 	DefaultDisableWorkingDirOverwrite       = true
 	DefaultDisableAffinityAssistant         = false
@@ -45,6 +46,7 @@ const (
 	DefaultRequireGitSSHSecretKnownHosts    = false
 	DefaultEnableTektonOciBundles           = false
 	DefaultEnableCustomTasks                = false
+	DefaultScopeWhenExpressionsToTask       = false
 	DefaultEnableAPIFields                  = StableAPIFields
 )
 
@@ -59,6 +61,7 @@ type FeatureFlags struct {
 	RequireGitSSHSecretKnownHosts    bool
 	EnableTektonOCIBundles           bool
 	EnableCustomTasks                bool
+	ScopeWhenExpressionsToTask       bool
 	EnableAPIFields                  string
 }
 
@@ -105,6 +108,9 @@ func NewFeatureFlagsFromMap(cfgMap map[string]string) (*FeatureFlags, error) {
 	if err := setFeature(requireGitSSHSecretKnownHostsKey, DefaultRequireGitSSHSecretKnownHosts, &tc.RequireGitSSHSecretKnownHosts); err != nil {
 		return nil, err
 	}
+	if err := setFeature(scopeWhenExpressionsToTask, DefaultScopeWhenExpressionsToTask, &tc.ScopeWhenExpressionsToTask); err != nil {
+		return nil, err
+	}
 	if err := setEnabledAPIFields(cfgMap, DefaultEnableAPIFields, &tc.EnableAPIFields); err != nil {
 		return nil, err
 	}

diff --git a/pkg/apis/config/feature_flags_test.go b/pkg/apis/config/feature_flags_test.go
@@ -38,6 +38,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				DisableHomeEnvOverwrite:          false,
 				DisableWorkingDirOverwrite:       false,
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
+				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 				EnableAPIFields:                  "stable",
 			},
 			fileName: config.GetFeatureFlagsConfigName(),
@@ -51,6 +52,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				RequireGitSSHSecretKnownHosts:    true,
 				EnableTektonOCIBundles:           true,
 				EnableCustomTasks:                true,
+				ScopeWhenExpressionsToTask:       true,
 				EnableAPIFields:                  "alpha",
 			},
 			fileName: "feature-flags-all-flags-set",
@@ -66,6 +68,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				DisableHomeEnvOverwrite:          true,
 				DisableWorkingDirOverwrite:       true,
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
+				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 			},
 			fileName: "feature-flags-enable-api-fields-overrides-bundles-and-custom-tasks",
 		},
@@ -78,6 +81,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				DisableHomeEnvOverwrite:          true,
 				DisableWorkingDirOverwrite:       true,
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
+				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 			},
 			fileName: "feature-flags-bundles-and-custom-tasks",
 		},
@@ -98,6 +102,7 @@ func TestNewFeatureFlagsFromEmptyConfigMap(t *testing.T) {
 		DisableHomeEnvOverwrite:          true,
 		DisableWorkingDirOverwrite:       true,
 		RunningInEnvWithInjectedSidecars: true,
+		ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 		EnableAPIFields:                  "stable",
 	}
 	verifyConfigFileWithExpectedFeatureFlagsConfig(t, FeatureFlagsConfigEmptyName, expectedConfig)
@@ -141,6 +146,8 @@ func TestNewFeatureFlagsConfigMapErrors(t *testing.T) {
 		fileName: "feature-flags-invalid-boolean",
 	}, {
 		fileName: "feature-flags-invalid-enable-api-fields",
+	}, {
+		fileName: "feature-flags-invalid-scope-when-expressions-to-task",
 	}} {
 		t.Run(tc.fileName, func(t *testing.T) {
 			cm := test.ConfigMapFromTestFile(t, tc.fileName)

diff --git a/pkg/apis/config/testdata/feature-flags-all-flags-set.yaml b/pkg/apis/config/testdata/feature-flags-all-flags-set.yaml
@@ -25,4 +25,5 @@ data:
   require-git-ssh-secret-known-hosts: "true"
   enable-tekton-oci-bundles: "true"
   enable-custom-tasks: "true"
+  scope-when-expressions-to-task: "true"
   enable-api-fields: "alpha"
diff --git a/pkg/apis/config/testdata/feature-flags-invalid-scope-when-expressions-to-task.yaml b/pkg/apis/config/testdata/feature-flags-invalid-scope-when-expressions-to-task.yaml
@@ -0,0 +1,21 @@
+# Copyright 2021 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: feature-flags
+  namespace: tekton-pipelines
+data:
+  scope-when-expressions-to-task: "im-not-a-boolean"