TEP-0090: Matrix - API Change and Feature Flag

[TEP-0090][tep-0090] proposed executing a `PipelineTask` in parallel `TaskRuns` and `Runs` with substitutions from combinations of `Parameters` in a `Matrix`. In this change, we add the `matrix` field to the `PipelineTask` specification, which takes a list of `Parameters`. We gate this field behind the `alpha` feature gate. We add validation and testing that the field is available in `alpha` only - we will add further validation in upcoming changes. We also add documentation the API change - we will incrementally add further details as we iterate on the implementation of [TEP-0090: Matrix][tep-0090]. [tep-0090]: https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md
tektoncd · Feb 28, 2022 · 4028cb2 · 4028cb2
1 parent 5adf274
commit 4028cb2
Show file tree

Hide file tree

Showing 14 changed files with 251 additions and 148 deletions.
diff --git a/docs/install.md b/docs/install.md
@@ -392,19 +392,20 @@ the `feature-flags` ConfigMap alongside your Tekton Pipelines deployment.
 
 Features currently in "alpha" are:
 
-| Feature                                                                                               | TEP                                                                                                         | Release                                                              | Individual Flag             |
-|:------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------- |:-------------------------------------------------------------------- |:--------------------------- |
-| [Bundles ](./pipelineruns.md#tekton-bundles)                                                          | [TEP-0005](https://github.com/tektoncd/community/blob/main/teps/0005-tekton-oci-bundles.md)                 | [v0.18.0](https://github.com/tektoncd/pipeline/releases/tag/v0.18.0) | `enable-tekton-oci-bundles` |
-| [`Runs` and `Custom Tasks`](./runs.md)                                                                | [TEP-0002](https://github.com/tektoncd/community/blob/main/teps/0002-custom-tasks.md)                       | [v0.19.0](https://github.com/tektoncd/pipeline/releases/tag/v0.19.0) | `enable-custom-tasks`       |
-| [Isolated `Step` & `Sidecar` `Workspaces`](./workspaces.md#isolated-workspaces)                       | [TEP-0029](https://github.com/tektoncd/community/blob/main/teps/0029-step-workspaces.md)                    | [v0.24.0](https://github.com/tektoncd/pipeline/releases/tag/v0.24.0) |                             |
-| [Hermetic Execution Mode](./hermetic.md)                                                              | [TEP-0025](https://github.com/tektoncd/community/blob/main/teps/0025-hermekton.md)                          | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) |                             |
-| [Graceful Termination](./pipelineruns.md#gracefully-cancelling-a-pipelinerun)                         | [TEP-0058](https://github.com/tektoncd/community/blob/main/teps/0058-graceful-pipeline-run-termination.md)  | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) |                             |
-| [`PipelineRun` Timeouts](./pipelineruns.md#configuring-a-failure-timeout)                             | [TEP-0046](https://github.com/tektoncd/community/blob/main/teps/0046-finallytask-execution-post-timeout.md) | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) |                             |
-| [Implicit `Parameters`](./taskruns.md#implicit-parameters)                                            | [TEP-0023](https://github.com/tektoncd/community/blob/main/teps/0023-implicit-mapping.md)                   | [v0.28.0](https://github.com/tektoncd/pipeline/releases/tag/v0.28.0) |                             |
-| [Windows Scripts](./tasks.md#windows-scripts)                                                         | [TEP-0057](https://github.com/tektoncd/community/blob/main/teps/0057-windows-support.md)                    | [v0.28.0](https://github.com/tektoncd/pipeline/releases/tag/v0.28.0) |                             |
-| [Remote Tasks](./taskruns.md#remote-tasks) and [Remote Pipelines](./pipelineruns.md#remote-pipelines) | [TEP-0060](https://github.com/tektoncd/community/blob/main/teps/0060-remote-resolutiond.md)                 |                                                                      |                             |
-| [Debug](./debug.md) | [TEP-0042](https://github.com/tektoncd/community/blob/main/teps/0042-taskrun-breakpoint-on-failure.md) | [v0.26.0](https://github.com/tektoncd/pipeline/releases/tag/v0.26.0) | |
-| [Step and Sidecar Overrides](./taskruns.md#overriding-task-steps-and-sidecars)| [TEP-0094](https://github.com/tektoncd/community/blob/main/teps/0094-specifying-resource-requirements-at-runtime.md)                 |                                                                      |                             |
+| Feature                                                                                               | TEP                                                                                                                  | Release                                                              | Individual Flag             |
+|:------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------|:----------------------------|
+| [Bundles ](./pipelineruns.md#tekton-bundles)                                                          | [TEP-0005](https://github.com/tektoncd/community/blob/main/teps/0005-tekton-oci-bundles.md)                          | [v0.18.0](https://github.com/tektoncd/pipeline/releases/tag/v0.18.0) | `enable-tekton-oci-bundles` |
+| [`Runs` and `Custom Tasks`](./runs.md)                                                                | [TEP-0002](https://github.com/tektoncd/community/blob/main/teps/0002-custom-tasks.md)                                | [v0.19.0](https://github.com/tektoncd/pipeline/releases/tag/v0.19.0) | `enable-custom-tasks`       |
+| [Isolated `Step` & `Sidecar` `Workspaces`](./workspaces.md#isolated-workspaces)                       | [TEP-0029](https://github.com/tektoncd/community/blob/main/teps/0029-step-workspaces.md)                             | [v0.24.0](https://github.com/tektoncd/pipeline/releases/tag/v0.24.0) |                             |
+| [Hermetic Execution Mode](./hermetic.md)                                                              | [TEP-0025](https://github.com/tektoncd/community/blob/main/teps/0025-hermekton.md)                                   | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) |                             |
+| [Graceful Termination](./pipelineruns.md#gracefully-cancelling-a-pipelinerun)                         | [TEP-0058](https://github.com/tektoncd/community/blob/main/teps/0058-graceful-pipeline-run-termination.md)           | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) |                             |
+| [`PipelineRun` Timeouts](./pipelineruns.md#configuring-a-failure-timeout)                             | [TEP-0046](https://github.com/tektoncd/community/blob/main/teps/0046-finallytask-execution-post-timeout.md)          | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) |                             |
+| [Implicit `Parameters`](./taskruns.md#implicit-parameters)                                            | [TEP-0023](https://github.com/tektoncd/community/blob/main/teps/0023-implicit-mapping.md)                            | [v0.28.0](https://github.com/tektoncd/pipeline/releases/tag/v0.28.0) |                             |
+| [Windows Scripts](./tasks.md#windows-scripts)                                                         | [TEP-0057](https://github.com/tektoncd/community/blob/main/teps/0057-windows-support.md)                             | [v0.28.0](https://github.com/tektoncd/pipeline/releases/tag/v0.28.0) |                             |
+| [Remote Tasks](./taskruns.md#remote-tasks) and [Remote Pipelines](./pipelineruns.md#remote-pipelines) | [TEP-0060](https://github.com/tektoncd/community/blob/main/teps/0060-remote-resolutiond.md)                          |                                                                      |                             |
+| [Debug](./debug.md)                                                                                   | [TEP-0042](https://github.com/tektoncd/community/blob/main/teps/0042-taskrun-breakpoint-on-failure.md)               | [v0.26.0](https://github.com/tektoncd/pipeline/releases/tag/v0.26.0) |                             |
+| [Step and Sidecar Overrides](./taskruns.md#overriding-task-steps-and-sidecars)                        | [TEP-0094](https://github.com/tektoncd/community/blob/main/teps/0094-specifying-resource-requirements-at-runtime.md) |                                                                      |                             |
+| [Matrix](./matrix.md)                                                                                 | [TEP-0090](https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md)                                      |                                                                      |                             |
 
 ## Configuring High Availability
 

diff --git a/docs/matrix.md b/docs/matrix.md
@@ -0,0 +1,21 @@
+<!--
+---
+linkTitle: "Matrix"
+weight: 11
+---
+-->
+
+# Matrix
+
+- [Overview](#overview)
+
+## Overview
+
+`Matrix` is used to fan out `Tasks` in a `Pipeline`. This doc will explain the details of `matrix` support in
+Tekton. 
+
+> :seedling: **`Matrix` is an [alpha](install.md#alpha-features) feature.**
+> The `enable-api-fields` feature flag must be set to `"alpha"` to specify `Matrix` in a `PipelineTask`.
+>
+> :warning: This feature is in a preview mode. 
+> It is still in a very early stage of development and is not yet fully functional.
diff --git a/docs/pipelines.md b/docs/pipelines.md
@@ -14,6 +14,7 @@ weight: 400
   - [Adding `Tasks` to the `Pipeline`](#adding-tasks-to-the-pipeline)
     - [Specifying `Resources` in `PipelineTasks`](#specifying-resources-in-pipelinetasks)
     - [Specifying `Parameters` in `PipelineTasks`](#specifying-parameters-in-pipelinetasks)
+    - [Specifying `Matrix` in `PipelineTasks`](#specifying-matrix-in-pipelinetasks)
     - [Specifying `Workspaces` in `PipelineTasks`](#specifying-workspaces-in-pipelinetasks)
     - [Tekton Bundles](#tekton-bundles)
     - [Using the `from` field](#using-the-from-field)
@@ -36,6 +37,7 @@ weight: 400
   - [Adding `Finally` to the `Pipeline`](#adding-finally-to-the-pipeline)
     - [Specifying `Workspaces` in `finally` tasks](#specifying-workspaces-in-finally-tasks)
     - [Specifying `Parameters` in `finally` tasks](#specifying-parameters-in-finally-tasks)
+    - [Specifying `matrix` in `finally` tasks](#specifying-matrix-in-finally-tasks)
     - [Consuming `Task` execution results in `finally`](#consuming-task-execution-results-in-finally)
     - [`PipelineRun` Status with `finally`](#pipelinerun-status-with-finally)
     - [Using Execution `Status` of `pipelineTask`](#using-execution-status-of-pipelinetask)
@@ -54,6 +56,7 @@ weight: 400
     - [Specifying the target Custom Task](#specifying-the-target-custom-task)
     - [Specifying a Custom Task Spec in-line (or embedded)](#specifying-a-custom-task-spec-in-line-or-embedded)
     - [Specifying parameters](#specifying-parameters-1)
+    - [Specifying matrix](#specifying-matrix)
     - [Specifying workspaces](#specifying-workspaces-1)
     - [Using `Results`](#using-results-1)
     - [Limitations](#limitations)
@@ -104,6 +107,8 @@ A `Pipeline` definition supports the following fields:
       - [`timeout`](#configuring-the-failure-timeout) - Specifies the timeout before a `Task` fails.
       - [`params`](#specifying-parameters-in-pipelinetasks) - Specifies the `Parameters` that a `Task` requires.
       - [`workspaces`](#specifying-workspaces-in-pipelinetasks) - Specifies the `Workspaces` that a `Task` requires.
+      - [`matrix`](#specifying-matrix-in-pipelinetasks) - Specifies the `Parameters` used to fan out a `Task` into
+        multiple `TaskRuns` or `Runs`.
   - [`results`](#emitting-results-from-a-pipeline) - Specifies the location to which the `Pipeline` emits its execution
     results.
   - [`description`](#adding-a-description) - Holds an informative description of the `Pipeline` object.
@@ -119,6 +124,8 @@ A `Pipeline` definition supports the following fields:
     - [`timeout`](#configuring-the-failure-timeout) - Specifies the timeout before a `Task` fails.
     - [`params`](#specifying-parameters-in-finally-tasks) - Specifies the `Parameters` that a `Task` requires.
     - [`workspaces`](#specifying-workspaces-in-finally-tasks) - Specifies the `Workspaces` that a `Task` requires.
+    - [`matrix`](#specifying-matrix-in-finally-tasks) - Specifies the `Parameters` used to fan out a `Task` into
+      multiple `TaskRuns` or `Runs`.
 
 [kubernetes-overview]:
   https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields
@@ -308,6 +315,32 @@ spec:
           value: /workspace/examples/microservices/leeroy-web
 ```
 
+### Specifying `Matrix` in `PipelineTasks`
+
+> :seedling: **`Matrix` is an [alpha](install.md#alpha-features) feature.**
+> The `enable-api-fields` feature flag must be set to `"alpha"` to specify `Matrix` in a `PipelineTask`.
+>
+> :warning: This feature is in a preview mode.
+> It is still in a very early stage of development and is not yet fully functional.
+
+You can also provide [`Parameters`](tasks.md#specifying-parameters) through the `matrix` field:
+
+```yaml
+spec:
+  tasks:
+    - name: browser-test
+      taskRef:
+        name: browser-test
+      matrix:
+        - name: browser
+          value:
+          - chrome
+          - safari
+          - firefox
+```
+
+For further information, read [`Matrix`](./matrix.md).
+
 ### Specifying `Workspaces` in `PipelineTasks`
 
 You can also provide [`Workspaces`](tasks.md#specifying-workspaces):
@@ -1186,6 +1219,39 @@ spec:
           value: "someURL"
 ```
 
+### Specifying `matrix` in `finally` tasks
+
+> :seedling: **`Matrix` is an [alpha](install.md#alpha-features) feature.**
+> The `enable-api-fields` feature flag must be set to `"alpha"` to specify `Matrix` in a `PipelineTask`.
+>
+> :warning: This feature is in a preview mode.
+> It is still in a very early stage of development and is not yet fully functional.
+
+Similar to `tasks`, you can also provide [`Parameters`](tasks.md#specifying-parameters) through `matrix`
+in `finally` tasks:
+
+```yaml
+spec:
+  tasks:
+    - name: tests
+      taskRef:
+        name: integration-test
+  finally:
+    - name: report-results
+      taskRef:
+        name: report-results
+      params:
+        - name: url
+          value: "someURL"
+      matrix:
+        - name: slack-channel
+          value: 
+          - "foo"
+          - "bar"
+```
+
+For further information, read [`Matrix`](./matrix.md).
+
 ### Consuming `Task` execution results in `finally`
 
 `finally` tasks can be configured to consume `Results` of `PipelineTask` from the `tasks` section:
@@ -1602,6 +1668,37 @@ spec:
           value: bah
 ```
 
+### Specifying matrix
+
+> :seedling: **`Matrix` is an [alpha](install.md#alpha-features) feature.**
+> The `enable-api-fields` feature flag must be set to `"alpha"` to specify `Matrix` in a `PipelineTask`.
+>
+> :warning: This feature is in a preview mode.
+> It is still in a very early stage of development and is not yet fully functional.
+
+If a custom task supports [`parameters`](tasks.md#specifying-parameters), you can use the
+`matrix` field to specify their values, if you want to fan:
+
+```yaml
+spec:
+  tasks:
+    - name: run-custom-task
+      taskRef:
+        apiVersion: example.dev/v1alpha1
+        kind: Example
+        name: myexample
+      params:
+        - name: foo
+          value: bah
+      matrix:
+        - name: bar
+          value:
+            - qux
+            - thud
+```
+
+For further information, read [`Matrix`](./matrix.md).
+
 ### Specifying workspaces
 
 If the custom task supports it, you can provide [`Workspaces`](workspaces.md#using-workspaces-in-tasks) to share data with the custom task.

diff --git a/pkg/apis/pipeline/v1beta1/openapi_generated.go b/pkg/apis/pipeline/v1beta1/openapi_generated.go
diff --git a/pkg/apis/pipeline/v1beta1/pipeline_types.go b/pkg/apis/pipeline/v1beta1/pipeline_types.go
@@ -188,10 +188,15 @@ type PipelineTask struct {
 	// outputs.
 	// +optional
 	Resources *PipelineTaskResources `json:"resources,omitempty"`
+
 	// Parameters declares parameters passed to this task.
 	// +optional
 	Params []Param `json:"params,omitempty"`
 
+	// Matrix declares parameters used to fan out this task.
+	// +optional
+	Matrix []Param `json:"matrix,omitempty"`
+
 	// Workspaces maps workspaces from the pipeline spec to the workspaces
 	// declared in the Task.
 	// +optional
@@ -282,6 +287,15 @@ func (pt PipelineTask) validateTask(ctx context.Context) (errs *apis.FieldError)
 	return errs
 }
 
+func (pt *PipelineTask) validateMatrix(ctx context.Context) (errs *apis.FieldError) {
+	if len(pt.Matrix) != 0 {
+		// This is an alpha feature and will fail validation if it's used in a pipeline spec
+		// when the enable-api-fields feature gate is anything but "alpha".
+		errs = errs.Also(ValidateEnabledAPIFields(ctx, "matrix", config.AlphaAPIFields))
+	}
+	return errs
+}
+
 // TaskSpecMetadata returns the metadata of the PipelineTask's EmbeddedTask spec.
 func (pt *PipelineTask) TaskSpecMetadata() PipelineTaskMetadata {
 	return pt.TaskSpec.Metadata

diff --git a/pkg/apis/pipeline/v1beta1/pipeline_validation.go b/pkg/apis/pipeline/v1beta1/pipeline_validation.go
@@ -71,6 +71,8 @@ func (ps *PipelineSpec) Validate(ctx context.Context) (errs *apis.FieldError) {
 	errs = errs.Also(validateTasksAndFinallySection(ps))
 	errs = errs.Also(validateFinalTasks(ps.Tasks, ps.Finally))
 	errs = errs.Also(validateWhenExpressions(ps.Tasks, ps.Finally))
+	errs = errs.Also(validateMatrix(ctx, ps.Tasks).ViaField("tasks"))
+	errs = errs.Also(validateMatrix(ctx, ps.Finally).ViaField("finally"))
 	return errs
 }
 
@@ -546,3 +548,10 @@ func validateGraph(tasks []PipelineTask) *apis.FieldError {
 	}
 	return nil
 }
+
+func validateMatrix(ctx context.Context, tasks []PipelineTask) (errs *apis.FieldError) {
+	for idx, task := range tasks {
+		errs = errs.Also(task.validateMatrix(ctx).ViaIndex(idx))
+	}
+	return errs
+}