Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEP-0090: Matrix - API Change and Feature Flag #4600

Merged
merged 1 commit into from
Feb 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 14 additions & 13 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -392,19 +392,20 @@ the `feature-flags` ConfigMap alongside your Tekton Pipelines deployment.

Features currently in "alpha" are:

| Feature | TEP | Release | Individual Flag |
|:------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------- |:-------------------------------------------------------------------- |:--------------------------- |
| [Bundles ](./pipelineruns.md#tekton-bundles) | [TEP-0005](https://github.com/tektoncd/community/blob/main/teps/0005-tekton-oci-bundles.md) | [v0.18.0](https://github.com/tektoncd/pipeline/releases/tag/v0.18.0) | `enable-tekton-oci-bundles` |
| [`Runs` and `Custom Tasks`](./runs.md) | [TEP-0002](https://github.com/tektoncd/community/blob/main/teps/0002-custom-tasks.md) | [v0.19.0](https://github.com/tektoncd/pipeline/releases/tag/v0.19.0) | `enable-custom-tasks` |
| [Isolated `Step` & `Sidecar` `Workspaces`](./workspaces.md#isolated-workspaces) | [TEP-0029](https://github.com/tektoncd/community/blob/main/teps/0029-step-workspaces.md) | [v0.24.0](https://github.com/tektoncd/pipeline/releases/tag/v0.24.0) | |
| [Hermetic Execution Mode](./hermetic.md) | [TEP-0025](https://github.com/tektoncd/community/blob/main/teps/0025-hermekton.md) | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) | |
| [Graceful Termination](./pipelineruns.md#gracefully-cancelling-a-pipelinerun) | [TEP-0058](https://github.com/tektoncd/community/blob/main/teps/0058-graceful-pipeline-run-termination.md) | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) | |
| [`PipelineRun` Timeouts](./pipelineruns.md#configuring-a-failure-timeout) | [TEP-0046](https://github.com/tektoncd/community/blob/main/teps/0046-finallytask-execution-post-timeout.md) | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) | |
| [Implicit `Parameters`](./taskruns.md#implicit-parameters) | [TEP-0023](https://github.com/tektoncd/community/blob/main/teps/0023-implicit-mapping.md) | [v0.28.0](https://github.com/tektoncd/pipeline/releases/tag/v0.28.0) | |
| [Windows Scripts](./tasks.md#windows-scripts) | [TEP-0057](https://github.com/tektoncd/community/blob/main/teps/0057-windows-support.md) | [v0.28.0](https://github.com/tektoncd/pipeline/releases/tag/v0.28.0) | |
| [Remote Tasks](./taskruns.md#remote-tasks) and [Remote Pipelines](./pipelineruns.md#remote-pipelines) | [TEP-0060](https://github.com/tektoncd/community/blob/main/teps/0060-remote-resolutiond.md) | | |
| [Debug](./debug.md) | [TEP-0042](https://github.com/tektoncd/community/blob/main/teps/0042-taskrun-breakpoint-on-failure.md) | [v0.26.0](https://github.com/tektoncd/pipeline/releases/tag/v0.26.0) | |
| [Step and Sidecar Overrides](./taskruns.md#overriding-task-steps-and-sidecars)| [TEP-0094](https://github.com/tektoncd/community/blob/main/teps/0094-specifying-resource-requirements-at-runtime.md) | | |
| Feature | TEP | Release | Individual Flag |
|:------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------|:----------------------------|
| [Bundles ](./pipelineruns.md#tekton-bundles) | [TEP-0005](https://github.com/tektoncd/community/blob/main/teps/0005-tekton-oci-bundles.md) | [v0.18.0](https://github.com/tektoncd/pipeline/releases/tag/v0.18.0) | `enable-tekton-oci-bundles` |
| [`Runs` and `Custom Tasks`](./runs.md) | [TEP-0002](https://github.com/tektoncd/community/blob/main/teps/0002-custom-tasks.md) | [v0.19.0](https://github.com/tektoncd/pipeline/releases/tag/v0.19.0) | `enable-custom-tasks` |
| [Isolated `Step` & `Sidecar` `Workspaces`](./workspaces.md#isolated-workspaces) | [TEP-0029](https://github.com/tektoncd/community/blob/main/teps/0029-step-workspaces.md) | [v0.24.0](https://github.com/tektoncd/pipeline/releases/tag/v0.24.0) | |
| [Hermetic Execution Mode](./hermetic.md) | [TEP-0025](https://github.com/tektoncd/community/blob/main/teps/0025-hermekton.md) | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) | |
| [Graceful Termination](./pipelineruns.md#gracefully-cancelling-a-pipelinerun) | [TEP-0058](https://github.com/tektoncd/community/blob/main/teps/0058-graceful-pipeline-run-termination.md) | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) | |
| [`PipelineRun` Timeouts](./pipelineruns.md#configuring-a-failure-timeout) | [TEP-0046](https://github.com/tektoncd/community/blob/main/teps/0046-finallytask-execution-post-timeout.md) | [v0.25.0](https://github.com/tektoncd/pipeline/releases/tag/v0.25.0) | |
| [Implicit `Parameters`](./taskruns.md#implicit-parameters) | [TEP-0023](https://github.com/tektoncd/community/blob/main/teps/0023-implicit-mapping.md) | [v0.28.0](https://github.com/tektoncd/pipeline/releases/tag/v0.28.0) | |
| [Windows Scripts](./tasks.md#windows-scripts) | [TEP-0057](https://github.com/tektoncd/community/blob/main/teps/0057-windows-support.md) | [v0.28.0](https://github.com/tektoncd/pipeline/releases/tag/v0.28.0) | |
| [Remote Tasks](./taskruns.md#remote-tasks) and [Remote Pipelines](./pipelineruns.md#remote-pipelines) | [TEP-0060](https://github.com/tektoncd/community/blob/main/teps/0060-remote-resolutiond.md) | | |
| [Debug](./debug.md) | [TEP-0042](https://github.com/tektoncd/community/blob/main/teps/0042-taskrun-breakpoint-on-failure.md) | [v0.26.0](https://github.com/tektoncd/pipeline/releases/tag/v0.26.0) | |
| [Step and Sidecar Overrides](./taskruns.md#overriding-task-steps-and-sidecars) | [TEP-0094](https://github.com/tektoncd/community/blob/main/teps/0094-specifying-resource-requirements-at-runtime.md) | | |
| [Matrix](./matrix.md) | [TEP-0090](https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md) | | |

## Configuring High Availability

Expand Down
21 changes: 21 additions & 0 deletions docs/matrix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<!--
jerop marked this conversation as resolved.
Show resolved Hide resolved
---
linkTitle: "Matrix"
weight: 11
---
-->

# Matrix

- [Overview](#overview)

## Overview

`Matrix` is used to fan out `Tasks` in a `Pipeline`. This doc will explain the details of `matrix` support in
Tekton.

> :seedling: **`Matrix` is an [alpha](install.md#alpha-features) feature.**
> The `enable-api-fields` feature flag must be set to `"alpha"` to specify `Matrix` in a `PipelineTask`.
>
> :warning: This feature is in a preview mode.
> It is still in a very early stage of development and is not yet fully functional.
97 changes: 97 additions & 0 deletions docs/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ weight: 400
- [Adding `Tasks` to the `Pipeline`](#adding-tasks-to-the-pipeline)
- [Specifying `Resources` in `PipelineTasks`](#specifying-resources-in-pipelinetasks)
- [Specifying `Parameters` in `PipelineTasks`](#specifying-parameters-in-pipelinetasks)
- [Specifying `Matrix` in `PipelineTasks`](#specifying-matrix-in-pipelinetasks)
- [Specifying `Workspaces` in `PipelineTasks`](#specifying-workspaces-in-pipelinetasks)
- [Tekton Bundles](#tekton-bundles)
- [Using the `from` field](#using-the-from-field)
Expand All @@ -36,6 +37,7 @@ weight: 400
- [Adding `Finally` to the `Pipeline`](#adding-finally-to-the-pipeline)
- [Specifying `Workspaces` in `finally` tasks](#specifying-workspaces-in-finally-tasks)
- [Specifying `Parameters` in `finally` tasks](#specifying-parameters-in-finally-tasks)
- [Specifying `matrix` in `finally` tasks](#specifying-matrix-in-finally-tasks)
- [Consuming `Task` execution results in `finally`](#consuming-task-execution-results-in-finally)
- [`PipelineRun` Status with `finally`](#pipelinerun-status-with-finally)
- [Using Execution `Status` of `pipelineTask`](#using-execution-status-of-pipelinetask)
Expand All @@ -54,6 +56,7 @@ weight: 400
- [Specifying the target Custom Task](#specifying-the-target-custom-task)
- [Specifying a Custom Task Spec in-line (or embedded)](#specifying-a-custom-task-spec-in-line-or-embedded)
- [Specifying parameters](#specifying-parameters-1)
- [Specifying matrix](#specifying-matrix)
- [Specifying workspaces](#specifying-workspaces-1)
- [Using `Results`](#using-results-1)
- [Limitations](#limitations)
Expand Down Expand Up @@ -104,6 +107,8 @@ A `Pipeline` definition supports the following fields:
- [`timeout`](#configuring-the-failure-timeout) - Specifies the timeout before a `Task` fails.
- [`params`](#specifying-parameters-in-pipelinetasks) - Specifies the `Parameters` that a `Task` requires.
- [`workspaces`](#specifying-workspaces-in-pipelinetasks) - Specifies the `Workspaces` that a `Task` requires.
- [`matrix`](#specifying-matrix-in-pipelinetasks) - Specifies the `Parameters` used to fan out a `Task` into
multiple `TaskRuns` or `Runs`.
- [`results`](#emitting-results-from-a-pipeline) - Specifies the location to which the `Pipeline` emits its execution
results.
- [`description`](#adding-a-description) - Holds an informative description of the `Pipeline` object.
Expand All @@ -119,6 +124,8 @@ A `Pipeline` definition supports the following fields:
- [`timeout`](#configuring-the-failure-timeout) - Specifies the timeout before a `Task` fails.
- [`params`](#specifying-parameters-in-finally-tasks) - Specifies the `Parameters` that a `Task` requires.
- [`workspaces`](#specifying-workspaces-in-finally-tasks) - Specifies the `Workspaces` that a `Task` requires.
- [`matrix`](#specifying-matrix-in-finally-tasks) - Specifies the `Parameters` used to fan out a `Task` into
multiple `TaskRuns` or `Runs`.

[kubernetes-overview]:
https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields
Expand Down Expand Up @@ -308,6 +315,32 @@ spec:
value: /workspace/examples/microservices/leeroy-web
```

### Specifying `Matrix` in `PipelineTasks`

> :seedling: **`Matrix` is an [alpha](install.md#alpha-features) feature.**
> The `enable-api-fields` feature flag must be set to `"alpha"` to specify `Matrix` in a `PipelineTask`.
>
> :warning: This feature is in a preview mode.
> It is still in a very early stage of development and is not yet fully functional.

You can also provide [`Parameters`](tasks.md#specifying-parameters) through the `matrix` field:

```yaml
jerop marked this conversation as resolved.
Show resolved Hide resolved
spec:
tasks:
- name: browser-test
taskRef:
name: browser-test
matrix:
- name: browser
value:
- chrome
- safari
- firefox
```

For further information, read [`Matrix`](./matrix.md).

### Specifying `Workspaces` in `PipelineTasks`

You can also provide [`Workspaces`](tasks.md#specifying-workspaces):
Expand Down Expand Up @@ -1186,6 +1219,39 @@ spec:
value: "someURL"
```

### Specifying `matrix` in `finally` tasks

> :seedling: **`Matrix` is an [alpha](install.md#alpha-features) feature.**
> The `enable-api-fields` feature flag must be set to `"alpha"` to specify `Matrix` in a `PipelineTask`.
>
> :warning: This feature is in a preview mode.
> It is still in a very early stage of development and is not yet fully functional.

Similar to `tasks`, you can also provide [`Parameters`](tasks.md#specifying-parameters) through `matrix`
in `finally` tasks:

```yaml
spec:
tasks:
- name: tests
taskRef:
name: integration-test
finally:
- name: report-results
taskRef:
name: report-results
params:
- name: url
value: "someURL"
matrix:
- name: slack-channel
value:
- "foo"
- "bar"
```

For further information, read [`Matrix`](./matrix.md).

### Consuming `Task` execution results in `finally`

`finally` tasks can be configured to consume `Results` of `PipelineTask` from the `tasks` section:
Expand Down Expand Up @@ -1602,6 +1668,37 @@ spec:
value: bah
```

### Specifying matrix

> :seedling: **`Matrix` is an [alpha](install.md#alpha-features) feature.**
> The `enable-api-fields` feature flag must be set to `"alpha"` to specify `Matrix` in a `PipelineTask`.
>
> :warning: This feature is in a preview mode.
> It is still in a very early stage of development and is not yet fully functional.

If a custom task supports [`parameters`](tasks.md#specifying-parameters), you can use the
`matrix` field to specify their values, if you want to fan:

```yaml
spec:
tasks:
- name: run-custom-task
taskRef:
apiVersion: example.dev/v1alpha1
kind: Example
name: myexample
params:
- name: foo
value: bah
matrix:
- name: bar
value:
- qux
- thud
```

For further information, read [`Matrix`](./matrix.md).

### Specifying workspaces

If the custom task supports it, you can provide [`Workspaces`](workspaces.md#using-workspaces-in-tasks) to share data with the custom task.
Expand Down
14 changes: 14 additions & 0 deletions pkg/apis/pipeline/v1beta1/openapi_generated.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 14 additions & 0 deletions pkg/apis/pipeline/v1beta1/pipeline_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -188,10 +188,15 @@ type PipelineTask struct {
// outputs.
// +optional
Resources *PipelineTaskResources `json:"resources,omitempty"`

// Parameters declares parameters passed to this task.
// +optional
Params []Param `json:"params,omitempty"`

// Matrix declares parameters used to fan out this task.
// +optional
Matrix []Param `json:"matrix,omitempty"`

// Workspaces maps workspaces from the pipeline spec to the workspaces
// declared in the Task.
// +optional
Expand Down Expand Up @@ -282,6 +287,15 @@ func (pt PipelineTask) validateTask(ctx context.Context) (errs *apis.FieldError)
return errs
}

func (pt *PipelineTask) validateMatrix(ctx context.Context) (errs *apis.FieldError) {
if len(pt.Matrix) != 0 {
// This is an alpha feature and will fail validation if it's used in a pipeline spec
// when the enable-api-fields feature gate is anything but "alpha".
errs = errs.Also(ValidateEnabledAPIFields(ctx, "matrix", config.AlphaAPIFields))
}
return errs
}

// TaskSpecMetadata returns the metadata of the PipelineTask's EmbeddedTask spec.
func (pt *PipelineTask) TaskSpecMetadata() PipelineTaskMetadata {
return pt.TaskSpec.Metadata
Expand Down
9 changes: 9 additions & 0 deletions pkg/apis/pipeline/v1beta1/pipeline_validation.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ func (ps *PipelineSpec) Validate(ctx context.Context) (errs *apis.FieldError) {
errs = errs.Also(validateTasksAndFinallySection(ps))
errs = errs.Also(validateFinalTasks(ps.Tasks, ps.Finally))
errs = errs.Also(validateWhenExpressions(ps.Tasks, ps.Finally))
errs = errs.Also(validateMatrix(ctx, ps.Tasks).ViaField("tasks"))
errs = errs.Also(validateMatrix(ctx, ps.Finally).ViaField("finally"))
return errs
}

Expand Down Expand Up @@ -546,3 +548,10 @@ func validateGraph(tasks []PipelineTask) *apis.FieldError {
}
return nil
}

func validateMatrix(ctx context.Context, tasks []PipelineTask) (errs *apis.FieldError) {
jerop marked this conversation as resolved.
Show resolved Hide resolved
jerop marked this conversation as resolved.
Show resolved Hide resolved
for idx, task := range tasks {
errs = errs.Also(task.validateMatrix(ctx).ViaIndex(idx))
}
return errs
}
Loading