Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEP-0118: Added Matrix.Include field in preview mode #6188

Merged
merged 1 commit into from
Feb 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 19 additions & 13 deletions docs/matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ weight: 406
- [Concurrency Control](#concurrency-control)
- [Parameters](#parameters)
- [Specifying both `params` and `matrix` in a `PipelineTask`](#specifying-both-params-and-matrix-in-a-pipelinetask)
- [Include](#Include)
- [Context Variables](#context-variables)
- [Results](#results)
- [Specifying Results in a Matrix](#specifying-results-in-a-matrix)
Expand All @@ -24,7 +25,7 @@ weight: 406
## Overview

`Matrix` is used to fan out `Tasks` in a `Pipeline`. This doc will explain the details of `matrix` support in
Tekton.
Tekton.

Documentation for specifying `Matrix` in a `Pipeline`:
- [Specifying `Matrix` in `Tasks`](pipelines.md#specifying-matrix-in-pipelinetasks)
Expand All @@ -40,12 +41,12 @@ A `Matrix` supports the following features:
* [Concurrency Control](#concurrency-control)
* [Parameters](#parameters)
* [Context Variables](#context-variables)
* [Results](#results)
* [Results](#results)

### Concurrency Control

The default maximum count of `TaskRuns` or `Runs` from a given `Matrix` is **256**. To customize the maximum count of
`TaskRuns` or `Runs` generated from a given `Matrix`, configure the `default-max-matrix-combinations-count` in
`TaskRuns` or `Runs` generated from a given `Matrix`, configure the `default-max-matrix-combinations-count` in
[config defaults](/config/config-defaults.yaml). When a `Matrix` in `PipelineTask` would generate more than the maximum
`TaskRuns` or `Runs`, the `Pipeline` validation would fail.

Expand Down Expand Up @@ -92,7 +93,7 @@ spec:
default:
- chrome
- safari
- firefox
- firefox
tasks:
- name: fetch-repository
taskRef:
Expand All @@ -110,7 +111,7 @@ spec:
...
```

A `Parameter` can be passed to either the `matrix` or `params` field, not both.
A `Parameter` can be passed to either the `matrix` or `params` field, not both.

For further details on specifying `Parameters` in the `Pipeline` and passing them to
`PipelineTasks`, see [documentation](pipelines.md#specifying-parameters).
Expand Down Expand Up @@ -148,9 +149,14 @@ spec:
...
```

### Include
> :warning: This feature is in a preview mode.
> It is still in a very early stage of development and is not yet fully functional.
The `Include` section in the `Matrix` field exists, but is not yet functional.

### Context Variables

Similarly to the `Parameters` in the `Params` field, the `Parameters` in the `Matrix` field will accept
Similarly to the `Parameters` in the `Params` field, the `Parameters` in the `Matrix` field will accept
[context variables](variables.md) that will be substituted, including:

* `PipelineRun` name, namespace and uid
Expand All @@ -161,7 +167,7 @@ Similarly to the `Parameters` in the `Params` field, the `Parameters` in the `Ma

#### Specifying Results in a Matrix

Consuming `Results` from previous `TaskRuns` or `Runs` in a `Matrix`, which would dynamically generate
Consuming `Results` from previous `TaskRuns` or `Runs` in a `Matrix`, which would dynamically generate
`TaskRuns` or `Runs` from the fanned out `PipelineTask`, is supported. Producing `Results` in from a
`PipelineTask` with a `Matrix` is not yet supported - see [further details](#results-from-fanned-out-pipelinetasks).

Expand Down Expand Up @@ -203,7 +209,7 @@ tasks:

Consuming `Results` from fanned out `PipelineTasks` will not be in the supported in the initial iteration
of `Matrix`. Supporting consuming `Results` from fanned out `PipelineTasks` will be revisited after array
and object `Results` are supported.
and object `Results` are supported.

## Fan Out

Expand Down Expand Up @@ -487,15 +493,15 @@ status:
name: platforms-and-browsers
taskRef:
apiVersion: cel.tekton.dev/v1alpha1
kind: CEL
kind: CEL
startTime: "2022-06-28T20:49:40Z"
completionTime: "2022-06-28T20:49:41Z"
conditions:
- lastTransitionTime: "2022-06-28T20:49:41Z"
message: 'Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0'
reason: Succeeded
status: "True"
type: Succeeded
type: Succeeded
childReferences:
- apiVersion: tekton.dev/v1alpha1
kind: Run
Expand Down Expand Up @@ -533,11 +539,11 @@ status:

## Retries

The `retries` field is used to specify the number of times a `PipelineTask` should be retried when its `TaskRun` or
`Run` fails, see the [documentation][retries] for further details. When a `PipelineTask` is fanned out using `Matrix`,
The `retries` field is used to specify the number of times a `PipelineTask` should be retried when its `TaskRun` or
`Run` fails, see the [documentation][retries] for further details. When a `PipelineTask` is fanned out using `Matrix`,
a given `TaskRun` or `Run` executed will be retried as much as the field in the `retries` field of the `PipelineTask`.

For example, the `PipelineTask` in this `PipelineRun` will be fanned out into three `TaskRuns` each of which will be
For example, the `PipelineTask` in this `PipelineRun` will be fanned out into three `TaskRuns` each of which will be
retried once:

```yaml
Expand Down
122 changes: 120 additions & 2 deletions docs/pipeline-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1500,6 +1500,65 @@ Each array element is supplied to the <code>PipelineTask</code> by substituting
The names of the <code>params</code> in the <code>Matrix</code> must match the names of the <code>params</code> in the underlying <code>Task</code> that they will be substituting.</p>
</td>
</tr>
<tr>
<td>
<code>include</code><br/>
<em>
<a href="#tekton.dev/v1.MatrixInclude">
[]MatrixInclude
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>Include is a list of MatrixInclude
Note this field is in preview mode and not yet supported</p>
</td>
</tr>
</tbody>
</table>
<h3 id="tekton.dev/v1.MatrixInclude">MatrixInclude
</h3>
<p>
(<em>Appears on:</em><a href="#tekton.dev/v1.Matrix">Matrix</a>)
</p>
<div>
<p>MatrixInclude allows passing in a specific combinations of Parameters into the Matrix.
Note this struct is in preview mode and not yet supported</p>
</div>
<table>
<thead>
<tr>
<th>Field</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<code>name</code><br/>
<em>
string
</em>
</td>
<td>
<p>Name the specified combination</p>
</td>
</tr>
<tr>
<td>
<code>params</code><br/>
<em>
<a href="#tekton.dev/v1.Param">
[]Param
</a>
</em>
</td>
<td>
<p>Params takes only <code>Parameters</code> of type <code>&quot;string&quot;</code>
The names of the <code>params</code> must match the names of the <code>params</code> in the underlying <code>Task</code></p>
</td>
</tr>
</tbody>
</table>
<h3 id="tekton.dev/v1.OnErrorType">OnErrorType
Expand Down Expand Up @@ -1528,7 +1587,7 @@ The names of the <code>params</code> in the <code>Matrix</code> must match the n
<h3 id="tekton.dev/v1.Param">Param
</h3>
<p>
(<em>Appears on:</em><a href="#tekton.dev/v1.Matrix">Matrix</a>, <a href="#tekton.dev/v1.PipelineRunSpec">PipelineRunSpec</a>, <a href="#tekton.dev/v1.PipelineTask">PipelineTask</a>, <a href="#tekton.dev/v1.ResolverRef">ResolverRef</a>, <a href="#tekton.dev/v1.TaskRunInputs">TaskRunInputs</a>, <a href="#tekton.dev/v1.TaskRunSpec">TaskRunSpec</a>)
(<em>Appears on:</em><a href="#tekton.dev/v1.Matrix">Matrix</a>, <a href="#tekton.dev/v1.MatrixInclude">MatrixInclude</a>, <a href="#tekton.dev/v1.PipelineRunSpec">PipelineRunSpec</a>, <a href="#tekton.dev/v1.PipelineTask">PipelineTask</a>, <a href="#tekton.dev/v1.ResolverRef">ResolverRef</a>, <a href="#tekton.dev/v1.TaskRunInputs">TaskRunInputs</a>, <a href="#tekton.dev/v1.TaskRunSpec">TaskRunSpec</a>)
</p>
<div>
<p>Param declares an ParamValues to use for the parameter called name.</p>
Expand Down Expand Up @@ -8997,6 +9056,65 @@ Each array element is supplied to the <code>PipelineTask</code> by substituting
The names of the <code>params</code> in the <code>Matrix</code> must match the names of the <code>params</code> in the underlying <code>Task</code> that they will be substituting.</p>
</td>
</tr>
<tr>
<td>
<code>include</code><br/>
<em>
<a href="#tekton.dev/v1beta1.MatrixInclude">
[]MatrixInclude
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>Include is a list of MatrixInclude
Note this field is in preview mode and not yet supported</p>
</td>
</tr>
</tbody>
</table>
<h3 id="tekton.dev/v1beta1.MatrixInclude">MatrixInclude
</h3>
<p>
(<em>Appears on:</em><a href="#tekton.dev/v1beta1.Matrix">Matrix</a>)
</p>
<div>
<p>MatrixInclude allows passing in a specific combinations of Parameters into the Matrix.
Note this struct is in preview mode and not yet supported</p>
</div>
<table>
<thead>
<tr>
<th>Field</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<code>name</code><br/>
<em>
string
</em>
</td>
<td>
<p>Name the specified combination</p>
</td>
</tr>
<tr>
<td>
<code>params</code><br/>
<em>
<a href="#tekton.dev/v1beta1.Param">
[]Param
</a>
</em>
</td>
<td>
<p>Params takes only <code>Parameters</code> of type <code>&quot;string&quot;</code>
The names of the <code>params</code> must match the names of the <code>params</code> in the underlying <code>Task</code></p>
</td>
</tr>
</tbody>
</table>
<h3 id="tekton.dev/v1beta1.OnErrorType">OnErrorType
Expand All @@ -9010,7 +9128,7 @@ The names of the <code>params</code> in the <code>Matrix</code> must match the n
<h3 id="tekton.dev/v1beta1.Param">Param
</h3>
<p>
(<em>Appears on:</em><a href="#tekton.dev/v1alpha1.RunSpec">RunSpec</a>, <a href="#tekton.dev/v1beta1.CustomRunSpec">CustomRunSpec</a>, <a href="#tekton.dev/v1beta1.Matrix">Matrix</a>, <a href="#tekton.dev/v1beta1.PipelineRunSpec">PipelineRunSpec</a>, <a href="#tekton.dev/v1beta1.PipelineTask">PipelineTask</a>, <a href="#tekton.dev/v1beta1.ResolverRef">ResolverRef</a>, <a href="#tekton.dev/v1beta1.TaskRunInputs">TaskRunInputs</a>, <a href="#tekton.dev/v1beta1.TaskRunSpec">TaskRunSpec</a>, <a href="#resolution.tekton.dev/v1beta1.ResolutionRequestSpec">ResolutionRequestSpec</a>)
(<em>Appears on:</em><a href="#tekton.dev/v1alpha1.RunSpec">RunSpec</a>, <a href="#tekton.dev/v1beta1.CustomRunSpec">CustomRunSpec</a>, <a href="#tekton.dev/v1beta1.Matrix">Matrix</a>, <a href="#tekton.dev/v1beta1.MatrixInclude">MatrixInclude</a>, <a href="#tekton.dev/v1beta1.PipelineRunSpec">PipelineRunSpec</a>, <a href="#tekton.dev/v1beta1.PipelineTask">PipelineTask</a>, <a href="#tekton.dev/v1beta1.ResolverRef">ResolverRef</a>, <a href="#tekton.dev/v1beta1.TaskRunInputs">TaskRunInputs</a>, <a href="#tekton.dev/v1beta1.TaskRunSpec">TaskRunSpec</a>, <a href="#resolution.tekton.dev/v1beta1.ResolutionRequestSpec">ResolutionRequestSpec</a>)
</p>
<div>
<p>Param declares an ParamValues to use for the parameter called name.</p>
Expand Down
61 changes: 61 additions & 0 deletions pkg/apis/pipeline/v1/openapi_generated.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 19 additions & 1 deletion pkg/apis/pipeline/v1/pipeline_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,24 @@ type Matrix struct {
// The names of the `params` in the `Matrix` must match the names of the `params` in the underlying `Task` that they will be substituting.
// +listType=atomic
Params []Param `json:"params,omitempty"`

// Include is a list of MatrixInclude which allows passing in specific combinations of Parameters into the Matrix.
// Note that Include is in preview mode and not yet supported.
// +optional
// +listType=atomic
Include []MatrixInclude `json:"include,omitempty"`
}

// MatrixInclude allows passing in a specific combinations of Parameters into the Matrix.
// Note this struct is in preview mode and not yet supported
type MatrixInclude struct {
// Name the specified combination
Name string `json:"name,omitempty"`

// Params takes only `Parameters` of type `"string"`
jerop marked this conversation as resolved.
Show resolved Hide resolved
// The names of the `params` must match the names of the `params` in the underlying `Task`
// +listType=atomic
Params []Param `json:"params,omitempty"`
}

// validateRefOrSpec validates at least one of taskRef or taskSpec is specified
Expand Down Expand Up @@ -275,7 +293,7 @@ func (pt PipelineTask) validateTask(ctx context.Context) (errs *apis.FieldError)

// IsMatrixed return whether pipeline task is matrixed
func (pt *PipelineTask) IsMatrixed() bool {
return pt.Matrix != nil && len(pt.Matrix.Params) > 0
return pt.Matrix != nil && (len(pt.Matrix.Params) > 0 || len(pt.Matrix.Include) > 0)
}

func (pt *PipelineTask) validateMatrix(ctx context.Context) (errs *apis.FieldError) {
Expand Down
Loading