TEP-0090: Matrix [Problem Statement]

[jerop]

https://github.com/jerop/community/blob/looping/teps/0090-matrix.md

This change adds the problem statement for Matrix. It scopes the
problem, describes the use cases, and identifies the goals, the
requirements for the solution, and related work in other continuous
delivery systems.

Today, users cannot supply varying Parameters to execute a
PipelineTask, that is, fan out a PipelineTasks.

To solve this problem, this TEP aims to enable executing the same
PipelineTask with different combinations of Parameters specified
in a matrix. TaskRuns or Runs will be created with variables
substituted with each combination of the Parameters in the matrix.

This matrix construct will enable users to specify concise but
powerful Pipelines. Moreover, it would improve the composability,
scalability, flexibility and reusability of Tekton Pipelines.

References:

• Task Loops Experimental Project
• Issues:
  • Support Task looping syntax inside Pipelines pipeline#2050
  • Variable to get a list of results of a task pipeline#4097

[jerop]

/assign @tlawrie

[tekton-robot]

@jerop: GitHub didn't allow me to assign the following users: tlawrie.

Note that only tektoncd members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @tlawrie

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tlawrie]

Hey @jerop thanks for bringing this up on the alternate WG. Is this TEP in reference to https://github.com/tektoncd/experimental/tree/main/task-loops#task-loop-extension? and is it about bringing that in as base functionality rather than via a customtask?

[bobcatfish]

/assign

[jerop]

Hey @jerop thanks for bringing this up on the alternate WG.

hi @tlawrie 👋🏾

Is this TEP in reference to https://github.com/tektoncd/experimental/tree/main/task-loops#task-loop-extension?

yes, this is related to that custom task

and is it about bringing that in as base functionality rather than via a customtask?

it's mostly about discussing where we want the looping construct to go -- it could start with making it stable and maybe eventually adding it to the pipelines api

wrote another tep to establish the graduation requirements and path for custom tasks in #523

in this pr, we're specifically defining the problem statement - will follow up with another pr with a proposal after we agree on the goals, non-goals, use cases and requirements

i saw this issue - boomerang-io/community#23 - from your project and i'm eager to hear the realworld use cases that you're working to solve using the counter task type discussed there

[bobcatfish]

Nice problem statement!!

/approve

[tlawrie]

Hi @jerop awesome. Ill look into this a bit further this week and also ask the boomerang-io/roadmap community around the use cases for looping.

I believe our main use case is around looping for a group of tasks. I.e. in a workflow loop task 3 + 4 + 5.

[tlawrie]

Hi @jerop my vote is to still keep in the generation of x TaskRuns based on a prior TaskRuns Results if possible. That is the use case that most relates to our usage.

[jerop]

Hi @jerop my vote is to still keep in the generation of x TaskRuns based on a prior TaskRuns Results if possible. That is the use case that most relates to our usage.

The use cases we have in the TEP right now could also use the dynamic generation using Results from previous TaskRuns (included a description of this in each of the use cases). However, we may benefit from taking an iterative approach where the initial version uses user-supplied Parameters only then we explore the dynamic generation as the next step considering the feedback that we gather. So the dynamic generation is definitely not out of the picture.

Added this item to the API WG agenda so we can discuss further on Monday (cc @imjasonh @bobcatfish @vdemeester)

[jerop]

/assign @pritidesai (thanks!)

[pritidesai]

@Tomcli please review the proposal here and let us know if we are missing anything based on kubeflow use case, thanks 🙏

[pritidesai]

/assign

[Tomcli]

thanks @pritidesai @jerop I think this is a great start to add the looping support. Looking forward to the new Matrix/looping syntax.

[jerop]

@bobcatfish @tlawrie @imjasonh @squee1945 @vdemeester @Tomcli @pritidesai thank you for the reviews and discussions at the API WG 🤗

Updated the TEP by adding the following requirements:

• The Parameters in the matrix can use Results from previous TaskRuns or Runs to dynamically generate TaskRuns or Runs from a given PipelineTask.
• Configuring the maximum number of TaskRuns or Runs generated in a given matrix should be supported, with a
  default value provided.

We will implement the feature iteratively, with the dynamic generation using Results coming later (will be mentioned in the proposal section in the next PR).

Also highlighting that dynamic generation of TaskRuns or Runs from a matrix means that different executions of a given Pipeline may generate different number of TaskRuns or Runs based on the size of the Results. It does not mean that there's a feed of Results that generate TaskRuns or Runs that may never stop - note that this is not possible because Tekton Pipelines is a DAG and parent PipelineTasks have to be completed (thus no longer producing Results) before a child PipelineTask is executed.

[bobcatfish]

sgtm!

[tlawrie]

@jerop fabulous work getting it all aligned. Thank you.

[pritidesai]

thank you @jerop for putting this together! Though I am listed as one of the authors, its all your efforts 🙏 and approving them 😄

/approve

I am very excited to design and implement this. I am a little nervous with the term combination of parameters specially when it's compared to experimental task loop (iterating over a list of items). But it can devised while designing it.

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bobcatfish, pritidesai, vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• teps/OWNERS [bobcatfish,pritidesai,vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[imjasonh]

/lgtm