This TEP describes an alternative way to run Pipelines composed of Ta…

…sks and Workspaces without any changes for an author and without the need for external storage to back the Workspaces. Without external storage, problems with scheduling of Pods that use the same volume is avoided.

teps/0046-pipelinerun-in-a-pod.md

@@ -0,0 +1,94 @@
+---
+status: proposed
+title: PipelineRun in a Pod
+creation-date: '2021-01-26'
+last-updated: '2021-01-26'
+authors:
+- '@jlpettersson'
+---
+
+# TEP-0046: PipelineRun in a Pod
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+    - [Goals](#goals)
+    - [Non-Goals](#non-goals)
+    - [Use Cases (optional)](#use-cases-optional)
+- [Requirements](#requirements)
+- [References (optional)](#references-optional)
+
+## Summary
+
+This TEP describes an alternative way to run a Pipeline composed of Tasks sharing data through Workspaces.
+
+The way we run Tasks in a Pipeline that share data through a Workspace today has several problems, some
+of them are related to the need to use Kubernetes Persistent Volumes to back Workspaces and to
+schedule the different TaskRun Pods in an appropriate way, especially to allow multiple of them to run
+concurrently while accessing the same Workspace volume.
+
+[#3638: Investigate if we can run whole PipelineRun in a Pod](https://github.com/tektoncd/pipeline/issues/3638)
+
+[Design doc: Task parallelism when using workspace](https://docs.google.com/document/d/1lIqFP1c3apFwCPEqO0Bq9j-XCDH5XRmq8EhYV_BPe9Y)
+
+## Motivation
+
+Currently the only way to run a Pipeline composed of Tasks is that each task is executed in its own Pod. If you combine
+Tasks in a Pipeline and they need to share data (beyond a simple
+[result](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-results)),
+you'll need to provision a PVC or do some other similar, cloud specific thing,
+to [make a volume available](https://github.com/tektoncd/pipeline/blob/master/docs/workspaces.md#specifying-volumesources-in-workspaces)
+that can be shared between them.
+
+Some problems with this:
+* PVCs add additional overhead, both in speed, in resource costs and in management.
+* PVCs is hard to share between multiple Pods concurrently - needs appropriate scheduling of the Pods.
+* PVCs is difficult to share in subsequently run Pods that might be scheduled to a different AZ than where the
+  volume is located.
+
+### Goals
+
+- Make it possible to run whole PipelineRun (including all TaskRuns) within a single Pod so that the workspace can be within the Pod:
+    - At runtime in a PipelineRun
+    - This should not change anything for the author of Tasks or Pipelines
+- No changes in the Pipeline or Task API for authors
+
+### Non-Goals
+
+- To fully handle the Pod resource requests and limits in an appropriate way for the PipelineRun Pod.
+
+### Use Cases (optional)
+
+- A user want to create a typical Build pipeline with git-clone, lint, code-build, unit-tests and multiple
+  parallel integration tests - all Tasks sharing the volume that was populated by git-clone and read by all
+  subsequent tasks to perform "validation". The user would like to see its PipelineRun run to completion as
+  a single unit INSTEAD OF worry about appropriately scheduled taskRun Pods in a busy cluster and need to
+  use Persistent Volumes for the files from git-clone.
+- A user want to run a Build pipeline that starts with two concurrent tasks, git-clone and another task that
+  fetches cached dependencies from a cache and then build the cloned code while reusing the cached dependencies
+  to speed up the build and see the full PipelineRun to be run in a single Pod, on a single Node WITHOUT NEEDING
+  to mount external volumes and worry that its Pipeline is composed of several Pods that might be scheduled to 
+  different cloud Availability Zones and reach a deadlocked state.
+- A user wants to create a Pipeline composed of Tasks from the catalog, to checkout code, run unit tests and upload results,
+  and does not want to incur the additional overhead (and performance impact) of creating
+  volume based workspaces to share data between them in a Pipeline. e.g. specifically
+  cloning with [git-clone](https://github.com/tektoncd/catalog/tree/master/task/git-clone/0.2),
+  running tests with [golang-test](https://github.com/tektoncd/catalog/tree/master/task/golang-test/0.1)
+  and uploading results with [gcs-upload](https://github.com/tektoncd/catalog/tree/master/task/gcs-upload/0.1).
+
+## Requirements
+
+- Pipeline can be composed of multiple Tasks and scheduled to run together in a single Pod:
+    - Must be able to share data without requiring
+      an external volume (i.e. probably the containers that make them up are run within
+      the same pod)
+- Tasks should be able to run concurrently if the Pipeline author want that
+
+## References (optional)
+
+* [Design doc: Task parallelism when using workspace](https://docs.google.com/document/d/1lIqFP1c3apFwCPEqO0Bq9j-XCDH5XRmq8EhYV_BPe9Y)
+* [#2586: Difficult to use parallel Tasks that share files using workspace](https://github.com/tektoncd/pipeline/issues/2586)
+* [#3049: Affinity assistant only takes first task's resource requirements into account when choosing a node](https://github.com/tektoncd/pipeline/issues/3049)
+* [#3052: Investigate to create a Custom Scheduler to schedule TaskRun pods](https://github.com/tektoncd/pipeline/issues/3052)
+* [#3638: Investigate if we can run whole PipelineRun in a Pod](https://github.com/tektoncd/pipeline/issues/3638)
+* [#3563: Task parallelism and Regional clusters - supported?](https://github.com/tektoncd/pipeline/issues/3563)

teps/README.md

@@ -147,3 +147,4 @@ This is the complete list of Tekton teps:
 |[TEP-0032](0032-tekton-notifications.md) | Tekton Notifications | proposed | 2020-11-18 |
 |[TEP-0035](0035-document-tekton-position-around-policy-authentication-authorization.md) | document-tekton-position-around-policy-authentication-authorization | implementable | 2020-12-09 |
 |[TEP-0036](0036-start-measuring-tekton-pipelines-performance.md) | Start Measuring Tekton Pipelines Performance | proposed | 2020-11-20 |
+|[TEP-0046](0046-pipelinerun-in-a-pod.md) | PipelineRun in a Pod | proposed | 2021-01-26 |