-
Notifications
You must be signed in to change notification settings - Fork 221
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This commit adds the proposal of adding CEL in WhenExpression Signed-off-by: Yongxuan Zhang [email protected]
- Loading branch information
1 parent
474a939
commit e721063
Showing
2 changed files
with
337 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,336 @@ | ||
| --- | ||
| status: implementable | ||
| title: CEL in WhenExpression | ||
| creation-date: '2023-09-28' | ||
| last-updated: '2023-09-28' | ||
| authors: | ||
| - '@jerop' | ||
| - '@chitrangpatel' | ||
| - '@Yongxuanzhang' | ||
| collaborators: [] | ||
| --- | ||
|
|
||
| # TEP-0145: CEL in WhenExpression | ||
|
|
||
| <!-- toc --> | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| - [Use Cases](#use-cases) | ||
| - [Goals](#goals) | ||
| - [Non Goals](#non-goals) | ||
| - [Related Work](#related-work) | ||
| - [Proposal](#proposal) | ||
| - [Syntax](#syntax) | ||
| - [Validation](#validation) | ||
| - [Example](#example) | ||
| - [Design Evaluation](#design-evaluation) | ||
| - [Simplicity](#simplicity) | ||
| - [Flexibility](#flexibility) | ||
| - [Conformance](#conformance) | ||
| - [User Experience](#user-experience) | ||
| - [Drawbacks](#drawbacks) | ||
| - [Alternatives](#alternatives) | ||
| - [Implementation Plan](#implementation-plan) | ||
| - [Upgrade and Migration Strategy](#upgrade-and-migration-strategy) | ||
| - [Implementation Pull Requests](#implementation-pull-requests) | ||
| - [References](#references) | ||
| <!-- /toc --> | ||
|
|
||
| ## Summary | ||
|
|
||
| This TEP proposes to add CEL (Common Expression Language) support in `when` expressions to enable more powerful conditional execution and improve the user experience by offering a more concise way of defining conditionals. | ||
|
|
||
|
|
||
| ## Motivation | ||
|
|
||
| CEL is a non-Turing complete language designed for simplicity, speed, safety, and portability. Tekton has adopted CEL expressions in [Trigger] to expose parts of the request, and some custom functions to make matching easier, and [CELCustomRun] is implemented to experiment CEL in Tekton Pipelines without adding CEL directly to the Tekton API surface. | ||
|
|
||
| There are several feature requests in the community which can be met if CEL is supported in Tekton Pipeline: | ||
| - [#3591] calls for supporting ANY match behaviour in `WhenExpressions` to execute the task if any `params` is not empty. | ||
| - [#3149] requires to use CEL to split the string instead of adding an additional task to do the work. | ||
|
|
||
| Take a look at current [WhenExpressions]: | ||
|
|
||
| ```yaml | ||
| when: | ||
| - input: "$(params.path)" | ||
| operator: in | ||
| values: ["README.md"] | ||
| ``` | ||
| Current `WhenExpression` requires users to define three fields: `input`, `operator` and `values`. but CEL expressions can achieve the same functionality with a single string. Additionally, the current operator only supports `in` and `notin`, which can be limiting for complex use cases. For example, users will need to figure out [hacky workarounds][#3591] to support any in their case. | ||
|
|
||
| For these reasons, `WhenExpressions` would be a good place to start introducing CEL in Tekton Pipelines. This would allow users to write more concise and powerful conditional expressions, and it could be extended to other fields in the future. | ||
|
|
||
| ### Use Cases | ||
|
|
||
| - Cover all the functionalities of current `WhenExpression` and offer a simpler way of having the conditional execution | ||
| - Support ANY: e.g. Execute the task with if any params is not empty | ||
| - Support numeric operators such as GreaterThan/LessThan, e.g. Execute the task if test coverage is large than a certain number. | ||
|
|
||
| ### Goals | ||
| - Support CEL in `WhenExpression`. | ||
|
|
||
| ### Non-Goals | ||
| - Support CEL in Pipeline API fields other than `WhenExpression`. | ||
|
|
||
| ### Related Work | ||
|
|
||
| Similar expression evalutation has been supported in other CI/CD tools. | ||
| - [Argo][Argo]: `Workflows` support conditional execution using a when property to specify whether to run or skip a `Step` ([example][argo example]). | ||
| - [GitHub Actions][GitHub Actions]: supports evaluating expressions in `if` of a `step`. | ||
| - [Jenkins][Jenkins]: executes the stage when the specified Groovy expression evaluates to true. | ||
| - [Spinnaker][Spinnaker]: Uses string expressions, `Stages` only run when expressions evaluate to `True`. | ||
|
|
||
|
|
||
| ## Proposal | ||
|
|
||
| We propose to add a `cel` field under `WhenExpression` and the type is `string`. Same as [CelCustomRun][CELCustomRun], we propose to use [cel-go][cel-go] for the expression evaluation. So users can craft any valid CEL expression as defined by the [cel-spec][cel-spec] language definition. | ||
|
|
||
|
|
||
| ### Syntax | ||
|
|
||
| In this section we will show how CEL can support the [use cases](#use-cases). | ||
|
|
||
| #### Use CEL to replace current WhenExpression | ||
| ```yaml | ||
| # current WhenWxpressions | ||
| when: | ||
| - input: "foo" | ||
| operator: "in" | ||
| values: ["foo", “bar”] | ||
| - input: "duh" | ||
| operator: "notin" | ||
| values: ["foo", “bar”] | ||
| # with cel | ||
| when: | ||
| - cel: "\”foo\” in [\"foo\", \“bar\”]” | ||
| - cel: "\”duh\” not in [\"foo\", \“bar\”]” | ||
| ``` | ||
|
|
||
| **Note:** The `WhenExpressions` contains a list of `WhenExpression`, if any `WhenExpression` is false it will return false. So we’re using ANY to evaluate the list of `WhenExpressions`. And this will remain the same for this proposal. For each WhenExpression, there will be one CEL expression which can fully replace the current when expression functionalities. | ||
|
|
||
| #### Use CEL to Support ANY operator | ||
|
|
||
| This is the workaround from [#3591][#3591], | ||
| ```yaml | ||
| when: | ||
| - input: "$(params.param1)$(params.param2)" | ||
| operator: notin | ||
| values: [""] | ||
| ``` | ||
|
|
||
| With CEL we can use the sraightforward expression to support this case. | ||
| ```yaml | ||
| when: | ||
| - cel: "\"$(params.param1)\" != \"\" || \"$(params.param2)\" != \"\"" | ||
| ``` | ||
|
|
||
| #### Use CEL to Support CreaterThan | ||
|
|
||
| This is not possible with current `WhenExpression`. With CEL we can do: | ||
|
|
||
| ```yaml | ||
| when: | ||
| - cel: "\"$(tasks.unit-test.results.test-coverage)\" >= 0.9" | ||
| ``` | ||
|
|
||
| [Note][numeric note] that currently there are no automatic arithmetic conversions for the numeric types (int, uint, and double), so the the emitted results must be the same numeric type to make sure the evluation works. | ||
|
|
||
|
|
||
| ### Validation | ||
|
|
||
| The new fields will be gated by a new feature flag, `enable-concise-api`, which defaults to `"false"` while the feature is in alpha. | ||
|
|
||
| The validation webhook will validate either the current when `input`+`operator`+`values` or `cel` is used, users cannot use both at the same time for 1 `WhenExpression`, but both can be used in the `WhenExpressions` list if the feature is enabled. | ||
|
|
||
|
|
||
| ### Example | ||
|
|
||
| This is an E2E example: | ||
|
|
||
| ```yaml | ||
| apiVersion: tekton.dev/v1 | ||
| kind: PipelineRun | ||
| metadata: | ||
| generateName: guarded-pr- | ||
| spec: | ||
| pipelineSpec: | ||
| params: | ||
| - name: path | ||
| type: string | ||
| description: The path of the file to be created | ||
| - name: branches | ||
| type: array | ||
| description: The list of branch names | ||
| workspaces: | ||
| - name: source | ||
| description: | | ||
| This workspace is shared among all the pipeline tasks to read/write common resources | ||
| tasks: | ||
| - name: create-file | ||
| when: | ||
| - cel: "\"$(params.path)\" == \"README.md\"" | ||
| workspaces: | ||
| - name: source | ||
| workspace: source | ||
| taskSpec: | ||
| workspaces: | ||
| - name: source | ||
| description: The workspace to create the readme file in | ||
| steps: | ||
| - name: write-new-stuff | ||
| image: ubuntu | ||
| script: 'touch $(workspaces.source.path)/README.md' | ||
| - name: check-file | ||
| params: | ||
| - name: path | ||
| value: "$(params.path)" | ||
| workspaces: | ||
| - name: source | ||
| workspace: source | ||
| runAfter: | ||
| - create-file | ||
| taskSpec: | ||
| params: | ||
| - name: path | ||
| workspaces: | ||
| - name: source | ||
| description: The workspace to check for the file | ||
| results: | ||
| - name: exists | ||
| description: indicates whether the file exists or is missing | ||
| steps: | ||
| - name: check-file | ||
| image: alpine | ||
| script: | | ||
| if test -f $(workspaces.source.path)/$(params.path); then | ||
| printf yes | tee $(results.exists.path) | ||
| else | ||
| printf no | tee $(results.exists.path) | ||
| fi | ||
| - name: echo-file-exists | ||
| when: | ||
| - cel: "\"$(tasks.check-file.results.exists)\" in [\"yes\"]" | ||
| taskSpec: | ||
| steps: | ||
| - name: echo | ||
| image: ubuntu | ||
| script: 'echo file exists' | ||
| finally: | ||
| - name: finally-task-should-be-executed | ||
| when: | ||
| - cel: "\"$(tasks.echo-file-exists.status)\" == \"Succeeded\"" | ||
| - cel: "\"$(tasks.status)\" == \"Succeeded\"" | ||
| - cel: "\"$(tasks.check-file.results.exists)\" == \"yes\"" | ||
| - cel: "\"$(params.path)\" == \"README.md\"" | ||
| taskSpec: | ||
| steps: | ||
| - name: echo | ||
| image: ubuntu | ||
| script: 'echo finally done' | ||
| params: | ||
| - name: path | ||
| value: README.md | ||
| - name: branches | ||
| value: | ||
| - main | ||
| - hotfix | ||
| workspaces: | ||
| - name: source | ||
| volumeClaimTemplate: | ||
| spec: | ||
| accessModes: | ||
| - ReadWriteOnce | ||
| resources: | ||
| requests: | ||
| storage: 16Mi | ||
| ``` | ||
|
|
||
| ## Design Evaluation | ||
|
|
||
| ### Simplicity | ||
|
|
||
| The proposal helps to reduce the verbosity in Tekton by using 1 string of CEL expression instead of 3 (input+operator+values). | ||
|
|
||
| ### Flexibility | ||
|
|
||
| CEL is a powerful expression language that can be used to perform a wide variety of tasks in CI/CD pipelines. This increases the flexibility of `WhenExpression` in Tekton to handle more complex conditions. | ||
|
|
||
| ### Conformance | ||
|
|
||
| The proposal will introduce an additive change to the current API. It does not impact conformance in the inital release. | ||
|
|
||
| ### User Experience | ||
|
|
||
| This proposal should help to improve the user experience by | ||
| - simplifying the syntax for `WhenExpression` | ||
| - support more complex conditional execution instead of hacky workaround | ||
|
|
||
| ### Drawbacks | ||
|
|
||
| Users need to be familar with CEL expression syntax to use it properly, this may be a learning curve but they can still choose to use current `WhenExpression` syntax for simple cases. | ||
|
|
||
|
|
||
| ## Alternatives | ||
|
|
||
| ### Support More Operators in current WhenExpression | ||
| Tekton can support more operators from [k8s apimachinery][k8s apimachinery] in `WhenExpression`'s `Operator`. | ||
|
|
||
| ## Implementation Plan | ||
|
|
||
| ### Update existing fields in v1 API | ||
|
|
||
| Add `CEL` to `WhenExpression`: | ||
| ```go | ||
| type WhenExpression struct { | ||
| // Input is the string for guard checking which can be a static input or an output from a parent Task | ||
| Input string `json:"input"` | ||
|
|
||
| // Operator that represents an Input's relationship to the values | ||
| Operator selection.Operator `json:"operator"` | ||
|
|
||
| // Values is an array of strings, which is compared against the input, for guard checking | ||
| // It must be non-empty | ||
| // +listType=atomic | ||
| Values []string `json:"values"` | ||
|
|
||
| // CEL is a CEL expression to evaluate, if follows the cel-spec language definition | ||
| CEL string `json:"cel",omitempty` | ||
| } | ||
| ``` | ||
|
|
||
| ### Upgrade and Migration Strategy | ||
|
|
||
| This feature will be introduced in alpha with a dedicated feature flag which is disabled by default. | ||
|
|
||
| When the feature is promoted to beta, the dedicated feature flag will be enabled by default. | ||
|
|
||
| When the feature is promoted to stable, the dedicated feature flag will be removed and the feature will be available by default. | ||
|
|
||
| When we move to v2 API, we can consider removing the `input`, `operator` and `values` and only keep `cel`. | ||
|
|
||
| ### Implementation Pull Requests | ||
|
|
||
| TODO | ||
|
|
||
| ## References | ||
| - [Tekon Trigger][Trigger] | ||
| - [CELCustomRun][CELCustomRun] | ||
| - [cel-go][cel-go] | ||
|
|
||
| [Trigger]: https://tekton.dev/docs/triggers/cel_expressions/ | ||
| [CELCustomRun]: https://github.com/tektoncd/experimental/tree/master/cel | ||
| [#3591]: https://github.com/tektoncd/pipeline/issues/3591 | ||
| [#3149]: https://github.com/tektoncd/pipeline/issues/3149 | ||
| [WhenExpressions]: https://github.com/tektoncd/pipeline/blob/main/docs/pipelines.md#guard-task-execution-using-when-expressions | ||
| [Argo]:https://github.com/argoproj/argo/tree/master/examples#conditionals | ||
| [argo example]: https://github.com/argoproj/argo/blob/master/examples/conditionals.yaml | ||
| [GitHub Actions]: https://docs.github.com/en/actions/learn-github-actions/expressions#example-expression-in-an-if-conditional | ||
| [Jenkins]: https://www.jenkins.io/doc/book/pipeline/syntax/#when | ||
| [Spinnaker]: https://www.spinnaker.io/guides/user/pipeline/expressions/#dynamically-skip-a-stage | ||
| [cel-go]:https://github.com/google/cel-go | ||
| [cel-spec]:https://github.com/google/cel-spec/blob/master/doc/langdef.md | ||
| [numeric note]: https://github.com/google/cel-spec/blob/master/doc/langdef.md#numeric-values | ||
| [k8s apimachinery]:https://pkg.go.dev/k8s.io/apimachinery/pkg/selection#Operator |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters