[feature] Allow time to live to be set on pipelines individually while supporting the persistance agent to do the GC.

[NikeNano]

Feature Area

There have been multiple issues related to time to live(ttl) for pipelines, #5545, #5234. A common solution is to use set_ttl_seconds_after_finish which don't respect the persistence agent and can resolve in that data is lost for the users.

In order to allow users to
configure the ttl for pipelines the suggested solution is to set TTL_SECONDS_AFTER_WORKFLOW_FINISH [https://github.com//issues/5234#issuecomment-805599988]
but this will set for all pipelines.

What feature would you like to see?

To support the persistence agent while allowing users to set time to live on pipeline level, I suggest that we add the option to the pipeline spec and
update the persistence agent to support this.

What is the use case or pain point?

Today either the users have to set the same TTL for all pipelines(through TTL_SECONDS_AFTER_WORKFLOW_FINISH in the persistiance agent) or
use set_ttl_seconds_after_finish which might result in data lose.

Is there a workaround currently?

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[NikeNano]

ping @Bobgy WDYT?

[zijianjoy]

Hello @NikeNano , may I ask for the use case for this feature? For example, what kind of users need to configure TTL for pipeline? Usually it is the cluster admin who control the lifecycle of workflow. And can the pipeline creators themselves have the control over TTL in your mind?

[NikeNano]

My experience is that I had some pipelines which I wanted to clean up quicker(especially recurring runs) and didn't wanted to set the same default for all pipelines(only have experience from smaller team where we have to do it all ourself).

Another use case is to release expensive hard where described here: #5234 (comment). Maybe this can be solved in other ways though?

[SebBlin]

As far as I understand it, latest releases of KFP come with an updated release of Argo, that as update the way to specify a TTL in workflow (see https://argoproj.github.io/argo-workflows/fields/#ttlstrategy).
I made a PR to correct it : #7141

[NikeNano]

As I understand this is now resolved so will close this.