pipeline_conf.set_ttl_seconds_after_finished(seconds=10) NOT WORKING

[prasadkyp7]

• How did you deploy Kubeflow Pipelines (KFP)?
  WE are using full kubeflow deployment on AWS Cloud
• KFP version:

Kubeflow UI is not showing us any version

• KFP SDK version:

kfp 1.4.0
kfp-pipeline-spec 0.1.6
kfp-server-api 1.4.1

Steps to reproduce

1.create a pipeline code
2. set the ttl using pipeline config
pipeline_conf = kfp.dsl.PipelineConf()
x=pipeline_conf.set_ttl_seconds_after_finished(seconds=10)
3. create the pipline using
kfp.compiler.Compiler().compile(loadTrain_pipeline,'loadTrain_pipeline_2.zip',pipeline_conf=x) and upload it in UI.
4. TEST this pipeline by creating a run.

EXPECTED RESULT:
As per setting TTL to 10 seconds , the pods created must be deleted after 10 sec.
But its not deleting the pods, in my case i am creating 1000's pods and as i am not deleting the successful pods Kubeflow UI is crashing.

[NikeNano]

I think this issue could guide you to a solution @prasadkyp7

[Bobgy]

+1 to NikeNano's answer

[prasadkyp7]

we upgraded kubeflow to latest version but still TTL is not working.
Need an urgent help on this

[Bobgy]

@prasadkyp7 can you add more details?
What config did you set? What version are you using etc?

Ref: #5234 (comment) this is my recommendation

[prasadkyp7]

@Bobgy i tried as you suggested , it worked.
But the problem here is . TTL is deleting the pods but not the run details. Because of this , run details SQL Table is becoming full.
when i tried to delete the pipelines using API it deletes both pods and run details. we want TTL to delete both pod and run details. can you suggest a way to do this ?

[Bobgy]

I see, not deleting runs in the DB is the expected behavior.
We don't have TTL feature for the DB. So far, we recommend building a Cron pipeline/job that calls KFP API to list, filter old runs and GC them. Do you see any blockers in that direction?