Question: Best practice for storage and cleanup in a Tekton CI pipeline

[djfinnoy]

I've patched together a pipeline using tasks from the catalog, and I'm noticing some areas where I'm probably not "doing it right".

The pipeline gets triggered via Github, and it simply clones the triggering repo, builds and pushes its dockerimage, and runs Helm to update an ArgoCD application resource.

The idea is that we'll have dozens of github repositories that call on this pipeline to deploy themselves into k8s.
Right now, I'm using a volumeClaimTemplate in the PipelineRun, and im noticing that this leaves a bit of clutter in the cluster;
Each PipelineRun leaves a PersistentVolumeClaim, after it completes; that's bad. Would it make more sense to have a permanent pvc, and if so, how to I ensure that it doesn't run out of space?

I'm also noticing that the PipelineRuns and their associated resources stick around indefinitely after completion; this is handy with respect to the Tekton Dashboard, but I'm thinking I'll need some sort of cleanup job as well.

Can anyoine point me toward resources / writeups on how to deal with storage and cleaning when using Tekton for CI?

[ghost]

/kind question

Thanks for the issue! I'm going to convert this over to a discussion thread so that it doesn't get chewed up by our issue automation.

Each PipelineRun leaves a PersistentVolumeClaim, after it completes; that's bad. Would it make more sense to have a permanent pvc, and if so, how to I ensure that it doesn't run out of space?

Worth noting: the PVCs created from a volumeClaimTemplate will be cleaned up when the PipelineRun is deleted. As you mentioned the runs get kept around because their logs and other info is useful for the dashboard. The Tekton Results project (github.com/tektoncd/results) has been started with the goal of moving that historical info into a database so that runs + their dependencies can be safely deleted without losing a bunch of data.

Using a permanent PVC would be another alternative way to go but Pipelines doesn't help out much with that approach - storage management is largely left up to the user because requirements around persistentce can vary dramatically from one org to another.

A totally different approach would be to "co-locate" the Tasks that share storage into a single Pod so that they can use an emptyDir to shuttle data around quickly. We're looking at that through a couple different ongoing designs but the one most closely related is tektoncd/community#318

I'm also noticing that the PipelineRuns and their associated resources stick around indefinitely after completion; this is handy with respect to the Tekton Dashboard, but I'm thinking I'll need some sort of cleanup job as well.

The Tekton Results project is designing a cleanup feature. In a nutshell: It'll observe your runs and consider them candidates for deletion after they complete (+ a grace period). The design doc is here: https://github.com/tektoncd/community/blob/main/teps/0052-tekton-results-automated-run-resource-cleanup.md