GC old tekton resources periodically

[Bobgy]

Discovered in GoogleCloudPlatform/kubeflow-distribution#128.

Currently all triggered PipelineRun and other tekton resources accumulate for ever.
We need to GC tekton resources.

Tekton community has an issue tektoncd/experimental#479 tracking this.
But there's no ready-to-use solution yet.

Before sth comes out of above issue, we should have at least a bash script that cleans up these resources.

[Bobgy]

/cc @jlewi

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/engprod	0.86
platform/gcp	0.86
kind/bug	0.67

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[Bobgy]

I wrote a very simple bash script for this purpose:

#!/bin/bash

set -e

# This script deletes k8s resources in batches of 100.
# usage: ./rm_k8s_res.sh <input-file-path>
# for example: ./rm_k8s_res.sh ./input.txt
# get the input file by
# ```
# kubectl get <resource-type> --sort-by=.metadata.creationTimestamp -o name > input.txt
# ```
# input file is a list of resource names to be deleted ordered by creation timestamp.
# if only want to delete old resources, you can manually remove the last section to keep latest resources.

input="$1"
resource="$2"
echo "Deleting $2 from file $input"
names=()
while IFS=" " read -r name age
do
  names+=($name)
  if [ ${#names[@]} -gt 100 ]; then
    echo "Deleting ${names[@]}"
    kubectl delete ${names[@]} || echo "already deleted"
    names=()
  fi
done < "$input"
if [ ${#names[@]} -gt 0 ]; then
  echo "Deleting ${names[@]}"
  kubectl delete ${names[@]} || echo "already deleted"
fi

[Bobgy]

After deleting until ~100 pipelineruns left in the cluster, https://kf-ci-v1.endpoints.kubeflow-ci.cloud.goog/tekton/#/namespaces/auto-deploy/pipelineruns can successfully load pipelineruns now.

[jlewi]

@Bobgy FYI this is the script we use to cleanup argo workflows.

testing/py/kubeflow/testing/cleanup_ci.py

Line 112 in d4394ea

def cleanup_workflows(args):

[jlewi]

#767 has a very simple program. The program can be used with arbitrary K8s resources and can delete things based on minimum age.

We need to turn that into a regularly running job with a suitable minimum age (e.g. 72 hours).

We can use the same pattern as we do for groups sync:
https://github.com/kubeflow/internal-acls/tree/master/google_groups/manifests

• Create a deployment that runs it periodically in a loop with some delay
• Run git-sync in a sidecar to fetch the config from the repo

We could also use a cron job.

[Bobgy]

@jlewi I think it will be easier to debug if the tool is called from a periodic prow job, that makes maintenance easier -- finding logs, health status, change commands, etc

[Bobgy]

I created #813 to dockerize the tool @jlewi built, next step would be adding a cronjob to run this in kf-releasing cluster.