Improve docker in docker cleanup

[BenTheElder]

Working on this today so we can more freely enable dind (along with #5436), in short:

• Move cleanup code out of bash so we can write something more sophisticated
• Implement more robust cleanup of images we aren't trying to cache.
  • We want to keep pulled base images around to reduce flakiness, but not built images

To implement more robust cleanup:

• We can't reliably detect if an image was pulled, but we can tell how old an image is. Base images should typically become old (say, older than 1 week), built images should not.
  • Pro: simple, catches images with tags we haven't hard-coded (unlike say, kube-build)
  • Con: we could let images slip if cleanup fails to run on a node for too long after a build. This could probably be solved in the future by periodically flushing ALL cache on the node. xref improve caching on prow #5436

/area images
/priority important-soon
/assign

[BenTheElder]

FYI @rramkumar1 I think we can enable dind for your job and then follow-up with this. We'll try setting up ingress-gce with it's own data root at least.

[rramkumar1]

@BenTheElder SGTM

[BenTheElder]

fixed in #6106

we might want to do some follow-ups though

[BenTheElder]

This works (#6106 (comment)), but it's not complete. We should also sometimes clean up older images, eg some 10 day old kubemark stuff falls through now because it's older than the threshold.

We could randomly delete all images with a low probability 🤔

[BenTheElder]

/close

Will open a new issue if we have problems, the current behavior is pretty great for now.