Discussion: Garden Core Docker Image Lifecycle

[stefreak]

This is a request for comments on a new Garden Docker container tagging scheme – if you see something that could be improved, please comment and I'll try to incorporate the feedback.

Discord link: https://discord.com/channels/817392104711651328/1088795679159222292/1088890739569729666

Problem description

The gardendev/garden image tags on dockerhub and the build process are currently a little bit hard to understand.

Here are some of the problems with it:

• Tag naming
  • There is a latest tag – but what does it refer to? The main branch? The latest stable release? Bonsai? Alpine or Debian?
  • There are a lot of tags that seem to be full git commit hashes. This is confusing when users want to find tags that are relevant to them.
  • There is a main tag – but again, is it alpine or debian? – oh there is also main-buster, so it must be alpine?
  • And then we also have latest-aws, latest-gcloud, …… not clear by looking at the tag when those will be updated
• The build process is very hard to understand (for me personally) – we could improve this on the way
• We have duplicate code in the cloud tool installation
  • This leads to fixes geting applied to one place but are forgot in another place (e.g. compare the Azure installer in full.Dockerfile with the Azure installer in azure.Dockerfile (installing kubelogin differently)
• We download & install tools without properly checking hashes / verifying public keys
  • For example aws-iam-authenticator: https://github.com/garden-io/garden/blob/main/support/aws-gcloud.Dockerfile#L7
  • Another example is IBM installer: https://github.com/garden-io/garden/blob/main/support/full.Dockerfile#L31
• Because we’re parametrizing base containers Dependabot can’t update the images automatically (e.g. if there is a new alpine system version in the node base image)
• Garden CLI sometimes installs containers into the kubernetes cluster (for example https://hub.docker.com/r/gardendev/k8s-util/tags), and references them by tag – this is not secure, because someone could maliciously replace existing tags, and therefore change the behaviour of previously released Garden binaries. (I know I am being paranoid here, because only Gardeners have access, but I am thinking about the safety of users who are targets for Government hackers).
  • Solution is actually easy: Garden should always refer to builtin-images via the SHA256 hash, e.g. gardendev/k8s-util:0.5.6@sha256:dce403dc7951e3f714fbb0157aaa08d010601049ea939517957e46ac332073ad instead of just gardendev/k8s-util:0.5.6 – this guarantees that if the garden binary has not been tampered with, then the containers have not been tampered with either.
• Some users are requesting unprivileged containers, e.g. for use in GitLab ci: add garden as node user #3865

Proposed Solution

• Always reference containers with the full SHA256 hash in core, to make it impossible to maliciously change the behaviour of previous garden releases
• Verify all the downloads in Dockerfiles using sha hashes or by verifying signatures using public keys, to make the images extra secure. We owe this attention of detail to our users and customers, who depend on us to keep their supply chain secure.
• Make Dockerfiles in the support/ directory unparametrized, so that Dependabot can update base images. Also refer to base images by sha hash there, so we have an audit trail of all the changes. Dependabot will take care of managing the sha hashes like we already do in Platform
• Merge all the Garden Dockerfiles into one multi-stage Dockerfile, so that we can refer to different stages without pushing a tag with the git commit hash.
  e.g.

FROM alpine…. as garden-alpine
…

FROM garden-alpine as garden-gcloud-alpine
…

FROM garden-gcloud-alpine as garden-gcloud-aws-alpine
…

• To keep it all DRY (and to avoid mistakes where we fix one image, but forget the other), add shared cloud tool install scripts that we can call from the different stages that need that specific cloud tool. e.g. install_aws.sh, install_azure.sh etc.
• Change our Tag naming conventions

Name	Garden version	Cloud tools	Default user
acorn-{flavor}, 0.12-{flavor}, 0.12.{patch-version}-{flavor}	Latest garden 0.12 stable	–	root
acorn-{flavor}-unprivileged, 0.12-{flavor}-unprivileged, 0.12.{patch-version}-{flavor}-unprivileged	Latest garden 0.12 stable	–	garden
acorn-edge-{flavor}, 0.12-edge-{flavor}	Latest garden 0.12 edge	–	root
acorn-edge-{flavor}-unprivileged, 0.12-edge-{flavor}-unprivileged	Latest garden 0.12 edge	–	garden
bonsai-{flavor}, 0.13-{flavor}, 0.13.{patch-version}-{flavor}	Latest garden 0.13 stable	–	root
bonsai-{flavor}-unprivileged, 0.13-{flavor}-unprivileged, 0.13.{patch-version}-{flavor}-unprivileged	Latest garden 0.13 stable	–	garden
bonsai-edge-{flavor},0.13-edge-{flavor}	Latest garden 0.13 edge	–	root
bonsai-edge-{flavor}-unprivileged, 0.13-edge-{flavor}-unprivileged	Latest garden 0.13 edge	–	garden

{flavor} is one of alpine, debian-buster, debian-bullseye

and then we apply this tagging scheme to the following images:

• gardendev/garden

• gardendev/garden-gcloud

• gardendev/garden-aws

• gardendev/garden-azure

• gardendev/garden-gcloud-aws

• gardendev/garden-gcloud-aws-azure

• etc…

• I propose that we remove the older tags once 0.12 is EOL

[edvald]

Thanks for writing this up! Just one note: I somehow had in mind that we'd end up dropping all the repos except gardendev/garden. Do we think the tags then become egregiously long? :P

[stefreak]

@edvald the repos and the tags are two different stories –

we can drop the Cloud provider tools containers, but even if we do that the current tags in gardendev/garden are still confusing for the reasons listed above.

I think the tags wouldn't be that long– some examples:

• gardendev/garden:bonsai-alpine
• gardendev/garden:0.12-edge-alpine

Some of them will be pretty long but they won't be the most commonly used:

• gardendev/garden:bonsai-edge-alpine-unprivileged

While it is a hassle to maintain the cloud provider tools variants, they are very handy for example when building a GitLab CI pipeline. My personal opinion is that we should actually keep them for this reason. But we should definitely make them easier to maintain– for example by making sure that we do not duplicate the installation procedures in our code and have some test coverage.

[vvagaytsev]

@stefreak has this been completely done via #4239? If so, then feel free to close this :)

This is also marked as a breaking change. Please put the necessary comments to the breaking changes list in the Bonsai migration guide.

[stefreak]

@vvagaytsev No, this is not fully finished yet; it is just the first step.

Will mention this 👍 Thank you

[vvagaytsev]

Ok, thank you!

[stefreak]

All done, only thing missing is "Verify all the downloads in Dockerfiles using sha hashes"

[vvagaytsev]

@stefreak any plans to implement the last step soon?

[stefreak]

@vvagaytsev Yeah it's only a minor change of the Dockerfiles actually 👍

[vvagaytsev]

@stefreak, cool, thanks! Let's try to get it done soon :)

[stefreak]

@vvagaytsev you can find the PR at #4826