Helm template validation in our CI/CD

[consideRatio]

Description

Rendering Helm templates is a cheap and easy test that I think would help us catch some potential problems.

We could also use the kubeval plugin for helm or the more rigorous --validate flag, but the latter requires access to a k8s cluster - either the one we actually use or a local CI cluster. These help us validate the rendered templates themselves.

Benefit

Doing this would ensure that we don't have any configuration in our charts that is going to clash with what Kubernetes expects - by validating quickly we can fail early rather than later on as a part of a deployment.

Implementation details

We could add a GitHub action that does a simple helm template command. This wouldn't be a rigorous validation, but would be a way to quickly check that the configuration was reasonable and didn't have errors.

We could use dummy values such as proxy.secretToken if needed to render without our secret config.

For a more rigorous test, we could also investigate using the helm kubeval plugin.

Here's how it is used in the mybinder.org deployment:

https://github.com/jupyterhub/mybinder.org-deploy/blob/2603737e02c6a32632a06ab0bad6608235cc86c0/.github/workflows/lint-validate.yml#L93-L136

Tasks to complete

• Decide which of the two approaches above is best to implement
• Implement the ideas in Helm template validation in our CI/CD #279 (comment)
• Add GitHub action to do this!

[choldgraf]

As a part of #790 we identified that implementing this would have been a quick way to avoid the problem we encountered of using an incorrect configuration structure. I'm going to bump this one on the queue a bit since it seems like a quick win.

[consideRatio]

I'm become quite opinionated at this point in time how to go about this based on some additional experience, this is my suggestion.

1. Whenever helm template is used in any way, we rely on dummy values as replacement of secrets if needed to get a valid config, which makes all of this something that can run without decrypting sensitive secrets etc.
2. We use helm template without --validate to start with to ensure we can render the templates at all.
3. As a stretch goal, we use helm template --validate against a temporary k8s cluster to validate the resources are valid as well against a modern version of k8s.
   • Note: CRDs needs to be registered ahead of time, otherwise the k8s api-server sais custom resources, for example in use by dask-gateway and cert-manager, will be invalid.

This is a task I'm willing to tackle, but would love @sgibson91's review assistance!

[sgibson91]

I'm loving part 1 🙌🏻 I think the mybinder.org-deploy repo has something along these lines (maybe just for linting though?) if you wanted to crib from there.

[choldgraf]

I believe that in our team meeting today @consideRatio agreed to pick this one up! Just encoding that decision here :-) let us know how we can help with this @consideRatio , I'm excited to improve the resilience of our deployment infrastructure!