Garden fails to find helm deployed resources where apiVersions don't match

[khaled]

Bug

Current Behavior

I've got a helm module that has an ingress. The ingress says:

{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}

My cluster runs Kubernetes 1.13, so helm install ends up rendering this as:

apiVersion: extensions/v1beta1

When I run garden dev, garden gets stuck on this:

⠼ frontend                  → Deploying version v-fe774671a4...
   ℹ frontend                  → Waiting for resources to be ready...

Bumping log level to 4, it turns out that garden is failing to find the ingress resource:

Status of Ingress frontend is "missing"                                                                                                                                   
    Kubernetes: Getting API resource info for group networking.k8s.io/v1beta1                                                                                                 
    Status of ConfigMap frontend is "ready"                                                                                                                                   
    Status of Service frontend is "ready"                                                                                                                                     
    Status of Deployment frontend is "ready"                                                                                                                                  
    Status of Ingress frontend is "missing"                                                                                                                                   
    Kubernetes: Getting API resource info for group networking.k8s.io/v1beta1                                                                                                 
    Status of ConfigMap frontend is "ready"                                                                                                                                   
    Status of Service frontend is "ready"                                                                                                                                     
    Status of Deployment frontend is "ready"                                                                                                                                  
    Status of Ingress frontend is "missing"                                                                                                                                   
    Kubernetes: Getting API resource info for group networking.k8s.io/v1beta1                                                                                                 
    Status of ConfigMap frontend is "ready"                                                                                                                                   
    Status of Service frontend is "ready"                                                                                                                                     
    Status of Deployment frontend is "ready"                                                                                                                                  
    Status of Ingress frontend is "missing"

Expected behavior

Garden is able to find the ingress resource.

Workaround

Remove the conditional from the ingress template :(

Suggested solution(s)

It appears that garden uses helm render under the hood, which unfortunately renders the ingress template as:

apiVersion: extensions/v1beta1

In other words, it apparently does not take into account the cluster's kubernetes version. Perhaps .Capabilities.KubeVersion.GitVersion does not have a value when helm render is run.

It turns out though that helm install --dry-run renders the template as we'd expect, i.e. it does take into account the actual cluster's version.

So it seems garden can use helm install --dry-run instead of helm render to get an accurate rendering.

Your environment

▶ garden --version
0.11.12

[eysi09]

Hi @khaled and my apologies for the late reply!

Indeed it looks like the helm template command that we use to render the charts can't access the cluster. From the docs:

Any values that would normally be looked up or retrieved in-cluster will be faked locally. Additionally, none of the server-side testing of chart validity (e.g. whether an API is supported) is done.

The problem with using helm install --dry-run is that chart render happens at build time whereas helm install should happen at runtime and assumes access to the cluster.

In general you should be able to do garden build with out a runtime environment.

I'm wondering how to best address this. Should Garden re-render the charts via the install command before deploying. Thoughts @edvald?

[edvald]

I'm not sure... garden build also generally requires a runtime environment, needs to resolve providers etc. So I think it might be fine to use helm install --dry-run.

[khaled]

I actually agree that garden build should not require an runtime environment, and its output should not be specific to a particular runtime environment (issue #1686, which I opened some time back, is somewhat related to this).

It seems to me that helm charts are fundamentally deployment artifacts, not build artifacts. Perhaps its a quirk of garden's design, but there really shouldn't be a need to render them at build time, I think.

[eysi09]

Yeah, I guess you could argue that the build step for Helm charts should be a no-op and the charts rendered in the deploy step. The former is much easier to implement though so I figure that's they way to go for now.