0.13: [Bug]: Bonsai and nginx ingress controller fails to update

[worldofgeese]

Note

This is a copy/paste issue from a Discord thread

Garden Bonsai (0.13) Bug

After upgrading to the 0.13 version of the CLI, running any command tries to update the nginx ingress controller and fails.

Steps to replicate:

• Running on Ubuntu 18
• Start with 0.12.56 version of garden
• Create a new kind cluster (kind version is 0.17.0 - https://kind.sigs.k8s.io/)
• Initialize garden resources in the cluster via garden plugins local-kubernetes cluster-init
• Upgrade to the 0.13.0 version of the CLI
• Run garden validate - this triggers upgrades of garden managed resources including the nginx controller

Expected results: No errors and components are updated

Actual results:

Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "ingress-nginx-admission-patch", Namespace: "ingress-nginx"
...TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil)}}: field is immutable
...
Failed resolving one or more providers:
- local-kubernetes

[thsig]

The problem here is that we're attempting to update immutable fields of resources that have already been deployed.

https://github.com/garden-io/garden/blob/main/core/src/plugins/kubernetes/system.ts#L137

For kind, it looks like we're using a kubernetes Deploy to deploy the ingress-related manifests:

https://github.com/garden-io/garden/blob/main/static/kubernetes/system/nginx-kind/nginx-kind-new.garden.yml#L4

Whereas the ingress controller is deployed with a helm Deploy when we're not using kind, microk8s or minikube:

https://github.com/garden-io/garden/blob/main/static/kubernetes/system/ingress-controller/garden.yml#L4

I may be wrong, but it looks like helm upgrade gracefully handles changes to immutable fields—or am I wrong there @stefreak?

https://github.com/garden-io/garden/blob/main/core/src/plugins/kubernetes/local/config.ts#L66

So my initial thought is that we have two options:

1. Wrap the kind manifests in a Helm chart and use a helm Deploy.
2. Stick to using a kubernetes Deploy, but detect the immutable field change error here, and force-deploy the ingress controller when that happens (notifying the user that we're about to do so with a warning message).

[thsig]

In the meantime, a workaround on kind would be to uninstall the system services using garden plugins kubernetes uninstall-garden-services on 0.12, and then running garden plugins local-kubernetes cluster-init on 0.13 to install the ingress controller from scratch (i.e. without attempting to patch the already-installed one).

[vvagaytsev]

Fixed in #5136 and #5562. Closing this.

Update. The fix was released in 0.13.20.