Flux isn't removing env vars from migrated helm deployments

[TheEdgeOfRage]

We have recently migrated some of our services from Helm to managing them as kustomizations in Flux (not HelmReleases). We used to pass some secrets under env in the Deployment, but have now switched to using envFrom and a secret in the kustomization manifests. However, the old env vars are still present on the Deployment, even after setting Flux up to manage the resources.

I have found this issue which seems to be related, but that one seems to have been addressed by this PR. I have tried removing these helm annotations and labels:

metadata:
  annotations:
    meta.helm.sh/release-name: core-api
    meta.helm.sh/release-namespace: default
  labels:
    app.kubernetes.io/managed-by: Helm

There is no managedFields metadata on the Deployment and here's what the metadata section looks like now:

metadata:
  annotations:
    deployment.kubernetes.io/revision: "2551"
  creationTimestamp: "2022-09-22T12:53:02Z"
  generation: 2670
  labels:
    app: core-api
    kustomize.toolkit.fluxcd.io/name: core-go
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: core-api
  namespace: core
  resourceVersion: "1308679872"
  uid: c2791336-2990-4ccf-b78a-ca04e9123035

Let me know what else I can provide to help :)

[stefanprodan]

Kubernetes does not trigger a rolling upgrade when some secret ref changes, this is by design. You need to use Kustomize secrets generator for that.

As for the old vars, Flux will not remove them because Helm doesn't uses kubectl as the manager. You need to tell Flux to override the Helm manager by setting the --override-manager flag.

[TheEdgeOfRage]

As for the old vars, Flux will not remove them because Helm doesn't uses kubectl as the manager. You need to tell Flux to override the Helm manager by setting the --override-manager flag.

Do I need to set that on the kustomize controller flags?

Kubernetes does not trigger a rolling upgrade when some secret ref changes, this is by design. You need to use Kustomize secrets generator for that.

And yeah, I'm aware of that, but that's not the issue :)

[stefanprodan]

Do I need to set that on the kustomize controller flags?

Yes, but first you need to find the name that Helm uses for the managed fields. To find it, run kubectl get -o yaml --show-managed-fields.

[TheEdgeOfRage]

Ohh, I didn't know that it's not showing the managed fields by default, thanks :D

And just in case somebody else stumbles upon this, here's the manager name for helm v3 deployments: helm

[stefanprodan]

Hmm I don't think Helm uses that, kube-controller-manager is a core Kubernetes component

[TheEdgeOfRage]

It did seem a bit suspicious... I'm not sure where that manager is coming from then, maybe somebody applied manual changes to the deployment at some point 🤷

[stefanprodan]

I suggest removing the flag from kustomize-controller after that deployment gets cleaned up.

[TheEdgeOfRage]

I realized where I messed up. I was looking at the pod's field managers instead of the deployment. On the deployment I see that it's helm and Go-http-client that are managing the fields that aren't getting removed. Now it makes sense 👍