Kustomize-controller shows path not found error if kustomization and resources are deleted in the same commit

[craigbryson]

Describe the bug

Related to discussion: #2622

In short I think the problem is resolved by v1beta2 kustomization API but there are still some errors in the logs that it might be worth investigating

Steps to reproduce

Ok so I've tested this a little with podinfo

Added the following Kustomization to my repo:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 2m0s
  sourceRef:
    kind: GitRepository
    name: fluxcd2-sync
  path: ./apps/podinfo
  prune: true
  validation: client
  decryption:
    provider: sops

Observed that the kustomization was applied and resources created:

flux-system podinfo 37s True Applied revision: main/dc943ef9adeb08b5ab35a65794f3e67a73ced977

k get all -n podinfo
NAME                           READY   STATUS    RESTARTS   AGE
pod/podinfo-597f79bcdf-rjtf7   1/1     Running   0          2m53s
pod/podinfo-597f79bcdf-tv8k7   1/1     Running   0          3m8s

NAME              TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.2.0.207   <none>        9898/TCP,9999/TCP   3m8s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/podinfo   2/2     2            2           3m8s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/podinfo-597f79bcdf   2         2         2       3m8s

NAME                                          REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/podinfo   Deployment/podinfo   3%/99%    2         4         2          3m8s

Deleted the Kustomization and /app/podinfo in the same commit. The Kustomization is successfully deleted and is no longer listed by kubectl get kustomizations -A but the podinfo namespace and resources remain.

I received a notification to say:

kustomization path not found: stat /tmp/podinfo2108903381/apps/podinfo: no such file or directory

And similarly the logs of the kustomize-controller shows:

{"level":"info","ts":"2022-04-08T08:01:11.789Z","logger":"controller.kustomization","msg":"Reconciliation finished in 241.478751ms, next run in 2m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"podinfo","namespace":"flux-system","revision":"main/dc943ef9adeb08b5ab35a65794f3e67a73ced977"}
{"level":"error","ts":"2022-04-08T08:01:17.916Z","logger":"controller.kustomization","msg":"Reconciliation failed after 61.552672ms, next try in 2m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"podinfo","namespace":"flux-system","revision":"main/254d75f6b3461f538a01cd55662a3001fe33eb70","error":"kustomization path not found: stat /tmp/podinfo2108903381/apps/podinfo: no such file or directory"}
{"level":"info","ts":"2022-04-08T08:01:18.389Z","logger":"controller.kustomization","msg":"garbage collection completed: Kustomization/flux-system/podinfo deleted","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"fluxcd2-sync","namespace":"flux-system"}

I then deleted the podinfo namespace manually and reapplied the kustomization, but this time as v1beta2 to see if the behaviour changed.

This time the resources were successfully deleted however the log still reports an error:

{"level":"info","ts":"2022-04-08T08:33:00.372Z","logger":"controller.kustomization","msg":"Reconciliation finished in 384.403406ms, next run in 2m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"podinfo","namespace":"flux-system","revision":"main/ab84602e6dc1510ef30128eb5697cffd2cbfbac5"}
{"level":"error","ts":"2022-04-08T08:34:03.412Z","logger":"controller.kustomization","msg":"Reconciliation failed after 89.821984ms, next try in 2m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"podinfo","namespace":"flux-system","revision":"main/baefd481cfd91cfb5b5c05b04b7ccd7084567824","error":"kustomization path not found: stat /tmp/podinfo1116633936/apps/podinfo: no such file or directory"}
{"level":"info","ts":"2022-04-08T08:34:03.926Z","logger":"controller.kustomization","msg":"garbage collection completed: Kustomization/flux-system/podinfo deleted","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"fluxcd2-sync","namespace":"flux-system"}

Expected behavior

When a kustomization is deleted and its base resources are also deleted in the same commit, the resources should be removed from cluster

Screenshots and recordings

No response

OS / Distro

n/a

Flux version

0.28.3

Flux check

n/a

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Kustomize-controller fails to remove managed resources if kustomization and manifests are deleted in the same commit

I think the title is misleading, the controller does not fail to remove resources in its latest version. We don't do backports to deprecated APIs, so I would close this with the resolution "upgrade to v1beta2". Please reword the title to be about logs or close this issue and open a new one.

[edgrz]

Hi @stefanprodan,

We are using now kustomize.toolkit.fluxcd.io/v1 after bumping flux to v2.1.2. However, we have been noticing same behaviour. So, we are deleting the kustomization along with the folder that contains all the files related with that one.

All the resources including namespace are deleted, but the kustomization itself is never deleted and states:

flux-system   app-X              112m   False   kustomization path not found: stat /tmp/kustomization-3005170571/kubernetes/gitops/app-X: no such file or directory

Only way I fully prune everything is by manually running kubectl delete kustomization ....

The kustomization itself has wait: true and prune: true, although I don't think it's relevant in that case since we are also removing the folder itself, not only the kustomization.