Sometimes Flux does not reconciliate - how to debug?

[manuschillerdev]

Thanks for your great template - I got a machine up and running in less than 10 minutes.

Having used ArgoCD for several years, It is the first time using flux. Often when I push changes within kubernetes/apps the flux/helm receive the webhook, and explicitly log that no differences are found.

When I run task cluster:reconciliate the changes

• sometimes get applied
• sometimes don't get applied

In cases they are not getting applied, I am trying to follow https://github.com/onedr0p/flux-cluster-template#-debugging and check the output of flux-system/helm-controller and flux-system/source-controller.

Sometimes there is a hint in the logs - often there isn't.

ArgoCD gives me proper Error Messages on Sync (errors in YAML Files, Connection errors, and so on) - but with Flux I am sometimes stuck without any information.

I am not sure if this is me being too inexperienced with flux, or if you have any hints on this specific setup on how to debug these cases any further - I'd be glad about any hint in the right direction :)

[onedr0p]

I would say that if the changes aren't getting applied it could be due to GitHub having issues or that you pushed invalid kubernetes configuration that flux cannot reconcile. The debugging link you sent is a great start and next time the changes don't get applied it would be great to share more info regarding the cluster state and in cluster flux resources if possible. Basically dumping the output of those commands in the debugging section here would be pretty helpful in finding what's wrong.

You need to keep in mind that the errors could be on the flux kustomization resource OR the helmrelease resource, checking all those resources and ensuring they are healthy is important. Looking at the flux controller logs might not be very helpful but rather look at all those two flux resources in your cluster.