[WIP] Integrate GitOps Engine into Flux

[2opremio]

TL;DR: Usefluxcd/flux:gitops-engine-903b4f82 to give it a try (I will be updating this image regularly)

This is the Flux counterpart of argoproj/argo-cd#3066

The code in this PR is not battle-tested yet, but it should be fully functional and is ready to test.

The GitOps Engine uses Argo as the reconciliation engine, and as such, it responds to the same annotations. Those currently are:

• argocd.argoproj.io/hook
• argocd.argoproj.io/hook-delete-policy
• argocd.argoproj.io/compare-options
• argocd.argoproj.io/sync-options
• argocd.argoproj.io/sync-wave

We still need to ponder whether it makes sense for Flux to respond to those annotations and keep the same behavior as Argo. What's certain is that if we keep those annotations, the argocd.argoproj.io prefix will be made Flux-specific or neutral.

The GitOps Engine provides a watcher-backed cache to query the Kubernetes API server. As a result, we expect Flux to put considerably less pressure on the API server and etcd.

I will be updating this PR description with the latest Flux docker image including this cases in case

TODO:

• Real world testing
• Get community feedback
• Compile a list of how the behavior changes with respect to vanilla Flux
• Come up with a release plan (Flux 2.0.0?)
• The --kubernetes-kubectl flag is currently ignored, until GitOps engine accepts a kubectl binary as a way to interact with the cluster
• Decide what to do about the Argo annotations, change the argocd.argoproj.io prefix to something Flux-specific or neutral
• Decide what to do about Healthchecks in general and custom Healthchecks in particular (since right now, and AFAIU, there is no way for Flux to supply them).
• Fix the interim go.mod hacks which were introduced

[kingdonb]

Closing. Flux v2 is based on the GitOps Toolkit SDK now instead.