[bitnami/argo-cd] Make it possible to run ArgoCD in HA mode

[maxnitze]

Description of the change

When the replicaCount of the application controller or the server deployments in the ArgoCD chart are set to a number higher than one, an additional env variable is set on the respective container ARGOCD_CONTROLLER_REPLICAS and ARGOCD_API_SERVER_REPLICAS.

In addition a change was added, that allows the application controller to be installed as a StatefulSet. I cannot find direct confirmation in the docs for it, but it seems the sharding of clusters, that is enabled when running with multiple application controller instances, is only working, when the pods are running as part of a stateful set. Otherwise they can't figure out their shard number and some clusters are not handled by any replica anymore (see here for example: argoproj/argo-cd#17016).

The "official" Helm chart also installs the application controller as a StatefulSet (see https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd/templates/argocd-application-controller). There are cases, where it is installed as a Deployment, but that's for a (currently in alpha) feature called Dynamic Cluster Distribution.

Also a validation was added, that multiple replicas of the application controller can only be set if the kind is changed to a StatefulSet (see "Benefits" for an explanation why).

Benefits

As per ArgoCD documentation on high availability, both the application controller and the server should have an environment variable set to the number of replicas in case they are run with multiple.

The application controller uses this value to "shard clusters across multiple controller replicas". The server uses it to "divide the limit of concurrent login requests (ARGOCD_MAX_CONCURRENT_LOGIN_REQUESTS_COUNT) between each replica." The repo-server has no such environment variable.

Especially on the application controller, if the variable is not set, then the different replica will concurrently try to do the same operations on the clusters. Which results in constantly out-of-sync applications and other issues with the apps.

Possible drawbacks

None for the application itself.

Is this change breaking?

There are some interesting cases though.

Installations with Multiple Replicas of the Application Controller

In case you already have an installation with multiple replicas of the application controller, the chart will fail to validate, if you have not set the .Values.controller.kind to "StatefulSet". But in that case your application would not be running properly anyways. I had a similar setup for a week, or so, and the controller pods failed every couple of minutes due to memory issues created by the fact, that they concurrently worked on my clusters.

Env variable is set already in extraEnvVars

In case a user set the respective env variable already in the extraEnvVars block, this leads to a duplicate entry:

spec:
  replicas: 3
  templates:
    spec:
      containers:
        - name: controller
          env:
            - name: ARGOCD_CONTROLLER_REPLICAS
              value: "3" # <-- from the .Values.controller.replicaCount
            # ...
            - name: ARGOCD_CONTROLLER_REPLICAS
              value: "3" # <-- from .Values.controller.extraEnvVars

This is not a problem, since according to the documentation the last occurrence of an environment variable takes precedence.

Env variable is set already in extraEnvVarsCM or extraEnvVarsSecret

This one is a bit more tricky. According to the docs, the environment variables from env take precedence over those from envFrom. So in theory this could overwrite a value set in the extraEnvVarsCM or extraEnvVarsSecret.

BUT: This is only an issue, if those values differ. And since this value should be the same value as the number of replicas, this is a misconfiguration, from my perspective. I don't actually know what the implications of a wrong value are though. I could not find detailed info on the sharding of the application controller, but reading the linked docs I assume this could lead to clusters not being served by any of the replicas!

Applicable issues

None

Additional information

There are some issues and PRs in the ArgoCD repo talking about this:

• The cluster xyz has no assigned shard argoproj/argo-cd#17124
• fix: infer correct shard in statefulset setup (#17124, #17016)  argoproj/argo-cd#17167
• v2.10.0-rc4: controller is configured to ignore cluster https://kubernetes.default.svc argoproj/argo-cd#17016 (comment)

Checklist

• Chart version bumped in Chart.yaml according to semver. This is not necessary when the changes only affect README.md files.
• Variables are documented in the values.yaml and added to the README.md using readme-generator-for-helm
• Title of the pull request follows this pattern [bitnami/<name_of_the_chart>] Descriptive title
• All commits signed off and in agreement of Developer Certificate of Origin (DCO)

[maxnitze]

Converted this back to a draft. I ran into some major issues with the sharding of the application controller today. Apparently it only really works, when the controller is installed as a StatefulSet :(

[fmulero]

Hi @maxnitze Thanks a lot for your contribution, this will be very useful for all users!

I left some comments, Could you give them a glance?

[fmulero]

$podLabels variable is not declared

[maxnitze]

Hey @fmulero ,

thanks for the review! Looking at the Pod partial I saw, that the whole indentation was wrong. In the file itself, but also the calls to nindent were not adapted accordingly. Fixed that as well as your other remarks :)

[fmulero]

Thanks a lot @maxnitze for your contribution.

LGTM