Undetected diffs when adding a key in the live manifest from the UI

[PonchonB]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Hello, and first of all thanks for your hard work on ArgoCD. It has been of great use for us to manage several applications. In particular it allows our dev teams to conveniently modify live manifests for some Kubernetes resources without leaving the ArgoCD web interface.

Usually when modifying the live manifest on an application resource within ArgoCD UI, the sync status of the application changes to "out of sync" and the APP DIFF shows the difference between the new live manifest and the desired manifest.

However when the edit to the live manifest involves adding a new key to a manifest (for instance the args key to a cronjob/deployment pod template spec or a new data key in a configmap), this edit seems undetected: the app still appears as synced with the target manifest and syncing the app/resource does not revert the key addition to the manifest.

Is it an expected behavior? Am I missing something?

To Reproduce

Minimal set-up to reproduce this issue:

1. ArgoCD basic setup.

• Leave all application controller parameters to their default value.

• one EKS cluster, a reproduce-bug-argocd-missing-diff namespace and proper IAM + Kubernetes RBAC configuration for the argocd user

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd
rules:
  - apiGroups:
      - '*'
    resources:
      - '*'
    verbs:
      - 'get'
      - 'list'
      - 'watch'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argocd
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: argocd
    namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd
  namespace: reproduce-bug-argocd-missing-diff
rules:
  - apiGroups:
      - ""
      - apps
      - autoscaling
      - rbac.authorization.k8s.io
      - extensions
      - keda.sh
      - policy
      - batch
      - networking.k8s.io
    resources:
      - '*'
    verbs:
      - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd
  namespace: reproduce-bug-argocd-missing-diff
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argocd
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: argocd
    namespace: default

• a source repository

• a simple ArgoCD application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-app-argocd-bug-reproduction
  namespace: argocd
spec:
  project: "default"
  source:
    repoURL: <source repository URL>
    targetRevision: <source branch>
    path: <path-to-Helm-chart-parent-directory>/test-app
  destination:
    name: <cluster name>
    namespace: "reproduce-bug-argocd-missing-diff"

2. Create a simple Helm chart with a configmap as only resource. Commit it and push it to a branch of your source repository:

test-app/Chart.yaml:

apiVersion: v2
name: test-app
description: Test application to reproduce ArgoCD missing diff bug in container args
kubeVersion: ">= v1.20-0"
type: application
version: 0.0.1
appVersion: "0.0.1"

test-app/templates/test-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-configmap
  namespace: reproduce-bug-argocd-missing-diff
data:
  TEST_KEY: "test-value"

3. In ArgoCD web interface, edit the live manifest of the application test-configmap resource to add a another data key:

Expected behavior

• The application sync status should be OUT OF SYNC

• In the DIFF tab of the configmap resource or in the APP DIFF panel of the application, we should see the diff with the desired manifest (that does not possess this other key). It's the case when altering any key present in the live manifest:

• When syncing the app, the added data key of the configmap should be removed.

Actual Behavior

• The application sync status is `SYNCED``

• No diff is detected between the live manifest and the desired manifest

• If syncing the application, nothing changes

Live Manifest:

Desired Manifest:

Diff:

Version

argocd: v2.8.2+dbdfc71
  BuildDate: 2023-08-24T20:05:39Z
  GitCommit: dbdfc712702ce2f781910a795d2e5385a4f5a0f9
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64```

[jesusmah]

Im seeing the same behaviour. When I add extra stuff to my kubernetes resource (a service in my case) through the UI (in my case straight through the OpenShift web UI), ArgoCD does not detect the diff. This seems to be quite dangerous cause it basically breaks the GitOps paradigm does not it?
Im using the following version:

{
    "Version": "v2.9.2+c5ea5c4",
    "BuildDate": "2023-12-01T19:21:49Z",
    "GitCommit": "c5ea5c4df52943a6fff6c0be181fde5358970304",
    "GitTreeState": "clean",
    "GoVersion": "go1.20.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 unknown",
    "HelmVersion": "v3.13.2+g2a2fb3b",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0",
    "ExtraBuildInfo": "{Vendor Information: Red Hat OpenShift GitOps version: v1.11.0}"
}

[emedvesApk]

Hi all,
i started encountering this issue on many different Kinds, when removing/adding keys in the manifests.
It happened after I applied some tuning to the argocd installation. I'm currently working with multiple versions 2.10+.

Looks like the issue is related to this configuration:

configs:
  params:
    controller.diff.server.side: "true"

As soon as i remove it, the diffs are detected as expected.
Can anyone confirm?

[dromadaire54]

Hi all,
We encounter same issue @PonchonB for configmap and secret. In the ApplicationController statefulset there's no log about the new key in the configmap or the secret. No pull request has been opened to fix this issue. @emedvesApk unfortunately, without the flag controller.diff.server.side: "true" is not working too.
The environment:

• Kind k8s: v0.22.0
• kubernetes version 1.29.2
• argocd v2.10.4

Finally it's worse, I tested on my local cluster the key deletion by git to the configmap and the status didn't change to outofsync and the resource is not updated in the cluster even if the resource is synced. It is updated only if you check the replace sync option in the UI. The priority of this bug is high because the gitops workflow is broken. It didn't happen when I added a key to the configmap or the secret.

[dan-catalano-vc]

Running into the same with configmaps & deployments. if values are live changed, server does no detect the diff. refresh & hard refresh do nothing. no logs in any of the pods. interestingly, changes once pushed to git will show as a diff in the cluster

k8s:

{
    "major": "1",
    "minor": "29+",
    "gitVersion": "v1.29.1-eks-508b6b3",
    "gitCommit": "91af9bf4f80caa30ef9c0ad9250d2053d084968b",
    "gitTreeState": "clean",
    "buildDate": "2024-01-29T20:58:44Z",
    "goVersion": "go1.21.6",
    "compiler": "gc",
    "platform": "linux/amd64"
  }

argocd:

{
    "Version": "v2.10.2+fcf5d8c",
    "BuildDate": "2024-03-01T21:24:51Z",
    "GitCommit": "fcf5d8c2381b68ab1621b90be63913b12cca2eb7",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.2+gc309b6f",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}

[jjaygohil]

Running into same issue.

[andrii-korotkov-verkada]

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and let us know if the issue is still present, please?

[matias-cova-skydropx]

Running into same issue.
argocd:
{
"Version": "v2.11.3+3f344d5",
"BuildDate": "2024-06-06T08:42:00Z",
"GitCommit": "3f344d54a4e0bbbb4313e1c19cfe1e544b162598",
"GitTreeState": "clean",
"GoVersion": "go1.21.9",
"Compiler": "gc",
"Platform": "linux/amd64",
"KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
"HelmVersion": "v3.14.4+g81c902a",
"KubectlVersion": "v0.26.11",
"JsonnetVersion": "v0.20.0"
}

[matias-cova-skydropx]

After upgrading from v2.11.3+3f344d5 to v2.12.0+ec30a48, the issue still persists.

argocd
{
"Version": "v2.12.0+ec30a48",
"BuildDate": "2024-08-05T13:25:54Z",
"GitCommit": "ec30a48bce7a60046836e481cd2160e28c59231d",
"GitTreeState": "clean",
"GoVersion": "go1.22.4",
"Compiler": "gc",
"Platform": "linux/amd64",
"KustomizeVersion": "v5.4.2 2024-05-22T15:19:38Z",
"HelmVersion": "v3.15.2+g1a500d5",
"KubectlVersion": "v0.29.6",
"JsonnetVersion": "v0.20.0"
}

[matias-cova-skydropx]

Seems the behavior is expected :/

#4537 (comment)

I'm using stable version and nothing changed it but that being said, the behavior is as expected, I hope argo add the feature, it would be a big win