directory: false always out of sync (app of apps)

[hawksight]

I asked in slack channel but didn't see any response. As this is an easy 'fix' for now I happy just to open the ticket and see if I have spotted a bug or not.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I have an application where I am using the directory and sub recurse keys to sync from a directory in git but only at the top level, not to recurse. I wanted to explicitly set the key to false so that I was confident that there would be no recursion to sub directories if they were present, rather than rely on assuming to default being no recursion.

What I've found is that, even though the application syncs, the config never takes effect and the application remains OutOfSync in the parent app. See image:

I followed the application specification here to see what the key could be defined as.

To Reproduce

Create an application with a definition that specifies the directory and recurse key to be false. Here is my example of cert-manager with some minor masking:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: stg-eu-cert-manager-config
  namespace: argocd
spec:
  destination:
    namespace: default
    server: XXXXXXXX
  project: stg-eu
  # Source of the application manifests
  source:
    repoURL: [email protected]:SoPost/gcloud-infrastructure.git
    targetRevision: master
    path: applications/manual/stg-eu/cert-manager-config
    directory:
      recurse: False
  syncPolicy:
    automated:
      prune: True
      selfHeal: False

Sync this config across with an App of app pattern. here is my staging example that syncs this to the cluster argocd runs in:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: stg-eu-apps-sync
  # You'll usually want to add your resources to the argocd namespace.
  namespace: argocd
  # Add a this finalizer ONLY if you want these to cascade delete.
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  # The project the application belongs to.
  project: stg-eu

  # Source of the application manifests
  source:
    repoURL: [email protected]:SoPost/gcloud-infrastructure.git
    targetRevision: HEAD
    path: applications/stg-eu

    # directory
    directory:
      recurse: true

  # Destination cluster and namespace to deploy the application
  destination:
    server: https://kubernetes.default.svc
    namespace: default

  # Sync policy
  syncPolicy:
    # manual:
    automated:
      prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
      selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
    syncOptions:     # Sync options which modifies sync behaviour
    # - Validate=false # disables resource validation (equivalent to 'kubectl apply --validate=true')

  # Ignore differences at the specified json pointers
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas

I had the same setup across 3 environment. All 3 are the same, but here's an example screenshot:

The child app sync fine and is healthy. The issue seems to be the parent app cannot apply the directory setting.

I tried false and False as the value, same result.

Expected behaviour

I expect an error if I have made an invalid config option.
Mostly I thought this would just sync across, but it does not seem to.

Screenshots

See above ^

Version

argocd: v1.7.7+33c93ae
  BuildDate: 2020-09-29T04:54:55Z
  GitCommit: 33c93aea0b9ee3d02fb9703cd82cecce3540e954
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.7.7+33c93ae
  BuildDate: 2020-09-29T04:56:23Z
  GitCommit: 33c93aea0b9ee3d02fb9703cd82cecce3540e954
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.17.8

Logs

From the argocd application controller relating you our us cluster app sync

time="2020-10-07T09:35:35Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=prd-us-apps-sync
time="2020-10-07T09:35:35Z" level=info msg="getRepoObjs stats" application=prd-us-apps-sync build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=9 unmarshal_ms=8 version_ms=0
time="2020-10-07T09:35:35Z" level=info msg="Skipping auto-sync: already attempted sync to f1dbb78af31fbe92952aa70aafe47538cb2dc5b0 with timeout 5s (retrying in 2.615509425s)" application=prd-us-apps-sync
time="2020-10-07T09:35:35Z" level=info msg="Updated sync status: Synced -> OutOfSync" application=prd-us-apps-sync dest-namespace=default dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
time="2020-10-07T09:35:35Z" level=info msg="Update successful" application=prd-us-apps-sync
time="2020-10-07T09:35:35Z" level=info msg="Reconciliation completed" application=prd-us-apps-sync dedup_ms=0 dest-name= dest-namespace=default dest-server="https://kubernetes.default.svc" diff_ms=21 fields.level=1 git_ms=9 health_ms=0 live_ms=1 settings_ms=0 sync_ms=0 time_ms=69

Solution(s)

For now I am going to remove the directory key entirely as I have no sub directories and should be safe.
But I would like to be able to specify the key so that in the future I can use sub directories.

I could also ignore the field, but that doesn't seem right in this instance.

Does anyone know what the default is? is it false? is it documented anywhere?

[jessesuen]

Does anyone know what the default is? is it false? is it documented anywhere?

Yes the default is to not recurse. And we use the omitempty json tag so when persisting, the field is dropped. So to workaround this, the solution is to omit the field entirely when you don't wish to recurse.

[jessesuen]

I marked this works-for-me because just like other kubernetes resources where default value fields are dropped, the application behaves the same way.

[hawksight]

Yes the default is to not recurse. And we use the omitempty json tag so when persisting, the field is dropped

Thanks @jessesuen for replying. Is this documented anywhere? Only i found it difficult to find anything that referenced to what you mentioned.

I think in a normal argocd application, there wouldn't be an issue, as the key is dropped then diffed and not cause a sync issue.
But with app-of-apps its going to continually re-sync the exact yaml file across in the declarative setup. So that includes with directory key which the child app would immediately drop I guess.

I guess it would be helpful for the app-of-apps pattern to have some reference on any default values, with a note on the dropping of config resulting in this issue. I just think others might hit similar issues. Happy to add some markdown to the docs if you can guide me as to where you think this would be best placed?

Side Note -> this might not be an issue with ApplicationSet as I think one of the goals is to replace the App-of-Apps pattern, but I'm not sure if I can go try that out just yet.

[PrachiMittal2016]

i got into same issue, I removed the directory key entirely from child apps, and app-of-apps have below setting:
directory:
recurse: false , This fixed the issue for me.

[ArthurHDRodrigues]

I've encountered the same problem on ArgoCD v2.12.6+4dab5bd.
The workaround worked for me.