BUG: Configuration Store: webhook cleanup job doesn't work in ArgoCD

[csun-cpointe]

Description

After successfully deploying the configuration store using ArgoCD, when delete the configuration-store app the webhook didn't get clean up.

Root Cause

helm.sh/hook: pre-delete is not supported by argocd.
Ref: Helm Hooks/hook: pre-delete

Definition of Done

• Fetch Latest Dev Branch of aiSSEMBLE
• Fix job to make sure webhook clean up job is successfully deleted.
• Steps to Reproduce section to verify webhook is cleaned up.

Test Instruction

• Clone Downstream project here

• Start a local argocd server by running this command:
  helm install aissemble-infrastructure oci://ghcr.io/boozallen/aissemble-infrastructure-chart --version 1.10.0-SNAPSHOT --set jenkins.enabled=false --set ingress-nginx.enabled=false

• Go to Rancher Desktop -> open main window -> Port Forwarding => forward argocd-server to port 8081

• Run argocd admin initial-password -n argocd (keep the password to login to argocd ui and cli console)

• Open the argocd UI http://localhost:8081/ and login as (admin/previous generated password)

• In the test-defect root directory run argocd login localhost:8081 (Use username: admin/password: previous generated password to login)

• Run below command to deploy the test-defect app

        argocd app create test-defect \
         --dest-namespace argocd \
         --dest-server https://kubernetes.default.svc \
         --repo https://github.com/jaebchoi/test-defect \
         --path test-defect-deploy/src/main/resources \
         --revision main \
         --values values.yaml \
         --values values-ci.yaml
  

• On the ArgoCD UI, verify that the test-defect app is shown

• Sync test-defect apps by running following command

   argocd app sync test-defect --revision main
  

• Make sure configuration-store app is healthy and sync and other apps finished syncing. Delete the test-defect or delete the configuration-store app from ArgoCD UI.

• Wait until configuration-store is completely deleted. Open the Rancher Desktop -> Open cluster dashboard -> Admission -> MutatingWebhookConfigurations, the [boozallen.aissemble-configuration-store.webhook](http://127.0.0.1:6120/c/local/explorer/admissionregistration.k8s.io.mutatingwebhookconfiguration/boozallen.aissemble-configuration-store.webhook) is still active

Steps to Reproduce

Clear, specific, and detailed steps taken to enable reproduction of the bug for investigation.

• Create a downstream project with baseline version 1.10.0-SNAPSHOT

mvn archetype:generate -U -DarchetypeGroupId=com.boozallen.aissemble \
  -DarchetypeArtifactId=foundation-archetype \
  -DarchetypeVersion=1.10.0-SNAPSHOT \
  -DgroupId=com.test \
  -DartifactId=test-defect \
  -DprojectGitUrl=test.url \
  -DprojectName=test-defect \
  && cd test-defect

• Add the SparkPipeline.json file to the test-defect-pipeline-models/src/main/resources/pipelines directory
• Add to the fermenter-mda plugin executions in test-defect-deploy/pom.xml

<execution>
    <id>configuration-store</id>
    <phase>generate-sources</phase>
    <goals>
        <goal>generate-sources</goal>
    </goals>
    <configuration>
        <basePackage>com.boozallen.aissemble.test</basePackage>
        <profile>configuration-store-deploy-v2</profile>
        <!-- The property variables below are passed to the Generation Context and utilized
             to customize the deployment artifacts. -->
        <propertyVariables>
            <appName>configuration-store</appName>
        </propertyVariables>
    </configuration>
</execution>

• Run mvn clean install until all the manual actions are complete

• Once the manual actions are complete, run mvn clean install -Dmaven.build.cache.skipCache=true once to get any remaining manual actions

• Download and unzip the attached 283-helper.zip and copy the src directory to the test-defect root directory

• Modify the helm templates for Argocd deployment:

  • In the test-defect-deploy/src/main/resources/values.yaml file, update the targetRevision and repo to reflect the correct value. e.g.:

   targetRevision: main
   repo: https://github.com/abc/test-defect
  

  • In the -deploy/src/main/resources/apps/configuration-store/ directory copy the content of the values-dev.yaml to the values-ci.yaml file
    • remove service and service.type from the values-ci.yaml file

        service:
            type: LoadBalancer
    

    • Mac User: In the -deploy/src/main/resources/apps/configuration-store/values-ci.yaml file update the volumePathOnNode to be /<pathToProject>/test-defect/src/main/resources/configurations
    • Window User: In the -deploy/src/main/resources/values-ci.yaml file update the volumePathOnNode to be /mnt/c/Users/YOUR_USER/PATH/TO/283-helper/apps/configuration-store/src/main/resources/configurations

• Create a repo for the project created

  • create a new test-defect repository from the github website
  • at the test-defect root directory run below commands
  • git init
  • git add .
  • `git commit -m "init ttest-defect base'
  • git branch -M main
  • git remote add origin https://github.com/<username>/<repo_name>.git
  • git push -u origina main

• Start a local argocd server

  • helm install aissemble-infrastructure oci://ghcr.io/boozallen/aissemble-infrastructure-chart --version 1.10.0-SNAPSHOT --set jenkins.enabled=false --set ingress-nginx.enabled=false
  • Rancher Deskip -> open main window -> Port Forwarding => forward argocd-server to port 8081
  • argocd admin initial-password -n argocd (keep the password to login to argocd ui and cli console)
  • Open the argocd UI http://localhost:8081/ and login as (admin/previous generated password)
  • In the test-defect root directory run argocd login localhost:8081 (Use username: admin/password: previous generated password to login)
  • Run below command to deploy the test-defect app

        argocd app create test-defect \
         --dest-namespace argocd \
         --dest-server https://kubernetes.default.svc \
         --repo https://github.com/csun-cpointe/test-defect \
         --path test-defect-deploy/src/main/resources \
         --revision main \
         --values values.yaml \
         --values values-ci.yaml
  

• On the ArgoCD UI, verify that the test-defect app is shown

• Sync test-defect apps

   argocd app sync test-defect --revision main
  

• Make sure configuration-store app is healthy and sync and other apps finished syncing. Delete the test-defect or delete the configuration-store app from ArgoCD UI.

• Wait until configuration-store is completely deleted. Open the Rancher Desktop -> Open cluster dashboard -> Admission -> MutatingWebhookConfigurations, the [boozallen.aissemble-configuration-store.webhook](http://127.0.0.1:6120/c/local/explorer/admissionregistration.k8s.io.mutatingwebhookconfiguration/boozallen.aissemble-configuration-store.webhook) is still active

Expected Behavior

A clear and concise description of the outcome you expected from the actions taken.
The configuration store webhook should also be deleted.

Actual Behavior

A clear and concise description of the outcome you experienced from the actions taken.
The configuration store webhook stays active

Additional Context

• Log output
• Screenshots (if applicable)
• Solution Baseline Version
• Environment details (local, cloud, Azure, AWS, etc.)

[jaebchoi]

DOD Completed with @cwoods-cpointe

[jaebchoi]

Keeping this in backlog as this is less critical as leftover webhook wouldn't affect k8s resource generation in argo CD.
Also ArgoCD have plan to support pre-delete hook See Here