How does arcocd support patching of existing resources

[raffaelespazzoli]

this is really more of a question rather than a feature request.
The use case for patching existing resources comes up pretty often when dealing with Kubernetes distributions that comes up with a some default settings.
It would be nice to have a way to declarative patch the default settings to a desired state. Examples are typically Kubernetes distribution-dependent, but one use case across the board is node labeling.

[simster7]

What about:

1. argocd app patch

$  argocd app patch
Examples:
	# Update an application's source path using json patch
	argocd app patch myapplication --patch='[{"op": "replace", "path": "/spec/source/path", "value": "newPath"}]' --type json

	# Update an application's repository target revision using merge patch
	argocd app patch myapplication --patch '{"spec": { "source": { "targetRevision": "master" } }}' --type merge

Usage:
  argocd app patch APPNAME [flags]

2. Deploying local manifest files by doing argocd app sync APPNAME --local [PATH_TO_LOCAL_DIRECTORY]. See the docs here: https://github.com/argoproj/argo-cd/blob/master/docs/user-guide/application_sources.md#development

Would either of these help?

[raffaelespazzoli]

@simster7 I am not trying to patch an application. I am asking if it is possible to patch an existing object.
For example let's say I need to create an Application which will patch node abc with label cde=123.
how do I do that?

[simster7]

Sorry meant to point you to argocd app patch-resource:

$ argocd app patch-resource
Usage:
  argocd app patch-resource APPNAME [flags]

Flags:
      --all                    Indicates whether to patch multiple matching of resources
      --group string           Group
  -h, --help                   help for patch-resource
      --kind string            Kind
      --namespace string       Namespace
      --patch string           Patch
      --patch-type string      Which Patching strategy to use: 'application/json-patch+json', 'application/merge-patch+json', or 'application/strategic-merge-patch+json'. Defaults to 'application/merge-patch+json' (default "application/merge-patch+json")
      --resource-name string   Name of resource

An example of this in action: https://github.com/argoproj/argocd-example-apps/tree/master/blue-green

You can also consider:

1. Updating your deployment manifests from your GitOps repo and performing a new sync. This is the preferred way to do it when using Argo.
2. Parametrizing the fields you want to change and editing them with argocd app set (the example above also uses this).
3. Overriding your manifests with argocd app sync APPNAME --local as mentioned above. Keep in mind that this is an anti-pattern and should only be done for development purposes. Furthermore, this will only affect the current deployment and changes will be lost after a sync.
4. As a last resort maybe consider patching the objects directly using kubectl patch.

[raffaelespazzoli]

@simster7 I am probably not explaining myself. A node object already exists and is not controlled by an argocd Application.
I'd like to be able to create an argocd Application that has the effect of patching a node.
So, basically we don't know what this node will look like and we want to change part of it, for example add a label.

[simster7]

Got it, I had misunderstood what you had meant.

The team should correct me if I'm wrong, but I am fairly certain that you won't be able to use Argo to patch resources that are not controlled by it—i.e. resources that are not declared on the deployment repo and deployed by Argo.

[raffaelespazzoli]

then if this is not possible, I'd like to formally request this feature to be added. Basically argocd should support a different templating model where the templates are actually patch fragments and they are applied and enforced to existing resources.

[alexmt]

This looks like a config management issue. Argo CD intentionally avoids making any changes in user-provided manifests and only inject one label to support resource pruning.

It is really difficult to implement config management in the right way so instead, Argo CD integrates with existing config management tools like helm/kustomize etc

[alexmt]

So I would suggest using kustomize to imlpement resource patching. @raffaelespazzoli please feel free to reopen ticket if necessary.

[raffaelespazzoli]

@alexmt I don't understand the answer you gave me. Kustomize for sure is not the answer to my problem because customize can only patch resources you own (and own the definition of).
I think having the ability to patch a resource you don't own is a good feature for a gitops operator. And I'd like to reopen this issue as a feature request. Ho do I do it?

[alexmt]

Sorry @raffaelespazzoli , did not realize you cannot reopen the ticket. Looks like I misunderstood your question too. Do you mean patching existing resources in a cluster that were not created by Argo CD originally? This should be supported as long as such resources can be modified using "kubectl apply".

You would have to create an application which includes resource manifest with apiVersion/name of an existing resource and add fields which you want to manage. Argo CD should detect that object exist and run kubectl apply agains it.

[raffaelespazzoli]

@alexmt So yes the request is to be able to patch resources that are pre-existing and not originally created by argocd. If I understand you suggestion, you are proposing to have an application that includes those resources in full (original fields plus the added fields that we wanted to patch).

There are two problems with this approach:

1. one might not know the full resource definition at the time of the writing of the application resource (think about the use case of adding a label to a node resource)
2. this would not work well with upgrades, because by giving argocd the full definition of a resource if the actual owner needs to change the resource (for an upgrade for example) argocd would not allow it and would reset the resource to the declared desired state in the application.

So, I believe, we need the concept of patching or if you will the idea that argocd does not fully own a resource but instead on some of the fields of that resource.

[alexec]

@raffaelespazzoli is your question answered now? Can I close this?

[raffaelespazzoli]

yes the answer was that there is not support for enforcing a patch on a preexisting resource. Then I asked to add this as a feature. If you think there is value in a feature like that then this issue should be left open.

[BostjanBozic]

This would definitely come in handy. Like is was mentioned, setting up node labels via ArgoCD would be great. Currently only option that I found so far is basically set up e.g. ansible playbook, which performs kubectl patch.

[raffaelespazzoli]

@BostjanBozic I created an operator to enforce a patch[1], you can have argocd create the CR that informs that operator how to create the patch.
[1]: https://github.com/raffaelespazzoli/resource-locker-operator

[BostjanBozic]

@raffaelespazzoli nice one! I will take a look. If I understand correctly you are basically feeding ArgoCD ResourceLocker CR, Argo sync it and then ResourceLocker actually patches resource?

[raffaelespazzoli]

correct.

…

On Thu, Jul 9, 2020 at 10:44 AM Bostjan Bozic ***@***.***> wrote: @raffaelespazzoli <https://github.com/raffaelespazzoli> nice one! I will take a look. If I understand correctly you are basically feeding ArgoCD ResourceLocker CR, Argo sync it and then ResourceLocker actually patches resource? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2437 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPERXHFHWOYXMSJQ3STGRLR2XJUNANCNFSM4I6U7M7A> .

-- ciao/bye Raffaele

[yuha0]

I think it would be nice to support patch. There are valid use cases. For example, some resources are just cannot be created by users in any circumstance, like kube-system and default namespaces, kubernetes.default.svc.cluster.local service...

If I want to add a label to such object, shouldn't GitOps concept cover this use case?

[peterbosalliandercom]

@alexmt Within openshift for example the default scc's (security context constraints) are owned by Openshift but there is a need to patch those scc with additional users and serviceaccounts (which is documented as a manual proces by doing oc adm or by oc patch (oc patch scc privileged --type=json -p '[{"op": "add", "path": "/users/0", "value":"system:serviceaccount:default:router"}]'). We need this to be possible within argocd because we want to add users and sa's dynamically based on the teams that have namespaces rolled out by argo.

[arikmaor]

I also would love this feature
My use case is configuring workload identity for Stackdriver Adapter that comes pre-installed in gke clusters

Currently, I have to manually run alongside argocd

kubectl annotate serviceaccount --namespace custom-metrics \
  custom-metrics-stackdriver-adapter \
  iam.gke.io/gcp-service-account=<google-service-account>@<project-id>.iam.gserviceaccount.com

[raffaelespazzoli]

@arikmaor take a look at: https://github.com/redhat-cop/resource-locker-operator

[kxr]

+1 to this feature request. resource-locker-operator seems like a viable option but it feels counter intuitive that we have to run a separate operator/controller to do this while argocd is present.

[rgordill]

+1. It will be very helpful if we can deploy a cluster, argocd and then everything can set up a complete operative cluster, including not only the applications but also patching what set up "as-is" in the original cluster deployment but has to be customized.

Examples: authentication methods, retention params in monitoring, api-server customizations, etc.

[raffaelespazzoli]

I created an operator to specifically solve this issue: https://github.com/redhat-cop/patch-operator

[r0bj]

Another use case. After new GKE cluster creation I want to set particular StorageClass as a default by adding annotation storageclass.beta.kubernetes.io/is-default-class=true. I cannot have entire StorageClass object stored in argocd (git) because there are fields managed by GKE, e.g. metadata.annotations.components.gke.io/component-version and maybe some other. So ideally, I would like to just add an annotation.

[priggad]

+1. Would be great to have all configuration after cluster bootstrap to be defined and managed in argocd.

[Ninsbean]

@alexmt Within openshift for example the default scc's (security context constraints) are owned by Openshift but there is a need to patch those scc with additional users and serviceaccounts (which is documented as a manual proces by doing oc adm or by oc patch (oc patch scc privileged --type=json -p '[{"op": "add", "path": "/users/0", "value":"system:serviceaccount:default:router"}]'). We need this to be possible within argocd because we want to add users and sa's dynamically based on the teams that have namespaces rolled out by argo.

Does anyone have a solution for this? I'm having this exact problem.

[raffaelespazzoli]

@Ninsbean I use the patch operator to solve for this issue: https://github.com/redhat-cop/patch-operator

[b-a-t]

Another use case. After new GKE cluster creation I want to set particular StorageClass as a default by adding annotation storageclass.beta.kubernetes.io/is-default-class=true. I cannot have entire StorageClass object stored in argocd (git) because there are fields managed by GKE, e.g. metadata.annotations.components.gke.io/component-version and maybe some other. So ideally, I would like to just add an annotation.

Exactly the same problem exists for AWS EKS default storageclass, if you want to redefine it to gp3 for example, you need to patch corresponding class annotation.

[iam-veeramalla]

This issue is solved by #9711 .

Note: Server-Side Apply is a feature that is not released yet. It is part of the milestone v2.5.
If you would like to test this feature you can either build Argo CD from master or use
docker.io/abhishekf5/argocd@sha256:ac5d7b74e71eb6453944d564e1b5f056a4f3d8c4447141cde0e9f540a7115afc

TL;DR
Enable Server-Side Apply and turn off Schema Validation(not required in the below example, but this is similar to kubectl apply --server-side --validate=false - required when you try to update anything in .spec) as shown below.

Server-Side Apply is a feature stable as part of k8s v1.22 and is introduced into Argo CD by @leoluz with PR #9711. However, this feature is not released. It is set to milestone v2.5.

Server-Side Apply helps users and controllers manage their resources through declarative configurations. Clients can create and modify their [objects](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/) declaratively by sending their fully specified intent.

A fully specified intent is a partial object that only includes the fields and values for which the user has an opinion. That intent either creates a new object or is [combined](https://kubernetes.io/docs/reference/using-api/server-side-apply/#merge-strategy), by the server, with the existing object.

GOAL:

We will try to label a simple nginx deployment that is already available on the cluster and not managed by Argo CD. Let's start with creating the nginx deployment.

1. Create a nginx deployment.
   kubectl create deployment nginx --image=nginx
2. Patch the object that only includes the fields and values for which the user has an opinion. We will patch the nginx deployment with label foo: bar.

• Place the below nginx deployment in a git repo.

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
   app: nginx
   foo: bar
name: nginx
namespace: default

3. Create an Argo CD Application that points to the git repo and path as shown below.

4. Finally you will see that the new label foo:bar is attached to the nginx deployment.

[leotomas837]

@iam-veeramalla

Thanks for the tip, especially that is now released. Yet that is based on the strategic merge patch I assume, is there any way to apply a json patch ? Json patches offer more complex patch possibilities that the strategic merge patch does not support and I am typically using one, which a lot of people are certainly using: a json patch to patch the aws-auth cm to map AWS roles to Kubernetes roles and configure accesses to the cluster. It adds a multiline string to a multiline string, with some go template code to manage different cases.
I am currently using the redhat patch-operator for this purpose. Argocd supports strategic merge patches from now on thanks to #9711 and it would be nice to support other patch types especially the json one.

[leotomas837]

Why is this closed, can the issue be re-opened ?

[b-a-t]

Why is this closed, can the issue be re-opened ?

Maybe cause ArgoCD now supports server-side applies?

[leotomas837]

Maybe cause ArgoCD now supports server-side applies?

@b-a-t
As mentioned, server side applies operate a strategic merge patch, see the Argocd doc.
Strategic merge patches handle only simple cases such as adding a new label for example (such case is mentioned in this thread). But this is far from handling all cases. Many very common cases are not supported by strategic merge patches, I shared one above from AWS.
JSON patches cover many more cases. Any plan to implement them i.e. servide side apply json patches ?

I found a workaround using Argocd CMP with helmfile, but that is not really intuitive. Another solution is to use the patch-operator. But it does not look maintained anymore and no enhancements are made to the chart even from external PRs.

[dellnoantechnp]

patch-operator project is not production ready.

Can any one find other solution ?

[crenshaw-dev]

Reopening, because I think @leotomas837's description of how SSA is limited to relatively simple patches makes sense.

[leotomas837]

patch-operator project is not production ready.

Can any one find other solution ?

There is another solution: creating an Argocd CMP with helmfile. As you can see in this helmfile.yaml example, helmfile supports json patches. Also add kubectl to the CMP sidecar.

Simply create an Argocd Application syncing a helmfile repository, and the trick is to use helmfile's Go template function called exec to get the yaml of the resources Argocd doesn't own with kubectl, then inject them in a raw helm chart into the helmfile.yaml (you can create your own raw helm chart, the link is just an example), and apply any helmfile json patch you like.

Don't forget to also create the right RBAC ressources to give the repo-server read access to the ressources.

Helmfile has a range of very useful other features anyway, have a look at it. I can patch any resource via Gitops in this way with Argocd using a helmfile CMP.

[dellnoantechnp]

I use Argo CD - Sync Options annotation resolved my problem.

Examples:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  annotations:
    argocd.argoproj.io/sync-options: ServerSideApply=true       # sync-options ServerSideApply
  name: k8s
  namespace: kubesphere-monitoring-system
spec:
  scrapeInterval: 30s    # change on 1m to 30s
  ......

Grafna dashboard configmap json: hurge json data

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Replace=true,Prune=true,PruneLast=true    # sync-options set Replace,Prune,PruneLast true
  name: kubernetes-dashboards
data:
  CoreDNS.json: |
    {"annotations":{"list":[{"builtIn .....
  Kubernetes-SkyDNS.json: |
    {"__inputs":[{"name":" .......
  Pod-Stats-Info-dashboard.json: |
    {"annotations":{"lis .......

It's greet.

[fredleger]

+1

Another valid use-case is to patch kubernetes API service with annotations as indicated here : https://docs.datadoghq.com/containers/kubernetes/control_plane/?tab=helm#EKS

Make sense to me to have everything in the same tool and repo

[gete76]

Let's say I want to add an initContainer to the aws-node DaemonSet on EKS. Is there a solution to do that? I don't believe the Argo strategic merge will work here because it requires a chart or templates to point to. What if we didn't deploy the object?

[christianh814]

Since we have SSA, but with the limitation described, should we close this one in favor of another issue?

Supporting, say, JSON patching would be an enhancement to the SSA functionality. But AFAIC, the SSA feature satisfies the original request