[Question] Possible to have multiple components that merge / patch on the same list? Last component applied overwrites any previous component patches

[jketcham]

I would like to have a base Deployment config with a series of components that can add values to tolerations and nodeAffinity selectors, but when I include two components that try to patch/merge on the same list, the last component that is applied overwrites the previous component's patch.

I'm essentially trying to achieve the same functionality as outlined in the docs for components.

base-deployment:

bases/deployment/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      serviceAccountName: app-sa

And two components, run-type-restricted and run-type-off-demand:

run-type-restricted

bases/deployment/components/run-type-restricted/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patchesStrategicMerge:
- patch.yaml

bases/deployment/components/run-type-restricted/patch.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      tolerations:
      - key: "run-type"
        operator: "Equal"
        value: "restricted"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: run-type
                operator: In
                values:
                - restricted

run-type-off-demand

bases/deployment/components/run-type-off-demand/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patchesStrategicMerge:
- patch.yaml

bases/deployment/components/run-type-off-demand/patch.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      tolerations:
      - key: "run-type"
        operator: "Equal"
        value: "off-demand"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: run-type
                operator: In
                values:
                - off-demand

Now when I try to use them in the base deployment for an app, the last component applied overwrites the previous component patches:

descendant-deployment

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../some-deployment

components:
- ../../../deployment/components/run-type-restricted
- ../../../deployment/components/run-type-off-demand

Actual output

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
  name: descendant-deployment-app
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: run-type
                operator: In
                values:
                - off-demand
      serviceAccountName: descendant-deployment-app-sa
      tolerations:
      - effect: NoSchedule
        key: run-type
        operator: Equal
        value: off-demand

Expected output

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
  name: descendant-deployment-app
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: run-type
                operator: In
                values:
                - restricted
            - matchExpressions:
              - key: run-type
                operator: In
                values:
                - off-demand
      serviceAccountName: descendant-deployment-app-sa
      tolerations:
      - effect: NoSchedule
        key: run-type
        operator: Equal
        value: restricted
      - effect: NoSchedule
        key: run-type
        operator: Equal
        value: off-demand

Is this something that should be possible? Or would I need to create multiple base deployments to achieve this?

I've tried this with json patching, but the spec defines that it's not possible to add to a list that doesn't exist yet, and it's not possible to define an empty list for affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms (a value is required).

Thanks for taking a look!

[k8s-ci-robot]

@jketcham: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jketcham]

/remove-lifecycle stale

[annasong20]

Very interesting question! If I'm understanding your setup correctly, this is a patch, not a components problem. You'd like to write patch overlays that are order-agnostic, but run into problems when:

1. using patchesJson6902 to add elements to tolerations because, as you said, tolerations needs to exist before you can add elements to it, but we cannot know if it already exists in the current setup assuming we are order-agnostic and blindly adding tolerations will overwrite existing elements
2. using patchesStrategicMerge to add elements to tolerations because its x-kubernetes-patch-strategy is "replace" instead of "merge", and so overlays will overwrite existing tolerations lists instead of adding to them.

Here are 2 workarounds that target the above problems.

1. @KnVerey proposed first initializing tolerations to the empty list [] either in the base or an initial overlay. In successive overlays, then, you can use patchesJson6902 to add elements.
2. This workaround is less practical and more of a theoretical exercise. @KnVerey found the field x-kubernetes-list-map-keys. For patchesStrategicMerge to merge instead of replace, I believe you could override the default openapi, via the openapi kustomization field, with one where the tolerations x-kubernetes-patch-strategy is merge and the x-kubernetes-list-map-keys consists of the tolerations fields (key, value, operator, effect). patchesStrategicMerge should then add each unique toleration instead of replacing. Note that tolerations patched in this way that are not identical but effectively the same won't be recognized as duplicates.

Hope this helps.

[jketcham]

Hi @annasong20 thanks for your detailed response! Yes this is a patch problem. The workarounds you proposed sound like they'd work for the tolerations, but I suppose I may be out of luck for the node affinities.
I'm no longer directly working on the project where I was running into this situation, and had since created another workaround, but this is good to know. I'll go ahead and close this issue.