Fix service-mirror template when running in HA mode

[mateiidavid]

Two clusters can be linked in HA mode. When HA values are used, the service-mirror deployment receives some pod affinity rules to ensure fair scheduling of pods across a cluster's nodes.

The service-mirror Deployment's template seems to be broken at the moment when using HA values. Affinity rules are incorrectly grouped under a top-level podAntiAffinity field. The Kubernetes API requires the rules to be grouped under a top-level affinity field. This change rectifies that by introducing the missing parent.

Fixes #11603

---

Tests

• The issue was introduced since we avoided including the entire linkerd.affinity subchart (i.e. partial). Instead of bringing in parts of the subchart, we are now copying the install values and including the entire partial template. Based on the values, it should populate the affinity rules.
• I encourage reviewers to do their own testing.

Before:

:; linkerd --context=k3d-target mc link --cluster-name target --api-server-address="https://192.168.224.4:6443" --ha | k apply -f -
secret/cluster-credentials-target created
secret/cluster-credentials-target created
link.multicluster.linkerd.io/target created
clusterrole.rbac.authorization.k8s.io/linkerd-service-mirror-access-local-resources-target created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-service-mirror-access-local-resources-target created
role.rbac.authorization.k8s.io/linkerd-service-mirror-read-remote-creds-target created
rolebinding.rbac.authorization.k8s.io/linkerd-service-mirror-read-remote-creds-target created
serviceaccount/linkerd-service-mirror-target created
error: error validating "STDIN": error validating data: ValidationError(Deployment.spec.template.spec): unknown field "podAntiAffinity" in io.k8s.api.core.v1.PodSpec; if you choose to ignore these errors, turn validation off with --validate=false

After:

:; bin/linkerd --context=k3d-target mc link --cluster-name target --api-server-address="https://192.168.224.4:6443" --ha | k apply -f -
secret/cluster-credentials-target configured
secret/cluster-credentials-target configured
link.multicluster.linkerd.io/target unchanged
clusterrole.rbac.authorization.k8s.io/linkerd-service-mirror-access-local-resources-target unchanged
clusterrolebinding.rbac.authorization.k8s.io/linkerd-service-mirror-access-local-resources-target unchanged
role.rbac.authorization.k8s.io/linkerd-service-mirror-read-remote-creds-target unchanged
rolebinding.rbac.authorization.k8s.io/linkerd-service-mirror-read-remote-creds-target unchanged
serviceaccount/linkerd-service-mirror-target unchanged
deployment.apps/linkerd-service-mirror-target configured
poddisruptionbudget.policy/linkerd-service-mirror-target configured
service/probe-gateway-target unchanged

We're not breaking non-ha:

:; bin/linkerd --context=k3d-target mc link --cluster-name target --api-server-address="https://192.168.224.4:6443" | k apply -f -
secret/cluster-credentials-target configured
secret/cluster-credentials-target configured
link.multicluster.linkerd.io/target unchanged
clusterrole.rbac.authorization.k8s.io/linkerd-service-mirror-access-local-resources-target unchanged
clusterrolebinding.rbac.authorization.k8s.io/linkerd-service-mirror-access-local-resources-target unchanged
role.rbac.authorization.k8s.io/linkerd-service-mirror-read-remote-creds-target unchanged
rolebinding.rbac.authorization.k8s.io/linkerd-service-mirror-read-remote-creds-target unchanged
serviceaccount/linkerd-service-mirror-target unchanged
deployment.apps/linkerd-service-mirror-target configured
service/probe-gateway-target unchanged

Render diff:

# < : bug.yaml
# > : fix.yaml
161,164c161,174
<       podAntiAffinity:
<         preferredDuringSchedulingIgnoredDuringExecution:
<         - podAffinityTerm:
<             labelSelector:
---
>       affinity:
>         podAntiAffinity:
>           preferredDuringSchedulingIgnoredDuringExecution:
>           - podAffinityTerm:
>               labelSelector:
>                 matchExpressions:
>                 - key: mirror.linkerd.io/cluster-name
>                   operator: In
>                   values:
>                   - target
>               topologyKey: topology.kubernetes.io/zone
>             weight: 100
>           requiredDuringSchedulingIgnoredDuringExecution:
>           - labelSelector:
170,179c180
<             topologyKey: topology.kubernetes.io/zone
<           weight: 100
<         requiredDuringSchedulingIgnoredDuringExecution:
<         - labelSelector:
<             matchExpressions:
<             - key: mirror.linkerd.io/cluster-name
<               operator: In
<               values:
<               - target
<           topologyKey: kubernetes.io/hostname
---
>             topologyKey: kubernetes.io/hostname
189c190
<         image: cr.l5d.io/linkerd/controller:stable-2.14.0
---
>         image: cr.l5d.io/linkerd/controller:git-006f6e2c
217c218
<     linkerd.io/created-by: linkerd/cli stable-2.14.0
---
>     linkerd.io/created-by: linkerd/cli git-006f6e2c

[adleong]

Would a test in multicluster/cmd/link_test.go for HA mode have caught this? can we add one?

[mateiidavid]

@adleong added a test, good suggestion! I was actually unaware we already had some fixture-based tests for link output. It might've saved us when we introduced the change, it maybe would've forced us to re-read the manifest and notice the missing key. Ah well, moving forward we will have a golden file to compare against.

I noticed we had an awkward newline when not using HA:

157-    spec:
158-
159:      containers:
160-      - args:
161-        - service-mirror
162-        - -log-level=info

Go somehow still renders the newline even if the subtemplate eats the whitespace correctly... using a guard is the only way I managed to fix this. It makes it a bit more complicated than it has to be but at least it looks good. Last few things I did:

• Use a lexical scope to copy the .Values tree, makes it look cleaner imo and we only copy it when we enable affinity
• Fixed a typo in the original fixture name (default service mirror values).

[alpeb]

LGTM 👍