linkerd labels are not propagated to infinispan pods

[cgmEdi]

In our k8s cluster we use the infinispan operator in version 2.3.6.Final (image: quay.io/infinispan/operator:2.3.6.Final).
With an infinispan in version 14.0.21.Final

        - name: INFINISPAN_OPERAND_VERSIONS
          value: |-
            [
              {
                "upstream-version": "13.0.8",
                "image": "quay.io/infinispan/server:13.0.8.Final"
              },
              {
                "upstream-version": "14.0.21",
                "image": "quay.io/infinispan/server:14.0.21.Final"
              }
            ]

We are trying to get the created infinispan pods meshed with an linkerd service mesh. When the pods are created and send to the linkerd admission webhook, the proxies sidecars get injected, as well as all the linkerd annotations but for some reason the labels that linkerd patches in, get ignored and the pods are left without those.

Example:

Source infinispan.yaml:

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  annotations:
    infinispan.org/podTargetLabels: kyverno.cgm.com/skip-policies
    infinispan.org/targetLabels: kyverno.cgm.com/skip-policies
  labels:
    kyverno.cgm.com/skip-policies: "true"
  name: infinispan-managed-idm
  namespace: team-idm
spec:
  replicas: 3
  service:
    type: DataGrid
  version: 14.0.21

Actually deployed pods:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 3c48c9126ce26d170ec9614ff8e59115cfcc2b03d95b6d1c3688a81921eb557b
    cni.projectcalico.org/podIP: 10.100.249.100/32
    cni.projectcalico.org/podIPs: 10.100.249.100/32
    config.linkerd.io/opaque-ports: '11222,11223'
    jaeger.linkerd.io/tracing-enabled: 'true'
    kubectl.kubernetes.io/restartedAt: '2024-01-09T10:21:42Z'
    linkerd.io/created-by: linkerd/proxy-injector stable-2.14.6
    linkerd.io/inject: enabled
    linkerd.io/proxy-version: stable-2.14.6
    linkerd.io/trust-root-sha256: 82f2f601d1408da362165ce1e48a1c69ac742362d7a5fed4230c74011ea734ab
    updateDate: '2024-01-09 10:15:02.14630544 +0000 UTC m=+24984.519383028'
    viz.linkerd.io/tap-enabled: 'true'
  creationTimestamp: '2024-01-09T10:22:47Z'
  generateName: infinispan-managed-idm-
  labels:
    app: infinispan-pod
    app.kubernetes.io/created-by: infinispan-managed-idm
    clusterName: infinispan-managed-idm
    controller-revision-hash: infinispan-managed-idm-fbff44984
    infinispan_cr: infinispan-managed-idm
    kyverno.cgm.com/skip-policies: 'true'
    statefulset.kubernetes.io/pod-name: infinispan-managed-idm-0

Expected Pods:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 3c48c9126ce26d170ec9614ff8e59115cfcc2b03d95b6d1c3688a81921eb557b
    cni.projectcalico.org/podIP: 10.100.249.100/32
    cni.projectcalico.org/podIPs: 10.100.249.100/32
    config.linkerd.io/opaque-ports: '11222,11223'
    jaeger.linkerd.io/tracing-enabled: 'true'
    kubectl.kubernetes.io/restartedAt: '2024-01-09T10:21:42Z'
    linkerd.io/created-by: linkerd/proxy-injector stable-2.14.6
    linkerd.io/inject: enabled
    linkerd.io/proxy-version: stable-2.14.6
    linkerd.io/trust-root-sha256: 82f2f601d1408da362165ce1e48a1c69ac742362d7a5fed4230c74011ea734ab
    updateDate: '2024-01-09 10:15:02.14630544 +0000 UTC m=+24984.519383028'
    viz.linkerd.io/tap-enabled: 'true'
  creationTimestamp: '2024-01-09T10:22:47Z'
  generateName: infinispan-managed-idm-
  labels:
    app: infinispan-pod
    app.kubernetes.io/created-by: infinispan-managed-idm
    clusterName: infinispan-managed-idm
    controller-revision-hash: infinispan-managed-idm-fbff44984
    infinispan_cr: infinispan-managed-idm
    kyverno.cgm.com/skip-policies: 'true'
    statefulset.kubernetes.io/pod-name: infinispan-managed-idm-0
    ####### the following are expected but missing ##########
    linkerd.io/control-plane-ns: cluster-base-service-mesh-linkerd
    linkerd.io/proxy-statefulset: infinispan-managed-idm
    linkerd.io/workload-ns: team-idm

For resources, that are not managed by the infinispan operator, the injection of labels works fine.

[ryanemerson]

@cgmEdi Thanks for raising the issue. Currently we don't provide a supported integration with Linkerd and I'm not familiar with this service-mesh.

I see that the labels infinispan.org/podTargetLabels specified in the Infinispan CR are applied to the Pod, so from an Infinispan operator POV things are working as expected.

The Infinispan CR deploys a StatefulSet to create the Infinispan pods, does Linkerd require that the labels are applied on the StatefulSet itself?

[cgmEdi]

As far as I understand it, the linkerd controller installs an admission webhook that is responsible for

1. injecting the linkerd proxy sidecar into pods (this is working fine with the infinispan pods)
2. injecting the linkerd annotations into pods (this is working fine with the infinispan pods)
3. injecting the linkerd labels into pods (this is not working for some reason)

I raised this issue because there might be a bug where the infinispan operator is ignoring admission webhook patches for labels, which is not directly related to linkerd but just wrong behavior.

In order to properly test this, one could write an admission webhook, which adds some arbitrary label to pods and check whether the pods generated by the statefulset of the infinispan operator get these labels after being deployed.

[ryanemerson]

I raised this issue because there might be a bug where the infinispan operator is ignoring admission webhook patches for labels, which is not directly related to linkerd but just wrong behavior.

The admission webhook would be interacting with the pods directly AFAIU, there's nothing for the Infinispan operator to ignore. Similarly, the Operator only ever modifies the StatefulSet definition, we don't update pod specs individually.

In order to properly test this, one could write an admission webhook, which adds some arbitrary label to pods and check whether the pods generated by the statefulset of the infinispan operator get these labels after being deployed.

Can you try this? I don't have the time to delve into this right now, but if you find there's an underlying issue I can use the reproducer and work on a fix.