Headless endpoint mirrors are incorrectly cleaned up as part of GC

[marwanad]

What is the issue?

When GC is triggered (which also happens at startup or when the link watch disconnects), the service mirror controller attempts to look for services that can be GC'ed. This is done by looping through the local mirrored services on the cluster, then extracting the name of the original service in the remote (by dropping the target name suffix).

linkerd2/multicluster/service-mirror/cluster_watcher.go

Lines 311 to 321 in 5760ed2

	for _, srv := range servicesOnLocalCluster {
	_, err := rcsw.remoteAPIClient.Svc().Lister().Services(srv.Namespace).Get(rcsw.originalResourceName(srv.Name))
	if err != nil {
	if kerrors.IsNotFound(err) {
	// service does not exist anymore. Need to delete
	if err := rcsw.localAPIClient.Client.CoreV1().Services(srv.Namespace).Delete(ctx, srv.Name, metav1.DeleteOptions{}); err != nil {
	// something went wrong with deletion, we need to retry
	errors = append(errors, err)
	} else {
	rcsw.log.Infof("Deleted service %s/%s while cleaning up mirror services", srv.Namespace, srv.Name)
	}

However, this check doesn't account for the headless endpoint service mirrors (the per pod cluster IP services). For example, if you have nginx-svc in the west cluster and two replicas, the source cluster will end up with nginx-svc-west, nginx-set-0-west and nginx-set-1-west. The logic would then parse the resource name for the latter two services as nginx-set-0 and nginx-set-1 which won't exist on the remote and ends up deleting them as part of GC.

The next sync would recreate those mirrors but you end up with downtime.

How can it be reproduced?

1. Setup Linkerd multicluster with headless services
2. Create a service in the remote cluster and observed the mirrored services in the local cluster
3. Restart the service mirror
4. Notice how the the endpoint service mirrors get recreated (the per pod cluster IP services)

Logs, error output, etc

time="2024-04-04T02:17:10Z" level=info msg="Received: OrphanedServicesGcTriggered: {}" apiAddress="https://{apiServer}:6443/" cluster=remote
time="2024-04-04T02:17:10Z" level=info msg="Deleted service {namespace}/{podName}-{partition}-{remote} while cleaning up mirror services" apiAddress="https://{apiServer}:6443/" cluster=remote

output of linkerd check -o short

N/A

Environment

• Kubernetes Version: v1.28.8
• Linkerd version: stable-2.14.8

Possible solution

Parsing the mirror.linkerd.io/headless-mirror-svc-name label from the mirrored headless endpoints could be a solution because this will have the root headless service name, then we can drop the remote suffix and keep the same logic.

Additional context

No response

Would you like to work on fixing this bug?

yes