Changes to server selector are not reflected in opaqueness

[adleong]

What is the issue?

If a Server is marking a Pod's port as opaque and then the Server's podSelector is updated to no longer select that Pod, then the Pod's port should no longer be marked as opaque. However, this update does not result in any updates from the destination API's Get stream and the port remains marked as opaque.

How can it be reproduced?

1. Create a Pod, a Service which selects that pod, and a Server which selects the pod and marks a port as opaque
2. Initiate a Get stream for the service and notice that the endpoint is marked with opaque transport:

go run controller/script/destination-client/main.go -method get -path test.emojivoto.svc.cluster.local:8080  --token '{"nodeName":"alex-worker"}'
INFO[0000] Add:
INFO[0000] labels: map[namespace:emojivoto service:test]
INFO[0000] - 10.42.0.163:8080
INFO[0000]   - labels: map[control_plane_ns:linkerd deployment:test pod:test-5d75589c45-wv5kz pod_template_hash:5d75589c45 serviceaccount:default zone:]
INFO[0000]   - protocol hint: UNKNOWN
INFO[0000]   - identity: dns_like_identity:{name:"default.emojivoto.serviceaccount.identity.linkerd.cluster.local"}  server_name:{name:"default.emojivoto.serviceaccount.identity.linkerd.cluster.local"}
INFO[0000]   - opaque transport port: 4143

3. Edit the Server's selector so that it no longer selects the Pod
4. Notice no updates on the Get stream.

Logs, error output, etc

This is because we ignore Server updates if the Server's selector does not select the pod:

https://github.com/linkerd/linkerd2/blob/main/controller/api/destination/watcher/endpoints_watcher.go#L1251

output of linkerd check -o short

N/A

Environment

reproduced on commit: 796bb85
k3d version v5.6.0
k3s version v1.27.4-k3s1 (default)

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None