Linkerd does not respect local traffic policy under certain circumstance

[yc185050]

What is the issue?

I have seen instances of linkerd routing traffic from one node to another for services whose traffic policy is set to local.
This appears to happen when a node is being rebooted

How can it be reproduced?

I was able to reproduce by creating a daemonset and a service with internalTrafficPolicy: Local on a multi-node cluster.

By using the destination-client to debug I found, when a pod is manually deleted, I saw the endpoints were updated two times: Add and Remove.

INFO[0038]                                              
INFO[0108] Add:                                         
INFO[0108] labels: map[]                                
INFO[0108] - 10.244.0.8:9376                            
INFO[0108]   - labels: map[control_plane_ns:linkerd daemonset:fluentd-elasticsearch pod:fluentd-elasticsearch-6mjjz serviceaccount:default zone:] 
INFO[0108]   - protocol hint: H2                        
INFO[0108]   - identity: dns_like_identity:{name:"default.default.serviceaccount.identity.linkerd.cluster.local"}  server_name:{name:"default.default.serviceaccount.identity.linkerd.cluster.local"} 
INFO[0108] - 10.244.2.7:9376                            
INFO[0108]   - labels: map[control_plane_ns:linkerd daemonset:fluentd-elasticsearch pod:fluentd-elasticsearch-2c59n serviceaccount:default zone:] 
INFO[0108]   - protocol hint: H2                        
INFO[0108]   - identity: dns_like_identity:{name:"default.default.serviceaccount.identity.linkerd.cluster.local"}  server_name:{name:"default.default.serviceaccount.identity.linkerd.cluster.local"} 
INFO[0108]                                              
INFO[0115] Remove:                                      
INFO[0115] - 10.244.0.8:9376                            
INFO[0115] - 10.244.2.7:9376                            
INFO[0115]   

The endpoints for remote pods got added first when the pod is being terminated and then got removed when the new pod is running.

This is causing issues if the request is sent before the remote endpoints get removed. So it is easier to reproduce if the node is being restarted as the gap is longer.

Logs, error output, etc

N/A

output of linkerd check -o short

❯ linkerd check -o short
Status check results are √

Environment

• Linkerd 2.14.10

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None

[yc185050]

#12312 fixes this issue. Is it possible to back port this to 2.15 and 2.14?