KEP-2086: Add internalTrafficPolicy: PreferLocal

[danwinship]

One-line PR description:
Adds PreferLocal option to internalTrafficPolicy

Issue link: #2086

Other comments:

OpenShift is already implementing this behavior as a hardcoded hack for DNS. The behavior of internalTrafficPolicy: Local is useless for us; OpenShift clusters do software updates regularly, and we don't want to cause rolling DNS failures across the cluster every time we update the CoreDNS DaemonSet.

I see in some previous discussions people talked about "SNAT semantics" as being a problem, but that only applies to external traffic policy; internal traffic doesn't get SNATted either way.

I left the second user story as still wanting Local semantics, not PreferLocal, since I guess some people may still want that sometimes...

cc @andrewsykim @robscott @aojea @Miciah @thockin

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: danwinship
To complete the pull request process, please assign caseydavenport after the PR has been reviewed.
You can assign the PR to them by writing /assign @caseydavenport in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-network/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aojea]

These was requested lately in slack and some other gihub issues ... and is a thing in Openshift, so there is proof of feasibility
LGTM

[robscott]

Is it going to confuse users if we only have PreferLocal for internal but not external traffic policy?

[danwinship]

Maybe... but adding externalTrafficPolicy: PreferLocal might also be confusing, since the current messaging around externalTrafficPolicy: Local is all about preserving source IP, which PreferLocal would have to not guarantee. There'd need to be some doc rewriting...

FWIW, OpenShift actually also has prefer-local external traffic policy as a local hack. A bit more detail here but it's basically the same as the internal case for DNS: we want to use externalTrafficPolicy: Local for efficiency, and don't particularly care about the no-SNAT thing, but care a lot about not having any disruption when endpoints change.

(Maybe PreferLocal would be less confusing if we said that it never preserves the source IP, just like Cluster, rather than sometimes preserving it and sometimes not? ie, it would SNAT the traffic even when sending it to a local endpoint.)

[robscott]

we want to use externalTrafficPolicy: Local for efficiency, and don't particularly care about the no-SNAT thing, but care a lot about not having any disruption when endpoints change.

I think this is a very common use of the feature.

Maybe PreferLocal would be less confusing if we said that it never preserves the source IP, just like Cluster, rather than sometimes preserving it and sometimes not? ie, it would SNAT the traffic even when sending it to a local endpoint.

💯 completely agree with this.

[andrewsykim]

Fwiw the original implementation of internalTrafficPolicy had PreferLocal as an option, but we removed it since we weren't sure how practical it would be w.r.t topology aware routing.

[danwinship]

Fwiw the original implementation of internalTrafficPolicy had PreferLocal as an option, but we removed it since we weren't sure how practical it would be w.r.t topology aware routing.

How so? I guess, if topology aware routing worked down to the node level, then you wouldn't need PreferLocal, since the topology would take care of it. Is that what you mean? Or did you just mean "what to do when traffic policy and topology conflict"? In that case, the KEPs say that internalTrafficPolicy: Local overrides topology for internal traffic, and externalTrafficPolicy: Local overrides topology for external traffic. I guess we'd have to clarify that that applies to PreferLocal too.

[danwinship]

So it seems like maybe I should close this PR, let internal traffic policy (with just Cluster vs Local) go to GA on its original schedule, and file a new PR adding PreferLocal for both internal and external policy?

[danwinship]

So it seems like maybe I should close this PR, let internal traffic policy (with just Cluster vs Local) go to GA on its original schedule, and file a new PR adding PreferLocal for both internal and external policy?

Right. Further discussion on #3016