[BUG] set-kube-service-host-fqdn at odds with k8s docs

[olix0r]

AKS 2022-07-17 includes the following release notes:

The annotation kubernetes.azure.com/set-kube-service-host-fqdn can now be added to pods to set the KUBERNETES_SERVICE_HOST variable to the domain name of the API server instead of the in-cluster service IP. This is useful in cases where the cluster egress is via a layer 7 firewall, like Azure Firewall with Application Rules.

This feature is at odds with the documented Kubernetes client discovery behavior:

While running in a Pod, the Kubernetes apiserver is accessible via a Service named kubernetes in the default namespace. Therefore, Pods can use the kubernetes.default.svc hostname to query the API server. Official client libraries do this automatically.

While client-go currently maintains support for the legacy environment-based configuration, other clients do not, rendering them unable to work properly in clusters that use egress firewalls.

Ideally, AKS should support the documented client discovery rules instead of forcing clients to support the (now undocumented) legacy discovery behavior. Alternatively, the Kubernetes docs should be updated to describe the heuristics whereby the KUBERNETES_SERVICE_HOST env var should be honored.

See also linkerd/linkerd2#9339

[olix0r]

Note also that client-go will sooner or later drop support for KUBERNETES_SERVICE_HOST, per https://github.com/kubernetes/client-go/blob/b8b620636cbbb3bf92696d3a9ba5efb7f60fe1f2/rest/config.go#L534-L540

[olix0r]

I've submitted a kubernetes website PR to correct the documentation kubernetes/website#36691