Fix for systemd-resolved DNS incompatibility #82
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vannrt
requested review from
dlipovetsky,
sarun87,
puneetguptanitj and
ojmhetar
and removed request for
dlipovetsky and
sarun87
vannrt
commented
Dec 7, 2018
| @@ -75,7 +75,7 @@ const ( | |||
| NodeadmKubeletSystemdDropinFilename = "20-nodeadm.conf" | |||
| NodeadmKubeletSystemdDropinTemplate = `[Service] | |||
| Environment="KUBELET_DNS_ARGS=--cluster-dns={{ .ClusterDNS }} --cluster-domain={{ .ClusterDomain }}" | |||
| Environment="KUBELET_EXTRA_ARGS=--max-pods={{ .MaxPods }} --fail-swap-on={{ .FailSwapOn }} --hostname-override={{ .HostnameOverride }} --kube-api-qps={{ .KubeAPIQPS }} --kube-api-burst={{ .KubeAPIBurst }} --feature-gates={{ .FeatureGates}} --eviction-hard={{ .EvictionHard }} --cpu-manager-policy={{ .CPUManagerPolicy }} --kube-reserved={{ .KubeReservedCPU }}" | |||
| Environment="KUBELET_EXTRA_ARGS=--resolv-conf=/run/systemd/resolve/resolv.conf --max-pods={{ .MaxPods }} --fail-swap-on={{ .FailSwapOn }} --hostname-override={{ .HostnameOverride }} --kube-api-qps={{ .KubeAPIQPS }} --kube-api-burst={{ .KubeAPIBurst }} --feature-gates={{ .FeatureGates}} --eviction-hard={{ .EvictionHard }} --cpu-manager-policy={{ .CPUManagerPolicy }} --kube-reserved={{ .KubeReservedCPU }}" | |||
There was a problem hiding this comment.
We should probably ask one of our customers how they were able to bypass this issue. Based on last night's demo, they seem to prep the machine's networking prior to running cctl.
This problem occurs because kube-dns on systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. Since 127.0.0.53 is a loopback address, dns queries never get past kube-dns causing our conformance tests to fail on DNS related issues. More discussion here: kubernetes/kubernetes#45828 Related issues: kubernetes/kubeadm#787 kubernetes/kubeadm#273 kubernetes/kubeadm#845 The upstream fix is now in v1.11. Without the fix, the kubedns and dnsmasq containers would copy the host's `/etc/resolv.conf`: ``` \# This file is managed by man:systemd-resolved(8). Do not edit. \# \# This is a dynamic resolv.conf file for connecting local clients to the \# internal DNS stub resolver of systemd-resolved. This file lists all \# configured search domains. \# \# Run "systemd-resolve --status" to see details about the uplink DNS servers \# currently in use. \# \# Third party programs must not access this file directly, but only through the \# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way, \# replace this symlink by a static file or a different symlink. \# \# See man:systemd-resolved.service(8) for details about the supported modes of \# operation for /etc/resolv.conf. nameserver 127.0.0.53 search platform9.sys ``` After the fix: ``` \# This file is managed by man:systemd-resolved(8). Do not edit. \# \# This is a dynamic resolv.conf file for connecting local clients directly to \# all known uplink DNS servers. This file lists all configured search domains. \# \# Third party programs must not access this file directly, but only through the \# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way, \# replace this symlink by a static file or a different symlink. \# \# See man:systemd-resolved.service(8) for details about the supported modes of \# operation for /etc/resolv.conf. nameserver 10.105.16.2 nameserver 10.105.16.4 search platform9.sys ```
vannrt
force-pushed
the
private/vteves/resolv-conf
branch
from
3b39821 to
f48df82
Compare
ojmhetar
reviewed
Dec 11, 2018
| @@ -75,7 +75,7 @@ const ( | |||
| NodeadmKubeletSystemdDropinFilename = "20-nodeadm.conf" | |||
| NodeadmKubeletSystemdDropinTemplate = `[Service] | |||
| Environment="KUBELET_DNS_ARGS=--cluster-dns={{ .ClusterDNS }} --cluster-domain={{ .ClusterDomain }}" | |||
| Environment="KUBELET_EXTRA_ARGS=--max-pods={{ .MaxPods }} --fail-swap-on={{ .FailSwapOn }} --hostname-override={{ .HostnameOverride }} --kube-api-qps={{ .KubeAPIQPS }} --kube-api-burst={{ .KubeAPIBurst }} --feature-gates={{ .FeatureGates}} --eviction-hard={{ .EvictionHard }} --cpu-manager-policy={{ .CPUManagerPolicy }} --kube-reserved={{ .KubeReservedCPU }}" | |||
| Environment="KUBELET_EXTRA_ARGS=--resolv-conf=/run/systemd/resolve/resolv.conf --max-pods={{ .MaxPods }} --fail-swap-on={{ .FailSwapOn }} --hostname-override={{ .HostnameOverride }} --kube-api-qps={{ .KubeAPIQPS }} --kube-api-burst={{ .KubeAPIBurst }} --feature-gates={{ .FeatureGates}} --eviction-hard={{ .EvictionHard }} --cpu-manager-policy={{ .CPUManagerPolicy }} --kube-reserved={{ .KubeReservedCPU }}" | |||
There was a problem hiding this comment.
Even though this is a temp workaround before 1.11, I think /run/systemd/solve/resolv.conf should be placed into a constant and substituted via this template like the other flags.
|
@vannrt As I understand it, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This problem occurs because kube-dns on systems using systemd-resolved
copy 127.0.0.53 from the host's /etc/resolv.conf.
Since 127.0.0.53 is a loopback address, dns queries never get past
kube-dns causing our conformance tests to fail on DNS related issues.
More discussion here: kubernetes/kubernetes#45828
Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845
The upstream fix is now in v1.11.
Without the fix, the kubedns and dnsmasq containers would copy the host's
/etc/resolv.conf:After the fix: