Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix multi-node cluster not working after restarting docker #2775

Merged
merged 5 commits into from
May 25, 2022
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 46 additions & 9 deletions images/base/files/usr/local/bin/entrypoint
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,12 @@ fi

grep_allow_nomatch() {
# grep exits 0 on match, 1 on no match, 2 on error
grep "$@" || [[ $? == 1 ]]
grep "$@"|| [[ $? == 1 ]]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is an unintended change and reverted it in 3rd patch, please let me know if this is on purpose

}

# regex_escape_ip converts IP address string $1 to a regex-escaped literal
regex_escape_ip(){
sed -e 's#\.#\\.#g' -e 's#\[#\\[#g' -e 's#\]#\\]#g' <<<"$1"
BenTheElder marked this conversation as resolved.
Show resolved Hide resolved
}

validate_userns() {
Expand Down Expand Up @@ -378,6 +383,22 @@ select_iptables() {
update-alternatives --set ip6tables "/usr/sbin/ip6tables-${mode}" > /dev/null
}

fix_certificate() {
BenTheElder marked this conversation as resolved.
Show resolved Hide resolved
local apiserver_crt_file="/etc/kubernetes/pki/apiserver.crt"
local apiserver_key_file="/etc/kubernetes/pki/apiserver.key"

# Skip if this Node doesn't run kube-apiserver
if [[ ! -f ${apiserver_crt_file} ]] || [[ ! -f ${apiserver_key_file} ]]; then
return
fi

# Deletes the certificate for kube-apiserver and generates a new one.
# This is necessary because the old one doesn't match the current IP.
echo 'INFO: clearing and regenerating the certificate for serving the Kubernetes API' >&2
rm -f ${apiserver_crt_file} ${apiserver_key_file}
kubeadm init phase certs apiserver --config /kind/kubeadm.conf
BenTheElder marked this conversation as resolved.
Show resolved Hide resolved
}

enable_network_magic(){
# well-known docker embedded DNS is at 127.0.0.11:53
local docker_embedded_dns_ip='127.0.0.11'
Expand Down Expand Up @@ -416,10 +437,18 @@ enable_network_magic(){
echo "ERROR: Have an old IPv4 address but no current IPv4 address (!)" >&2
exit 1
fi
# kubernetes manifests are only present on control-plane nodes
sed -i "s#${old_ipv4}#${curr_ipv4}#" /etc/kubernetes/manifests/*.yaml || true
# this is no longer required with autodiscovery
sed -i "s#${old_ipv4}#${curr_ipv4}#" /var/lib/kubelet/kubeadm-flags.env || true
if [[ "${old_ipv4}" != "${curr_ipv4}" ]]; then
# kubernetes manifests are only present on control-plane nodes
sed_ipv4_command="s#$(regex_escape_ip "${old_ipv4}")#${curr_ipv4}#g"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that when I did the regex fix I also:

  • stopped computing this string repeatedly
  • changed it to a global replace (!) instead of just replacing the first instance

the existing sed pattern carried forward from the existing script was not good :(
it should be reasonable now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
Reminded by it, I also added word boundary to sed pattern to avoid mismatch like

# sed "s/1\.1\.1\.2/1.1.1.3/g" <<< "1.1.1.22"
1.1.1.32

sed -i "${sed_ipv4_command}" /etc/kubernetes/manifests/*.yaml || true
sed -i "${sed_ipv4_command}" /etc/kubernetes/controller-manager.conf || true
sed -i "${sed_ipv4_command}" /etc/kubernetes/scheduler.conf || true
sed -i "${sed_ipv4_command}" /kind/kubeadm.conf || true
# this is no longer required with autodiscovery
sed -i "${sed_ipv4_command}" /var/lib/kubelet/kubeadm-flags.env || true
# certificate must match the new IP
fix_certificate || true
BenTheElder marked this conversation as resolved.
Show resolved Hide resolved
fi
fi
if [[ -n $curr_ipv4 ]]; then
echo -n "${curr_ipv4}" >/kind/old-ipv4
Expand All @@ -435,10 +464,18 @@ enable_network_magic(){
if [[ -z $curr_ipv6 ]]; then
echo "ERROR: Have an old IPv6 address but no current IPv6 address (!)" >&2
fi
# kubernetes manifests are only present on control-plane nodes
sed -i "s#${old_ipv6}#${curr_ipv6}#" /etc/kubernetes/manifests/*.yaml || true
# this is no longer required with autodiscovery
sed -i "s#${old_ipv6}#${curr_ipv6}#" /var/lib/kubelet/kubeadm-flags.env || true
if [[ "${old_ipv6}" != "${curr_ipv6}" ]]; then
sed_ipv6_command="s#$(regex_escape_ip "${old_ipv6}")#${curr_ipv6}#g"
# kubernetes manifests are only present on control-plane nodes
sed -i "${sed_ipv6_command}" /etc/kubernetes/manifests/*.yaml || true
BenTheElder marked this conversation as resolved.
Show resolved Hide resolved
sed -i "${sed_ipv6_command}" /etc/kubernetes/controller-manager.conf || true
sed -i "${sed_ipv6_command}" /etc/kubernetes/scheduler.conf || true
sed -i "${sed_ipv6_command}" /kind/kubeadm.conf || true
# this is no longer required with autodiscovery
sed -i "${sed_ipv6_command}" /var/lib/kubelet/kubeadm-flags.env || true
# certificate must match the new IP
fix_certificate || true
fi
fi
if [[ -n $curr_ipv6 ]]; then
echo -n "${curr_ipv6}" >/kind/old-ipv6
Expand Down