Fix issue 6294 - "All pods simultaneously restart during worker scaling" #6520

holmesb · 2020-08-10T20:03:29Z

If no_proxy_exclude_workers is true, workers will be excluded from the no_proxy variable. This prevents docker engine restarting when scaling workers.

/kind bug

What this PR does / why we need it:
See issue #6294

Which issue(s) this PR fixes:
Fixes #6294

Does this PR introduce a user-facing change?:

NONE

k8s-ci-robot · 2020-08-10T20:03:32Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2020-08-10T20:03:37Z

Hi @holmesb. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hafe · 2020-08-11T09:26:46Z

Can you use a wildcard.domain in the no_proxy config? That won't change when scaling thus no restart

holmesb · 2020-08-11T12:58:58Z

Can you use a wildcard.domain in the no_proxy config? That won't change when scaling thus no restart

no_proxy contains nodes' single-label name ({{ hostvars[item]['ansible_hostname'] }}) and FQDN ({{ hostvars[item]['ansible_hostname'] }}.{{ dns_domain }}). Adding wildcard domain will fix this without disrupting FQDN traffic, but you'd still have a list of node single-label names, unless you are willing to make the assumption they are not used and can be removed? A risky breaking change if you're not sure.

Regardless, the wildcard domain should be in no_proxy, but I think this is a seperate issue.

hafe · 2020-08-11T13:24:58Z

I was thinking you override the no_proxy var with a wildcard value and that's it. Just reconfig and no code changes. But I might be completely erong...

holmesb · 2020-08-11T14:34:21Z

In addition to the single-label names assumption, the generated no_proxy contains IPs too:
{{ hostvars[item]['access_ip'] | default(hostvars[item]['ip'] | default(fallback_ips[item])) }}

So you'd have to hope connections aren't established to these as well. If they are, they'll go via the proxy and fail. In which case, your suggestion would mean proxy users end-up micro-managing their no_proxy var to keep single-label name and IP connections working. This PR is non-breaking and changes nothing unless the user opts-in (changes no_proxy_exclude_workers to true).

We've scaled our production cluster without downtime with this fix.

EppO · 2020-09-22T16:17:03Z

hi @holmesb, can you sign the CLA please?
/ok-to-test

EppO · 2020-09-22T16:18:05Z

some markdown issues have been flagged by the CI:

docs/proxy.md:20:115 MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]
docs/proxy.md:21:118 MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]

hafe · 2020-09-22T17:50:52Z

You don't add a commit to fix an unmerged commit. Amend and force push instead

holmesb · 2020-09-22T18:51:16Z

No markdown erros now @EppO, but is there any option now to correct the author of the first commit? Or should I create a fresh PR?

EppO · 2020-09-22T19:25:06Z

you can rebase your branch on top of master and change the commit message/author. In your branch, run these commands:

git fetch origin/master
git rebase -i master

this will show you the list of commits, select reword for the first commit at the top and then squash or fixup for the second one.

EppO · 2020-09-22T21:01:33Z

author info is still messed up:

brendan authored and holmesb committed

you can now just do git commit --amend --author="your name " now and git push force again

holmesb · 2020-09-23T10:54:07Z

Thanks @EppO, hopefully fixed the author\committer now. CI failed because
"Failed to download metadata for repo 'updates-modular". Is this a glitch with the CI rather than this PR?

floryut · 2020-09-23T15:12:53Z

Thanks @EppO, hopefully fixed the author\committer now. CI failed because
"Failed to download metadata for repo 'updates-modular". Is this a glitch with the CI rather than this PR?

CI issue indeed, I've retried the job.

EppO · 2020-09-23T18:16:17Z

/lgtm

floryut · 2020-09-25T13:09:22Z

Fine with the change, but can't we have something simpler ?
I mean the code is the same except groups['k8s-cluster']/groups['kube-master']

holmesb · 2020-09-25T15:31:36Z

I've removed the second for loop @floryut

holmesb · 2020-09-28T15:30:53Z

Do we need to give the CI another kick @floryut ?

…e no_proxy variable. This prevents docker engine restarting when scaling workers. Signed-off-by: holmesb <[email protected]>

holmesb · 2020-10-06T13:32:52Z

CI looks good now @floryut, be great to get this merged.
/assign @Atoms

Atoms · 2020-10-06T13:45:10Z

/approve

k8s-ci-robot · 2020-10-06T13:45:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Atoms, holmesb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Atoms]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

floryut · 2020-10-09T08:07:18Z

/lgtm
🎉

hafe · 2020-10-09T17:08:16Z

Looks like you add a file with weird line (windows?) Line endings. Can you please fix that?

…ster * 'master' of github.com:kubernetes-sigs/kubespray: (1632 commits) Chmod kubeconfig to avoid group-readable (kubernetes-sigs#6800) Update bunch of dependencies (kubernetes-sigs#6801) If no_proxy_exclude_workers is true, workers will be excluded from the no_proxy variable. This prevents docker engine restarting when scaling workers. (kubernetes-sigs#6520) crio: ensure service is started and enabled (kubernetes-sigs#6753) Do not install etcd and etcdctl on master with scale.yml playbook. (kubernetes-sigs#6798) Fix csi-snapshotter timeout option. Fix ebs-external-attacher-role ClusterRole. (kubernetes-sigs#6776) Fix cinder & external_openstack cacert deployment (kubernetes-sigs#6745) Added Comment line above checksum section to add clarification about Kubespray's version support and testing (kubernetes-sigs#6785) Update nginx ingress controller to 0.40.1 (kubernetes-sigs#6786) Use v2.14.1 as base image for CI (kubernetes-sigs#6773) Add oomichi to reviewers (kubernetes-sigs#6796) Update triage/support label references to kind/support (kubernetes-sigs#6792) Update kube-router to 1.1.0 (kubernetes-sigs#6793) harden reset to work in more cases (kubernetes-sigs#6781) Add extra arguments variables for openstack and vsphere cloud controller manager daemonsets (kubernetes-sigs#6783) Update cilium with minor fix for CVE (kubernetes-sigs#6784) Add `plugins/mitogen` to `.gitignore` (kubernetes-sigs#6774) Remove arch from flannel image tag (kubernetes-sigs#6765) nginx ingress: fix yaml for multiple nodeselectors (kubernetes-sigs#6768) Forgotten debian10 test during nightly tests (kubernetes-sigs#6769) ...

* 'master' of https://github.com/kubernetes-sigs/kubespray: change owner to root for bin_dir directory (kubernetes-sigs#6814) Modify imagepullpolicy (kubernetes-sigs#6816) fix: add tags for set facts nodelocaldns (kubernetes-sigs#6813) Make reset work for crio (kubernetes-sigs#6812) Added option to force apiserver and respective client certificate to … (kubernetes-sigs#6403) cleanup kubelet_deployment_type (kubernetes-sigs#6815) allow pre-existing floating IPs to be specified with k8s_master_fips (kubernetes-sigs#6755) Fix line-spacing in no_proxy.yml (kubernetes-sigs#6810) Fix handler naming issue for Kubeadm | kubelet (kubernetes-sigs#6803) Disable dashboard by default (kubernetes-sigs#6804) Chmod kubeconfig to avoid group-readable (kubernetes-sigs#6800) Update bunch of dependencies (kubernetes-sigs#6801) If no_proxy_exclude_workers is true, workers will be excluded from the no_proxy variable. This prevents docker engine restarting when scaling workers. (kubernetes-sigs#6520)

…e no_proxy variable. This prevents docker engine restarting when scaling workers. (kubernetes-sigs#6520) Signed-off-by: holmesb <[email protected]>

k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 10, 2020

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Aug 10, 2020

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 10, 2020

k8s-ci-robot requested review from EppO and floryut August 10, 2020 20:03

holmesb mentioned this pull request Aug 10, 2020

Fix issue 6294 - "All pods simultaneously restart during worker scaling" #6305

Closed

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 22, 2020

holmesb force-pushed the bh/6294_fix branch from aa9e15d to 4750b06 Compare September 22, 2020 19:39

holmesb force-pushed the bh/6294_fix branch from 4750b06 to 6d0f309 Compare September 23, 2020 07:17

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Sep 23, 2020

k8s-ci-robot assigned EppO Sep 23, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 23, 2020

holmesb force-pushed the bh/6294_fix branch from 6d0f309 to 2fe8580 Compare September 25, 2020 15:29

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 25, 2020

If no_proxy_exclude_workers is true, workers will be excluded from th…

3e50cdb

…e no_proxy variable. This prevents docker engine restarting when scaling workers. Signed-off-by: holmesb <[email protected]>

holmesb force-pushed the bh/6294_fix branch from 2fe8580 to 3e50cdb Compare October 2, 2020 08:09

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 6, 2020

k8s-ci-robot assigned floryut Oct 9, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 9, 2020

k8s-ci-robot merged commit 1301e69 into kubernetes-sigs:master Oct 9, 2020

holmesb mentioned this pull request Oct 9, 2020

Fix line-spacing in no_proxy.yml #6810

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issue 6294 - "All pods simultaneously restart during worker scaling" #6520

Fix issue 6294 - "All pods simultaneously restart during worker scaling" #6520

holmesb commented Aug 10, 2020

k8s-ci-robot commented Aug 10, 2020

k8s-ci-robot commented Aug 10, 2020

hafe commented Aug 11, 2020

holmesb commented Aug 11, 2020

hafe commented Aug 11, 2020

holmesb commented Aug 11, 2020 •

edited

Loading

EppO commented Sep 22, 2020

EppO commented Sep 22, 2020

hafe commented Sep 22, 2020

holmesb commented Sep 22, 2020

EppO commented Sep 22, 2020

EppO commented Sep 22, 2020

holmesb commented Sep 23, 2020

floryut commented Sep 23, 2020

EppO commented Sep 23, 2020

floryut commented Sep 25, 2020

holmesb commented Sep 25, 2020

holmesb commented Sep 28, 2020 •

edited

Loading

holmesb commented Oct 6, 2020 •

edited

Loading

Atoms commented Oct 6, 2020

k8s-ci-robot commented Oct 6, 2020

floryut commented Oct 9, 2020

hafe commented Oct 9, 2020

Fix issue 6294 - "All pods simultaneously restart during worker scaling" #6520

Fix issue 6294 - "All pods simultaneously restart during worker scaling" #6520

Conversation

holmesb commented Aug 10, 2020

k8s-ci-robot commented Aug 10, 2020

k8s-ci-robot commented Aug 10, 2020

hafe commented Aug 11, 2020

holmesb commented Aug 11, 2020

hafe commented Aug 11, 2020

holmesb commented Aug 11, 2020 • edited Loading

EppO commented Sep 22, 2020

EppO commented Sep 22, 2020

hafe commented Sep 22, 2020

holmesb commented Sep 22, 2020

EppO commented Sep 22, 2020

EppO commented Sep 22, 2020

holmesb commented Sep 23, 2020

floryut commented Sep 23, 2020

EppO commented Sep 23, 2020

floryut commented Sep 25, 2020

holmesb commented Sep 25, 2020

holmesb commented Sep 28, 2020 • edited Loading

holmesb commented Oct 6, 2020 • edited Loading

Atoms commented Oct 6, 2020

k8s-ci-robot commented Oct 6, 2020

floryut commented Oct 9, 2020

hafe commented Oct 9, 2020

holmesb commented Aug 11, 2020 •

edited

Loading

holmesb commented Sep 28, 2020 •

edited

Loading

holmesb commented Oct 6, 2020 •

edited

Loading