Tolerate failed coredns svc errors on kubeadm init/upgrade #6244

mattymo · 2020-06-08T15:30:12Z

Also deletes kube-dns svc in kube-system namespace which is unused

k8s-ci-robot · 2020-06-08T15:30:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mattymo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mattymo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Miouge1 · 2020-06-16T20:21:38Z

roles/kubernetes/master/tasks/kubeadm-secondary-legacy.yml

@@ -38,6 +38,9 @@
  when:
    - inventory_hostname != groups['kube-master']|first
    - not kubeadm_already_run.stat.exists
+    - kubeadm_init.rc != 0


CI fails with the following error:

fatal: [instance-2]: FAILED! => {"msg": "The conditional check 'kubeadm_init.rc != 0' failed. The error was: error while evaluating conditional (kubeadm_init.rc != 0): 'dict object' has no attribute 'rc'\n\nThe error appears to be in '/builds/kargo-ci/kubernetes-sigs-kubespray/roles/kubernetes/master/tasks/kubeadm-secondary-legacy.yml': line 33, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: kubeadm | Init other uninitialized masters\n ^ here\n"}

fejta-bot · 2020-09-14T20:50:49Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

EppO · 2020-09-14T21:59:35Z

kubeadm-secondary-legacy.yml is now gone, need to rebase this

/remove-lifecycle stale

Also deletes kube-dns svc in kube-system namespace which is unused Change-Id: I61798a258efbc3f6ee72cd57f68a776db063d417

champtar · 2020-12-23T15:25:29Z

roles/kubernetes/master/tasks/kubeadm-secondary.yml

+  failed_when:
+    - kubeadm_join_control_plane.rc != 0
+    - '"field is immutable" not in kubeadm_join_control_plane.stderr'
+    - '"unable to create/update the DNS service" not in kubeadm_join_control_plane.stdaerr'


typo stdaerr

@mattymo ping

woopstar · 2021-01-08T08:20:14Z

@mattymo should we merge this?

champtar · 2021-03-24T22:21:07Z

Is this still relevant ?

k8s-ci-robot · 2021-03-24T23:38:31Z

@mattymo: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

klippo · 2021-06-21T11:30:58Z

Is this still relevant ?

We're currently experiencing this issue with Kubespray 2.15 for unknown reasons

k8s-triage-robot · 2021-11-09T17:20:29Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

juliohm1978 · 2021-11-16T18:33:03Z

Coming in from the discussion at #7083. This is still relevant.

However, the patch here seems to be "simply ignore" the kubeadm return code when the error message relates to the issue. Is this actully safe? I mean... kubeadm dies and stops mid-upgrade. I'm not familiar with all of kubeadm phases, but can we be sure this is not skipping some important phase that kubeadm should be executing after that?

k8s-triage-robot · 2021-12-16T19:20:53Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2022-01-15T19:32:29Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2022-01-15T19:32:39Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mvilche · 2022-02-24T04:36:40Z

I have same problem when upgrade 1.22.3 to 1.23.3
the fix exist in master branch?

Monpoke · 2022-10-07T07:56:05Z

We have the same issue, upgrading from kubespray 1.18.2 to 1.19.1.

mattymo requested review from EppO, Miouge1, floryut and riverzhang June 8, 2020 15:30

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 8, 2020

k8s-ci-robot requested a review from bozzo June 8, 2020 15:30

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 8, 2020

Miouge1 reviewed Jun 16, 2020

View reviewed changes

k8s-ci-robot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Sep 14, 2020

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 14, 2020

Tolerate failed coredns svc errors on kubeadm init/upgrade

8f0daa9

Also deletes kube-dns svc in kube-system namespace which is unused Change-Id: I61798a258efbc3f6ee72cd57f68a776db063d417

mattymo force-pushed the kubeadm_coredns_svc branch from 6fdc0ed to 8f0daa9 Compare November 4, 2020 08:04

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 4, 2020

champtar reviewed Dec 23, 2020

View reviewed changes

This was referenced Jan 19, 2021

FATAL post-upgrade error: unable to create/update the DNS service: services "kube-dns" not found #7083

Closed

Remove deletion of coredns deployment. #7211

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 24, 2021

k8s-ci-robot assigned riverzhang Jun 3, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 3, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 9, 2021

dlouks mentioned this pull request Nov 16, 2021

unable to create/update the DNS service: Service \"kube-dns\" is invalid: #7745

Closed

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 16, 2021

k8s-ci-robot closed this Jan 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tolerate failed coredns svc errors on kubeadm init/upgrade #6244

Tolerate failed coredns svc errors on kubeadm init/upgrade #6244

mattymo commented Jun 8, 2020

k8s-ci-robot commented Jun 8, 2020

Miouge1 Jun 16, 2020

fejta-bot commented Sep 14, 2020

EppO commented Sep 14, 2020

champtar Dec 23, 2020

champtar Aug 11, 2021

woopstar commented Jan 8, 2021

champtar commented Mar 24, 2021

k8s-ci-robot commented Mar 24, 2021

klippo commented Jun 21, 2021

k8s-triage-robot commented Nov 9, 2021

juliohm1978 commented Nov 16, 2021

k8s-triage-robot commented Dec 16, 2021

k8s-triage-robot commented Jan 15, 2022

k8s-ci-robot commented Jan 15, 2022

mvilche commented Feb 24, 2022

Monpoke commented Oct 7, 2022

Tolerate failed coredns svc errors on kubeadm init/upgrade #6244

Tolerate failed coredns svc errors on kubeadm init/upgrade #6244

Conversation

mattymo commented Jun 8, 2020

k8s-ci-robot commented Jun 8, 2020

Miouge1 Jun 16, 2020

Choose a reason for hiding this comment

fejta-bot commented Sep 14, 2020

EppO commented Sep 14, 2020

champtar Dec 23, 2020

Choose a reason for hiding this comment

champtar Aug 11, 2021

Choose a reason for hiding this comment

woopstar commented Jan 8, 2021

champtar commented Mar 24, 2021

k8s-ci-robot commented Mar 24, 2021

klippo commented Jun 21, 2021

k8s-triage-robot commented Nov 9, 2021

juliohm1978 commented Nov 16, 2021

k8s-triage-robot commented Dec 16, 2021

k8s-triage-robot commented Jan 15, 2022

k8s-ci-robot commented Jan 15, 2022

mvilche commented Feb 24, 2022

Monpoke commented Oct 7, 2022