coredns_dual setup is broken in Kubespray v2.24.0

[jeanfabrice]

What happened?

Upgrade Kubespray to v2.24.0 then apply upgrade_cluster.yml playbook on a coredns_dual enabled Kubernetes cluster.

Upgrade failed on the first control-plane node with:

TASK [kubernetes-apps/ansible : Kubernetes Apps | Lay Down Secondary CoreDNS Template] ****************************************************************************************************************
ok: [nuc10i7fnh-m1] => (item={'name': 'coredns', 'src': 'coredns-deployment.yml', 'file': 'coredns-deployment-secondary.yml', 'type': 'deployment'})
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.
ok: [nuc10i7fnh-m1] => (item={'name': 'coredns', 'src': 'coredns-svc.yml', 'file': 'coredns-svc-secondary.yml', 'type': 'svc'})
ok: [nuc10i7fnh-m1] => (item={'name': 'dns-autoscaler', 'src': 'dns-autoscaler.yml', 'file': 'coredns-autoscaler-secondary.yml', 'type': 'deployment'})
fatal: [nuc10i7fnh-m1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'src'. 'dict object' has no attribute 'src'\n\nThe error appears to be in '/Users/jeanfabrice/Personal/Homelab/contrib/kubespray/roles/kubernetes-apps/ansible/tasks/coredns.yml': line 30, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Kubernetes Apps | Lay Down Secondary CoreDNS Template\n  ^ here\n"}

Looks like root cause is 7b7c9f5#diff-b1eb8bcdd1f42fddbdc93abe17d47e7c93f2d4c7f37a0bef7d2292f2039ff691R39 committed 4 days ago where:

- { name: coredns, file: coredns-poddisruptionbudget.yml, type: poddisruptionbudget, condition: coredns_pod_disruption_budget }

should rather be:

- { name: coredns,  src: coredns-poddisruptionbudget.yml, file: coredns-poddisruptionbudget.yml, type: poddisruptionbudget, condition: coredns_pod_disruption_budget }

What did you expect to happen?

Cluster to be upgraded with no issue

How can we reproduce it (as minimally and precisely as possible)?

Set dns_mode: coredns_dual in group_vars/k8s_cluster/k8s-cluster.yml, then install or upgrade a cluster.

OS

MacOS Sonoma

Version of Ansible

$ ansible --version
ansible [core 2.15.5]
  config file = /Users/jeanfabrice/Personal/Homelab/contrib/kubespray/ansible.cfg
  configured module search path = ['/Users/jeanfabrice/Personal/Homelab/contrib/kubespray/library']
  ansible python module location = /Users/jeanfabrice/.virtualenvs/homelab/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/jeanfabrice/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/jeanfabrice/.virtualenvs/homelab/bin/ansible
  python version = 3.9.18 (main, Oct 16 2023, 07:38:39) [Clang 15.0.0 (clang-1500.0.40.1)] (/Users/jeanfabrice/.virtualenvs/homelab/bin/python)
  jinja version = 3.1.2
  libyaml = True

Version of Python

Python 3.9.18

Version of Kubespray (commit)

64447e7

Network plugin used

calico

Full inventory with variables

No response

Command used to invoke ansible

No response

Output of ansible run

No response

Anything else we need to know

No response

[jeanfabrice]

need to double-check what the issue is exactly. Closing for now

[jeanfabrice]

Reopening. Issue edited with correct analysis about the root cause

[arusa]

I had the same problem and solved it here: https://github.com/kubernetes-sigs/kubespray/pull/10815/files

But since upgrading my cluster coredns was broken and no pods were able to resolve any hostnames anymore.

After a very long debugging session I found the issue, but I have no clue how that could have happened, but maybe it is somehow related to this issue.

My problem was that the coredns service had this configured:

ports:
- name: dns
  port: 53
  protocol: UDP
  targetPort: dns-tcp

instead of

  targetPort: dns

Just if this is related or someone has the same problem.

[jeanfabrice]

Thank you @arusa!
Can you create a dedicated issue for the coredns service issue? It looks like your observed service doesn't match the content of https://github.com/kubernetes-sigs/kubespray/blob/v2.24.0/roles/kubernetes-apps/ansible/templates/coredns-svc.yml.j2

[VannTen]

That highlights that we're missing CI for the coredns_dual case... :(

[VannTen]

Can you check if the linked PR fix the issue ?

[amogilny]

This issue also happens on coredns setup.
Confirm that even after applying merged patch coredns setup is broken.

ports:
- name: dns
  port: 53
  protocol: UDP
  targetPort: dns-tcp