Release Proposal v2.23

[yankay]

Deprecation / Removal

• Ubuntu 16 and 18 are no longer tested (Remove end of life ubuntu versions in CI #10107, @MrFreezeex)
• Drop support for ansible-core 2.11 and update tests dependencies (Upgrade tests dependencies #10034, @MrFreezeex)
• Drop Kubernetes 1.24 support (project: drop Kubernetes 1.24 support #10234, @MrFreezeex)

Feature / Major Changes

• Make kubernetes v1.27.5 default ([kubernetes] Make 1.27.5 default #10392, @mzaian)
• Add kubernetes v1.27.4 ([kubernetes] Make 1.27.4 default #10359, @mzaian)
• Add Kubernetes 1.27.2 ([kubernetes] support 1.27.2 #9976, @mzaian)
• Add hashes for 1.27.3 1.26.6, 1.25.11 ([kubernetes] upgrade versions to address CVE-2023-2728 #10220, @mzaian)
• Add hashes for 1.27.4 1.26.7, 1.25.12 ([kubernetes] Add hashes for kubernetes 1.27.4, 1.26.7, 1.25.12 #10300, @mzaian)
• Add CPU Management Policies on the Node (Add CPU Management Policies on the Node #10309, @yankay)
• Add Debian 12(bookworm) support (Add Debian 12(bookworm) support and CI #10221, @tu1h)
• Add download.timeout to update download timeout value ([#10148] The download.timeout can be changed. #10149, @yjqg6666)
• Add corresponding coredns versions to all the supported kubernetes releases. ([feature] Correct CoreDNS versions for kubernetes releases #10233, @mzaian)
• Add growpart azure enabled (add growpart azure enabled #10241, @pedro-peter)
• Add ingressClass resource for ingress_nginx by default (Add ingressclass for ingress_nginx #10091, @peschmae)
• Add kubelet topology manager policy on the node (kubelet_topology_manager_scope and kubelet_topoloy_manager_policy) (Add kubelet topology manager policy on the node #10370, @tu1h)
• Add labels to kube-vip static pods (Add labels to kube-vip static pods #10139, @liupeng0518)
• Add node_taints to aws_inventory script (Add node_taints to aws_inventory script (#10168) #10170, @mstoetzer)
• Add option to set SSL_CERT_FILE for offline installation using custom CA for https proxy (Add option to set SSL_CERT_FILE for offline installation using custom CA for https proxy #10215, @HappyFX)
• Add terraform support for NIFCLOUD (Added terraform support for NIFCLOUD #10227, @ystkfujii)
• Add the huawei cloud controller as external cloud controller (Add huawei cloud controller #10198, @dabeck)
• Show detected ansible version when it isn't compatible with kubespray (Print the found version when it is incorrect #10109, @jcpunk)
• Allow to override etcd listen-metrics-urls configuration (using etcd_listen_metrics_urls variable) (Allow to override etcd listen-metrics-urls configuration #10332, @forselli-stratio)
• Don't let find search filesystem mounts in docker build run step (Don't let find search filesystem mounts in docker build run step #10131, @tomodachi)
• Permit custom names for API server lb/proxy containers (Permit custom names for API server lb/proxy containers. #10166, @jcpunk)
• Permit skipping helm update (Add flag to prevent running helm update #10169, @jcpunk)
• Split defaults main file into 2 files (checksums and version) (Split defaults main file #10121, @electrocucaracha)
• System upgrade for Debian-family nodes is available with system_upgrade=true (Add system-upgrade to upgrade-cluster playbook #10184, @sathieu)
• Update download_hash.sh script (Update download_hash.sh script #10120, @electrocucaracha)
• Use a uniform way to get the local path of the binaries (Use a uniform way to get the local path of the binaries #10211, @ErikJiang)
• Disable fapolicyd service (Disable fapolicyd service #10081, @epif4nio)
• Upgrade the load balancer ( nginx and haproxy ) image version to Nginx 1.25, Haproxy 2.8. (Update load balancers Versions to Nginx 1.25, Haproxy 2.8 #10409, @yankay)
• [etcd] Default version to 3.5.7 for kubernetes 1.27 (Update etcd version on README #10410, @mzaian)

Applications

• [argocd] update argocd to v2.7.4 ([argocd] update argocd to v2.7.4 #10226, @mzaian)
• [argocd] update argocd to v2.8.0 ([argocd] update argocd to v2.8.0 #10364, @mzaian)
• [argocd] Add argocd_install_url option to allow changing argocd url ([argocd] allow change argocd url #10176, @liupeng0518)
• [helm] upgrade to 3.12.1 ([helm] upgrade to 3.12.1 #10225, @mzaian)
• [helm] upgrade to 3.12.3 ([helm] upgrade to 3.12.3 #10365, @mzaian)
• [helm] add python dependency check for helm-apps ([flatcar] add python dependency check for helm-apps #10192, @palmeXx)
• [krew] add krew_no_upgrade_check ([krew] add krew_no_upgrade_check #10175, @liupeng0518)
• [coredns] Bump coredns version to 1.10.1 (bump coredns version to 1.10.1 #10199, @eminaktas)
• [coredns] Bump nodelocaldns version to 1.22.20 (bump nodelocaldns version to 1.22.20 #10200, @eminaktas)
• [cert-manager] This introduces a new variable for the cert-manager implementation that will allow one to pass in extra arguments to the cert-manager controller.(cert-manager controller args: #10049, @phunyguy)
• Update Helm (v3.12.2) / Skopeo (v1.13.0) and yq (v4.34.2) (Bump versions #10295, @tu1h)
• Upgrade many tool versions (Helm, crun, kata, youki, gvisor, skopeo, Calico, Cilium etc...) (Upgrade versions #9798, @electrocucaracha)
• [local_path_provisioner] Fix invalid podhelper yaml (local_path_provisioner: fix invalid podhelper yaml #10237, @MrFreezeex)
• Update metrics server to v0.6.4 ([feat] Update metrics server to v0.6.4 #10400, @mzaian)

Container-Managers

• [containerd] Make containerd 1.7.5 default ([containerd] add hashes for 1.7.4-5 #10397, @mzaian)
• [containerd] Support containerd v1.7.2 ([containerd]: bump to 1.7.2 #10219, @Dentrax)
• [containerd] Support containerd 1.7.3 ([containerd] Make containerd 1.7.3 default #10368, @mzaian)
• [containerd] containerd config_path enable mirrors config using new variable containerd_registries_mirrors (deprecate and remove containerd_insecure_registries for containrd and nerdctl_extra_flags and insecure_registry setting for nerdctl (Fix containerd config_path mirrors and remove nerdctl insecure_registry #10196, @yckaolalala)
• [crio] Add crio_insecure_registries option for specifying insecure_registries of crio (Add insecure_registry config to crio.conf #10142, @qlijin)
• [crio] runroot now needs to be setup in storage.conf instead of crio.conf (Move runroot from crio.conf to storage.conf #10372, @floryut)
• [crio] Fix etcdctl copy operation (fix etcdctl copy operation in crio #10242, @ErikJiang)
• [Kata] Set/keep owner/group root/root when unarchiving kata-containers (Set/keep owner/group root/root when unarchiving kata-containers #10338, @rybnico)
• [youki] Fix youki binary download url (not requiring 'v' in version) (Fix youki binary download url #10337, @ErikJiang)

Network

• [calico] Use configmap to configure calico cni config (using configmap to configure calico cni config #10177, @cyclinder)
• [calico] Update calico v3.25.2 (Update calico v3.25.2 #10414, @mzaian)
• [calico] Add calico version to v3.26.0 ([calico] add hashes for v3.26.0 #10224, @mzaian)
• [calico] Add calico version to v3.26.1 ([calico] add hashes for v3.26.1 #10235, @mzaian)
• [calico] Clean up calicoctl_alternate_download_url and calicoctl.mirrors (Clean up calicoctl_alternate_download_url #10271, @yckaolalala)
• [cilium] Add custom rules to clusterrole for cilium operator ([cilium] add custom vars for clusterrole cilium operator #10267, @jeremythuon)
• [cilium] Upgrade to version 1.13.4 (Upgrade cilium version to 1.13.4 #10269, @yulng)
• [Cilium] Do not mount tls when 'cilium_hubble_tls_generate' is false (fix: not mount tls when 'cilium_hubble_tls_generate' is false #10357, @charlychiu)
• [Cilium] Update cilium to 1.13.3 (Update cilium to latest (1.13.3) #10158, @jcpunk)
• [flannel] Only create /var/lib/calico when needed (Don't create calico CNI dir when not using calico #10156, @jcpunk)
• [flannel] Bump flannel version to v0.22.0 and flannel-cni-plugin version to v1.1.2. Also, changes flannel repository from flannelcni to flannel (bump flannel version to v0.22.0 and flannel-cni-plugin version to v1.1.2 #10205, @eminaktas)
• [flannel] Remove unused flannel_cni_download_url (Remove unused flannel_cni_download_url #10188, @oomichi)
• [kube-ovn]: update version v1.11.5 ([kube-ovn]: update version v1.11.5 #10125, @yankay)
• [multus] Fix loop_control template error when item is None (fix(multus): loop_control template error when item is None #10347, @nicolas-goudry)

API Change

• Unless the pod security standard versions are changed on intentionally, as default it will be the same major version with Kubernetes version. ([podSecurityConfiguration]: fix apiVersion and change default policy … #10210, @ugur99)
• Upgrade ansible to 7.0 and ansible-core to 2.14.x (Upgrade ansible #10190, @MrFreezeex) ⚠️ (See Notes 2)

Documentation

• Add github container registry (github_image_repo) to docs/offline-environment.md (DOCU: Add github container registry to docs/offline-environment.md #10265, @blackliner)
• Update doc for ansible-core 2.14 support and clarify issues running older python versions (Update doc after ansible-core upgrade to 2.14 #10261, @MrFreezeex)
• Update links for aws_alb_ingress_controller (link update for aws_alb_ingress_controller #10264, @kundan2707)
• Update links in ingress-controller and kuberentes-apps (Updates the broken links in ingress-controller and kubernetes-apps un… #10239, @vaibhav2107)
• Update Calico to lowercase and fix broken calico link in README (Update Calico to lowercase and fix broken calico link in README #10232, @Xieql)
• Document containerd command to restart nginx-proxy container when adding control plane node (docs: add containerd command to restart nginx-proxy #10406, @nicolas-goudry)

Failing Test

• Increase metallb wait timeout from 30sec to 2min (metallb: increase wait timeout from 30s to 2m #10260, @MrFreezeex)
• Update CentOS 7 image and test fedora 37 and 38 instead of fedora 35 and 36 (Use supported version of fedora in CI #10108, @MrFreezeex)

Bug or Regression

• Fix Dockerfile for newest directory layout (Fix Dockerfile for newest directory layout #10128, @dabeck)
• Fix Flatcar bootstrap issues (yaml module missing and ntp issue) (fix: flatcar bootstrap #10363, @tenni-paws)
• Fix argocd install not working using the kubespray docker image (fix argocd install not working using the kubespray docker image #10371, @cortex3)
• Fix correctly mount ssl ca directories (fix: correctly mount ssl ca directories #9794, @maxime1907)
• Fix etcdctl copy operation (Fix etcdctl copy operation #10230, @ErikJiang)
• Fix gce-pd-csi driver (fix gce-pd-csi driver #10208, @ashishsinghdev)
• Fix grep command without -w option causing prefix matched while adding one etcd member (bugfix: fix grep command without -w option causing prefix matched whi… #10291, @yangsenzk)
• Fix hcloud-cloud-controller-manager not working in certain setups (fix hcloud-cloud-controller-manager not working in certain setups #10297, @cortex3)
• Fix helm (kubelet-csr-approver) installation on redhat distro (helm: fix pyyaml package on RH distros #10204, @MrFreezeex)
• Fix kubelet-csr-approver usage with upgrade-cluster.yml and missing package with helm role (Kubelet csr approver fixes #10165, @j4m3s-s)
• Fix nginxingress-class template (missing newline) (fix missing newline in template which breaks nginx-ingress deployment #10174, @richard-fairthorne)
• Fix problem migration problem with k8s 1.27 (Fix problem migration to k8s 1.27 #10136, @batazor)
• Fix reset_confirmation not working when inputing correct value (reset_confirmation in reset.yml #10288, @somewho)
• Fix wrong path in manage-offline-files script (fix manage-offline-files script - wrong path #9886, @Medosopher)
• Fix an issue where using Rocky Linux 8 as OS for Vagrant for testing purposes causing etcd to fail on start. (Stop firewalld for rockylinux8 on Vagrant #10252, @nltimv)
• Fix ansible-lint galaxy rule (project: fix galaxy ansible-lint rule #10277, @MrFreezeex)
• Fix ansible-lint key-order error (project: resolve ansible-lint key-order rule #10314, @MrFreezeex)
• Fix outdated tag and experimental ansible-lint rules (Fix outdated tag and experimental ansible-lint rules #10254, @MrFreezeex)
• Fix dockerfile build error (Fix dockerfile build error by add some package #10127, @yankay)
• Fix metrics-server deployment to run with kubernetes 1.26+ (Fix metrics-server for k8s 1.26 #10183, @mzaian)
• Fix undefined reset_confirmation_prompt variable in reset play (fix: define undefined variable for reset #10303, @Mishavint)
• Fix CIS Kubernetes V1.23 Benchmark item number 4.1.9 to enhance security (Change kubelet-config.yaml and kubelet.env file permissions from 640 to 600) (improve security with CIS Kubernetes V1.23 Benchmark item number 4.1.9 #10304, @satandyh)
• Fix parsing of RHSM proxy configuration (fix parsing of RHSM proxy configuration (#10060) #10228, @tmurakam)
• Fix var-spacing ansible rule (project: fix var-spacing ansible rule #10266, @MrFreezeex)
• Fix specify owner to kube_owner in task of copy cni plugins (fix: specify owner to kube_owner in task of copy cni plugins #10407, @NierYYDS)
• Fix typo kubelet_topoloy_manager_policy => kubelet_topology_manager_policy (Fix Typo kubelet_topology_manager_policy #10384, @hangscer8)
• Fix recover_control_plane playbook (also add debian 12 with cilium as a new nightly test) (Fixup recover control plane playbook + add debian12/cilium test #10411, @floryut)
• Fix nameserver inline comments in /etc/resolv.conf (Fix nameserver inline comments in /etc/resolv.conf #10415, @yankay)
• Added systemd_resolved_disable_stub_listener variable to disable systemd-resolved's stub listener, defaults to true on Flatcar. (Add systemd_resolved_disable_stub_listener #9875, @cosandr)
• Remove auto_attach and syspurpose in RHEL subscription Organization ID/Activation Key registration. (Fix RHEL subscription activation key by removing auto_attach and syspurpose #10258, @yckaolalala)
• Replace "crio_packages" with "crio_bin_files" (remove cri-o using crio_bin_files #10182, @yckaolalala)
• Update MetalLB deployment, wait for resource. (Update MetalLB deployment, wait for resource. #9995, @Jeroen0494)
• Upgrade ansible to 7.0 and ansible-core to 2.14.x in Dockerfile (Upgrade ansible in Dockerfile #10259, @yckaolalala)
• Fix typo kubelet_topoloy_manager_policy => kubelet_topology_manager_policy (Fix Typo kubelet_topology_manager_policy #10384, @hangscer8) ⚠️ (See Notes 1)
• Change maximal_ansible_version to 2.15(exclusive) (Change maximal_ansible_version to 2.15 #10395, @yankay)
• Install etcdutl file by default (Install etcdutl file by default #10385, @liupeng0518)

Other (Cleanup or Flake)

• [CI] Add CI VM for debian12 (Add CI VM for debian12 #10222, @yankay)
• [CI] Removes Ansible reinstall from build pipeline (Removes Ansible reinstall in CI pipeline #10032, @luksi1)
• [CI] cleanup stale packet namespace automatically (tests: cleanup stale packet namespace automatically #10245, @MrFreezeex)
• [CI] fix tf-elastx_cleanup fail ([CI] fix tf-elastx_cleanup fail #10133, @yankay)
• [CI] Sanitize branch name in testing before using it in kubernetes label for packet-ci (tests/packet-ci: sanitize branch name for kubernetes labels #10315, @MrFreezeex)
• Add an exception for youki in download_hash script (not requiring 'v' in youki version #10346, @ErikJiang)
• Drop support for Kubernetes 1.24.x (move min version to 1.25.x) (Update KUBESPRAY_VERSION and kube_version_min_required (with hashes cleanup) for v2.22.0 #10126, @yankay)
• Ensure host entries from /etc/host are absent when populate_inventory_to_hosts_file is false (clean up /etc/hosts file if populate_inventory_to_hosts_file is false #10144, @rptaylor)
• Exclude terraform.tfstate backups in .gitignore (exclude terraform.tfstate backups in .gitignore #10216, @rptaylor)
• Ping is no longer reported as a changed task (Running ping doesn't change state #10160, @jcpunk)
• Reading mounted volumes no longer considered a changed task (Locate mount names isn't a change to the system #10161, @jcpunk)
• Resolve ansible-lint name errors (Resolve ansible-lint name errors #10253, @MrFreezeex)
• Update KUBESPRAY_VERSION for v2.22.1 (Update KUBESPRAY_VERSION in CI and Readme for v2.22.1 #10201, @yankay)

Supported Components

• Core
  • kubernetes v1.27.5
  • etcd v3.5.7
  • docker v20.10 (see note)
  • containerd v1.7.5
  • cri-o v1.27 (experimental: see CRI-O Note. Only on fedora, ubuntu and centos based OS)
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.25.2
  • cilium v1.13.4
  • flannel v0.22.0
  • kube-ovn v1.11.5
  • kube-router v1.5.1
  • multus v3.8
  • weave v2.8.1
  • kube-vip v0.5.12
• Application
  • cert-manager v1.11.1
  • coredns v1.10.1
  • ingress-nginx v1.8.1
  • krew v0.4.4
  • argocd v2.8.0
  • helm v3.12.3
  • metallb v0.13.9
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.22.0
  • gcp-pd-csi-plugin v1.9.2
  • local-path-provisioner v0.0.24
  • local-volume-provisioner v2.5.0

Known issues

N/A

Notes

1. Variable kubelet_topoloy_manager_policy change to kubelet_topology_manager_policy, please update your inventory
2. Upgrade ansible to 7.0 and ansible-core to 2.14.x

[dibi-codes]

I would love to see #10198 merged ❤️

[mzaian]

I'm adding 1.27.5 to this release together with more updated versions PR is here: #10392

[yankay]

#10395 is also want to merged

[mzaian]

Please consider #10397 and #10400

[yankay]

HI @floryut @mzaian @oomichi @MrFreezeex

The v2.22 will be released when all the block issues are merged.

The PR block the release:

• Add huawei cloud controller #10198
• [kubernetes] Make 1.27.5 default #10392
• Change maximal_ansible_version to 2.15 #10395
• [containerd] add hashes for 1.7.4-5 #10397
• [feat] Update metrics server to v0.6.4 #10400
• Update load balancers Versions to Nginx 1.25, Haproxy 2.8 #10409
• Fix Typo kubelet_topology_manager_policy #10384
• Update etcd version on README #10410
• Fixup recover control plane playbook + add debian12/cilium test #10411
• Update calico v3.25.2 #10414

The PR not block the release:

• [calico] Make version 3.26.1 default #10416
• Fix nameserver inline comments in /etc/resolv.conf #10415

[sathieu]

It would be great to upgrade etcd. We were hit by etcd-io/etcd#15090 (there is #10047 which is outdated).

[floryut]

HI @floryut @mzaian @oomichi @MrFreezeex

The v2.22 will be released when all the block issues are merged.

The PR block the release:

• Add huawei cloud controller #10198
• [kubernetes] Make 1.27.5 default #10392
• Change maximal_ansible_version to 2.15 #10395
• [containerd] add hashes for 1.7.4-5 #10397
• [feat] Update metrics server to v0.6.4 #10400
• Update load balancers Versions to Nginx 1.25, Haproxy 2.8 #10409
• Fix Typo kubelet_topology_manager_policy #10384

Thank you, I was away for the last 2 weeks, but I'll check all PR/issues I've missed 🙇

But again, thank you man!

[mzaian]

It would be great to upgrade etcd. We were hit by etcd-io/etcd#15090 (there is #10047 which is outdated).

Please check my answer here #10047 (comment)

I created this PR for updating the readme #10410

@yankay Please consider this in the release

[floryut]

@yankay Would need this one to be merged before releasing, fixing the recover CP bug 🙇
#10411

[mzaian]

Last call and I hope it's possible #10414

[yankay]

@floryut @mzaian

The kubespray will be released tomorrow(2023-9-8),
Is there any concern for it ?

Thanks

[floryut]

@floryut @mzaian

The kubespray will be released tomorrow(2023-9-8), Is there any concern for it ?

Thanks

All good, one comment though: maybe regenerate release notes if you could ? or if not I'll add manually the few PR merged since

[mzaian]

@floryut @mzaian

The kubespray will be released tomorrow(2023-9-8), Is there any concern for it ?

Thanks

I just created this #10416 as per request. It seems to be stable enough so we can make the release with it.

/cc @yankay

[yankay]

@floryut @mzaian
The kubespray will be released tomorrow(2023-9-8), Is there any concern for it ?
Thanks

All good, one comment though: maybe regenerate release notes if you could ? or if not I'll add manually the few PR merged since

Thanks @floryut
The release notes have been regenerated, and merged into in the issue :-)
And I will update the release notes again just before the release.

[yankay]

@floryut @mzaian
The kubespray will be released tomorrow(2023-9-8), Is there any concern for it ?
Thanks

I just created this #10416 as per request. It seems to be stable enough so we can make the release with it.

/cc @yankay

HI @mzaian

Because the calico is the default CNI, If we merge it now, we may not have time to find possible issues before release.
So should we change it after the v2.23 is released? And we will have a lot of time to test it.

What do you think about that :-)

[mzaian]

Hello @yankay

I agree please go ahead without this PR.

@floryut @mzaian
The kubespray will be released tomorrow(2023-9-8), Is there any concern for it ?
Thanks

I just created this #10416 as per request. It seems to be stable enough so we can make the release with it.
/cc @yankay

HI @mzaian

Because the calico is the default CNI, If we merge it now, we may not have time to find possible issues before release. So should we change it after the v2.23 is released? And we will have a lot of time to test it.

What do you think about that :-)

Hello @yankay

I agree please go ahead without this PR.

[yankay]

Thanks @mzaian :-)

[floryut]

Thank you @yankay 🎉