Release Proposal v2.13.0

[Miouge1]

Below is a proposal for release notes:

Announcements

We are looking for maintainers, reach out in #5432.

Deprecation / Removal

• Removed support for Fedora 28 (EOL since May 2019)
• Removed support for Fedora Atomic (see EOL announcement)
• CoreOS is deprecated (See upstream announcement)
• docker_container_storage_setup variable is deprecated (was supported only on RHEL and Centos 7)

Major changes:

• Ansible v2.9 required
• Adds support for Fedora 30 and 31
• Adds Flatcar Linux support (fork of CoreOS, see update instructions) (Add flatcar #5607)
• Adds support for Centos 8 (CentOS 8 CI #5842 docker: Fix docker install on CentOS/RHEL 8 #5820)
• Add experimental support for Fedora CoreOS (Fedora CoreOS support #5657)
• Helm 3 is the new default if helm_enabled: true (Set helm 3.0 as default #5503)
• Kubernetes dashboard v2.0 ([2.12] kube-proxy need conntrack (#5478) #5828). The kubernetes-dashboard team recommends to install the dashboard in a separate namespace, to keep compatibility kubespray defaults to kube-system but this can be change using the new dashboard_namespace variable
• contrib/terraform/openstack: Add az_list_node variable to specify different AZs for nodes (add az_list_node variable to specify different AZs for kubelets #5413) (action is required if you don't use the default "nova" AZ)

Applications

• Add external cloud provider for Openstack (External OpenStack Cloud Controller Manager implementation #5491)
• Add external cloud provider and CSI for vSphere (vSphere CPI & CSI implementation #5959)
• Add CSI controller for Openstack Cinder (Deploy Cinder CSI driver to provision volumes over OpenStack #5184)
• Add CSI controller for Azure disk (Azure Disk CSI deployment #5833)
• Add CSI controller for Google Cloud (GCP Persistent Disk CSI Driver deployment #5857)
• Add CSI controller for AWS EBS (AWS EBS CSI implementation #5549)
• Add support for AWLS ALB Ingress controller (Add AWS ALB Ingress Controller #5489)

Container managers

• Add configuration for containerd runtimes (Add variables to configure Containerd default runtime, untrusted runt… #5497)
• Added proxy support for containerd (Add proxy support to containerd, improves no_proxy #5583)
• CRI-O packages from Kubic repositories(Centos, debian and fedora CRI-O repo #6008)

Network

N/A

Other note worthy changes

• Add support to download all on one host and distribute on cluster nodes (Refactor download role #5697)
• PodSecurityPolicies configurable (Customize PodSecurityPolicies from inventory #5920)
• contrib/terraform/openstack: Allow free form work node defintion (Terraform/OpenStack: Allow free form worker node definition #5952)
• contrib/terraform/openstack: Allow use of existing router (Terraform/OpenStack: Enable usage of an existing router #5890)
• Support configuring the Calico iptables insert mode (Support configuring the Calico iptables insert mode #5473)
• Allows to configure additionnal zones on nodelocaldns for domains not resolved by upstream_dns_servers. (Add external zones in nodelocaldns configuration #5591)
• Improved deploy time in large clusters (Gather just the necessary facts #5955 and assembly fallback_ips and no_proxy var only one time on localhost and… #5957)
• Improved support for mitogen (add strategy mitogen_linear when installed mitogen #5985)
• Calico usage report disabled now by default (Add option to enable usage reports to calico servers #6030)

Component versions:

• Kubernetes v1.17.5
• Etcd 3.3.12
• Docker 18.06
• containerd 1.2.13
• Cri-O 1.17
• CNI-plugins v0.8.5
• Calico v3.13.2
• Cilium 1.7.2
• Contiv 1.2.1
• Flannel 0.12.0
• Kube-Router 0.4.0
• Multus 3.4.1
• kube-ovn 1.10
• Weave 2.6.2
• CoreDNS 1.6.5
• nodelocaldns 1.15.12
• Helm 3.1.2
• nginx-ingress 0.30.0
• cert-manager 0.11.1
• Kubernetes Dashboard v2.0.0
• Oracle OCI: v0.7.0

Known issues

TBD

Notes

N/A

[floryut]

@Miouge1 nodelocaldns is up to 1.15.10 not 1.15.8, weave up to 2.6.2 and Cilium up to 1.7.2 (README not up to date but will be following my PR)
Hopefully Kub dashboard will release their final 2.0 soon... (Milestone already past 10 days)

[floryut]

(cert-manager 0.11.1 instead of .0)

[floryut]

• Kubernetes version up to 1.17.5
• You are missing kube-ovn to 1.1.0
• helm version up to 3.1.2
• CoreDNS down to 1.6.7 (due to bug with k8s migration) -> PR soon

[EppO]

* `Kuberentes` version up to 1.17.5
* You are missing `kube-ovn` to 1.1.0
* `helm` version up to 3.1.2
* `CoreDNS` down to 1.6.7 (due to bug with k8s migration) -> PR soon

Actually, for etcd and CoreDNS versions, I would align kubespray with kubeadm's defaults (kubeadm config images list), except huge bug fixed in upstream. WDYT?

[floryut]

* `Kubernetes` version up to 1.17.5
* You are missing `kube-ovn` to 1.1.0
* `helm` version up to 3.1.2
* `CoreDNS` down to 1.6.7 (due to bug with k8s migration) -> PR soon

Actually, for etcd and CoreDNS versions, I would align kubespray with kubeadm's defaults (kubeadm config images list), except huge bug fixed in upstream. WDYT?

That would be coreDNS:1.6.5 and etcd:3.4.3 right ?
That could be nicer as 1.17 bundle corefile-migration:1.0.4 which support coreDNS up to 1.6.5.

@EppO Regarding etcd, you already tried to upgrade it to latest 3.3.x without luck, did you try with 3.4.x ?

[EppO]

* `Kuberentes` version up to 1.17.5
* You are missing `kube-ovn` to 1.1.0
* `helm` version up to 3.1.2
* `CoreDNS` down to 1.6.7 (due to bug with k8s migration) -> PR soon

Actually, for etcd and CoreDNS versions, I would align kubespray with kubeadm's defaults (kubeadm config images list), except huge bug fixed in upstream. WDYT?

That would be coreDNS:1.6.5 and etcd:3.4.3 right ?
That could be nicer as 1.17 bundle corefile-migration:1.0.4 which support coreDNS up to 1.6.5.

@EppO Regarding etcd, you already tried to upgrade it to latest 3.3.x without luck, did you try with 3.4.x ?

I tried 3.3.20 on my systems because I thought I got hit by this etcd bug. Works well on my side but it's true that my previous etcd upgrade attempt on Kubespray CI was not very conclusive.
I can give it a try with etcd 3.4.3, see how it behaves.

[floryut]

* `Kubernetes` version up to 1.17.5
* You are missing `kube-ovn` to 1.1.0
* `helm` version up to 3.1.2
* `CoreDNS` down to 1.6.7 (due to bug with k8s migration) -> PR soon

Actually, for etcd and CoreDNS versions, I would align kubespray with kubeadm's defaults (kubeadm config images list), except huge bug fixed in upstream. WDYT?

That would be coreDNS:1.6.5 and etcd:3.4.3 right ?
That could be nicer as 1.17 bundle corefile-migration:1.0.4 which support coreDNS up to 1.6.5.
@EppO Regarding etcd, you already tried to upgrade it to latest 3.3.x without luck, did you try with 3.4.x ?

I tried 3.3.20 on my systems because I thought I got hit by this etcd bug. Works well on my side but it's true that my previous etcd upgrade attempt on Kubespray CI was not very conclusive.
I can give it a try with etcd 3.4.3, see how it behaves.

I'd say go for it, I've changed my coreDNS PR to 1.6.5, we'll bump it back to 1.6.7 when 1.18 is set as default version 👍

[floryut]

Following all that, CoreDNS is now down to 1.6.5 and nodelocaldns up to 1.5.12

[spaced]

context: flatcar/coreos/fcos/atomic:
to be correct:

• fcos supersedes coreos (redhat/fedora)
• flatcar is a fork of coreos by kinvok
• atomic is removed

you write it like: coreos is deprecated, take flatcar.
i appreciate flatcar, and what the small kinvok team is doing, but i would be carefully with recommendation for a production ready kubernetes.

[Miouge1]

@spaced I've added the "Removed support for Fedora Atomic" and rewrote the coreos & flatcar part.

Kubespray is not recommending any specific migration path from coreos, that's up to each organization to see what fits them best.

[EppO]

Working on the etcd part, hope to have something ready soon. kubeadm/docker deployment modes work out of the box, still need to fix the host deployment mode to handle both 3.3.x/3.4.x use cases

[LuckySB]

what about:
speedup kubespray on setup with 10+ nodes
PR: #5957 #5955

Refactor role download: Now you may download all on one host and distribute on cluster nodes via kubespray.
PR: #5697

[Miouge1]

@LuckySB those PRs will be included in the v2.13 release (since it will be a git tag from master at that point in time).

In the release note we try to list things that have user facing consequences (new features, new versions etc...).
Maybe we can add "Performance improvement" or "Improvement in large scale clusters" or something of that nature? Maybe you have an idea?

[spaced]

@Miouge1

rewrote the coreos & flatcar part.

much better and neutral now, i like it.
just a small typo: CoreOS instead of Coreos

[LuckySB]

@LuckySB those PRs will be included in the v2.13 release (since it will be a git tag from master at that point in time).

In the release note we try to list things that have user facing consequences (new features, new versions etc...).
Maybe we can add "Performance improvement" or "Improvement in large scale clusters" or something of that nature? Maybe you have an idea?

Maybe just add in "Other note worthy changes" ?

[Miouge1]

Fixed CoreOS and added those PRs @LuckySB

[huxcrux]

Under application it would be good to differentiate cloud provider and CSI controllers, example how it could look like:

Applications
Add external cloud provider for Openstack (#5491)
Add external cloud provider and CSI for vSphere (#5959)
Add CSI controller for Openstack Cinder (#5184)
Add CSI controller for Azure disk (#5833)
Add CSI controller for Google Cloud (#5857)
Add CSI controller for AWS EBS (#5549)
Add support for AWLS ALB Ingress controller (#5489)

Also since booth docker-storage and contiv still lacks any maintainers will those be added as deprecated?

[Miouge1]

Good idea @bl0m1

I've updated the list of application changes to split cloud controllers and CSI.

I've added docker_container_storage_setup var as deprecated since that's what triggers docker-storage. It supports only centos7 and RHEL7, and the upstream project doesn't look too active.

Finally I've opened #6059 to discuss how to go about the contiv situation.

[EppO]

do people use something else than overlay2 these days anyway? device-mapper was such a PITA to setup...

[Miouge1]

@EppO AFAIK lots of overlay2, if someones wants to maintain the docker-storage role and bring it up to speed I don't have a problem bringing it back for prime time.

Since v2.13.0 is released, I'll close this issue.