Migrate to Kubernetes 1.19+ and Rancher 2.5

[asoltesz]

Reasons

• VolumeSnapshot API for backup tools
• Other, non-beta API versions (K3S compatibility)
• Newer, more powerful Rancher

VolumeSnapshot API

Currently, VolumeSnapshot based backups cannot be implemented because both Stash and Velero requires the v1beta1 API version and that is not available on Kubernetes v1.15.x that we target. Note: the v1alpha1 API version is there but that is ignored by both.

The VolumeSnapshot v1beta1 API is first available in Kubernetes 1.17.

IMPORTANT NOTE: Neither Velero 1.4, nor Stash 0.90 implements consistent filesystem backups by using VolumeSnapshots and Restic in a combined manner. Without this, consistent, online, point-in-time, offsite backup of the VolumeSnapshotContent cannot be implemented. In theory, Velero has a mechanism for this but it needs a special volume plugin which is not available for neither Rook/Ceph, nor Hetzner Cloud Volumes.

Restic based backups are inconsistent when the workload is online (files may be changed by the workload while the Restic backup is progressing ahead on a large volume filesystem)

Via the VolumeSnapshot API, point-in-time, online volume backups would become available for at least Rook/Ceph.

Stash issue: stashed/stash#1099
Velero issue: vmware-tanzu/velero#2671

Other, non-beta API versions (K3S compatibility)

K3S is a lightweight Kubernetes distribution for Rancher. It is ideal for running on low-end hardware and cheaper VMs.

However, K3S drops all of the old API versions where there are production ones (e.g.: apps/v1)

SolaKube could work against K3S clusters if all components were updated to new APIs.

Newer, more powerful Rancher

Rancher 2.5 is more advanced than the 2.3 version we use for Kubernetes 1.15 cluster building (RKE).

Upgrading to 2.5 would allow using the new Cluster Explorer feature and other advancements in Rancher.

Issues / checklist

• All deployers, Helm charts and deployment descriptors moved to API levels supported by Kubernetes 1.19
• Test cluster builds with Rancher 2.5 and Kubernetes 1.19.4
• FirewallD rules for required ports

FirewallD issues (port requirements)

Rancher port requirements may have changed because I can only make cluster deployment finish in Rancher if I completely disable firewalld on the CentOS 7 nodes.

[asoltesz]

Currently, the 0.7 (develop) can only succeed in provisioning a cluster if we completely disable FirewallD (comment it out in provision.yaml).

Issue is discussed here: rancher/rancher#28840

Rejected firewalld traffic can be investigated like this