General fixes

[r-tierney]

This was tested on Debian bookworm with kubernetes version 1.26 and 1.27, calico v3.25

UdpIdleTimeout has been deprecated:

• https://github.com/kubernetes-sigs/kubespray/pull/9925/files
• https://github.com/kubernetes/kubernetes/pull/112133/files

Kubernetes has moved the registry to:

• registry.k8s.io
  Reference as seen in the red banner on their website:
• https://kubernetes.io/

Calico requires a v before the version number without it you get a 404 Example:

• https://docs.projectcalico.org/archive/v3.18/manifests/calico.yaml
• https://docs.projectcalico.org/archive/3.18/manifests/calico.yaml

container-runtime remote has been deprecated as the only possible value was remote

• Reference: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
  Error i received:

Apr 27 15:04:22 kubec01-mel kubelet[3790]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-co>
Apr 27 15:04:22 kubec01-mel kubelet[3790]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
Apr 27 15:04:22 kubec01-mel kubelet[3790]: E0427 15:04:22.062236    3790 run.go:74] "command failed" err="failed to parse kubelet flag: unknown flag: --container-runtime"

discovery token from kubetool didnt work ( found that i needed to change rsa to pkey ) as we can see from 2 different clusters using the command with rsa gives the same result.

kubec01-rya.ops:/home/users/ryant# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
kubec01-rya.ops:/home/users/ryant# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl pkey -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
adcc248bb5ab39eb750a85f941ccfc6bfd1eef133a5aca57989ccda0eacedbdd
kubec01-rya.ops:/home/users/ryant#

kubec01-san:/home/users/ryant# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
kubec01-san:/home/users/ryant# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl pkey -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
84dcf2e02d0c93f1cfb2fc7af1f7757dcd204c26a6ca286a620a2bd8dc6796c2
kubec01-san:/home/users/ryant#

I found that on Debian the kubelet would constantly crash as the kubelet's default cgroupDriver on Debian is set to systemd

ryant@kubec01-san:~$ cat /var/lib/kubelet/config.yaml | grep -i cgroup
cgroupDriver: systemd
ryant@kubec01-san:~$

and this modules default sets containerd's cgroup_driver to cgroupfs if its not running on redhat ( found in init.pp )
The fix for Debian ( Should this just be the default for both Debian and Redhat now? ) as recommended by kubernetes reference

class { '::kubernetes':
    cgroup_driver => 'systemd',
}

The above change sets the following in containerd's config which causes kubelet and containerd to work on Debian

ryant@kubec01-san:~$ cat /etc/containerd/config.toml | grep -i 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options' -A 1
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true
ryant@kubec01-san:~$

And lastly calico required the mount to be shared:
Error reported

Apr 28 17:00:56 kubec01-san kubelet[11687]: E0428 17:00:56.926812   11687 remote_runtime.go:302] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to generate container \"866d1ce88b3fc72e4a455ec485dfee0e3ef7cdeb84bc8146dfb949d7c87ce267\" spec: failed to generate spec: path \"/sys/fs/\" is mounted on \"/sys\" but it is not a shared mount" podSandboxID="85df3c3dfa95f1878a4b328f93cbeb021a1df366fdc8214440cd64d44451a364"

The fix ( This solution requires the puppet module mount_core ):

  # For calico requires mount to be shared
  mount { "/" : atboot => yes, options => "rshared", name => "/", ensure => mounted, remounts => true, pass => "0" } ~>
  exec { "/usr/bin/mount --make-rshared /" : refreshonly => true }

Fixes #584

[puppet-community-rangefinder]

kubernetes is a class

Breaking changes to this file MAY impact these 5 modules (near match):

• jetstack-fluent_bit
• jetstack-tarmak
• jetstack-prometheus
• jetstack-calico
• jetstack-kubernetes_addons

This module is declared in 0 of 580 indexed public Puppetfiles.

---

These results were generated with Rangefinder, a tool that helps predict the downstream impact of breaking changes to elements used in Puppet modules. You can run this on the command line to get a full report.

Exact matches are those that we can positively identify via namespace and the declaring modules' metadata. Non-namespaced items, such as Puppet 3.x functions, will always be reported as near matches only.

[CLAassistant]

All committers have signed the CLA.

[r-tierney]

Discovery token fixed in:
1871b5c

Using pkey instead of rsa

[jordanbreen28]

@r-tierney this is brilliant - can you rebase the PR?

[r-tierney]

Thanks @jordanbreen28, I've updated this branch from main

[jordanbreen28]

@r-tierney apologies... I missed the notification.
Can you rebase once again and clean up the merge commits? Then we can get this progressed.
Thanks

[r-tierney]

@jordanbreen28 Sure thing, updating now

[r-tierney]

rebase complete

[jordanbreen28]

Nice one @r-tierney - I'll merge in once green!
thanks again for this massive effort.

[r-tierney]

Not a problem at all, glad to help.

The issue which took the longest to troubleshoot was actually this modules default setting for the cgroup_driver located on line 741 of init.pp which had it set to cgroupfs by default instead of systemd and would cause a conflict with kubelet as kubelets default setting on Debian is systemd.

With the pods and kubelet crashlooping it took some time to work out that was the issue as without the kubelet running it's hard to run a kubectl describe etc to figure out why a pod is crashlooping.

I understand changing the default for a setting like this may break those not running systemd so I left it out of this pull request but thought I'd mention it anyway and leave the decision up to your team whether or not to change it or add a mention in some docs somewhere.

[jordanbreen28]

Not a problem at all, glad to help.

The issue which took the longest to troubleshoot was actually this modules default setting for the cgroup_driver located on line 741 of init.pp which had it set to cgroupfs by default instead of systemd and would cause a conflict with kubelet as kubelets default setting on Debian is systemd.

With the pods and kubelet crashlooping it took some time to work out that was the issue as without the kubelet running it's hard to run a kubectl describe etc to figure out why a pod is crashlooping.

I understand changing the default for a setting like this may break those not running systemd so I left it out of this pull request but thought I'd mention it anyway and leave the decision up to your team whether or not to change it or add a mention in some docs somewhere.

Yeah the removal of cgroupfs as the default driver would need to be part of a major release due to the high possibility it may break things, we would need to document this also.
Systemd is now the recommended for both debian and rhel based distros, so should probably be progressed in the next major release.

If you want to go ahead and create a seperate PR for that, I will try to ensure its included in the next major release (which should be in the next week or two due to puppet 8).

Anyways, happy to merge this! 🥇