Duplicate metrics in K3s

[cyc1ingsir]

In k3s-overrides scraping of the ports 10251 and 10252 is defined.
Running a K3s installation in version v1.22.3+k3s1, port 10252 seems to be no longer accessible. I've set up the K3s via (https://github.com/k3s-io/k3s-ansible/) on a Rasperrry Pi.
All I could find for the kube-controller-manager in the logs was a --secured-port=10257. I haven't tested using this port though.
When searching for the root cause I discovered this issue:
k3s-io/k3s#2262
Conclusion of the discussion is, that is is advisable to scrape port 10249 only since it includes all metrics. I haven't found out how one could do this.
The symptoms when using version are that lot's of KubeControllerManagerDown - alerts are firing: e.g. when looking at the Kubernetes cluster monitoring (via Prometheus) - Dashboard.
It will show up when k3s: {enabled: true, is selected inside vars.jsonnet

Regarding the secured-ports, this answer might be helpful too:
k3s-io/k3s#3619 (comment)

[ToMe25]

After some testing with k3s v1.20.12+k3s1 it seems like out of the ports 10249, 10250, 10251, and 10252 10250 is the only one that is handled by k3s.

Why did I test with k3s v1.20?
Because the version of kube-prometheus used in this project only supports kubernetes versions up to v1.20 iirc.
At least I am pretty certain it doesn't support newer versions at least, not sure if it officially supports v1.20 tho.

Anyways, since it seems like all four ports had the same content I would recommend only scaping 10250 then, rather then the non existing 10249.
Also port 10250 is already being scraped.

This was tested with k3d 5.1.0, because the Raspberry Pis my cluster ran on recently died.

It seems like this is the reason why 2 targets are permanently offline, which was mentioned more then once before.
So just removing the service monitors and discovery services for those should make it work fine.