Upgrade kube-prometheus-stack to 67.11.0

[anders-elastisys]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

Application Developer notice

Prometheus has been upgraded to version 3.0. This includes changes to the Prometheus UI. Prometheus V3 comes with some changes that may affect existing PromQL expressions in alerts or dashboards. Please have a look at the Prometheus V3 migration guide.

Security notice

Upgraded Prometheus to v3.1.0 to address CVE-2024-45337

What does this PR do / why do we need this PR?

Noticed that the kube-prometheus-stack was falling behind a bit, this PR upgrades the Helm chart to v67.5.0 which also upgrades Prometheus to v3..
I checked the v3 migration guide and I did not see that we are currently using any of breaking flags or configurations in our default Welkin config, but please verify if this is used in some environments.

This fixes some ARP metrics and a log issue caused by this in the node-exporter (this is mentioned in the linked issue).

Alertmanager in the Mangement cluster is not upgraded, instead the image version is fixed to previous v0.26.0 due to v0.27.0 deprecating the v1 API endpoint, which is still used by Thanos.
Once we upgrade Thanos to v0.35 or higher, the v2 endpoint will be default (see related upstream issue) and we can remove the image override.

• Fixes Upgrade Kube-prometheus-stack-60.0.0 #2166

Information to reviewers

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - link to change
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[OlleLarsson]

Did you account for the change that they mention here? We seem to set these to false in the wc kps config

[Xartos]

Question: Does this comparison work with versions?

Wouldn't this just need to be

Suggested change

	if [[ ! "${current_version}" < "$(echo -e "${new_version}\n${current_version}" | sort -V | tail -n1)" ]]; then
	if [[ "${current_version}" != "${new_version}" ]]; then

[anders-elastisys]

The comparison should work in most cases since it sorts lexicographically and the kps helm release will be a couple of major versions behind in most cases. But I like your suggestion, then it should be possible to do downgrade as well. PTAL 78f5a5c

[Elias-elastisys]

Do we need to care about the PromQL change to the dot token?

Just looking quickly I found usage in both alerts and dashboards, I'm sure they are everywhere.

compliantkubernetes-apps/helmfile.d/charts/prometheus-alerts/templates/records/k8s.rules.yaml

Line 130 in c8783f3

"workload", "$1", "owner_name", "(.*)"

compliantkubernetes-apps/helmfile.d/charts/grafana-dashboards/dashboards/backup-dashboard.json

Line 94 in c8783f3

"expr": "min((time()-kube_job_status_completion_time{job_name=~\"harbor-backup-cronjob-.*\", cluster=~\"$cluster\"})/3600)",

Have you checked if all dashboards behave the same?

[anders-elastisys]

Have you checked if all dashboards behave the same?

I did a quick check comparing some dashboards that contains this regex and did not see any noticeable difference.
Difficult to verify all cases where this is found, I am doubting that we have newlines in any label values that would cause problems here, but if we are unsure we could do as suggested in the migration doc and use [^\n] for all occurrences.

[anders-elastisys]

@OlleLarsson I missed that, thanks, PTAL bb54e21

[Xartos]

I did a quick check comparing some dashboards that contains this regex and did not see any noticeable difference.
Difficult to verify all cases where this is found, I am doubting that we have newlines in any label values that would cause problems here, but if we are unsure we could do as suggested in the migration doc and use [^\n] for all occurrences.

Do you know if the upstream dashboards/rules have done this?

[anders-elastisys]

@OlleLarsson I missed that, thanks, PTAL bb54e21

Nvm, this got reverted in v64, I will go back to how it was before, since this also causes our unit tests to fail due to schema issues.

[anders-elastisys]

Do you know if the upstream dashboards/rules have done this?

Not from what I have seen, you can see that the regular expressions used in the upstream kube-prometheus-stack alerts and dashboards part of this PR did not get changed to address this new change, so I do not think we should be affected.

[OlleLarsson]

@OlleLarsson I missed that, thanks, PTAL bb54e21

Nvm, this got reverted in v64, I will go back to how it was before, since this also causes our unit tests to fail due to schema issues.

Great, super that you noticed that they reverted the changes in the release after where they introduced the changes 😄!

[viktor-f]

I think this looks good. Note that I have not looked very closely at the changelogs.

[aarnq]

Mainly reviewing the migration.

[anders-elastisys]

I updated this PR a bit, moved the migration script to next version since v0.43 is being released atm.
I also upgraded the chart to latest minor version of v67.11 to upgrade Prometheus to v3.1.0 since there was a critical CVE in v3.0. Tested upgrading to this new chart version, did not see any major changes, there were some changes to some alerts upstream that used the le function, but I did not see that we use this in any of our alerts.

[anders-elastisys]

@Xartos @aarnq @OlleLarsson since you all left comments, do you want to take another look before I merge?

[aarnq]

LGTM