Bump prometheus-operator version #2948

alexandre-allard · 2020-11-27T09:57:48Z

Component: charts, salt, build

Context:
Addon versions bump.

Summary:
Replace prometheus-operator which is deprecated by kube-prometheus-stack in version 12.2.3

Acceptance criteria: Currently testing if everything still OK.

bert-e · 2020-11-27T09:57:50Z

Hello alexandre-allard-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2020-11-27T09:57:54Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

gdemonet

I see a mention of Helm 3 in one of your commits... If we need it, then it's something we should at least document in the developer book, and likely also update in CI, I guess?

CHANGELOG.md

charts/render.py

TeddyAndrieux · 2020-11-27T10:50:24Z

charts/render.py

    template = subprocess.check_output([
-        'helm', 'template',
-        '--name', args.name,
+        'helm', 'template', args.name,
        '--namespace', args.namespace,
        '--values', args.values,
+        '--include-crds',
        args.path,
    ])


Should we do an if/else here to check if we are with helm2 or helm3 so that the script is compatible with helm2 as well (or we just expect all charts to be compatible with helm3, I don't know if helm3 has this kind of backward compatibility guaranteed)

On my system, both binaries are named helm, so I don't see how we can handle the two versions at the same time (except if we ask developers to have both, with specific paths or aliases, i.e.: helm2 and helm3).
I think we may need to test if all charts are OK with helm 3, especially since helm 2 is now considered as deprecated.

We could easily have both,
But right if all charts are OK with helm3 we should get rid of helm2, I agree

bert-e · 2020-11-27T14:06:27Z

Conflict

There is a conflict between your branch improvement/update-prometheus-operator-addon and the
destination branch development/2.7.

Please resolve the conflict on the feature branch (improvement/update-prometheus-operator-addon).

 $ git fetch
 $ git checkout origin/improvement/update-prometheus-operator-addon
 $ git merge origin/development/2.7
 $ # <intense conflict resolution>
 $ git commit
 $ git push origin HEAD:improvement/update-prometheus-operator-addon

bert-e · 2020-11-27T17:40:41Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

This chart has been renamed into `kube-prometheus-stack`. We update the chart to the version 12.2.3. ``` rm -rf charts/prometheus-operator helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm fetch -d charts --untar prometheus-community/kube-prometheus-stack ```

Update the customization configuration file for `kube-prometheus-stack`. Also override the prefix name to have the same as before with `prometheus-operator` chart. Add node-exporter extra args to override default behavior ignoring loop devices and other kind of devices from disk stats.

kube-prometheus-stack chart is no longer compatible with helm 2, so we need to use helm 3 to render it correctly. So we slightly adapt the render script to work with helm 3

These alert rules come from the new version of kube-prometheus-stack charts. We drop them so we can add them back in an another Salt formula to allow customization by the users at runtime.

Since chart has been bumped, we need to regen it to apply changes. ``` ./charts/render.py prometheus-operator \ charts/kube-prometheus-stack.yaml \ charts/kube-prometheus-stack/ \ --namespace metalk8s-monitoring \ --service-config grafana \ metalk8s-grafana-config \ metalk8s/addons/prometheus-operator/config/grafana.yaml \ metalk8s-monitoring \ --service-config prometheus \ metalk8s-prometheus-config \ metalk8s/addons/prometheus-operator/config/prometheus.yaml \ metalk8s-monitoring \ --service-config alertmanager \ metalk8s-alertmanager-config \ metalk8s/addons/prometheus-operator/config/alertmanager.yaml \ metalk8s-monitoring \ --service-config dex \ metalk8s-dex-config \ metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \ --drop-prometheus-rules charts/drop-prometheus-rules.yaml \ > salt/metalk8s/addons/prometheus-operator/deployed/chart.sls ```

Update the node-exporter rules with the new alert rules coming from the kube-prometheus-stack version 12.2.3

grafana 6.7.4 to 7.2.1 alertmanager v0.20.0 to v0.21.0 k8s-sidecar 0.1.20 to 1.1.0 kube-state-metrics v1.9.5 to v1.9.7 node-exporter v0.18.1 to v1.0.1 prometheus v2.16.0 to v2.22.1 prometheus-config-reload v0.38.1 to v0.43.2 prometheus-operator v0.38.1 to v0.43.2

We need to regenerate these files as we bumped the Prometheus Operator version. The alerting & recording rules are generated using: ``` $ ./tools/rule_extractor/rule_extractor.py \ -i <control-plane-ip> -p 8443 -t rules ```

alexandre-allard · 2020-12-03T14:28:47Z

/approve

bert-e · 2020-12-03T14:28:57Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.7

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2020-12-04T19:16:24Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.7

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

bert-e · 2020-12-04T19:17:17Z

Queue out of order

The changeset has received all authorizations to enter the merge queue,
however the merge queue is currently in an incoherent state. In order
to protect this pull-request, I have have not added the changeset to the
queue.

Please contact a member of release engineering.

The following options are set: approve

bert-e · 2020-12-07T08:08:20Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.7

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2020-12-08T12:42:00Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.7

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

bert-e · 2020-12-08T12:43:00Z

Queue out of order

The changeset has received all authorizations to enter the merge queue,
however the merge queue is currently in an incoherent state. In order
to protect this pull-request, I have have not added the changeset to the
queue.

Please contact a member of release engineering.

The following options are set: approve

bert-e · 2020-12-08T13:12:00Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.7

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2020-12-08T13:14:00Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.7

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

alexandre-allard requested a review from a team as a code owner November 27, 2020 09:57

gdemonet reviewed Nov 27, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Nov 27, 2020

View reviewed changes

charts/render.py Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Nov 27, 2020

View reviewed changes

alexandre-allard force-pushed the improvement/update-prometheus-operator-addon branch 2 times, most recently from 1e99e4e to 57c6203 Compare November 27, 2020 12:58

alexandre-allard force-pushed the improvement/update-prometheus-operator-addon branch from 57c6203 to 1579ae3 Compare November 27, 2020 17:40

alexandre-allard force-pushed the improvement/update-prometheus-operator-addon branch 2 times, most recently from b726cc9 to 361a45b Compare November 30, 2020 10:49

alexandre-allard added topic:monitoring Everything related to monitoring of services in a running cluster kind:debt Technical debt labels Nov 30, 2020

alexandre-allard force-pushed the improvement/update-prometheus-operator-addon branch from 361a45b to 4b22969 Compare November 30, 2020 15:52

alexandre-allard added 8 commits December 1, 2020 15:17

charts: Update render script for helm 3

052d304

kube-prometheus-stack chart is no longer compatible with helm 2, so we need to use helm 3 to render it correctly. So we slightly adapt the render script to work with helm 3

charts: Drop new node-exporter rules

6190de3

These alert rules come from the new version of kube-prometheus-stack charts. We drop them so we can add them back in an another Salt formula to allow customization by the users at runtime.

salt: Add new node-exporter prometheus rules to custom rules

4ee6879

Update the node-exporter rules with the new alert rules coming from the kube-prometheus-stack version 12.2.3

Add CHANGELOG entry for kube-prometheus-stack version change

1c02a66

Update alert rules extraction

25ef2bc

We need to regenerate these files as we bumped the Prometheus Operator version. The alerting & recording rules are generated using: ``` $ ./tools/rule_extractor/rule_extractor.py \ -i <control-plane-ip> -p 8443 -t rules ```

alexandre-allard force-pushed the improvement/update-prometheus-operator-addon branch from 4b22969 to 25ef2bc Compare December 1, 2020 14:18

gdemonet approved these changes Dec 1, 2020

View reviewed changes

alexandre-allard added the complexity:medium Something that requires one or few days to fix label Dec 3, 2020

bert-e merged commit cfedbd9 into development/2.7 Dec 8, 2020

bert-e deleted the improvement/update-prometheus-operator-addon branch December 8, 2020 13:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump prometheus-operator version #2948

Bump prometheus-operator version #2948

alexandre-allard commented Nov 27, 2020 •

edited

Loading

bert-e commented Nov 27, 2020

bert-e commented Nov 27, 2020

gdemonet left a comment

TeddyAndrieux Nov 27, 2020

alexandre-allard Nov 27, 2020 •

edited

Loading

TeddyAndrieux Nov 27, 2020

bert-e commented Nov 27, 2020

bert-e commented Nov 27, 2020

alexandre-allard commented Dec 3, 2020

bert-e commented Dec 3, 2020

bert-e commented Dec 4, 2020

bert-e commented Dec 4, 2020

bert-e commented Dec 7, 2020

bert-e commented Dec 8, 2020

bert-e commented Dec 8, 2020

bert-e commented Dec 8, 2020

bert-e commented Dec 8, 2020

Bump prometheus-operator version #2948

Bump prometheus-operator version #2948

Conversation

alexandre-allard commented Nov 27, 2020 • edited Loading

bert-e commented Nov 27, 2020

Hello alexandre-allard-scality,

bert-e commented Nov 27, 2020

Waiting for approval

gdemonet left a comment

Choose a reason for hiding this comment

TeddyAndrieux Nov 27, 2020

Choose a reason for hiding this comment

alexandre-allard Nov 27, 2020 • edited Loading

Choose a reason for hiding this comment

TeddyAndrieux Nov 27, 2020

Choose a reason for hiding this comment

bert-e commented Nov 27, 2020

Conflict

bert-e commented Nov 27, 2020

Waiting for approval

alexandre-allard commented Dec 3, 2020

bert-e commented Dec 3, 2020

In the queue

bert-e commented Dec 4, 2020

bert-e commented Dec 4, 2020

Queue out of order

bert-e commented Dec 7, 2020

In the queue

bert-e commented Dec 8, 2020

bert-e commented Dec 8, 2020

Queue out of order

bert-e commented Dec 8, 2020

In the queue

bert-e commented Dec 8, 2020

alexandre-allard commented Nov 27, 2020 •

edited

Loading

alexandre-allard Nov 27, 2020 •

edited

Loading