Template and store replicas count as service configuration in Metalk8s cluster #2258

Ebaneck · 2020-02-25T08:44:13Z

Component:

'salt', 'charts',

Context:

See #2255

Summary:

Template and render the charts to consume service configuration values directly.

Acceptance criteria:

Closes: #2255

bert-e · 2020-02-25T08:44:15Z

Hello ebaneck,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2020-02-25T08:44:22Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge feature/2255-template-and-store-replicas-as-service-configuration into
development/2.5
w/2.6/feature/2255-template-and-store-replicas-as-service-configuration will be merged into development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2020-02-25T08:44:25Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

charts/render.py

bert-e · 2020-02-25T09:22:27Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@gdemonet

salt/_modules/metalk8s_service_configurator.py

buildchain/buildchain/salt_tree.py

salt/_modules/metalk8s_service_configurator.py

TeddyAndrieux · 2020-02-25T10:44:37Z

salt/_modules/metalk8s_service_configurator.py

+import logging
+import re
+
+from salt.exceptions import CommandExecutionError


Please respect pylint import suggestion

C: 8, 0: third party import "import yaml" should be placed before "from salt.exceptions import CommandExecutionError" (wrong-import-order)

Maybe we should setup pylint for Salt tree (or _modules at least)?
That would automatically flag unused import and import ordering.

That way we can enforce it in CI and focus our brain's bandwidth on functional issues.

Agree we should but we need some extra stuff about "salt" stuffs like all the __opts__ __salt__ __pillar__ ... but anyway agree we should have salt module checking in the CI so linting + unit test

If this is something we really want to enforce, it would make sense to add it in CI. Especially since not everyone here uses pylint + the special Salt plugin to lint their Salt modules (I don't).

I thin there is a pylint plugin for Salt, otherwise we can simply ignore Salt stuff by tweaking Pylint conf (you can probably consider some stuff as blackbox).

https://github.com/saltstack/salt-pylint (Personnaly never tested)

I think @jbertran tried it (but I may remember wrong)

buildchain/buildchain/salt_tree.py

charts/render.py

salt/_modules/metalk8s_service_configurator.py

salt/metalk8s/addons/addons-service-config/configmap.sls

salt/metalk8s/addons/addons-service-config/deployed.sls

bert-e · 2020-02-25T11:10:32Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

salt/metalk8s/addons/prometheus-operator/deployed/service-configuration.sls

salt/_modules/metalk8s_service_configuration.py

salt/metalk8s/orchestrate/downgrade/init.sls

salt/metalk8s/addons/prometheus-operator/deployed/service-configuration.sls

salt/metalk8s/orchestrate/upgrade/init.sls

…uster Initially, we do not deploy and manage service configurations using configmaps. This commit adds generic configmaps with service related configurations such that the end user can configure cluster settings directly by editing the related configmaps

This commit adds a salt module used to query a configmap object within a metalk8s cluster Once called with a service config parameter, this module returns the value of the configuration

This commit adds argparse to our render script and introduces a new argument `--service-config` which can be called several times accepting only two arguments(service name & service configmap name)

This hack is required to bypass the fact that dumping Jinja to YAML does not work with the render script because of the presence of unquote special characters e.g `{%`. If we try to escape thess special characters, we end up with type(str) always and in some special conditions, we need for example replicas count to be of type(int)

The prometheus-operator chart is rendered using the following command: ./charts/render.py prometheus-operator --namespace metalk8s-monitoring charts/prometheus-operator.yaml --service-config grafana metalk8s-grafana-config --service-config prometheus metalk8s-prometheus-config --service-config alertmanager metalk8s-alertmanager-config charts/prometheus-operator/ > salt/metalk8s/addons/prometheus-operator/deployed/chart.sls Service configuration values are passed using `--service-config` argument. When rendering this chart, we must pass the following services(grafana, prometheus, alertmanager) including the configmap that holds their respective service & cluster configurations Closes: #2255

TeddyAndrieux

LGTM

Ebaneck · 2020-02-27T13:50:57Z

/approve

Ebaneck · 2020-02-27T14:26:55Z

/reset

bert-e · 2020-02-27T14:27:40Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: approve

bert-e · 2020-02-27T14:32:03Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge feature/2255-template-and-store-replicas-as-service-configuration into
development/2.5
w/2.6/feature/2255-template-and-store-replicas-as-service-configuration will be merged into development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

bert-e · 2020-02-27T14:32:11Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

The following options are set: approve

bert-e · 2020-02-27T14:32:48Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

The following options are set: approve

stale

bert-e · 2020-02-28T00:02:04Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@gdemonet

The following options are set: approve

stale

Ebaneck · 2020-02-28T00:02:48Z

/status

bert-e · 2020-02-28T00:02:50Z

Status

Status report is not available.

The following options are set: approve

bert-e · 2020-02-28T03:33:09Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.5
✔️ development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2020-02-28T03:33:17Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.5
✔️ development/2.6

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

Please check the status of the associated issue None.

Goodbye ebaneck.

Ebaneck requested a review from a team February 25, 2020 08:44

slaperche-scality reviewed Feb 25, 2020

View reviewed changes

charts/render.py Outdated Show resolved Hide resolved

gdemonet previously requested changes Feb 25, 2020

View reviewed changes

charts/render.py Outdated Show resolved Hide resolved

charts/render.py Outdated Show resolved Hide resolved

charts/render.py Outdated Show resolved Hide resolved

charts/render.py Outdated Show resolved Hide resolved

charts/render.py Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Feb 25, 2020

View reviewed changes

salt/_modules/metalk8s_service_configurator.py Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Feb 25, 2020

View reviewed changes

salt/_modules/metalk8s_service_configurator.py Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Feb 25, 2020

View reviewed changes

buildchain/buildchain/salt_tree.py Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Feb 25, 2020

View reviewed changes

salt/_modules/metalk8s_service_configurator.py Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Feb 25, 2020

View reviewed changes

salt/_modules/metalk8s_service_configurator.py Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Feb 25, 2020

View reviewed changes

NicolasT previously requested changes Feb 25, 2020

View reviewed changes

Ebaneck force-pushed the feature/2255-template-and-store-replicas-as-service-configuration branch 3 times, most recently from c1b9b89 to 2afbfcf Compare February 25, 2020 22:18

Ebaneck changed the title ~~WIP: Template and store replicas count as service configuration in Metalk8s cluster~~ Template and store replicas count as service configuration in Metalk8s cluster Feb 25, 2020

Ebaneck requested review from gdemonet and NicolasT February 25, 2020 22:37

Ebaneck force-pushed the feature/2255-template-and-store-replicas-as-service-configuration branch 3 times, most recently from dd39d83 to e5c6401 Compare February 26, 2020 07:15

TeddyAndrieux reviewed Feb 26, 2020

View reviewed changes

salt/metalk8s/addons/prometheus-operator/deployed/service-configuration.sls Outdated Show resolved Hide resolved

Ebaneck force-pushed the feature/2255-template-and-store-replicas-as-service-configuration branch 2 times, most recently from c4bf2ef to 71257ae Compare February 26, 2020 12:29

Ebaneck marked this pull request as ready for review February 26, 2020 12:44

Ebaneck requested a review from TeddyAndrieux February 26, 2020 12:55

Ebaneck force-pushed the feature/2255-template-and-store-replicas-as-service-configuration branch from 71257ae to 0d02d6e Compare February 26, 2020 14:11

Ebaneck force-pushed the feature/2255-template-and-store-replicas-as-service-configuration branch 2 times, most recently from da93185 to fb7a2f4 Compare February 27, 2020 10:52

TeddyAndrieux requested changes Feb 27, 2020

View reviewed changes

salt/_modules/metalk8s_service_configuration.py Outdated Show resolved Hide resolved

salt/_modules/metalk8s_service_configuration.py Outdated Show resolved Hide resolved

Ebaneck force-pushed the feature/2255-template-and-store-replicas-as-service-configuration branch from fb7a2f4 to d4aa36c Compare February 27, 2020 11:14

Ebaneck requested a review from TeddyAndrieux February 27, 2020 12:48

TeddyAndrieux requested changes Feb 27, 2020

View reviewed changes

salt/metalk8s/orchestrate/downgrade/init.sls Show resolved Hide resolved

salt/metalk8s/addons/prometheus-operator/deployed/service-configuration.sls Show resolved Hide resolved

salt/metalk8s/orchestrate/upgrade/init.sls Show resolved Hide resolved

Ebaneck and others added 5 commits February 27, 2020 14:19

salt: add a salt module to query configmap

5e0a343

This commit adds a salt module used to query a configmap object within a metalk8s cluster Once called with a service config parameter, this module returns the value of the configuration

chart: Fix the render.py script to use argparse

a01ae41

This commit adds argparse to our render script and introduces a new argument `--service-config` which can be called several times accepting only two arguments(service name & service configmap name)

Ebaneck force-pushed the feature/2255-template-and-store-replicas-as-service-configuration branch from d4aa36c to 6f96f83 Compare February 27, 2020 13:21

TeddyAndrieux approved these changes Feb 27, 2020

View reviewed changes

scality deleted a comment from bert-e Feb 28, 2020

bert-e merged commit 6f96f83 into development/2.5 Feb 28, 2020

bert-e deleted the feature/2255-template-and-store-replicas-as-service-configuration branch February 28, 2020 03:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Template and store replicas count as service configuration in Metalk8s cluster #2258

Template and store replicas count as service configuration in Metalk8s cluster #2258

Ebaneck commented Feb 25, 2020 •

edited

Loading

bert-e commented Feb 25, 2020

bert-e commented Feb 25, 2020

bert-e commented Feb 25, 2020

bert-e commented Feb 25, 2020

TeddyAndrieux Feb 25, 2020

slaperche-scality Feb 25, 2020 •

edited

Loading

TeddyAndrieux Feb 25, 2020

gdemonet Feb 25, 2020

slaperche-scality Feb 25, 2020

TeddyAndrieux Feb 25, 2020

slaperche-scality Feb 25, 2020

bert-e commented Feb 25, 2020

TeddyAndrieux left a comment

Ebaneck commented Feb 27, 2020

Ebaneck commented Feb 27, 2020

bert-e commented Feb 27, 2020

bert-e commented Feb 27, 2020

bert-e commented Feb 27, 2020

bert-e commented Feb 27, 2020

bert-e commented Feb 28, 2020

Ebaneck commented Feb 28, 2020

bert-e commented Feb 28, 2020

bert-e commented Feb 28, 2020

bert-e commented Feb 28, 2020

Template and store replicas count as service configuration in Metalk8s cluster #2258

Template and store replicas count as service configuration in Metalk8s cluster #2258

Conversation

Ebaneck commented Feb 25, 2020 • edited Loading

bert-e commented Feb 25, 2020

Hello ebaneck,

bert-e commented Feb 25, 2020

Integration data created

bert-e commented Feb 25, 2020

Waiting for approval

bert-e commented Feb 25, 2020

Waiting for approval

TeddyAndrieux Feb 25, 2020

Choose a reason for hiding this comment

slaperche-scality Feb 25, 2020 • edited Loading

Choose a reason for hiding this comment

TeddyAndrieux Feb 25, 2020

Choose a reason for hiding this comment

gdemonet Feb 25, 2020

Choose a reason for hiding this comment

slaperche-scality Feb 25, 2020

Choose a reason for hiding this comment

TeddyAndrieux Feb 25, 2020

Choose a reason for hiding this comment

slaperche-scality Feb 25, 2020

Choose a reason for hiding this comment

bert-e commented Feb 25, 2020

Waiting for approval

TeddyAndrieux left a comment

Choose a reason for hiding this comment

Ebaneck commented Feb 27, 2020

Ebaneck commented Feb 27, 2020

bert-e commented Feb 27, 2020

Reset complete

bert-e commented Feb 27, 2020

Integration data created

bert-e commented Feb 27, 2020

Waiting for approval

bert-e commented Feb 27, 2020

Waiting for approval

bert-e commented Feb 28, 2020

Waiting for approval

Ebaneck commented Feb 28, 2020

bert-e commented Feb 28, 2020

Status

bert-e commented Feb 28, 2020

In the queue

bert-e commented Feb 28, 2020

Ebaneck commented Feb 25, 2020 •

edited

Loading

slaperche-scality Feb 25, 2020 •

edited

Loading