csc: Add ability to update CSC defaults during upgrade/downgrade #2513

Ebaneck · 2020-05-04T09:36:07Z

Component:

'salt', 'kubernetes', 'csc'

Context:

See #2488

Summary:

This PR adds the ability to update default CSC values during upgrade and downgrade while keeping the CSC Configmap intact. This works because we are able to merge the user customizations defined within Configmaps into the defaults packaged alongside each chart and then use the resulting dict object to render & deploy Metalk8s addons.

Flow:

Create CSC defaults for each Addon e.g alertmanager.yaml
Import the CSC defaults within the charts using the render.py script
Call a salt module that takes arguments csc ConfigMap name, namespace and default csc
The module should merge the configs read in csc ConfigMap to those of default CSC and use the resulting object to deploy the Addon

Acceptance criteria:

no regression

Closes: #2488

bert-e · 2020-05-04T09:36:10Z

Hello ebaneck,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2020-05-04T09:36:20Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/2488-make-csc-upgrade-able into
development/2.5
w/2.6/improvement/2488-make-csc-upgrade-able will be merged into development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2020-05-04T09:36:21Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2020-05-04T14:10:07Z

History mismatch

Merge commit #ffa183d799bcbde3875b3e21e20b9e1265188f67 on the integration branch
w/2.6/improvement/2488-make-csc-upgrade-able is merging a branch which is neither the current
branch improvement/2488-make-csc-upgrade-able nor the development branch
development/2.6.

It is likely due to a rebase of the branch improvement/2488-make-csc-upgrade-able and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

Since we make use of csc defaults in the charts, the render utility script should import the csc defaults using jinja syntax

charts/render.py

salt/_modules/metalk8s_service_configuration.py

salt/metalk8s/addons/prometheus-operator/deployed/service-configuration.sls

salt/metalk8s/addons/dex/deployed/service-configuration.sls

This commit refactors service configuration configmaps to become empty objects that will hold and persist user customizable settings

…nfigurations This chart is re-rendered using: ``` $ ./charts/render.py prometheus-operator --namespace metalk8s-monitoring charts/prometheus-operator.yaml --service-config grafana metalk8s-grafana-config --service-config prometheus metalk8s-prometheus-config --service-config alertmanager metalk8s-alertmanager-config charts/prometheus-operator/ > salt/metalk8s/addons/prometheus-operator/deployed/chart.sls ```

This chart is re-rendered using: ``` $ ./charts/render.py dex --namespace metalk8s-auth charts/dex.yaml --service-config dex metalk8s-dex-config charts/dex/ > salt/metalk8s/addons/dex/deployed/chart.sls ```

This commit makes use of the salt module `dictupdate.merge` ensuring that default CSC values are merged with user specified CSC while still providing a fallback configuation for extreme scenarios. This above merge is implemented to ensure that we can do the following: 1. Deploy metalk8s addons with minimum default service values 2. Metalk8s administrators can update and add new configurations values 3. In cases of no Administrator configurations, the default values will be used to deploy the services. 4. In cases where CSC configmaps are removed, we can fall back to default values closes: #2488

This commit adds csc configuration files for dex, alertmanager, prometheus and grafana to their respective addon folders. These default configurations will be read during addon rendering by salt and then the default values will be used to deploy the respective services.

Initially, we expect default CSC ConfigMaps to contain default values and so we read these default values e.g `replicas` to perform csc persistence tests But with our new approach, default CSC ConfigMaps are deployed without default service values, so we need to adapt the test accordingly

Ebaneck · 2020-05-05T16:28:36Z

/reset

bert-e · 2020-05-05T16:28:45Z

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e · 2020-05-05T16:28:52Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/2488-make-csc-upgrade-able into
development/2.5
w/2.6/improvement/2488-make-csc-upgrade-able will be merged into development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2020-05-05T16:28:53Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

alexandre-allard

LGTM,

Ebaneck · 2020-05-06T13:00:41Z

/approve

bert-e · 2020-05-06T13:00:53Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.5
✔️ development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2020-05-06T14:06:20Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.5
✔️ development/2.6

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

Please check the status of the associated issue None.

Goodbye ebaneck.

Ebaneck requested review from alexandre-allard and a team May 4, 2020 09:36

Ebaneck force-pushed the improvement/2488-make-csc-upgrade-able branch from 71211fe to f4f899d Compare May 4, 2020 14:10

Ebaneck force-pushed the improvement/2488-make-csc-upgrade-able branch 3 times, most recently from 8109c12 to 11926d0 Compare May 4, 2020 23:12

salt: render utility should import csc defaults

dc751ab

Since we make use of csc defaults in the charts, the render utility script should import the csc defaults using jinja syntax

Ebaneck force-pushed the improvement/2488-make-csc-upgrade-able branch from 11926d0 to 05721b0 Compare May 5, 2020 11:03

Ebaneck marked this pull request as ready for review May 5, 2020 11:06

Ebaneck commented May 5, 2020

View reviewed changes

charts/render.py Show resolved Hide resolved

alexandre-allard reviewed May 5, 2020

View reviewed changes

Ebaneck added 3 commits May 5, 2020 17:47

kubernetes: create empty service configuration configmaps

020387c

This commit refactors service configuration configmaps to become empty objects that will hold and persist user customizable settings

charts: re-render the dex charts to use default service configurations

e21326e

This chart is re-rendered using: ``` $ ./charts/render.py dex --namespace metalk8s-auth charts/dex.yaml --service-config dex metalk8s-dex-config charts/dex/ > salt/metalk8s/addons/dex/deployed/chart.sls ```

Ebaneck force-pushed the improvement/2488-make-csc-upgrade-able branch from 05721b0 to 619ebba Compare May 5, 2020 15:52

Ebaneck requested a review from alexandre-allard May 5, 2020 15:54

Ebaneck force-pushed the improvement/2488-make-csc-upgrade-able branch from 619ebba to e12ce23 Compare May 5, 2020 16:23

Ebaneck added 3 commits May 5, 2020 18:23

scality deleted a comment from bert-e May 5, 2020

alexandre-allard approved these changes May 6, 2020

View reviewed changes

bert-e merged commit f5d4ccd into development/2.5 May 6, 2020

bert-e deleted the improvement/2488-make-csc-upgrade-able branch May 6, 2020 14:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

csc: Add ability to update CSC defaults during upgrade/downgrade #2513

csc: Add ability to update CSC defaults during upgrade/downgrade #2513

Ebaneck commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

Ebaneck commented May 5, 2020

bert-e commented May 5, 2020

bert-e commented May 5, 2020

bert-e commented May 5, 2020

alexandre-allard left a comment

Ebaneck commented May 6, 2020

bert-e commented May 6, 2020

bert-e commented May 6, 2020

csc: Add ability to update CSC defaults during upgrade/downgrade #2513

csc: Add ability to update CSC defaults during upgrade/downgrade #2513

Conversation

Ebaneck commented May 4, 2020

bert-e commented May 4, 2020

Hello ebaneck,

bert-e commented May 4, 2020

Integration data created

bert-e commented May 4, 2020

Waiting for approval

bert-e commented May 4, 2020

History mismatch

Ebaneck commented May 5, 2020

bert-e commented May 5, 2020

Reset complete

bert-e commented May 5, 2020

Integration data created

bert-e commented May 5, 2020

Waiting for approval

alexandre-allard left a comment

Choose a reason for hiding this comment

Ebaneck commented May 6, 2020

bert-e commented May 6, 2020

In the queue

bert-e commented May 6, 2020