Improve robustness of salt-master and upgrade #3125

TeddyAndrieux · 2021-02-17T16:18:08Z

Component:

'salt', 'lifecycle'

Context:

Upgrade get a lot of flakies on environment that are a bit slow

Summary:

Backport 9ee0c53
Bump timeout waiting for containers after local container engine upgrade from 60s to 120s
Backport 73835be
Increase Salt master sock_pool_size (from 1 to 15) and worker_threads (from 5 to 10)

After kubelet upgrade all static pod may restart, so we need to wait for some "useful" pods to be running before going further in the upgrade process. NOTE: That before this PR we already sleep 20 seconds, so this commit only add a proper check after this sleep (cherry picked from commit 9ee0c53)

In upgrade script, bump the default to wait for a container to be ready from default 60s to 120s, to avoid some flakiness on environment that are a bit slow during upgrade process

bert-e · 2021-02-17T16:18:10Z

Hello teddyandrieux,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2021-02-17T16:18:15Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.8/improvement/improve-upgrade-robustness with contents from improvement/improve-upgrade-robustness
and development/2.8.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.8/improvement/improve-upgrade-robustness origin/development/2.8
 $ git merge origin/improvement/improve-upgrade-robustness
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.8/improvement/improve-upgrade-robustness

bert-e · 2021-02-17T16:22:57Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

TeddyAndrieux · 2021-02-17T18:01:11Z

/approve

bert-e · 2021-02-17T18:01:18Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

salt/metalk8s/salt/minion/running.sls

bert-e · 2021-02-17T18:45:17Z

History mismatch

Merge commit #d562b23cde07622357e2bd752b87ca700c3d228f on the integration branch
w/2.8/improvement/improve-upgrade-robustness is merging a branch which is neither the current
branch improvement/improve-upgrade-robustness nor the development branch
development/2.8.

It is likely due to a rebase of the branch improvement/improve-upgrade-robustness and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: approve

TeddyAndrieux · 2021-02-17T18:45:41Z

/reset

bert-e · 2021-02-17T18:45:49Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: approve

bert-e · 2021-02-17T18:45:55Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.8/improvement/improve-upgrade-robustness with contents from improvement/improve-upgrade-robustness
and development/2.8.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.8/improvement/improve-upgrade-robustness origin/development/2.8
 $ git merge origin/improvement/improve-upgrade-robustness
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.8/improvement/improve-upgrade-robustness

The following options are set: approve

Time to time, especially on really slow platform, we got failure because salt state execution timeout. Increase salt-master default timeout to 20 (cherry picked from commit 73835be)

Time to time salt-master get overloaded because he receive to much query, for example during upgrade and one environment a bit slow some salt states may timeout and make the upgrade fail. To avoid that kind of issue just bump the `sock_pool_size` on salt master (from 1 to 15) to avoid blocking waiting for zeromq communications and also bump the `worker_threads` on salt master (from 5 to 10) to avoid some failure if you have too many communication with the salt master (e.g.: because of upgrade + storage operator) Sees: saltstack/salt#53147

bert-e · 2021-02-17T18:56:28Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

gdemonet

Code is OK, just an edit to make for the changelog entry (sorry 😇)

CHANGELOG.md

bert-e · 2021-02-17T18:58:48Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@gdemonet

The following options are set: approve

bert-e · 2021-02-17T19:00:37Z

History mismatch

Merge commit #59d096901b694558ab7a9bc40f9a77b3962f6c3c on the integration branch
w/2.8/improvement/improve-upgrade-robustness is merging a branch which is neither the current
branch improvement/improve-upgrade-robustness nor the development branch
development/2.8.

It is likely due to a rebase of the branch improvement/improve-upgrade-robustness and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: approve

TeddyAndrieux · 2021-02-17T19:00:55Z

/reset

bert-e · 2021-02-17T19:01:02Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: approve

bert-e · 2021-02-17T19:01:07Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.8/improvement/improve-upgrade-robustness with contents from improvement/improve-upgrade-robustness
and development/2.8.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.8/improvement/improve-upgrade-robustness origin/development/2.8
 $ git merge origin/improvement/improve-upgrade-robustness
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.8/improvement/improve-upgrade-robustness

The following options are set: approve

gdemonet

bert-e · 2021-02-17T19:37:58Z

Build failed

The build for commit did not succeed in branch w/2.8/improvement/improve-upgrade-robustness.

The following options are set: approve

bert-e · 2021-02-18T09:16:15Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.7
✔️ development/2.8

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2021-02-18T10:52:35Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.7
✔️ development/2.8

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

Please check the status of the associated issue None.

Goodbye teddyandrieux.

TeddyAndrieux added 2 commits February 17, 2021 16:44

script/upgrade: Bump timeout to wait containers to 120

4403116

In upgrade script, bump the default to wait for a container to be ready from default 60s to 120s, to avoid some flakiness on environment that are a bit slow during upgrade process

TeddyAndrieux added topic:lifecycle Issues related to upgrade or downgrade of MetalK8s complexity:medium Something that requires one or few days to fix topic:salt Everything related to SaltStack in our product labels Feb 17, 2021

TeddyAndrieux requested a review from a team February 17, 2021 16:18

TeddyAndrieux force-pushed the improvement/improve-upgrade-robustness branch from 43c6ddf to a29fe17 Compare February 17, 2021 16:18

gdemonet reviewed Feb 17, 2021

View reviewed changes

salt/metalk8s/salt/minion/running.sls Outdated Show resolved Hide resolved

TeddyAndrieux force-pushed the improvement/improve-upgrade-robustness branch from a29fe17 to 8bc1f3e Compare February 17, 2021 18:45

TeddyAndrieux added 2 commits February 17, 2021 19:51

salt: Increase salt-master default timeout to 20

d6c226f

Time to time, especially on really slow platform, we got failure because salt state execution timeout. Increase salt-master default timeout to 20 (cherry picked from commit 73835be)

TeddyAndrieux force-pushed the improvement/improve-upgrade-robustness branch from 8bc1f3e to 59d0969 Compare February 17, 2021 18:53

gdemonet suggested changes Feb 17, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

changelog: Changelog entry for upgrade robustness improvements

debe85e

TeddyAndrieux force-pushed the improvement/improve-upgrade-robustness branch from 59d0969 to debe85e Compare February 17, 2021 19:00

gdemonet approved these changes Feb 17, 2021

View reviewed changes

bert-e merged commit debe85e into development/2.7 Feb 18, 2021

bert-e deleted the improvement/improve-upgrade-robustness branch February 18, 2021 10:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve robustness of salt-master and upgrade #3125

Improve robustness of salt-master and upgrade #3125

TeddyAndrieux commented Feb 17, 2021

bert-e commented Feb 17, 2021

bert-e commented Feb 17, 2021

bert-e commented Feb 17, 2021

TeddyAndrieux commented Feb 17, 2021

bert-e commented Feb 17, 2021

bert-e commented Feb 17, 2021

TeddyAndrieux commented Feb 17, 2021

bert-e commented Feb 17, 2021

bert-e commented Feb 17, 2021

bert-e commented Feb 17, 2021

gdemonet left a comment

bert-e commented Feb 17, 2021

bert-e commented Feb 17, 2021

TeddyAndrieux commented Feb 17, 2021

bert-e commented Feb 17, 2021

bert-e commented Feb 17, 2021

gdemonet left a comment

bert-e commented Feb 17, 2021

bert-e commented Feb 18, 2021

bert-e commented Feb 18, 2021

Improve robustness of salt-master and upgrade #3125

Improve robustness of salt-master and upgrade #3125

Conversation

TeddyAndrieux commented Feb 17, 2021

bert-e commented Feb 17, 2021

Hello teddyandrieux,

bert-e commented Feb 17, 2021

Conflict

bert-e commented Feb 17, 2021

Waiting for approval

TeddyAndrieux commented Feb 17, 2021

bert-e commented Feb 17, 2021

Waiting for approval

bert-e commented Feb 17, 2021

History mismatch

TeddyAndrieux commented Feb 17, 2021

bert-e commented Feb 17, 2021

Reset complete

bert-e commented Feb 17, 2021

Conflict

bert-e commented Feb 17, 2021

Waiting for approval

gdemonet left a comment

Choose a reason for hiding this comment

bert-e commented Feb 17, 2021

Waiting for approval

bert-e commented Feb 17, 2021

History mismatch

TeddyAndrieux commented Feb 17, 2021

bert-e commented Feb 17, 2021

Reset complete

bert-e commented Feb 17, 2021

Conflict

gdemonet left a comment

Choose a reason for hiding this comment

bert-e commented Feb 17, 2021

Build failed

bert-e commented Feb 18, 2021

In the queue

bert-e commented Feb 18, 2021