salt: Set salt-master timeout to 10 seconds #3033

TeddyAndrieux · 2021-01-13T17:29:04Z

Component:

'salt'

Context:

Time to time we get some failure during orchestrate execution.

For example, if pillar calculation take too much time when executing a salt states on a salt-minion we may get an error:

During handling of the above exception, another exception occurred:
              
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2106, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/saltmod.py", line 322, in state
    cmd_ret = __salt__["saltutil.cmd"](tgt, fun, **cmd_kw)
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 1657, in cmd
    fcn_ret = _exec(client, tgt, fun, arg, timeout, tgt_type, ret, kwarg, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 1609, in _exec
    for ret_comp in _cmd(**cmd_kwargs):
  File "/usr/lib/python3.6/site-packages/salt/client/__init__.py", line 868, in cmd_iter
    **kwargs
  File "/usr/lib/python3.6/site-packages/salt/client/__init__.py", line 1207, in get_iter_returns
    jid, list(minions - found), "list", **kwargs
  File "/usr/lib/python3.6/site-packages/salt/client/__init__.py", line 254, in gather_job_info
    **kwargs
  File "/usr/lib/python3.6/site-packages/salt/client/__init__.py", line 361, in run_job
    raise SaltClientError(general_exception)
salt.exceptions.SaltClientError: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

Also if for whatever reason a salt-minion is not able to send an event to a job before timeout we get an error

----------
          ID: <state id>
    Function: salt.state
      Result: False
     Comment: Run failed on minions: <minion id>
     Started: 17:13:27.036267
    Duration: 2313.207 ms
     Changes:   
              <minion id>:
                  False

Summary:

By default salt master timeout is set to 5 seconds, and time to time
it's not sufficient, as pillar compute may take some time and also it
happens that some time a salt-minion take a bit of time to answer a job
listing when executing a salt states

bert-e · 2021-01-13T17:29:07Z

Hello teddyandrieux,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2021-01-13T17:29:14Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/increase-salt-master-timeout into
development/2.7
w/2.8/improvement/increase-salt-master-timeout will be merged into development/2.8

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

By default salt master timeout is set to 5 seconds, and time to time it's not sufficient, as pillar compute may take some time and also it happens that some time a salt-minion take a bit of time to answer a job listing when executing a salt states

bert-e · 2021-01-13T17:29:16Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2021-01-13T17:29:31Z

History mismatch

Merge commit #579966e2682abb1f2c9bfb74c4a813b3db549f64 on the integration branch
w/2.8/improvement/increase-salt-master-timeout is merging a branch which is neither the current
branch improvement/increase-salt-master-timeout nor the development branch
development/2.8.

It is likely due to a rebase of the branch improvement/increase-salt-master-timeout and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

TeddyAndrieux · 2021-01-13T17:29:39Z

/reset

bert-e · 2021-01-13T17:29:48Z

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e · 2021-01-13T17:29:56Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/increase-salt-master-timeout into
development/2.7
w/2.8/improvement/increase-salt-master-timeout will be merged into development/2.8

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2021-01-13T17:29:58Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

gdemonet

Any reason we'd want this to be configurable at some point in time (not for this PR)? I don't see any, except maybe to increase it even further when needed.

TeddyAndrieux · 2021-01-13T17:39:03Z

Any reason we'd want this to be configurable at some point in time (not for this PR)? I don't see any, except maybe to increase it even further when needed.

Yes maybe it could make sense, but anyway since it's a salt config a user can easily overide this value with another config file "99-toto.conf" 😄

TeddyAndrieux · 2021-01-14T09:27:02Z

/approve

bert-e · 2021-01-14T09:27:13Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.7
✔️ development/2.8

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2021-01-14T09:27:20Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.7
✔️ development/2.8

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6

Please check the status of the associated issue None.

Goodbye teddyandrieux.

TeddyAndrieux added kind:bug Something isn't working complexity:medium Something that requires one or few days to fix topic:salt Everything related to SaltStack in our product labels Jan 13, 2021

TeddyAndrieux requested a review from a team January 13, 2021 17:29

TeddyAndrieux force-pushed the improvement/increase-salt-master-timeout branch from 579966e to 2caba9e Compare January 13, 2021 17:29

gdemonet approved these changes Jan 13, 2021

View reviewed changes

bert-e merged commit 2caba9e into development/2.7 Jan 14, 2021

bert-e deleted the improvement/increase-salt-master-timeout branch January 14, 2021 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

salt: Set salt-master timeout to 10 seconds #3033

salt: Set salt-master timeout to 10 seconds #3033

TeddyAndrieux commented Jan 13, 2021

bert-e commented Jan 13, 2021

bert-e commented Jan 13, 2021

bert-e commented Jan 13, 2021

bert-e commented Jan 13, 2021

TeddyAndrieux commented Jan 13, 2021

bert-e commented Jan 13, 2021

bert-e commented Jan 13, 2021

bert-e commented Jan 13, 2021

gdemonet left a comment

TeddyAndrieux commented Jan 13, 2021

TeddyAndrieux commented Jan 14, 2021

bert-e commented Jan 14, 2021

bert-e commented Jan 14, 2021

salt: Set salt-master timeout to 10 seconds #3033

salt: Set salt-master timeout to 10 seconds #3033

Conversation

TeddyAndrieux commented Jan 13, 2021

bert-e commented Jan 13, 2021

Hello teddyandrieux,

bert-e commented Jan 13, 2021

Integration data created

bert-e commented Jan 13, 2021

Waiting for approval

bert-e commented Jan 13, 2021

History mismatch

TeddyAndrieux commented Jan 13, 2021

bert-e commented Jan 13, 2021

Reset complete

bert-e commented Jan 13, 2021

Integration data created

bert-e commented Jan 13, 2021

Waiting for approval

gdemonet left a comment

Choose a reason for hiding this comment

TeddyAndrieux commented Jan 13, 2021

TeddyAndrieux commented Jan 14, 2021

bert-e commented Jan 14, 2021

In the queue

bert-e commented Jan 14, 2021