[3007.x][BUG] Non-root users can not call functions after upgrade to 3007.0

[lee-harmonic]

Description
After upgrading to 3007.0, non-root users in the salt group (and in publisher_acl) can not start jobs. Permissions to ipc prevent access and are automatically reset.

Error messages:

lee@host:~$ salt '*.*' test.ping     
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds -   File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 532, in salt_main
    client.run()

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 816, in cmd_cli
    self.pub_data = self.run_job(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 388, in run_job
    pub_data = self.pub(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1905, in pub
    if listen and not self.event.connect_pub(timeout=timeout):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
    self.subscriber = salt.utils.asynchronous.SyncWrapper(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 77, in __init__
    self.obj = cls(*args, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
    return publish_client(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
    return salt.transport.tcp.PublishClient(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 219, in __init__
    super().__init__(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in __init__
    super().__init__()

Authentication error occurred.

Group membership and permissions:

lee@host:~$ groups
lee salt
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw------- 1 salt salt 0 Mar 12 09:53 master_event_pub.ipc
srw------- 1 salt salt 0 Mar 12 09:53 master_event_pull.ipc
srw------- 1 salt salt 0 Mar 12 09:53 publish_pull.ipc
srw------- 1 salt salt 0 Mar 12 09:53 workers.ipc

Setting permissions:

lee@host:~$ sudo chmod g+rw /var/run/salt/master/*
lee@host:~$ ls -l /var/run/salt/master/           
total 0
srw-rw---- 1 salt salt 0 Mar 12 09:53 master_event_pub.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 master_event_pull.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 publish_pull.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 workers.ipc
lee@host:~$ salt '*.*' test.ping
Authentication error occurred.

Logs for salt master now have:

[WARNING ] Authentication failure of type "user" occurred.

Restarting salt master:

lee@host:~$ sudo service salt-master restart
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw------- 1 salt salt 0 Mar 14 10:30 master_event_pub.ipc
srw------- 1 salt salt 0 Mar 14 10:30 master_event_pull.ipc
srw------- 1 salt salt 0 Mar 14 10:30 publish_pull.ipc
srw------- 1 salt salt 0 Mar 14 10:30 workers.ipc

The log from the restart has the following line:

[ERROR   ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None

Setup
Contents of /etc/salt/master.d/auth.conf:

publisher_acl:
  lee:
    - .*

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
Follow instructions at https://docs.saltproject.io/salt/user-guide/en/latest/topics/security.html#publisher-acls to set up publisher-acl and directory permissions.

Expected behavior

Non-root user can start jobs such as test.ping and permissions to do so are not reset when (re)starting the salt-master service. Was working before upgrade to 3007.0.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.0
 
Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.5 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-18-amd64
        system: Linux
       version: Debian GNU/Linux 12.5 bookworm

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[david-pulkowski]

We started noticing issues with the Publisher_ACL since switching to salt 3006
#66067

[alexholodak]

I ran into this exact issue after upgrading from salt-master 3006.7 to 3007.0.
I think I may have stumbled upon a workaround that solved the problem for me:

I did not have the python-is-python3 package installed. This made it so python commands would run as python3. Previously only python3 commands would run.

On Debian/Ubuntu:
sudo apt install python-is-python3

This change seems to have fixed the issue. I no longer see the TCP Publish Client error anymore.
I can now run salt '*' test.ping and other commands fine.

@lee-harmonic - does this work for you?

[Jesperbelt]

I'm encountering the same issues described earlier. Reverting back to version 3005.5 enables me to utilize local or domain-signed-in users. I've experimented with versions 3006.7 and 3007.0, both with and without root user access. However, the 'publisher_acl' feature doesn't operate as anticipated in these versions.

Thanks for bringing it up, @alexholodak, but it doesn't address the problem we've discussed previously.

[dmurphy18]

@alexholodak python not being available, and having to use python3 is very much an OS issue, for example: arch has python has python3, but older OS's got rid of python since it implies python2, so don't see having to install python-is-python3 as a Salt issue but a user and their OS's in use issue, about forgetting python referred to python2.

But thanks for the tip, for those needing the work-around for their environments.

@lee-harmonic Wondering if this is related to 3006.x plus using user salt, rather than root as in earlier releases, and missed a place where the change to default user salt affects things. Wonder if you change to user: root in the /etc/salt/master and minion configuration files, and restart and see if the problem still occurs, if not, indicates issue is default user: salt in configuration file.

[dmurphy18]

@lee-harmonic Can you check if the problem occurs with user: root as previously mentioned, it would help a lot in identifying the cause.

[jamest-pin]

@dmurphy18
We are having this same issue after upgrading from 3005 to 3007. Our CI runner can no longer run salt commands.

Triple-checked all the settings as per https://docs.saltproject.io/en/latest/ref/publisheracl.html#publisher-acl-system

Everything works fine as salt user. We were already running the salt-master service as user salt prior to updating.

Installing python-is-python3 had no effect.

buildkite-agent@salt:/srv$ salt 'salt' test.ping
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds -   File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 815, in cmd_cli
    self.pub_data = self.run_job(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 387, in run_job
    pub_data = self.pub(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1904, in pub
    if listen and not self.event.connect_pub(timeout=timeout):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
    self.subscriber = salt.utils.asynchronous.SyncWrapper(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in __init__
    self.obj = cls(*args, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
    return publish_client(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
    return salt.transport.tcp.PublishClient(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 220, in __init__
    super().__init__(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in __init__
    super().__init__()

[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
TypeError: argument must be an int, or have a fileno() method.
Traceback (most recent call last):
  File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 830, in cmd_cli
    for fn_ret in self.get_cli_event_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1653, in get_cli_event_returns
    for ret in self.get_iter_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1187, in get_iter_returns
    for raw in ret_iter:
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1104, in get_returns_no_block
    raw = self.event.get_event(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 651, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 553, in _get_event
    raw = self.subscriber.recv(timeout=wait)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 138, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 146, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 527, in run_sync
    return future_cell[0].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 375, in recv
    events, _, _ = select.select([self._stream.socket], [], [], 0)
TypeError: argument must be an int, or have a fileno() method.
Traceback (most recent call last):
  File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 830, in cmd_cli
    for fn_ret in self.get_cli_event_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1653, in get_cli_event_returns
    for ret in self.get_iter_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1187, in get_iter_returns
    for raw in ret_iter:
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1104, in get_returns_no_block
    raw = self.event.get_event(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 651, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 553, in _get_event
    raw = self.subscriber.recv(timeout=wait)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 138, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 146, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 527, in run_sync
    return future_cell[0].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 375, in recv
    events, _, _ = select.select([self._stream.socket], [], [], 0)
TypeError: argument must be an int, or have a fileno() method.
buildkite-agent@salt:/srv$

[jamest-pin]

We figured this out, turns out that the update messes up the permissions.

Before (not working, see above)

$ /var/run/salt/master$ ls -al
total 0
drwxr-xr-x 2 salt salt 120 Jul 24 08:50 .
drwxr-xr-x 4 root root  80 Jul 24 08:39 ..
srw------- 1 salt salt   0 Jul 24 08:50 master_event_pub.ipc
srw------- 1 salt salt   0 Jul 24 08:50 master_event_pull.ipc
srw------- 1 salt salt   0 Jul 24 08:50 publish_pull.ipc
srw------- 1 salt salt   0 Jul 24 08:50 workers.ipc

after (working):

$ /var/run/salt/master$ ls -al
total 0
drwxr-xr-x 2 salt salt 120 Jul 24 08:50 .
drwxr-xr-x 4 root root  80 Jul 24 08:39 ..
srw-rw---- 1 salt salt   0 Jul 24 08:50 master_event_pub.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 master_event_pull.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 publish_pull.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 workers.ipc

The command, for anyone in doubt:

cd /var/run/salt/master
sudo chmod g+rw *.ipc

[dmurphy18]

@jamest-pin Not seeing the difference between before and after in #66228 (comment), and thinking the fixes in #66218 will fix this, once 3006.9 is out, the fix is merged forward to the 3007.x branch, hence need to release a 3007.2, but 3006.9 comes first

[jamest-pin]

@dmurphy18 the difference is the rw under the group permissions. Meaning that user accounts in the salt group have read/write access to the files in the 'after' snapshot, after running the chmod command to fix them.

[dmurphy18]

This problem also exists with recently released Salt 3006.9

[root@dhcp-10-47-15-216 david]# l /var/run/salt/master/*
srw-------. 1 salt salt 0 Aug 12 14:50 /var/run/salt/master/publish_pull.ipc
srw-------. 1 salt salt 0 Aug 12 14:50 /var/run/salt/master/master_event_pull.ipc
srw-------. 1 salt salt 0 Aug 12 14:50 /var/run/salt/master/master_event_pub.ipc
srw-------. 1 salt salt 0 Aug 12 14:50 /var/run/salt/master/workers.ipc
[root@dhcp-10-47-15-216 david]# salt-run --versions-report
Salt Version:
          Salt: 3006.9

Will fix the problem in branch 3006.x and allow for the merge forward to fix in 3007.x branch

[dmurphy18]

@lee-harmonic I edited and restarted the salt-master using publisher_acl (https://docs.saltproject.io/salt/user-guide/en/latest/topics/security.html#id6), and have rw on the master_event_pub.ipc (3006.x)

[root@dhcp-10-47-15-216 david]# cat /etc/salt/master.d/user.conf 
publisher_acl:
  david:
    - .*

[root@dhcp-10-47-15-216 david]# l /var/run/salt/master/
total 0
drwxr-xr-x. 3 root root  60 Aug 13 11:51 ..
srw-------. 1 salt salt   0 Aug 13 11:53 publish_pull.ipc
srw-------. 1 salt salt   0 Aug 13 11:53 master_event_pull.ipc
srw-rw----. 1 salt salt   0 Aug 13 11:53 master_event_pub.ipc
srw-------. 1 salt salt   0 Aug 13 11:53 workers.ipc
drwxr-xr-x. 2 salt salt 120 Aug 13 11:53 .
[root@dhcp-10-47-15-216 david]# salt-run --versions-report
Salt Version:
          Salt: 3006.9

After upgrading to Salt 3007.1 it failed, there was hardening in #64063, for 3006.0 which requires publisher_acl or external_auth, so investigating further in 3007.x branch

[dmurphy18]

Assigning this to @dwoz

[rimskij]

is this the same bug&?

2024-09-04 13:20:10,118 [salt.utils.parsers:1062][WARNING ][16404] Master received a SIGTERM. Exiting.
2024-09-04 13:20:20,103 [salt.config :2034][DEBUG ][76631] Reading configuration from /etc/salt/master
2024-09-04 13:20:20,151 [salt.config :2197][DEBUG ][76631] Including configuration from '/etc/salt/master.d/reactor.conf'
2024-09-04 13:20:20,151 [salt.config :2034][DEBUG ][76631] Reading configuration from /etc/salt/master.d/reactor.conf
2024-09-04 13:20:20,300 [salt.utils.verify:599 ][WARNING ][76631] Insecure logging configuration detected! Sensitive data may be logged.
2024-09-04 13:20:42,109 [salt.utils.parsers:1062][WARNING ][76631] Master received a SIGINT. Exiting.
2024-09-04 13:23:55,086 [salt.config :2034][DEBUG ][79193] Reading configuration from /etc/salt/master
2024-09-04 13:23:55,135 [salt.config :2197][DEBUG ][79193] Including configuration from '/etc/salt/master.d/reactor.conf'
2024-09-04 13:23:55,135 [salt.config :2034][DEBUG ][79193] Reading configuration from /etc/salt/master.d/reactor.conf
2024-09-04 13:23:55,283 [salt.utils.verify:599 ][WARNING ][79193] Insecure logging configuration detected! Sensitive data may be logged.
2024-09-04 13:24:49,965 [salt.transport.tcp:312 ][WARNING ][79249] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds - File "/usr/bin/salt-master", line 11, in
sys.exit(salt_master())

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 86, in salt_master
master.start()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/daemons.py", line 223, in start
self.master.start()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 844, in start
salt.engines.start_engines(self.opts, self.process_manager)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/init.py", line 59, in start_engines
proc_mgr.add_process(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py", line 531, in add_process
process.start()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py", line 1100, in start
super().start()

File "/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)

File "/opt/saltstack/salt/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)

File "/opt/saltstack/salt/lib/python3.10/multiprocessing/context.py", line 281, in _Popen
return Popen(process_obj)

File "/opt/saltstack/salt/lib/python3.10/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)

File "/opt/saltstack/salt/lib/python3.10/multiprocessing/popen_fork.py", line 71, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)

File "/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py", line 995, in wrapped_run_func
return run_func()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/init.py", line 104, in run
self.engineself.fun

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in call
ret = self.loader.run(run_func, *args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
return _func_or_method(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/reactor.py", line 31, in start
salt.utils.reactor.Reactor(opts).run()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py", line 995, in wrapped_run_func
return run_func()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/reactor.py", line 214, in run
with salt.utils.event.get_event(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 127, in get_event
return MasterEvent(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 928, in init
super().init(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 265, in init
self.connect_pub()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
self.subscriber = salt.utils.asynchronous.SyncWrapper(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in init
self.obj = cls(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
return publish_client(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
return salt.transport.tcp.PublishClient(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 220, in init
super().init(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in init
super().init()

2024-09-04 13:24:50,029 [salt.utils.parsers:1062][WARNING ][79193] Master received a SIGINT. Exiting

[jamest-pin]

@rimskij

is this the same bug&?
TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc:

yes

[Yarakson]

so i've done all the permission and still not able to ping to the minion02 from the master. Every Machine is running

altinsher@salt-master:~$ sudo salt 'minion02' test.ping
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds - File "/usr/bin/salt", line 11, in
sys.exit(salt_main())

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
client.run()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
for full_ret in cmd_func(**kwargs):

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 815, in cmd_cli
self.pub_data = self.run_job(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 387, in run_job
pub_data = self.pub(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 1904, in pub
if listen and not self.event.connect_pub(timeout=timeout):

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
self.subscriber = salt.utils.asynchronous.SyncWrapper(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in init
self.obj = cls(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
return publish_client(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
return salt.transport.tcp.PublishClient(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 220, in init
super().init(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in init
super().init()

[jamest-pin]

@Yarakson

so i've done all the permission and still not able to ping to the minion02 from the master. Every Machine is running

altinsher@salt-master:~$ sudo salt 'minion02' test.ping [WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds - File "/usr/bin/salt", line 11, in sys.exit(salt_main())

Check the permissions again. Be aware it resets the perms again whenever the salt master service is restarted.

This fixes it for me (note the different paths /var/log and /var/run

sudo chmod g+rw /var/log/salt/master
sudo chmod g+rw /var/run/salt/master/*.ipc

[Yarakson]

@jamest-pin

@Yarakson

so i've done all the permission and still not able to ping to the minion02 from the master. Every Machine is running
altinsher@salt-master:~$ sudo salt 'minion02' test.ping [WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds - File "/usr/bin/salt", line 11, in sys.exit(salt_main())

Check the permissions again. Be aware it resets the perms again whenever the salt master service is restarted.

This fixes it for me (note the different paths /var/log and /var/run

sudo chmod g+rw /var/log/salt/master
sudo chmod g+rw /var/run/salt/master/*.ipc

Didn't know that every reset resets the perms, so thx for that. Regardless i'ts still not working, the config should be fine:

altinsher@salt-master:~$ sudo salt 'minion02' test.ping
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds - File "/usr/bin/salt", line 11, in
sys.exit(salt_main())

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
client.run()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
for full_ret in cmd_func(**kwargs):

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 815, in cmd_cli
self.pub_data = self.run_job(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 387, in run_job
pub_data = self.pub(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 1904, in pub
if listen and not self.event.connect_pub(timeout=timeout):

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
self.subscriber = salt.utils.asynchronous.SyncWrapper(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in init
self.obj = cls(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
return publish_client(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
return salt.transport.tcp.PublishClient(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 220, in init
super().init(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in init
super().init()

[ERROR ] Request client send timedout
Salt request timed out. The master is not responding. You may need to run your command with --async in order to bypass the congested event bus. With --async, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use salt-run jobs.lookup_jid to look up the results of the job in the job cache later.
altinsher@salt-master:~$ ls -l /var/run/salt/master/
total 0
srwxrwxr-x 1 root salt 0 Nov 18 13:11 master_event_pub.ipc
srw-rw---- 1 root salt 0 Nov 18 13:11 master_event_pull.ipc
srw-rw---- 1 root salt 0 Nov 18 13:11 publish_pull.ipc
srw-rw---- 1 root salt 0 Nov 18 13:11 workers.ipc

[jamest-pin]

@Yarakson you may need a chown salt:salt in there
here are mine

$ ls -l /var/run/salt/master
total 0
srw-rw---- 1 salt salt 0 Oct 28 10:53 master_event_pub.ipc
srw-rw---- 1 salt salt 0 Oct 28 10:53 master_event_pull.ipc
srw-rw---- 1 salt salt 0 Oct 28 10:53 publish_pull.ipc
srw-rw---- 1 salt salt 0 Oct 28 10:53 workers.ipc

[Yarakson]

@jamest-pin I did what you suggested

altinsher@salt-master:~$ sudo ls -l /var/run/salt/master
total 0
srwxr-xr-x 1 salt salt 0 Nov 18 13:11 master_event_pub.ipc
srwxr-xr-x 1 salt salt 0 Nov 18 13:11 master_event_pull.ipc
srwxr-xr-x 1 salt salt 0 Nov 18 13:11 publish_pull.ipc
srwxr-xr-x 1 salt salt 0 Nov 21 06:55 workers.ipc

Still:

altinsher@salt-master:~$ sudo salt 'minion02' test.ping
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds - File "/usr/bin/salt", line 11, in
sys.exit(salt_main())

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
client.run()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
for full_ret in cmd_func(**kwargs):

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 815, in cmd_cli
self.pub_data = self.run_job(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 387, in run_job
pub_data = self.pub(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/init.py", line 1904, in pub
if listen and not self.event.connect_pub(timeout=timeout):

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
self.subscriber = salt.utils.asynchronous.SyncWrapper(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in init
self.obj = cls(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
return publish_client(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
return salt.transport.tcp.PublishClient(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 220, in init
super().init(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in init
super().init()

[ERROR ] An un-handled exception was caught by Salt's global exception handler:
TypeError: argument must be an int, or have a fileno() method.

Does the minion also need salt:salt?:

altinsher@salt-master:~$ ls -l /var/run/salt/
total 0
drwxr-xr-x 2 salt salt 120 Nov 21 07:21 master
drwxr-xr-x 2 root root  80 Nov 18 14:34 minion

[jamest-pin]

@Yarakson

Does the minion also need salt:salt?:

probably, just trial and error until it works, that's what I did

[Yarakson]

@jamest-pin
I finally can ping my minions and it works like you said with the salt:salt user and group. The problem for me was that the systemservice script that the master executes after every reset, resets the user to root. It helps to change the ExecStartPost Permission to salt:salt
old script:
[Unit]
Description=The Salt Master Server
Documentation=man:salt-master(1) file:///usr/share/doc/salt/html/contents.html https://docs.saltproject.io/en/latest>After=network.target

[Service]
LimitNOFILE=100000
Type=notify
NotifyAccess=all
ExecStart=/usr/bin/salt-master
ExecStartPost=/usr/bin/chown -R root:salt /var/run/salt/master
ExecStartPost=/usr/bin/chmod 755 /var/cache/salt /var/cache/salt/master /var/cache/salt/master/jobs /var/run/salt /v>
[Install]
WantedBy=multi-user.target