salt-ssh return exit code 0 if salt-call fails to run

[marmarek]

Description of Issue/Question

When salt-ssh fails to run requested state, it still returns code 0.

bash-4.4# salt-ssh f29min-test test.ping
f29min-test:
    ----------
    retcode:
        0
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_62a99a_salt/salt-call", line 15, in <module>
            salt_call()
          File "/var/tmp/.root_62a99a_salt/py2/salt/scripts.py", line 395, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_62a99a_salt/py2/salt/cli/call.py", line 5, in <module>
            import salt.utils.parsers
          File "/var/tmp/.root_62a99a_salt/py2/salt/utils/parsers.py", line 27, in <module>
            import salt.config as config
          File "/var/tmp/.root_62a99a_salt/py2/salt/config/__init__.py", line 98, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * .5
          File "/var/tmp/.root_62a99a_salt/py2/salt/config/__init__.py", line 88, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.root_62a99a_salt/py2/salt/grains/core.py", line 44, in <module>
            import salt.utils.dns
          File "/var/tmp/.root_62a99a_salt/py2/salt/utils/dns.py", line 32, in <module>
            import salt.modules.cmdmod
          File "/var/tmp/.root_62a99a_salt/py2/salt/modules/cmdmod.py", line 34, in <module>
            import salt.utils.templates
          File "/var/tmp/.root_62a99a_salt/py2/salt/utils/templates.py", line 32, in <module>
            import salt.utils.http
          File "/var/tmp/.root_62a99a_salt/py2/salt/utils/http.py", line 41, in <module>
            import salt.loader
          File "/var/tmp/.root_62a99a_salt/py2/salt/loader.py", line 28, in <module>
            import salt.utils.event
          File "/var/tmp/.root_62a99a_salt/py2/salt/utils/event.py", line 85, in <module>
            import salt.transport.ipc
          File "/var/tmp/.root_62a99a_salt/py2/salt/transport/ipc.py", line 21, in <module>
            from tornado.locks import Semaphore
          File "/var/tmp/.root_62a99a_salt/py2/tornado/locks.py", line 18, in <module>
            from concurrent.futures import CancelledError
        ImportError: No module named concurrent.futures
    stdout:
bash-4.4# echo $?
0

If I try salt-ssh f29min-test grains.items I get additionally:

    _error:
        Failed to return clean data

The actual reason why it failed is already addressed in #48289, the reason is irrelevant to this bug.

This is yet another issue in this family: #7876, #28300, #23102, #41296, ...
This is similar to #18510, but that is mostly salt/salt-call, not salt-ssh.

Steps to Reproduce Issue

Full log with -l all
salt-ssh-all.log

Versions Report

Salt Version:
           Salt: 2018.3.2
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15 (default, Oct 15 2018, 15:24:06)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 17.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 5.0.2
            ZMQ: 4.1.6
 
System Versions:
           dist: fedora 28 Twenty Eight
         locale: UTF-8
        machine: x86_64
        release: 4.14.74-1.pvops.qubes.x86_64
         system: Linux
        version: Fedora 28 Twenty Eight

[gdm85]

This is not fixed in the following versions available via Ubuntu:

• 3000.2 (testing)
• 2018.3.4 (19.10 Eoan Ermine)
• 2017.7.4 (current LTS Bionic Beaver 18.04)

Installing via pip3 also gives an exit code of 0 (!) with the following on-screen errors:

Building wheels for collected packages: salt-ssh, PyYAML
  Running setup.py bdist_wheel for salt-ssh ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-f2wswfxg/salt-ssh/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp21s60geqpip-wheel- --python-tag cp36:
  3000.3
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Failed building wheel for salt-ssh
  Running setup.py clean for salt-ssh
  Running setup.py bdist_wheel for PyYAML ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-f2wswfxg/PyYAML/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpowcglr6vpip-wheel- --python-tag cp36:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Failed building wheel for PyYAML
  Running setup.py clean for PyYAML
Failed to build salt-ssh PyYAML
Installing collected packages: MarkupSafe, Jinja2, PyYAML, msgpack, urllib3, chardet, certifi, idna, requests, salt-ssh
  Running setup.py install for PyYAML ... done
  Running setup.py install for salt-ssh ... done
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 certifi-2020.4.5.1 chardet-3.0.4 idna-2.9 msgpack-0.6.2 requests-2.23.0 salt-ssh-3000.3 urllib3-1.25.9

The installed salt-ssh also gives an exit code of 0 with multiple failures. @s0undt3ch is it possible that this is not yet fixed, or there was a regression?

[s0undt3ch]

I just merged the PR that, apparently, fixed this.
Can you try latest master branch?

[s0undt3ch]

Building wheels for collected packages: salt-ssh, PyYAML
  Running setup.py bdist_wheel for salt-ssh ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-f2wswfxg/salt-ssh/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp21s60geqpip-wheel- --python-tag cp36:
  3000.3
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel' 

What exactly is trying to build a salt-ssh package?

[gdm85]

What exactly is trying to build a salt-ssh package?

Sorry, I don't understand your question; the output snippet you mentioned is from the pip3 install command:

Installing via pip3 also gives an exit code of 0 (!) with the following on-screen errors:

I am not familiar with the process but I guess it is preparing "wheels" for salt-ssh. This SO answer sheds some light: https://stackoverflow.com/a/44862371
It might be something missing in the requirements.

[gdm85]

I just merged the PR that, apparently, fixed this.

Which PR? Can't see it: https://github.com/saltstack/salt/pulls?q=is%3Apr+is%3Aclosed

Also, I have to ask: no regression test for this? :)

[xmsk]

I think this is still an issue. While some salt-ssh commands give a non-zero exit code on failure I ran into various cases where a 0 exit code was returned. One such example I have tried to replicate in this repository https://github.com/xmsk/salt-ssh-exit-code.

The issue leading to the 0 exit code is trying to access non-existing pillar data in a state file, namely I have the state

file append:
  file.append:
    - name: /tmp/salt-append.txt
    - text: >
        {{ pillar['key'] }}

where pillar['key'] is undefined. In fact I have no pillar data defined at all.

Running salt-ssh -i '*' state.apply gives the following output

(local-venv) [] $ salt-ssh '*' state.apply 
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/home/xmsk/coding/test/salt/salt/salt/utils/templates.py", line 497, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/home/xmsk/coding/test/salt/salt/local-venv/lib/python3.9/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/xmsk/coding/test/salt/salt/local-venv/lib/python3.9/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/xmsk/coding/test/salt/salt/local-venv/lib/python3.9/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 5, in top-level template code
jinja2.exceptions.UndefinedError: 'salt.loader_context.NamedLoaderContext object' has no attribute 'key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xmsk/coding/test/salt/salt/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/home/xmsk/coding/test/salt/salt/salt/utils/templates.py", line 504, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'salt.loader_context.NamedLoaderContext object' has no attribute 'key'
[CRITICAL] Rendering SLS 'base:complex' failed: Jinja variable 'salt.loader_context.NamedLoaderContext object' has no attribute 'key'
minion:
    - Rendering SLS 'base:complex' failed: Jinja variable 'salt.loader_context.NamedLoaderContext object' has no attribute 'key'
(local-venv) [] $ echo $?
0

Versions Report

I'm running salt-ssh from master

Salt Version:
          Salt: 3003rc1+1.g4af72e7875
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 20 2021, 18:40:11)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.0.3
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: arch  
        locale: utf-8
       machine: x86_64
       release: 5.11.2-arch1-1
        system: Linux
       version: Arch Linux 

[lkubb]

I'm reasonably sure with #64515, #64576 and #64542 merged that this can finally be closed, so marked it on the PR.

The actual fix for the initial description is the last one I assume since it makes any JSON decode failure an exception in cases where JSON is to be expected (see test: https://github.com/saltstack/salt/pull/64542/files#diff-ae71f89f956935339d33b0b39496d0c77d51b532f65075d89b42d7feec716467).