Salt minions exit with non-zero exit code for Puppet run even though puppet returned successfully.

[BenjaminSchiborr]

When calling the puppet module for salt, puppet executes successfully (Retcode 0 or 2). The returned status code in the cli return is also successful (as in 0). However, the retcode of the job on the event bus will not necessarily be 0. Therefore salt will evaluate the puppet call as a failure.

The reason for this is two-fold.

Puppet exits successfully with either a 0 or a 2 as a return code. This issue occurs when the return code is 2.

The puppet module (modules/puppet.py) successfully intercepts that:

    ret = __salt__['cmd.run_all'](repr(puppet), python_shell=puppet.useshell)
    if ret['retcode'] in [0, 2]:
        ret['retcode'] = 0
    else:
        ret['retcode'] = 1

    return ret

Because of this the retcode in the json output on the shell will be 0 and successful.

However, this retcode is not the one that salt uses to evaluate success of the overall job. This is done in cmdmod (modules/cmdmod.py):
Here, the return code is defined as the following:

        ret['pid'] = proc.process.pid
        ret['retcode'] = proc.process.returncode
        ret['stdout'] = out
        ret['stderr'] = err

As you can see the return code of the actual process is being taken and not of the module that calls puppet.

Ergo, when calling the puppet agent, puppet.py changes it's retcode from 2 to 0, while cmdmod takes the retcode directly from the process and does not change it. The salt call fails.

Apprach to fix the issue

One approach to fix this issue is to capsule the actual puppet call inside puppet.py. That way the process itself will already exit with the correct status code. See PR #43765 for a possible fix.

Steps to Reproduce Issue

• Have puppet setup
• Call "salt 'your_server' puppet.run tags=your_tag"
• See the line: "ERROR: Minions returned with non-zero exit code" even though the retcode in the output on the CLI is 0

References

The issue is related to #27506 and #20850

Versions Report

Salt Version:
Salt: 2016.11.5

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 0.5.4
gitpython: 0.3.2 RC1
ioflo: Not Installed
Jinja2: 2.9.6
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 0.9.1
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.2.3
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 14.0.1
RAET: Not Installed
smmap: 0.8.2
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5

System Versions:
dist: Ubuntu 14.04 trusty
machine: x86_64
release: 4.4.0-79-generic
system: Linux
version: Ubuntu 14.04 trusty