If (python-) pip is not installed, some state requisites will not work

[tj90241]

Description of Issue

This one is a real headscratcher... I observed that unless and onlyif requisites simply did not work when I installed 2019.2.1 on my hosts. When I ran the simpliest of states with either these requisites, the master would observe a minion timeout and the minion's log would be plastered with a NameError and dozens of AttributeErrors.

Digging into the logs, the NameError stacktrace suggested that pip is imported whenever a an unless or onlyif requisite is used... and if not present, it is not handled gracefully!

When running a state with these requisites and when python-pip is not installed on the host this is observed:

From the master:

$ sudo salt api6 state.apply what
[sudo] password for tyler: 
api6:
    Minion did not return. [No response]

In the minion logs:

2019-09-29 15:54:41,749 [salt.loader      :1584][ERROR   ][2376] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 43, in <module>
    import pip
ModuleNotFoundError: No module named 'pip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1560, in _load_module
    mod = spec.loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 407, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 907, in load_module
  File "<frozen importlib._bootstrap_external>", line 732, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 58, in <module>
    del pip
NameError: name 'pip' is not defined
2019-09-29 15:54:41,807 [salt.loader      :1584][ERROR   ][2376] Failed to import states pkgrepo, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1560, in _load_module
    mod = spec.loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 407, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 907, in load_module
  File "<frozen importlib._bootstrap_external>", line 732, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/salt/states/pkgrepo.py", line 93, in <module>
    from salt.exceptions import CommandExecutionError, SaltInvocationError
  File "/usr/lib/python3/dist-packages/salt/__init__.py", line 8, in <module>
    import warnings
  File "/usr/lib/python3.7/warnings.py", line 554, in <module>
    _processoptions(sys.warnoptions)

Setup

Ensure pip is not installed on the host:

$ python3
Python 3.7.3 (default, Apr  3 2019, 05:39:12) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pip'
>>> 

Create a simple state:

this-should-work:
  module.run:
    - name: test.ping
    - onlyif:
      - /bin/true

Steps to Reproduce Issue

Apply the state to a minion when pip is not installed; it will timeout.

$ sudo salt api6 state.apply what
api6:
    Minion did not return. [No response]

Remove the onlyif or unless requisite and the state works as expected.

or: Install python3-pip on the minion, bounce the salt-minion, and voila...

tyler@api6:~$ sudo apt-get install --no-install-recommends python3-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  python-pip-whl python3-distutils python3-lib2to3
Recommended packages:
  build-essential python3-dev python3-setuptools python3-wheel
The following NEW packages will be installed:
  python-pip-whl python3-distutils python3-lib2to3 python3-pip
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,980 kB of archives.
After this operation, 3,772 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://ftp.us.debian.org/debian buster/main amd64 python-pip-whl all 18.1-5 [1,591 kB]
Get:2 http://ftp.us.debian.org/debian buster/main amd64 python3-lib2to3 all 3.7.3-1 [76.7 kB]
Get:3 http://ftp.us.debian.org/debian buster/main amd64 python3-distutils all 3.7.3-1 [142 kB]
Get:4 http://ftp.us.debian.org/debian buster/main amd64 python3-pip all 18.1-5 [171 kB]
Fetched 1,980 kB in 1s (2,739 kB/s)   
Selecting previously unselected package python-pip-whl.
(Reading database ... 34701 files and directories currently installed.)
Preparing to unpack .../python-pip-whl_18.1-5_all.deb ...
Unpacking python-pip-whl (18.1-5) ...
Selecting previously unselected package python3-lib2to3.
Preparing to unpack .../python3-lib2to3_3.7.3-1_all.deb ...
Unpacking python3-lib2to3 (3.7.3-1) ...
Selecting previously unselected package python3-distutils.
Preparing to unpack .../python3-distutils_3.7.3-1_all.deb ...
Unpacking python3-distutils (3.7.3-1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../python3-pip_18.1-5_all.deb ...
Unpacking python3-pip (18.1-5) ...
Setting up python-pip-whl (18.1-5) ...
Setting up python3-lib2to3 (3.7.3-1) ...
Setting up python3-distutils (3.7.3-1) ...
Setting up python3-pip (18.1-5) ...
tyler@api6:~$ sudo systemctl restart salt-minion
tyler@api6:~$ 

Again: from the master after having run the above:

$ sudo salt api6 state.apply what
[sudo] password for tyler: 
api6:
----------
          ID: this-should-work
    Function: module.run
        Name: test.ping
      Result: True
     Comment: Module function test.ping executed
     Started: 16:07:50.023357
    Duration: 10201.699 ms
     Changes:   
              ----------
              ret:
                  True

Summary for api6
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  10.202 s

Versions Report

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: 2.1.11
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Apr  3 2019, 05:39:12)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist: debian 10.1 
         locale: UTF-8
        machine: x86_64
        release: 5.2.0-0.bpo.2-amd64
         system: Linux
        version: debian 10.1 

[tj90241]

I guess the easiest thing to do is add python3-pip to the list of Depends:?

$ apt-cache show salt-common | grep Depends
Depends: python3-apt, python3-dateutil, python3-jinja2, python3-msgpack (>= 0.4), python3-pkg-resources, python3-requests, python3-tornado4 (>= 4.2.1), python3-tornado4 (<< 5), python3-yaml, python3-systemd, python3-psutil, python3-distro, python3:any
Depends: python3-apt, python3-dateutil, python3-jinja2, python3-msgpack, python3-pkg-resources, python3-psutil, python3-requests (>= 1.0.0), python3-tornado4 (>= 4.2) | python3-tornado (<< 5), python3-tornado4 (>= 4.2) | python3-tornado (>= 4.2), python3-yaml, python3:any

[tj90241]

Or maybe not; something weird may be going on here. After installing pip on my hosts, it takes ~10sec to run the state with the onlyif requisite, ~150ms to run the state without:

Sample state per above:

$ cat /srv/salt/what.sls 
this-should-work:
  module.run:
    - name: test.ping
    - onlyif:
      - /bin/true

With the requisite:

$ sudo salt api6 state.apply what
api6:
----------
          ID: this-should-work
    Function: module.run
        Name: test.ping
      Result: True
     Comment: Module function test.ping executed
     Started: 16:19:47.230856
    Duration: 10006.305 ms
     Changes:   
              ----------
              ret:
                  True

Summary for api6
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  10.006 s

and without:

$ sudo salt api6 state.apply what
api6:
----------
          ID: this-should-work
    Function: module.run
        Name: test.ping
      Result: True
     Comment: Module function test.ping executed
     Started: 16:20:12.040838
    Duration: 157.621 ms
     Changes:   
              ----------
              ret:
                  True

Summary for api6
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 157.621 ms

[max-arnold]

See #53570 and #54755

[tj90241]

Thanks!