[BUG] EL9 3005~rc1-2 salt minion fails to run with FIPS enabled

[imjustvisiting]

Description
3005~rc1-2 salt minion fails to run on a Rocky 9 machine

Setup
Installed the RC1~2 RPMs on a baremetal workstation running Rocky 9.0:

# cat /etc/redhat-release 
Rocky Linux release 9.0 (Blue Onyx)

# rpm -qa | grep salt
salt-3005~rc1-2.el9.x86_64
salt-minion-3005~rc1-2.el9.x86_64

Steps to Reproduce the behavior
Any attempt to run the salt minion results in a trace:

# salt-call test.ping
Traceback (most recent call last):
  File "cffi/api.py", line 183, in _typeof
KeyError: 'const uint8_t*'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "salt/utils/pyinstaller/rthooks/pyi_rth_subprocess.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/pyinstaller/rthooks/_overrides.py", line 10, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/vt.py", line 32, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/crypt.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/loader/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/event.py", line 63, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/channel/client.py", line 13, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/crypt.py", line 54, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Cipher/__init__.py", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Cipher/_mode_ecb.py", line 29, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Util/_raw_api.py", line 88, in <module>
  File "cffi/api.py", line 266, in new
  File "cffi/api.py", line 186, in _typeof
  File "cffi/api.py", line 171, in _typeof_locked
  File "cffi/cparser.py", line 552, in parse_type
  File "cffi/cparser.py", line 555, in parse_type_and_quals
  File "cffi/cparser.py", line 336, in _parse
  File "cffi/cparser.py", line 53, in _get_parser
  File "pycparser/c_parser.py", line 110, in __init__
  File "pycparser/ply/yacc.py", line 3256, in yacc
  File "pycparser/ply/yacc.py", line 2961, in signature
UnboundLocalError: local variable 'sig' referenced before assignment
[63524] Failed to execute script 'pyi_rth_subprocess' due to unhandled exception!
[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
UnboundLocalError: local variable 'sig' referenced before assignment
Traceback (most recent call last):
  File "cffi/api.py", line 183, in _typeof
    result = self._parsed_types[cdecl]
KeyError: 'const uint8_t*'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "salt/utils/pyinstaller/rthooks/pyi_rth_subprocess.py", line 7, in <module>
    from salt.utils.pyinstaller.rthooks._overrides import PyinstallerPopen
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/pyinstaller/rthooks/_overrides.py", line 10, in <module>
    import salt.utils.vt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/vt.py", line 32, in <module>
    import salt.utils.crypt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/crypt.py", line 8, in <module>
    import salt.loader
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/loader/__init__.py", line 23, in <module>
    import salt.utils.event
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/utils/event.py", line 63, in <module>
    import salt.channel.client
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/channel/client.py", line 13, in <module>
    import salt.crypt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "salt/crypt.py", line 54, in <module>
    from Cryptodome.Cipher import AES, PKCS1_OAEP, PKCS1_v1_5 as PKCS1_v1_5_CIPHER
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Cipher/__init__.py", line 27, in <module>
    from Cryptodome.Cipher._mode_ecb import _create_ecb_cipher
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Cipher/_mode_ecb.py", line 29, in <module>
    from Cryptodome.Util._raw_api import (load_pycryptodome_raw_lib,
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "Cryptodome/Util/_raw_api.py", line 88, in <module>
    uint8_t_type = ffi.typeof(ffi.new("const uint8_t*"))
  File "cffi/api.py", line 266, in new
    cdecl = self._typeof(cdecl)
  File "cffi/api.py", line 186, in _typeof
    result = self._typeof_locked(cdecl)
  File "cffi/api.py", line 171, in _typeof_locked
    type = self._parser.parse_type(cdecl)
  File "cffi/cparser.py", line 552, in parse_type
    return self.parse_type_and_quals(cdecl)[0]
  File "cffi/cparser.py", line 555, in parse_type_and_quals
    ast, macros = self._parse('void __dummy(\n%s\n);' % cdecl)[:2]
  File "cffi/cparser.py", line 336, in _parse
    ast = _get_parser().parse(fullcsource)
  File "cffi/cparser.py", line 53, in _get_parser
    _parser_cache = pycparser.CParser()
  File "pycparser/c_parser.py", line 110, in __init__
    self.cparser = yacc.yacc(
  File "pycparser/ply/yacc.py", line 3256, in yacc
    signature = pinfo.signature()
  File "pycparser/ply/yacc.py", line 2961, in signature
    digest = base64.b16encode(sig.digest())
UnboundLocalError: local variable 'sig' referenced before assignment

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

That looks like a broken dependency. Does it do the same if you just run salt-call --versions?

[imjustvisiting]

Yes. salt-call returns the same trace with the --versions (or any) argument.

[imjustvisiting]

I dug into this a bit more...

The "crypto" messages clued me into it possibly being an issue with FIPS, and I have verified that this is the case. After disabling FIPS, the minion seems to run fine -- so really the bug to report is that the "salt-minion does not work with FIPS enabled".

I used fips-mode-setup --check to verify that FIPS was enabled, and then fips-mode-setup --disable to turn it off so I could continue testing 3005 on EL9.

[OrangeDog]

I assume FIPS mode was working before?

[imjustvisiting]

This was my first attempt at 3005 on EL9, so I can't say if it ever worked with FIPS enabled.

I'm okay relabeling this as a feature request if FIPS wasn't previously supported.

[dmurphy18]

@imjustvisiting Thanks for the issue but as per all issues, a versions report should be in the issue, esp. for the salt-master that you are using, understanding the salt-minion versions used, but wondering if we have and issue between the minion and the master, given the error is occurring in dependencies pulled in.

[OrangeDog]

@dmurphy18 A versions report is not possible.

salt-call returns the same trace with the --versions (or any) argument.

As this was the RC package, all the dependencies in question would be the bundled ones.

As the error occurs before any remote calls, the versions on the master should be irrelevant (but is also the same RC package).

[dmurphy18]

I wondering if the salt-master is pre-cve and fixes for signing are interfering and hence the 'sig' issue. Hence the need to know what version of salt-master.

Also since Centos 9 etc has Openssl v3.x, how has fips been enabled on the machine, for example: kernel changes flagged with fips=1. Given there is no dracut-fips on Centos 9, digging in to just how is fips enabled given I see

[root@Unknown david]# fips-mode-setup --check
Installation of FIPS modules is not completed.
FIPS mode is disabled.
[root@Unknown david]#

Resolving issue on Centos 9 since have the VM up with 3005rc2-1, before checking Rocky and fips

Noting: salt-call will go out to the salt-master, need to use --local to ensure only local machine involved

[OrangeDog]

@imjustvisiting Presumably FIPS has been enabled with fips-mode-setup --enable?

[dmurphy18]

got it duplicated with

salt-call --local test.versions

which should make it easier to fix since no salt-master involved
and yes Centos 9 to enable fips

fips-mode-setup --enable

and reboot for changes to take effect

[dmurphy18]

Problem appears to be due to use of pycparser v2.17 and use of md5 which doesn't exist on RHEL 9 along with SHA-1. From pycparser ply/yacc.py

2941     # Compute a signature over the grammar
2942     def signature(self):
2943         try:
2944             from hashlib import md5
2945         except ImportError:
2946             from md5 import md5
2947         try:
2948             sig = md5()

Need to update requirements file for Salt.

[Ch3LL]

Fixed by: #62420