Salt installation fail if base package not installed on rhel

[TeddyAndrieux]

Component:

'salt', 'deployment'

What happened:

If for whatever reason some Salt dependencies are not installed on the system prior to bootstrap the salt-minion installation failed:

# /srv/scality/metalk8s-2.8.0-dev/bootstrap.sh 
> Determine the OS... done [0s]
> Checking that BootstrapConfiguration is present... done [0s]
> Pre-minion system tests... done [6s]
> Configure internal repositories... done [3s]
> Check mandatory packages presence... done [0s]
> Disabling Salt minion service... done [0s]
> Stopping Salt minion service... done [0s]
> Installing mandatory packages... fail [24s]

Failure while running step 'Installing mandatory packages'

Command: install_packages salt-minion-3002.2 genisoimage

Output:

<< BEGIN >>
Loaded plugins: product-id, search-disabled-repos, subscription-manager,
              : versionlock
Package genisoimage-1.1.11-25.el7.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package salt-minion.noarch 0:3002.2-1.el7 will be installed
--> Processing Dependency: salt = 3002.2-1.el7 for package: salt-minion-3002.2-1.el7.noarch
--> Running transaction check
---> Package salt.noarch 0:3002.2-1.el7 will be installed
--> Processing Dependency: yum-utils for package: salt-3002.2-1.el7.noarch
--> Running transaction check
---> Package yum-utils.noarch 0:1.1.31-54.el7_8 will be installed
--> Processing Dependency: python-kitchen for package: yum-utils-1.1.31-54.el7_8.noarch
--> Running transaction check
---> Package python-kitchen.noarch 0:1.1.1-5.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package            Arch       Version             Repository              Size
================================================================================
Installing:
 salt-minion        noarch     3002.2-1.el7        metalk8s-saltstack      39 k
Installing for dependencies:
 python-kitchen     noarch     1.1.1-5.el7         metalk8s-saltstack     267 k
 salt               noarch     3002.2-1.el7        metalk8s-saltstack      10 M
 yum-utils          noarch     1.1.31-54.el7_8     rhel-7-server-rpms     122 k

Transaction Summary
================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 11 M
Installed size: 66 M
Downloading packages:
warning: /srv/scality/metalk8s-2.8.0-dev/packages/redhat/7/metalk8s-saltstack-el7/python-kitchen-1.1.1-5.el7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for python-kitchen-1.1.1-5.el7.noarch.rpm is not installed
--------------------------------------------------------------------------------
Total                                               22 MB/s |  11 MB  00:00     
Retrieving key from file:///srv/scality/metalk8s-2.8.0-dev/packages/redhat/7/metalk8s-saltstack-el7/RPM-GPG-KEY-metalk8s-saltstack-7_1


The GPG keys listed for the "metalk8s-saltstack" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.


 Failing package is: python-kitchen-1.1.1-5.el7.noarch
 GPG Keys are configured as: file:///srv/scality/metalk8s-2.8.0-dev/packages/redhat/7/metalk8s-saltstack-el7/RPM-GPG-KEY-metalk8s-saltstack-7_1

<< END >>

This script will now exit

It fail because as part of metalk8s-saltstack we include some "base" dependencies packages signed with CentOS base gpg key and this key is not part of this repo and by default not installed on rhel machines

What was expected:

Working bootstrap even if some "base" dependencies packages are not yet installed (even if the CentOS base gpg key is not accepted yet)

Steps to reproduce

• Create a rhel7 machine
• Make sure (for example) python-kitchen is not installed on the machine
• Bootstrap
• 💥

Resolution proposal (optional):

First note that those base packages are part of the default saltstack repo in a specific "base" directory with the CentOS gpg key and note also that those package are all available in default base repo of CentOS and RHEL

2 different solutions:
1- We embed the CentOS base gpg key as part of metalk8s-saltstack repo and we add this one in the repo definition
2- We do not embed any base package in metalk8s-saltstack, since MetalK8s rely on base packages to be available on the host before bootstrap, anyway it does not make sense to include those packages also in metalk8s-saltstack

IMHO we should go for 2- it does not make sense to include those base packages as part of our metalk8s-saltstack repo

---

Workaround:

As a simple workaround
1- either install the dependencies manually from base repository (e.g.: for python-kitchen)

yum install --disablerepo="metalk8s-saltstack" python-kitchen

2- either import CentOS base gpg key (e.g. from saltstack repo)

rpm --import https://repo.saltstack.com/py3/redhat/7/x86_64/3002/base/RPM-GPG-KEY-CentOS-7

Even if it's may require more manual steps I suggest to use the 1- method as it will install RHEL supported packages

[NicolasT]

I think solution 2 makes sense. However, this does raise a question: why are these packages in the SaltStack repo in the first place? Is this some (required) version not found in some CentOS/RHEL 7.x (for some x) versions?

[TeddyAndrieux]

Not entirely sure, for me it was just for being able to install salt packages only from saltstack repo but I just tested and it seems you still have some dependencies not part of this repo .... so, I need to check

[TeddyAndrieux]

After some discussion on Salt Slack it shouldn't not be needed in any "basic" CentOs/RHEL 7

Reason for this "base" directory:

SaltStack was a struggling startup, it had a pending sale with a High Frequency Trading company who had ripped things out of their Centos 7 images, in order to trade faster etc. and in order for Salt to work, StackStack had to provide those packages which should have been there in a standard Centos 7 install. Hence the base sub-directory containing them, and once provided, had to be maintained and changes tracked through all the different updates to Centos 7.