Bootstrap fails with locked Yum database

[gdemonet]

Component: salt

What happened:

Locally, using Vagrant, we hit an issue during bootstrap (most often during the Deploy early-stage bootstrap node step), showing errors when trying to install packages, e.g.:

    bootstrap: Failure while running step 'Deploying early-stage bootstrap node in local mode (this may take a while)'
[...]
    bootstrap: Output:
    bootstrap: 
    bootstrap: << BEGIN >>
    bootstrap: warning: /vagrant/_build/root/packages/redhat/metalk8s-kubernetes-el7/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
    bootstrap: [ERROR   ] Command '[u'systemd-run', u'--scope', u'yum', u'-y', u'install', u'yum-plugin-versionlock']' failed with return code: 1
    bootstrap: [ERROR   ] stdout: Running scope as unit run-6189.scope.
    bootstrap: Loaded plugins: fastestmirror
    bootstrap: Loading mirror speeds from cached hostfile
    bootstrap:  * base: distrib-coffee.ipsl.jussieu.fr
    bootstrap:  * extras: centos.crazyfrogs.org
    bootstrap:  * updates: ftp.pasteur.fr
    bootstrap: Resolving Dependencies
    bootstrap: --> Running transaction check
    bootstrap: ---> Package yum-plugin-versionlock.noarch 0:1.1.31-52.el7 will be installed
    bootstrap: --> Finished Dependency Resolution
    bootstrap: 
    bootstrap: Dependencies Resolved
    bootstrap: 
    bootstrap: ================================================================================
    bootstrap:  Package                      Arch         Version             Repository  Size
    bootstrap: ================================================================================
    bootstrap: Installing:
    bootstrap:  yum-plugin-versionlock       noarch       1.1.31-52.el7       base        36 k
    bootstrap: 
    bootstrap: Transaction Summary
    bootstrap: ================================================================================
    bootstrap: Install  1 Package
    bootstrap: 
    bootstrap: Total size: 36 k
    bootstrap: Installed size: 53 k
    bootstrap: Downloading packages:
    bootstrap: Running transaction check
    bootstrap: Running transaction test
    bootstrap: Transaction test succeeded
    bootstrap: Running transaction
    bootstrap: Traceback (most recent call last):
    bootstrap:   File "/bin/yum", line 29, in <module>
    bootstrap:     yummain.user_main(sys.argv[1:], exit_code=True)
    bootstrap:   File "/usr/share/yum-cli/yummain.py", line 375, in user_main
    bootstrap:     errcode = main(args)
    bootstrap:   File "/usr/share/yum-cli/yummain.py", line 281, in main
    bootstrap:     return_code = base.doTransaction()
    bootstrap:   File "/usr/share/yum-cli/cli.py", line 817, in doTransaction
    bootstrap:     resultobject = self.runTransaction(cb=cb)
    bootstrap:   File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1852, in runTransaction
    bootstrap:     self.skipped_packages, rpmdb_problems, cmdline)
    bootstrap:   File "/usr/lib/python2.7/site-packages/yum/history.py", line 939, in beg
    bootstrap:     yum.misc.getloginuid()))
    bootstrap:   File "/usr/lib/python2.7/site-packages/yum/sqlutils.py", line 168, in executeSQLQmark
    bootstrap:     return cursor.execute(query, params)
    bootstrap: sqlite3.OperationalError: database is locked

What was expected: No failure at all, or if that was an actual configuration issue outside of the bootstrap script control, a better error message with help on how to resolve said issue.

Steps to reproduce: Not sure, since flaky, but happens on fresh Vagrant deployment.

[gdemonet]

Not sure if it's really a flaky issue. Three attempts to deploy locally on a fresh Vagrant always yielded this issue, and re-running the bootstrap always fixed it.

[alexandre-allard]

Yup, I think I've an idea why it's failing, may be a concurrent access on the yum db because of the ordering of salt states in the repo formula.
I'll try something and open a PR if it fixes the issue.

[Ebaneck]

Yup, I think I've an idea why it's failing, may be a concurrent access on the yum db because of the ordering of salt states in the repo formula.
I'll try something and open a PR if it fixes the issue.

I would agree there is an existence of a race condition but this will not really explain why we do not have this same issue in the CI.

no??

[slaperche-scality]

I would agree there is an existence of a race condition but this will not really explain why we do not have this same issue in the CI.

no??

A race condition is flaky by definition.
The fact that it hasn't happened yet doesn't mean that we don't have the issue in the CI.