F39 Copr builders fail Mock build with: Parsing armored OpenPGP packet(s) failed

[praiskup]

This is a blind shot, just to check whether there's anyone among the DNF team that could help us resolve this puzzle.

If users disable the Mock's bootstrap chroot feature - which means Mock uses the up2date DNF on F39 host with --installroot - the DNF on the host randomly fails with:

Red Hat Enterprise Linux - BaseOS               3.1 MB/s | 3.1 kB     00:00    
Traceback (most recent call last):
  File "/usr/bin/dnf-3", line 62, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 208, in user_main
    errcode = main(args)
              ^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 130, in cli_run
    ret = resolving(cli, base)
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 183, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.12/site-packages/dnf/cli/cli.py", line 238, in do_transaction
    self.gpgsigcheck(install_pkgs)
  File "/usr/lib/python3.12/site-packages/dnf/cli/cli.py", line 294, in gpgsigcheck
    self._get_key_for_package(po, fn)
  File "/usr/lib/python3.12/site-packages/dnf/base.py", line 2539, in _get_key_for_package
    keys = dnf.crypto.retrieve(keyurl, repo)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/crypto.py", line 137, in retrieve
    keyinfos = rawkey2infos(handle)
               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/crypto.py", line 127, in rawkey2infos
    keys = libdnf.repo.Key.keysFromFd(key_fo.fileno())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/site-packages/libdnf/repo.py", line 341, in keysFromFd
    return _repo.Key_keysFromFd(fileDescriptor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Parsing armored OpenPGP packet(s) failed

Copr build error: Build failed

So far it has been reported against RHEL 7 and RHEL 8 repositories.

[jrohel]

According to traceback, librepo found an ASCII armored OpenPGP public key block (-----BEGIN PGP PUBLIC KEY BLOCK-----) in the file. The block was passed to librpm's pgpParsePkts function to decode OpenPGP keys. The pgpParsePkts function returned an error.

Looks like a bad (unsupported?) file or a bug in librpm (or the Sequoia library - librpm uses it).

The problem is with RHEL 7 and RHEL 8 repositories?
Hmm. In RHEL 7 (and probably RHEL 8 as well) librepo uses GpgME. And there are less stringent requirements for correctness and validity of keys. Fedora 39 uses the newer librpm, which provides OpenPGP key parsing using the Sequoia library. And librepo is compiled to use librpm instead of GpgME.

[praiskup]

For the first time this happened to me personally, in the epel-9-aarch64 chroot:

[SKIPPED] rust-srpm-macros-epel-26.3-1.el9.noarch.rpm: Already downloaded      
Red Hat Enterprise Linux - BaseOS               3.5 MB/s | 3.6 kB     00:00    
Traceback (most recent call last):
  File "/usr/bin/dnf-3", line 62, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 208, in user_main
    errcode = main(args)
              ^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 130, in cli_run
    ret = resolving(cli, base)
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/cli/main.py", line 183, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.12/site-packages/dnf/cli/cli.py", line 238, in do_transaction
    self.gpgsigcheck(install_pkgs)
  File "/usr/lib/python3.12/site-packages/dnf/cli/cli.py", line 294, in gpgsigcheck
    self._get_key_for_package(po, fn)
  File "/usr/lib/python3.12/site-packages/dnf/base.py", line 2539, in _get_key_for_package
    keys = dnf.crypto.retrieve(keyurl, repo)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/crypto.py", line 137, in retrieve
    keyinfos = rawkey2infos(handle)
               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/dnf/crypto.py", line 127, in rawkey2infos
    keys = libdnf.repo.Key.keysFromFd(key_fo.fileno())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/site-packages/libdnf/repo.py", line 341, in keysFromFd
    return _repo.Key_keysFromFd(fileDescriptor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Parsing armored OpenPGP packet(s) failed

FTR, the reason for bootstrap=off is configured in this Copr project to test that the option works, not for any technical reasons (IOW, in the Mock team, we want to reveal situations like this).

[praiskup]

I was able to extract the chroot as a tarball, and reproduce with (RHN subscribed) DNF, reproducer is here:
https://download.copr.fedorainfracloud.org/archive/issues/dnf-issue-2107/

It fails in about ~50% of attempts.

$ sudo ./reproduce 
[sudo] password for praiskup: 
+ rm -rf /tmp/root
+ tar xf backup.tar.gz -C /tmp
+ /usr/bin/dnf-3 --installroot /tmp/root --releasever 8 --setopt=deltarpm=False --setopt=allow_vendor_change=yes --allowerasing --disableplugin=local --disableplugin=spacewalk --disableplugin=versionlock install python3-dnf python3-dnf-plugins-core subscription-manager --setopt=tsflags=nocontexts
No matches found for the following disable plugin patterns: local, spacewalk, versionlock
Last metadata expiration check: 0:42:13 ago on Pá 26. července 2024, 09:56:32.
Module yaml error: Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 118 col 9]
....
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/site-packages/libdnf/repo.py", line 341, in keysFromFd
    return _repo.Key_keysFromFd(fileDescriptor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Parsing armored OpenPGP packet(s) failed

Actually, I'm able to reproduce boh on F39, and on my own F40 box with pre-release DNF:

$ rpm -q rpm-sequoia rpm dnf libdnf libsolv
rpm-sequoia-1.7.0-1.fc40.x86_64
rpm-4.19.1.1-1.fc40.x86_64
dnf-4.21.0-20240725005559.6.gb23e3fbd.fc40.noarch
libdnf-0.73.2-20240725005557.3.gb245193e.fc40.x86_64
libsolv-0.7.30-20240725005551.0.g27aa6a72.fc40.x86_64

[r0x0d]

Just hit that in both epel 8 and epel 9 buildroot in copr:

epel 8: https://download.copr.fedorainfracloud.org/results/@oamg/convert2rhel/epel-8-x86_64/07872513-convert2rhel/builder-live.log.gz
epel 7: https://download.copr.fedorainfracloud.org/results/@oamg/convert2rhel/epel-7-x86_64/07872843-convert2rhel/builder-live.log.gz

[praiskup]

Yes, this became more frequent for some reason.

[praiskup]

@jrohel & @m-blaha thank you for fixing it! Any idea if the fix ever appears in EL8+? The thing is that Mock uses DNF stack from EL8 for building EL8 packages, etc. Nb. if Mock is affected, very likely end-users are affected as well.

Or, do we have a work-around?

[praiskup]

Sorry, my mistake, @jrohel corrected me off-list; this only ever affects distributions using sequoia. For Mock, that means chroots that switched to sequoia (if mock bootstrap ON), and build hosts using seqoia (if bootstrap is OFF).