Grafana Agent still unable to install on FIPS-mode CentOS 9

[andrewimeson]

What's wrong?

This was first reported in #4267, and an attempt to fix was made in #4268, but the release since that PR was merged still is not able to be installed on a FIPS-mode EL system.

Steps to reproduce

# On a test CentOS 9 Stream box, enable FIPS mode (warning, irreversible)
fips-mode-setup --enable
shutdown -r now

And then with an /etc/yum.repos.d/grafana.repo with GPG checking turned on or off:

[grafana]
baseurl = https://rpm.grafana.com
enabled = 1
gpgcheck = 0
gpgkey = https://rpm.grafana.com/gpg.key
name = grafana
repo_gpgcheck = 0
sslcacert = /etc/pki/tls/certs/ca-bundle.crt
sslverify = 1

$ sudo dnf install grafana-agent-flow -y
Last metadata expiration check: 2:29:44 ago on Wed 12 Jul 2023 02:58:43 PM UTC.
Dependencies resolved.
=============================================================================================================================================================================
 Package                                          Architecture                         Version                                   Repository                             Size
=============================================================================================================================================================================
Installing:
 grafana-agent-flow                               x86_64                               0.34.3-1                                  grafana                                51 M

Transaction Summary
=============================================================================================================================================================================
Install  1 Package

Total download size: 51 M
Installed size: 177 M
Downloading Packages:
grafana-agent-flow-0.34.3-1.amd64.rpm                                                                                                         17 MB/s |  51 MB     00:03
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                         17 MB/s |  51 MB     00:03
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                     1/1
  Installing       : grafana-agent-flow-0.34.3-1.x86_64                                                                                                                  1/1
Error unpacking rpm package grafana-agent-flow-0.34.3-1.x86_64
  Verifying        : grafana-agent-flow-0.34.3-1.x86_64                                                                                                                  1/1

Failed:
  grafana-agent-flow-0.34.3-1.x86_64

Error: Transaction failed

System information

CentOS 9 Stream kernel 5.14.0-333.el9.x86_64

Software version

Grafana Flow Agent 0.34.3-1

Configuration

No response

Logs

No response

[andrewimeson]

Here's a comparison of a signed package from the CentOS repos and the Grafana Agent Flow

$ rpm -v --checksig vim-enhanced-8.2.2637-20.el9.x86_64.rpm
vim-enhanced-8.2.2637-20.el9.x86_64.rpm:
    Header V3 RSA/SHA256 Signature, key ID 8483c65d: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA256 Signature, key ID 8483c65d: OK

$ rpm -v --checksig grafana-agent-flow-0.34.3-1.amd64.rpm
grafana-agent-flow-0.34.3-1.amd64.rpm:
    Header V4 RSA/SHA512 Signature, key ID 2cf3c0c6: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK

[andrewimeson]

Looking at other projects, it may also need %_binary_filedigest_algorithm sha256 in the RPM macros.

[mattdurham]

Can you check against the boringcrypto suffixed images/binaries?

[andrewimeson]

@mattdurham doing a dnf repository-packages grafana list I don't see any RPM packages containing "boring"

Looking at 566b5b9, I don't see any changes to packaging. Since it's failing on installation, I don't think that even if there was a binary built with boringcrypto and packaged in an RPM it would help this stage of the problem.

The problem appears to be with the RPM, not the executable.

[andrewimeson]

I tried to reproduce this locally by creating a GPG key and building and signing an RPM with the ~/.rpmmacros from this repository, but the result RPM installs fine for me on a FIPS-mode test machine with both dnf localinstall and rpm -i.

The error from trying to install grafana-agent-flow might be instructive

[root@fips-test ~]# rpm -i grafana-agent-flow-0.35.1-1.amd64.rpm
error: unpacking of archive failed on file /etc/grafana-agent-flow.river;64c165ff: cpio: Digest mismatch
error: grafana-agent-flow-0.35.1-1.x86_64: install failed

Interestingly, the grafana (not grafana-agent[-flow]) package installs with no issues.

[andrewimeson]

I think it is the lack of %_binary_filedigest_algorithm. My RPM environment must have a different default and that's why my attempt to reproduce the issue by copying the .rpmmacros failed.

[root@fips-test ~]# rpm -qp --qf "%{filedigestalgo}\n" grafana-agent-flow-0.35.1-1.amd64.rpm
(none)
[root@fips-test ~]# rpm -qp --qf "%{filedigestalgo}\n" grafana-10.0.3-1.x86_64.rpm
8
[root@fips-test ~]# rpm -qp --qf "%{filedigestalgo}\n" hello-world-1.0.0-1.el9.noarch.rpm
8

Setting _binary_filedigest_algorithm to sha256 (indirectly, via FPM) is how Elastic Beats (elastic/beats#27103) fixed this issue, so I'm going to submit a PR to do the same.

[andrewimeson]

Something didn't work with my change (#4611 / #4622)

[root@fips-test ~]# rpm -qp --qf "%{filedigestalgo}\n" grafana-agent-flow-0.35.2-1.amd64.rpm
(none)

I know that the change worked when building from a CentOS 9 box, because if I set _binary_filedigest_algorithm to an invalid value that command would report (none) on my test package.

The documentation on this stuff is scattered, and most of it isn't from the RPM project 😖

I guess I'll have to build the container that the RPMs are building in and try and reproduce from there - unless there is some other hypothesis.

[rfratto]

The RPMs that get published are re-signed by the publish process, which might be making your change a no-op. @julienduchesne knows the most about the RPM publish process, but he's on PTO until the 31st.

[andrewimeson]

@rfratto I think the GPG signing is completely separate from the file digest creation (you can have the latter without the former), but I can't say for sure.

I built the build-image/Dockerfile, setup the rpmmacros file, built my test package, and it's properly setting the file digest, so I remain confused.

[andrewimeson]

I take that back - rpmsign / rpm --sign is responsible for creating the digest, not just signing with GPG, so your theory seems like a good explanation.

I'll set this aside for a while and wait for Julien to take a look when he's back.

[julienduchesne]

Seems like the fix didn't work in #4268 since the digest is not OK either on the releases on this repo or on packages.grafana.net. Using @andrewimeson's command:

julienduchesne@Juliens-MacBook-Pro ~ % rpm -qp --qf "%{filedigestalgo}\n" ~/Downloads/rpm_Packages_grafana-agent-0.34.3-1.amd64.rpm 
(none)
julienduchesne@Juliens-MacBook-Pro ~ % rpm -qp --qf "%{filedigestalgo}\n" ~/Downloads/rpm_Packages_grafana-agent-0.35.2-1.amd64.rpm 
(none)
julienduchesne@Juliens-MacBook-Pro ~ % rpm -qp --qf "%{filedigestalgo}\n" ~/Downloads/grafana-agent-0.35.2-1.amd64.rpm 
(none)

The first two files are from packages.grafana.net and the last one is from the repo's Github release. I've verified that there is no re-signing in our internal process uploading from Github releases -> packages.grafana.net so once the sig is fixed in the release process here: https://github.com/grafana/agent/blob/main/packaging/grafana-agent/rpm/gpg-sign.sh, then it should work

[rfratto]

@andrewimeson How did you try to test your fix? If you run USE_CONTAINER=1 make dist-agent-flow-packages-amd64 locally, it'll place the RPM and DEB in dist/, replacing our build environment exactly, including shelling out to a Docker container with our build tools.

[andrewimeson]

@rfratto I tried it by building the build-image/Dockerfile, execing into the container, creating a GPG key, building a "hello world" RPM, putting the gpg-sign.sh file in and modifying it to use my test key and sign the hello world package, and then verifying the filedigestalgo of the RPM.

I'll try with it with make, thanks for the tip.

[andrewimeson]

I am having a hard time getting a build environment working with the make route.

This may be exceeding my ability to troubleshoot given my lack of knowledge on the interworking of the Grafana agent build system.

[andrewimeson]

I just tested it, and the installation of v0.35.4 now works! Thank you!