Sign Logstash RPM w/ SHA256 header for FIPS-enabled Operating Systems

[n0othing]

The Logstash RPM will fail to install on RHEL 8 w/ FIPS mode enabled because it isn't signed with a SHA256 header (see BZ#1581990 [1].

[robbie@rob-nix-06 ~]$ sudo rpm -ivh logstash-7.10.2-x86_64.rpm
warning: logstash-7.10.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
        package logstash-1:7.10.2-1.x86_64 does not verify: no digest

It's possible to get around this by installing with --nodigest --nofiledigest arguments [2], but such a workaround may not be acceptable for certain organizations.

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/8.0_release_notes/index
[2] https://access.redhat.com/solutions/4460971

[SeanathanVT]

@yaauie I see the commit referenced on 2021-07-09, but as of my writing this (and logstash-7.17.5), there is no SHA256 Digest value in the RPM package -- only SHA1 and MD5. We're moving to RHEL 8 and we're experiencing the same "no digest" issue as @n0othing. For comparison, the SHA256 Digest value does appear in the Elasticsearch and Kibana RPM packages (also 7.17.5) alongside the SHA1 and MD5 Digest values.

[root@sat01 Elastic_Stack]# rpm --checksig --verbose elastic-7_x/Packages/e/elasticsearch-7.17.5-x86_64.rpm
elastic-7_x/Packages/e/elasticsearch-7.17.5-x86_64.rpm:
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
    MD5 digest: OK
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]# rpm --checksig --verbose elastic-7_x/Packages/l/logstash-7.17.5-x86_64.rpm
elastic-7_x/Packages/l/logstash-7.17.5-x86_64.rpm:
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
    Header SHA1 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
    MD5 digest: OK
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]# rpm --checksig --verbose elastic-7_x/Packages/k/kibana-7.17.5-x86_64.rpm
elastic-7_x/Packages/k/kibana-7.17.5-x86_64.rpm:
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
    MD5 digest: OK
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]#
[root@sat01 Elastic_Stack]# rpm --checksig --verbose curator-5/Packages/e/elasticsearch-curator-5.8.4-1.x86_64.rpm
curator-5/Packages/e/elasticsearch-curator-5.8.4-1.x86_64.rpm:
    Header V4 RSA/SHA256 Signature, key ID d88e42b4: NOKEY
    Header SHA1 digest: OK
    V4 RSA/SHA256 Signature, key ID d88e42b4: NOKEY
    MD5 digest: OK
[root@sat01 Elastic_Stack]#

Edit: Same issue with elasticsearch-curator-5.8.4-1.x86_64.rpm. Will look around on that project's Issue board (unless there is some internal Elastic coordination on this stuff).

[MikePaquette]

A possibly related issue has been raised by a customer on December 29, 2022

We are unable to install Logstash on a RHEL8 host in FIPS mode. We can install the logstash rpm on a fresh RHEL8 host that is not in FIPS mode. We are using the logstash rpm from the Elastic repo. We've also tried to manually download the rpm and install. In both cases the hash of the rpm is correct and validated so we know we have the authentic rpm.

I'm attaching the log output from the attempted install of the rpm. You can see the below error at line xxxx of the file.

When digging in to the issue with RedHat support we found the following error while it was unpacking the rpm:

D: create 100644 1 ( 0, 0) 268 /etc/default/logstash;63aacf71
ufdio: 1 writes, 268 total bytes in 0.000018 secs
fdio: 5 reads, 404 total bytes in 0.000168 secs
########################################
error: unpacking of archive failed on file /etc/default/logstash;63aacf71: cpio: Digest mismatch

The recent error explanation "Digest mismatch" is slightly different than the error explanation cited in the original comment, which was "no digest"

cc: @jsvd @n0othing

[jsvd]

Logstash packages already have header and payload sha256 payloads:

❯ rpm --checksig --verbose logstash-8.5.3-x86_64.rpm | grep SHA 
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: OK

Compared to elasticsearch, Logstash is only missing this "Payload SHA256 ALT digest":

❯ rpm --checksig --verbose elasticsearch-8.5.3-x86_64.rpm | grep SHA 
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 ALT digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: OK

I'll take a look to see what's missing.

[SeanathanVT]

Logstash packages already have header and payload sha256 payloads:

❯ rpm --checksig --verbose logstash-8.5.3-x86_64.rpm | grep SHA 
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: OK

Compared to elasticsearch, Logstash is only missing this "Payload SHA256 ALT digest":

❯ rpm --checksig --verbose elasticsearch-8.5.3-x86_64.rpm | grep SHA 
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 ALT digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: OK

I'll take a look to see what's missing.

Just to note, you're checking against the Elastic Stack 8 RPMs while the issue we have experienced is on the Elastic Stack 7 baseline.

The version I referenced above was 7.17.5 and I believe there have been a couple Elastic Stack 7 releases since then. We will obviously be moving to Elastic Stack 8 in the future but for now we're stuck with Elastic Stack 7.

Thank you for investigating!

[jsvd]

Recent 7.17.* Logstash RPMs already have the payload digest,

❯ rpm --checksig --verbose /Users/joaoduarte/Downloads/logstash-7.17.5-x86_64.rpm
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    Header SHA1 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    MD5 digest: OK

vs:

❯ rpm --checksig --verbose /Users/joaoduarte/Downloads/logstash-7.17.8-x86_64.rpm
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    MD5 digest: OK

@SeanathanVT any chance you could do a quick check with 7.17.8?

[Jaraxal]

Customer reports this is still an issue with 8.6.0.

[azabbett]

To add to @Jaraxal 's report, customer reports two issues that negatively affect their security posture:

• Cannot install LS on RHEL8 FIPS enabled VM without bypassing file digest
• Have to allow untrusted shared libraries

Customer's first priority is to install on RHEL8 FIPS enabled without bypassing digest

The only way they can successfully install Logstash after implementing FIPS at the RHEL8 OS level is by bypassing the file digest: rpm -ivhU --nodigest --nofiledigest
When they just use

yum -yv install /etc/deploy/installers/elastic/logstash-8.6.0-x86_64.rpm

they get

Installing:
logstash x86_64 1:8.6.0-1 @commandline 325 M

Transaction Summary
======================================================================================================
Install 1 Package

Total size: 325 M
Installed size: 568 M
Downloading Packages:
Using rpmkeys executable at /usr/bin/rpmkeys to verify signatures
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: logstash-1:8.6.0-1.x86_64 1/1
Installing : logstash-1:8.6.0-1.x86_64 1/1
Error unpacking rpm package logstash-1:8.6.0-1.x86_64
Errors occurred during transaction.
Verifying : logstash-1:8.6.0-1.x86_64 1/1
Completion plugin: Generating completion cache...
Installed products updated.
Failed: logstash-1:8.6.0-1.x86_64

Failed:
logstash-1:8.6.0-1.x86_64

Error: Transaction failed

In order to run Logstash they needed to change it's temp directory by modifying jvm.options. However, they also need to explicitly allow untrusted shared libraries as Logstash attempts to open a randomized shared library from it's temp directory at startup. This is not ideal as it opens up the OS to any executable referencing any untrusted shared library. There is no way to pre-trust these libraries as Logstash writes them to its temp directory and the filenames are randomized.

[jsvd]

wrt digest mismatch, Logstash artifact tasks aren't setting the _binary_filedigest_algorithm value to build RPMs, so by default it's MD5, which isn't accepted when FIPS is enabled in RHEL.
Building a custom RPM after patch:

~/elastic/logstash 8.6 ❯ git diff
diff --git a/rakelib/artifacts.rake b/rakelib/artifacts.rake
index f69fe2d44..2fe1aa16f 100644
--- a/rakelib/artifacts.rake
+++ b/rakelib/artifacts.rake
@@ -640,6 +640,7 @@ namespace "artifact" do
         out.attributes[:rpm_user] = "root"
         out.attributes[:rpm_group] = "root"
         out.attributes[:rpm_os] = "linux"
+        out.attributes[:rpm_digest] = "sha256"
         out.config_files << "/etc/logstash/startup.options"
         out.config_files << "/etc/logstash/jvm.options"
         out.config_files << "/etc/logstash/log4j2.properties"

Installs correctly:

logstash-8.6.1-x86_64.rpm:
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: OK

[joaoduarte@jsvd-fips ~]$ rpm --checksig -v logstash-8.6.2-SNAPSHOT-x86_64.rpm 
logstash-8.6.2-SNAPSHOT-x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK

[joaoduarte@jsvd-fips ~]$ sudo rpm -Uvh logstash-8.6.1-x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:8.6.1-1               ################################# [100%]
error: unpacking of archive failed on file /etc/default/logstash;63da885d: cpio: Digest mismatch
error: logstash-1:8.6.1-1.x86_64: install failed

[joaoduarte@jsvd-fips ~]$ sudo rpm -Uvh logstash-8.6.2-SNAPSHOT-x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:8.6.2~SNAPSHOT-1      ################################# [100%]