Add new settings to optionally disable hot reload distinguished names validation

[parislarkins]

Description

This change adds two new boolean settings for the Security plugin:

• plugins.security.ssl.http.enforce_cert_reload_dn_verification
• plugins.security.ssl.transport.enforce_cert_reload_dn_verification

Both options will default to True, which will maintain the current behaviour where performing a hot reload requires the Distinguished Names (IssuerDN, SubjectDN, and SAN) in the new certificate to match the current certificate.

When set to False, the Distinguished Names validation will be skipped when hot reloading the respective certificate (HTTP or Transport).

The current behaviour is preventing us from hot-reloading new Let's Encrypt certificates that were signed by a different intermediate CA than the original certificate, meaning we need to perform a rolling restart of each of our clusters in order to rotate the certificates. According to Let's Encrypt, one of their root CAs is expected to expire as soon as 2030 (https://letsencrypt.org/certificates/), so an approach that allowed different intermediate CAs but still rejected changed root CAs will still require us to perform a rolling restart for our clusters when the root CAs are inevitably rotated.

We do not believe adding the ability to disable this validation for hot-reloads creates any added security risk, as long as it is properly considered considering organisational context. There is no similar validation performed when changing certificates by restarting OpenSearch, and an actor that could trigger a hot-reload (requiring them to modify the certificate files on the OpenSearch node) would also likely be able to bypass the validation by restarting OpenSearch.

For example, in our operational context only the super admin user can trigger a hot-reload. The certificate for the super admin user is stored only on the OpenSearch nodes, so any malicious actor who had the super admin certificate and ability to modify the OpenSearch certificate files would also have the ability to restart OpenSearch. This makes the validation performed when triggering a hot-reload irrelevant, as any malicious actor could simply restart OpenSearch instead.

• Category: New feature

Issues Resolved

• Resolves [BUG] Exception when hot-reloading Let's Encrypt certs issued by alternate intermediate CAs opensearch-project/security#4509, although this approach covers a slightly larger scope by allowing skipping the validation entirely, and may not be appropriate for all deployments.

Testing

Unit tests have been added that cover reloading certificates to a certificate signed by a different Certificate Authority (different root and signing CA), and validating that this is rejected when the new settings are set to true, and accepted when the settings are set to false.

Check List

• New functionality includes testing
• New functionality has been documented - draft: Update hot reload section to show how DN validation can be disabled documentation-website#1
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.