Config options to enable TLS from files in a mounted directory instead of from a secret

[thelabdude]

One emerging pattern in the K8s ecosystem I've noticed is to have a cert-manager extension that issues a unique TLS certificate for each pod. Typically how this works is the cert-manager extension watches for specific annotations on the pod spec (in the STS) and then handles the certificate request per pod; once the cert is issued, the extension mounts the cert files to a directory on the pod. The files could come from a unique secret per pod or via a CSI driver that loads the files to a specified volume mount.

The current TLS solution in the Solr operator looks for a shared secret containing the keystore and supporting files; all pods share the same cert. With the cert-manager extension approach described above, the secret containing the TLS files for each pod is unique. Hence, the current TLS options in the operator won't work. The Solr operator needs to know this automated TLS mounting process is happening so it can configure Solr to use the provided keystore / truststore, as well as enable TLS in various places, such as on the probes as well as setting the urlScheme in ZK. The operator also needs to know to send requests to Solr using https.

So I'm thinking of adding a mountedTLSDir option to the existing SolrTLSOptions struct to indicate to the Solr operator it needs to enable TLS for Solr using the files in the specified directory and not look for a shared secret containing the TLS files.

When mountedTLSDir is used, the cert issuing process is opaque to the Solr operator, it just cares about the keystore, truststore, and store password files being in the PKCS12 format and being mounted on each Solr pod. The mountedTLSDir will be struct with defaults for the expected file names, e.g. keystore.p12, truststore.p12, and keystore-password. However, we need to be flexible about the names of the various files in the mount dir (although I think we can require the keystore/truststore to be in PKCS12 format)