Support 'certificate' in the ssl.verification_mode options for TLS options.

[ph]

We support the full or the none values for the verification_mode in our TLS options for the outputs. It is always a bad idea to use none because this is the equivalent of allowing everything. But sometimes due to the nature of your network, it would be nice to be able to test a certificate without having to test the hosts.

Adding a certificate value would mean that we will check the validity of the certificate, the CA but we would ignore the hostname check.

[ph]

I am not sure the whole story about that but we have a commented certificate option.

[CamiloSierraH]

Adding a certificate value would mean that we will check the validity of the certificate, the CA but we would ignore the hostname check.

+1

[mattpr]

+1

Kind of pointless to bother with TLS if we completely ignore the validity of the certificates or that they are signed by a recognized CA.

The hostname validation seems a bit silly to me in a devops environment since we often connect using whatever is spat out of service discovery which might be an IP or ephemeral hostname. Different machines might connect to the same host via different IP/hostname as well especially in a multi-cloud environment.

Having to generate new internal certificates every time we move something around because of this hostname validation requirement seems like a bit of an odd default.

Hostname validation makes sense (to me) when we are talking about public services that are connected to by many potentially unknown clients...and in those cases we would generally have certificates signed by a commercial CA so that our certificates are signed by a trusted CA for those many unknown clients. Since the trusted CA in these cases sign certificates for everybody, you need to use hostname verification or there can't be any trust.

However when we are talking about servers talking to servers within single organizations, often the CA will be a private/internal CA. Verifying the signer is the internal CA is already a big verification as it implies that although you might connect to the wrong server you are connecting to a server controlled by the organization and not subject to a MITM attack. Further verification could optionally be done by requiring that CN or SAN on the server cert contain a specific string (e.g. "logstash-server").

Choosing between strict hostname verification and not even checking that the cert was signed by a trusted CA are both bad choices for many use cases. Of course we can work around this by just adding entries to our hosts file to make sure everyone is connecting to the logstash-server using the same hostname...but isn't this forcing a tighter coupling between service discovery and beats than should be necessary? Additional complexity without real value. After all we aren't talking about public services and commercial CAs in most cases, right? Would be good to at least have the option to do something in the middle.

If beats is really ignoring output.* > ssl.certificate_authorities when ssl.verification_mode: none then I would expect a big fat warning message saying that the specified certificate authorities will be ignored and all certificates will be accepted...otherwise one might expect that this setting is still being used to validate certificates even if other verification (e.g. hostname) is disabled.

Additional comments on the topic here:
https://discuss.elastic.co/t/tls-verification-mode-custom-validation-options/171532

[Matt-Yorkley]

This would be a great enhancement, and would bring the beats options inline with the verification_mode: certificate settings in Elasticsearch, Kibana, and Logstash. As it is, that setting is fairly useless elsewhere if beats don't support it.

[ltning]

What's happening here? Seems like a no-brainer, and not having this causes a lot of headaches. In an environment with an in-house, trusted CA and a cluster of receiving ends, having hostname verification is pretty much meaningless as long as the chain is being validated. Point is - leave it to the user.

[anyasabo]

For another example use case: in ECK, we allow users to provide us with their own certificate chain. This may be valid for the host name they are using to connect to Elasticsearch (perhaps a public host name that they route through a proxy), but is not necessarily valid for the Kubernetes internal host name that we use to connect to Elasticsearch from a Beat inside of the cluster. Currently we need to disable SSL verification entirely to enable this use case. Being able to use an intermediate amount of verification would be helpful.

You can see a bit about how we implement similar functionality to allow our operator to connect to ES here if it's helpful.

Related: elastic/cloud-on-k8s#2415 (comment)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)