Expand remote audit logging info

[elasticmachine]

Original comment by @markwalkom:

Here we're a little light on some details that might make it easier for users to understand what is happening.

eg;

• How does this integrate into SSL/TLS secured clusters?
• Are we using transport protocol? It seems implied that we do but it'd be nice to be explicit (I checked the code but wasn't 100% sure).
• Pending ^; Can HTTP be used instead?
• What are these "other settings" that are mentioned below;

Additional settings may be passed to the remote client by placing them under the shield.audit.index.client namespace

[elasticmachine]

Original comment by @jaymode:

+1 the documentation is very sparse about this feature in comparison to the rest.

How does this integrate into SSL/TLS secured clusters?

For remote indexing, you need to configure the client with the appropriate truststore and keystore. See below

Are we using transport protocol? It seems implied that we do but it'd be nice to be explicit (I checked the code but wasn't 100% sure).

Yes we startup a transport client to the remote cluster

Pending ^; Can HTTP be used instead?

No

What are these "other settings" that are mentioned below;

These other settings are any additional settings that need to be passed to the transport client such as shield.user or SSL config...

[elasticmachine]

Original comment by @ppf2:

We still don't have instructions on setting up remote auditing with SSL today in Xpack on 5.x.
(And the settings are also missing from the X-pack Settings settings page). So I am trying to figure out what the settings look like on 5.x.

So far, I am not able to get this working on 5.2. I am using certgen to create certs for both clusters so they should be signed by the same ca.

On live cluster:

xpack.ssl.key: /Users/pius/Elastic/ElasticStack_5_2/5.2.0/elasticsearch-5.2.0_ssl/config/x-pack/certificate-bundle/127.0.0.1_live/127.0.0.1_live.key
xpack.ssl.certificate: /Users/pius/Elastic/ElasticStack_5_2/5.2.0/elasticsearch-5.2.0_ssl/config/x-pack/certificate-bundle/127.0.0.1_live/127.0.0.1_live.crt

xpack.security.audit.index.client.hosts: 127.0.0.1:9333
xpack.security.audit.index.client.cluster.name: es_5_0_2_ssl_marvel
xpack.security.audit.index.client.xpack.security.user: elastic:changeme
xpack.security.audit.index.client.xpack.ssl.certificate_authorities: ["/Users/pius/Elastic/ElasticStack_5_2/5.2.0/elasticsearch-5.2.0_ssl/config/x-pack/certificate-bundle/ca/ca.crt"]
xpack.security.audit.index.client.xpack.security.transport.ssl.enabled: true

Remote cluster has the same reference to the ca:

xpack.ssl.certificate_authorities: ["/Users/pius/Elastic/ElasticStack_5_2/5.2.0/elasticsearch-5.2.0_marvel_ssl/config/x-pack/certificate-bundle/ca/ca.crt"]

It throws an error as if the cert is bad.

[2017-02-24T12:38:24,687][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [security-audit-client-ssl_node] exception caught on transport layer [[id: 0x09d82287, L:0.0.0.0/0.0.0.0:55203 ! R:/127.0.0.1:9333]], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]

So I then tested the SSL connection using a standalone java transport client using the same cert, key, and ca files, and it works fine:

                .put("cluster.name", "es_5_0_2_ssl_marvel")
                .put("xpack.security.user", "elastic:changeme")
                .put("xpack.ssl.key", "/Users/pius/Elastic/ElasticStack_5_2/5.2.0/elasticsearch-5.2.0_ssl/config/x-pack/certificate-bundle/127.0.0.1_live/127.0.0.1_live.key")
                .put("xpack.ssl.certificate", "/Users/pius/Elastic/ElasticStack_5_2/5.2.0/elasticsearch-5.2.0_ssl/config/x-pack/certificate-bundle/127.0.0.1_live/127.0.0.1_live.crt")
                .put("xpack.ssl.certificate_authorities", "/Users/pius/Elastic/ElasticStack_5_2/5.2.0/elasticsearch-5.2.0_ssl/config/x-pack/certificate-bundle/ca/ca.crt")
                .put("xpack.security.transport.ssl.enabled", "true")
        .build())
        .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("127.0.0.1"), 9333));

What am I missing here ? :) @jaymode

[elasticmachine]

Original comment by @jaymode:

@ppf2 you're missing the xpack.security.audit.index.client.xpack.ssl.certificate and xpack.security.audit.index.client.xpack.ssl.key setting for the auditing transport client.

[elasticmachine]

Original comment by @ppf2:

Thanks @jaymode. Ah, works now, thx!

@debadair
We now have multiple customers running into this. While I could write a KB article on this, this information really belongs in documentation for they are required for proper set up of remote auditing for SSL enabled clusters.

Specifically, please update the documentation with the following:

1. Update the forwarding audit logs section to provide information on all the other options that are required when the remote cluster is SSL-enabled.

The additional options are:

xpack.security.audit.index.client.xpack.ssl.certificate
xpack.security.audit.index.client.xpack.ssl.key
xpack.security.audit.index.client.xpack.ssl.certificate_authorities
xpack.security.audit.index.client.xpack.security.transport.ssl.enabled (set to true)

2. Then for completeness, also add all xpack.security.audit.index.client.* settings (and any other ones that are not covered below - @jaymode anything else?) to the list of available security settings.

xpack.security.audit.index.client.hosts
xpack.security.audit.index.client.cluster.name
xpack.security.audit.index.client.xpack.security.user
xpack.security.audit.index.client.xpack.ssl.certificate
xpack.security.audit.index.client.xpack.ssl.key
xpack.security.audit.index.client.xpack.ssl.certificate_authorities
xpack.security.audit.index.client.xpack.security.transport.ssl.enabled
xpack.security.audit.index.client.transport.sniff

[elasticmachine]

Original comment by @debadair:

Will do!

[elasticmachine]

Original comment by @inqueue:

Support is still seeing multiple reports of users struggling to setup remote audit logging just on these two points alone:

• Are we using transport protocol? It seems implied that we do but it'd be nice to be explicit (I checked the code but wasn't 100% sure).

Yes we startup a transport client to the remote cluster

• Pending ^; Can HTTP be used instead?

No

Covering these items in the docs would be extremely helpful @debadair.

[elasticmachine]

Original comment by @fxdgear:

bump @debadair @ppf2 I'm on site with a customer and having issues getting this to work.

has there been any progress on getting the docs up to date?

[tvernum]

@lcawl Bump - it would be good to at least update the docs to clarify that hosts should point to the transport port rather than the http port (actully, the port can just be left blank if the remote cluster is using the default port of 9300).

[sgelastic]

We should also point out that user account on the elasticsearch user on the monitoring cluster must have permissions to create .security-audit.

Many users have more conservative role based permissions, so it's helpful to provide this info for them.