[Stack Monitoring] Remove internal collectors

[ycombinator]

The work described in this issue is a breaking change. As such, it is intended to be released only in a major release (i.e. no backports).

We now have the ability to monitor Elasticsearch, Kibana, Logstash, Beats, and APM Server with Metricbeat. The previous method of monitoring stack products, using collection code internal to each product that shipped monitoring data to a custom Monitoring Bulk API endpoint, is now deprecated. And in the next major release (likely 8.0.0), all code related to this internal collection approach should be removed.

This issue tracks the removal of one specific item related to the internal collection approach: the code within Logstash that periodically collects monitoring metrics to construct logstash_stats and logstash_state documents, and ships them to the custom Monitoring Bulk API in Elasticsearch (POST _monitoring/bulk). Note that settings related to this code (xpack.monitoring.*) should also be removed, including from documentation.

By removing this internal collection code Logstash will no longer be able to monitor itself (for the purposes of the Stack Monitoring application in Kibana). Instead, users will have to run Metricbeat along side Logstash and enable the logstash-xpack module. This module knows how to poll various Logstash APIs to periodically collect monitoring metrics and ship them to the monitoring Elasticsearch cluster.

While removing the internal collection code, be careful to make sure that the various Logstash Monitoring APIs continue to function as-is, as they are required for Metricbeat collection.

[andsel]

@ycombinator what do you think of using the APM's instrumenting agent to monitor metrics of stack's components?

[ycombinator]

@andsel For now, we plan to use Metricbeat but definitely would be interesting to consider the APM agent in the future as well. I suggest chatting with @cachedout about it, as he's the area lead for the @elastic/stack-monitoring team.

[cachedout]

Hi @andsel. I've thought a bit about this but I'd definitely be interested to hear more about specifically how you see APM instrumentation fitting into Logstash monitoring.

[andsel]

Mine are only plain thoughts. APM has a client lib to instrument Java code (with the agent) and LogStash at the end is a Java application. So we could benefit from standard monitoring metrics, to investigate the behaviour of the heap, including pauses from GC, but we can use it to replace the internal collectors to count the events per second processed by every plugin. Going deeper, we can know how much time an event spend in every plugin, and probably with percentiles knowing which is the more time consuming events, discover the latency that an event experience during the provessing.

[cachedout]

@andsel Thanks for the explanation. I agree that this is worthy of more investigation but I think the first step here would be to identify the potential enhancements that an APM-based approach would have over the current pipeline viewer.

[cachedout]

@jsvd I'd just like to bring this issue to your attention and see if we can get it assigned to a Logstash developer soon. Let me know what you think. Thanks!