New Plugin: logstash-output-prometheus

[SpencerMalone]

Hey folks! I'm making this issue based on the guidelines in the logstash docs, although some of the chatter on other issues made it sound like there might be a newer method to do so? If that new method is public, feel free to redirect me there.

Anyways, I have a simple output plugin over @ https://github.com/SpencerMalone/logstash-output-prometheus which outputs data to an instance of https://github.com/prometheus/client_ruby run by the plugin. This is slightly different from your average logstash output plugin because it is serving a new page that a Prometheus instance (or instances) can then scrape. I chose to go this route instead of relying on the Pushgateway (a Prometheus feature which lets you use the more traditional push model) because of the caveats and use cases mentioned at https://prometheus.io/docs/practices/pushing/

Lemme know what feedback you have, or what the next steps should be!

[SpencerMalone]

Tagging @andrewvc based on some prior open issues?

[yaauie]

Hey @SpencerMalone. First, thank you for taking the time to create this, and for going the extra mile to follow up with pings when we failed to respond in a timely manner. It's easy for things to get lost in the plugin ecosystem, since there is just a small team of us at @elastic trying to stay on top of the 200+ plugins in the logstash-plugins org (some of which also have excellent external maintainers of their own). We would be glad to help you out with this plugin as we have capacity, and to make it discoverable for those who would benefit from its use.

I've taken a look at your implementation, and would be glad to open issues on your repository and to help you chase down solutions, but first I would like to validate some of my own assumptions. I see that if multiple prometheus outputs in a single Logstash process are configured with the same port (e.g., in multiple logstash pipelines), the implementation attempts to share a single prometheus registry between them (launching a server backed by the shared registry). Since the server will fail to launch if another server is already bound to the port, I'd appreciate a little more info on why the registry is being shared (e.g., support for pipeline reloads, single server with metrics from many output plugins, multiple pipelines etc.).

[SpencerMalone]

Thanks for picking this up! No worries on the timing, I know how it goes. Lemme try to give you some context on the implementation there, as well as some further background for the choice in how the plugin is configured.

single server with metrics from many output plugins

is my OG use case. My primary use with this plugin is that we have a great internal pipeline for our nginx data from filebeat -> kafka -> logstash -> es, and I wanted to hop into that and add an extra set of logstash instances that are going kafka -> logstash -> prometheus easily. We set it up on our instances so that we essentially have two prometheus registries:

1. A registry of metadata from the logstash instances. This is stuff like the time taken on each stage for each event, or number of events dropped due to age (since prometheus does not allow for backfilling, we've chosen to drop events that are delayed by over a threshold).
2. A registry of actual event data, so like nginx_request_total and nginx_request_duration_seconds. This is where most of our engineers will add new outputs as they find new bits of data that they care about.

I wanted to avoid a unique port per logstash pipeline output requirement because it can be kind of a PITA to tell prometheus to scrape lots of ports from a single service, and while that is a configuration management problem, it felt to me like enough of a problem that it was worth offering an easy sidestep from within the plugin.

I also tried to build the configuration API to feel similar to the statsd output plugin, as we were previously using the statsd output plugin -> https://github.com/prometheus/statsd_exporter -> prometheus.

[yaauie]

@SpencerMalone I'm going to ruminate on this and circle back later in the week.

From what I can tell, there are a number of ways we'll need to improve the plugin to make it able to handle features like pipeline reloading, pipeline shutdowns etc., and we should make it more resilient to clashes in metrics (e.g., same metric defined in two different outputs currently will crash the pipeline with the second output). It's also susceptible to race conditions in its current form where two pipelines pointing to the same port are started simultaneously, which can be worked around fairly easily.

I'll futz around with it as a "back burner" project this week and get back to you.

[PedroMSantosD]

Do you have a metrics screenshot?
I'm curious on whether this will let me scrape the data from whatever logstash is receiving (completely skipping ES for my alerting), or if this is about exporting logstash-internal-metrics

Thanks,

[SpencerMalone]

This is about exposing arbitrary data received by Logstash, which I think is what you want.

[Srivalle]

Hi
I am not able to install logstash-output-prometheus . Could you please help in installation. . I am using Ubuntu16.04
ubuntu@ip-xxxxx:/logstash-output-prometheus$ ./script/build
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:29: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:30: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Fetching gem metadata from https://rubygems.org/........
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Could not find gem 'logstash-core-plugin-api' in any of the gem sources listed in your Gemfile.
ubuntu@ip-xxxx:/logstash-output-prometheus$

[strainu]

@SpencerMalone @yaauie is there still interest in developing this plug-in?

[SpencerMalone]

I've not heard anything, in subsequent years I moved away from this pattern, although it was nice in the transition period. Nowadays I mostly rely on mtail for unstructured logging, or writing my own exporter that parses different log files for structured logging (in the rare circumstances that I need metrics retrievable only from logging).

I'm happy to hand over the maintainer flag if you have goals that involve this.