integrate alerting system with kibana

[Arun-Ghosh]

Hi ,

It will be better if there is an alerting system with kibana. SMTP or SMPP any one or both will be good.

[martinseener]

This will not happen, since Kibana is totally client-side. Iam right now also in the process of integrating such an alerting-system but iam doing this with the nagios output from logstash. An alerting-system, even possible to send mails with javascript, would not make sense, since a user has to open kibana, so kibana can send the mail - and then it makes no sense anymore.

better try to create a totally autonomous server-side version with logstash itself (nagios, nagios_nsca for ex.) or even with munin and munin-limits for more historic alerts! (what we do here at moment!)

[VAdamec]

I wrote some plugins for Nagios, which directly call elastic cluster (exact queries). It's quite easy todo even for noob like me.

[chenryn]

There should be some webpage to configure alert rules and watch the history for logstash, and they would all store in some elasticsearch indices. But I don't think it's the vision of kibana.

[martinseener]

@VAdamec can you post some of those in a gist or something? would like to see them and maybe do the same lik you!?

[lokivog]

I would like to make a suggestion. In order to make an altering system to work with Kibana/logstash logs that is robust and easily configured for any type of log event stored in elasticsearch. I would suggest designing an external alerting system/application (separate from Kibana) using a pull approach that queries elasticsearch for specific logs using custom queries. Meaning the system is designed in a way so that an alert is configured by specifying a query, fetch time interval, an alert type. The system should be designed so any type of alert module can be added as a plugin into the system. Ex. Email, Twitter, IRC, etc. Each alert log allows you to specify which alert types to trigger. Not sure how this would scale but something to think about.

[martinseener]

@lokivog basically a great idea but i hate it to have 502833723 tools for everything. why not just designing a generic nagios check which can handle different queries? much easier to accomplish and also useable for icinga, shinken and so on. nagios itself can call pagerduty, write mails or post a DM on twitter.... let us not reinvent the wheel!

[lokivog]

@martinseener good point and agreed. Ideally take an existing system that already has the core alerting functionality. Should have mentioned that.

[martinseener]

@lokivog but to combine your idea with @VAdamec query scripts, we can build such a generic check which can handle lots of stuff out of the box. and example for such a good script could be Bucardos check_postgresql which has actions for pre-defined checks

[mikmik]

You can use Logstash to catch inputs and output them wherever you want. This might fulfill your requirement Arun...
Lots of input and output plugins already in existence, plus you can create your own.

[Arun-Ghosh]

Thanks a lot ... Yes ... I already implemented with logstash ...

[tomkregenbild]

I used Splunk for a long time and when I started using kibana in a different organisation I immediately felt that an alert system was missing. With Splunk we where able to create alerts that not only sent mails with saved reports or search results on a specified condition but also trigger scripts from the server side in order to response automatically to certain events in our system. Using alerts we were able to create a powerful monitoring architecture that based on logs and not as its data.

I know that Kibana is client side but a server side scheduling mechanism for running periodic searches with the ability to alert in case the search find specific results or any results at all will make Kibana a very powerful system.

[TiNico22]

+1
now K4 is server side also. It should be possible to add it on the server side.
with K3, a "simple" workaround with some scripting is

• build your request /dasboard from K3 you want to use as alerting if things are matching
• catch the request to ES
• script (curl) this request in a crontab and parse the result to decide when to send an email or other stuff.

[sahilthapar]

👍

[TiNico22]

from closed #1675
It could be great to visualize alert in a specific App in Kibana 4

[kiranos]

+1 it would be great to be getting alerts if a criteria would match.

[drfh]

I propose that an API or "standard" be created for interrogating Kibanna on the status of queries. That way you can setup the queries on Kibanna and then let an external alerting system take it from there.

Also, if an "Alerts" tab was created then the user could use it to view/direct what they want the external alerting application to do. It seems that as long as the (query -> reporting) it triggered from another program then @martinseener would still be right about Kibanna being client side (No design change necessary). I would ask to have only a tab to visualize and edit what the user would like to happen and also display what apps are registered with Kibanna:Alerting to process what is directed.

[drfh]

Another way of looking at it is that Kibanna would be a great interface/app to direct logging related actions.
** There is a fault with this in that (per proper programming) you should not have to poll to create a trigger for an event. The event should be the trigger and lead to the process organically. The only polling should be on logstash input.
That being said it would be better to have some integration in the future with elasticsearch or even logstash, to (notify/kick off) the alerting system when very simple criteria are met (example: more that 25 of a type of error in last 5 min). That could then lead to the actual (query -> reporting) being run.

Another way of doing this is to have the future integration I referred to just create a +1 when a tag/value is seen on elasticsearch, and the send that to a messaging brokerage/queue (example: RabbitMQ) and then let the alerting app use that as a trigger to do it's work. That way it can have multiple instances as well resulting in alerts getting created during critical outages.

[VAdamec]

Sorry for late response, setup with just querying ES was quite unstable (it's probably because I'm really no programer) and it's still under development. My actual attempt is to use some sort of fast replicated KV/service system in the middle (now trying Consul) which are filled via periodic calls (cron/jenkins/nagios or consul service check) and alerts are fired on value/service change (Consul watch) instead on relying of direct ES calls.

[tfladsrud]

+1. Alerting from Kibana 4 (by e-mail or SMS, or both) would be great. Heard rumors that this is something that eventually comes in Kibana 4 , is this correct?
We have project in our organization where we look at ELK as the system for monitoring and reviewing our logs, and one important feature many of my colleagues ask for is alerting in Kibana.

[dzubchik]

+1 We are currently searching for log aggregation system with alerting system in our company. It would be great to have such feature in kibana.

[mikmik]

Hi Guys, I originally introduced this idea of an integrated Alerting System as I thought it could be standard in ELK.
This idea seems to have legs (see above), and our thoughts have progressed slightly:

We are thinking of pairing ELK with a bespoke "Alerting System" that would allow you to:

• Set the events you are interested to capture
• Set lists and/or groups of Users you want to alert
• Select the Alert outputs you want to pair to this event

It seems that JMX beans would be ideal to monitor/capture the events themselves. Any monitoring tool e.g. Zabbix/Nagios etc. could subscribe to them, so would this specific Alerting System.

The outputs would be "plugins" written progressively as needed, starting with e-Mail, SMS, Voice apps etc.

Does anyone know if such a system exists as a stand alone in the FOSS world ?
I am interested to contribute and build one if not... target Q2 2015.

Comments welcome.

[mylesagray]

Voicing my support for this as well, we are looking for a logging aggregation system that will do alerting based on statistical methods like std-dev. etc.

[kaos]

We're also looking at a integrated solution for alerting with kibana.

[dworld]

+1
I hope I can configure and view alerting in Kibana

[parabolic]

+1 that would be just perfect!

[jesseangell]

+1 -- it makes a lot of sense for this to be in Kibana 4 based on saved searches.

[sweetest]

We've used ElasticSearch for a long time and recently implemented alerting system before we use kibana. In my opinion, it would cause substantial complexity to kibana itself. Simple alerting can be handy, but that's not the one that you want. You may want to specify query, interval, threshold log count etc(similar to Splunk's alert). But handling all those together in kibana seems impossible to me since it has to manage all those stuff with unlimited numbers of alert rules.

[d1rtym0nk3y]

+1

[rocksfrow]

+1 -- although currently using elastalert with success. The trick is to make sure you monitor elastalert with a separate process as it can crash due to non-properly configured rules and unexpected data.

[sergiolr100]

+1

[nejczupan]

+1 @rocksfrow what are you using as separate process?

[metabsd]

+1

[elasticmachine]

Pinging @elastic/kibana-stack-services

[Dave-c-Ross]

Not sure where this issue is going ... SMTP is already part of xpack actions, and SMPP ... nah

[ymao1]

Closing because Alerting is part of Kibana