[nagios] Add Nagios package #2824

kush-elastic · 2022-03-15T12:43:58Z

Enhancement

What does this PR do?

Generated the skeleton of Nagios integration package.
Added 3 data stream (Logs, Host and Service)
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
Added dashboards and visualizations.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/nagios directory.
Run the following command to run tests.
elastic-package test

Related issues

Relates Create Nagios Integration #2626

Screenshots

Integration Page

Nagios XI Overview Dashboard

elasticmachine · 2022-03-15T12:59:19Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-31T08:05:03.004+0000
Duration: 16 min 40 sec

Test stats 🧪

Test	Results
Failed	0
Passed	34
Skipped	0
Total	34

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

mtojek

I don't have much experience with Nagios, especially from the configuration site and I'm not following the product since a few years. Based on your PR, I assume that we're tracking the "Nagios XI" integration, not the "Nagios"? Should we rename the integration to "nagios_xi"?

The first round, so mostly general comments:

Please rename "logs" to "events" as this is about catching notifications and alerts.
Are there more commands that Nagios provides, not just the "host check"? If so, maybe we need to add the "commands" data stream instead.
Could we split the "services" into a few different data streams? I appreciate the idea of having a multi-select box, but we'd rather go with multiple data streams if possible. Feel free to propose any way of dividing them.

elasticmachine · 2022-03-15T13:15:53Z

Pinging @elastic/integrations (Team:Integrations)

kush-elastic · 2022-03-17T12:15:21Z

Hey Marcin,
Thanks for the Review.
We found a few things while changes were requested in Demo Call.

Please rename "logs" to "events" as this is about catching notifications and alerts.

Done

Are there more commands that Nagios provides, not just the "host check"? If so, maybe we need to add the "commands" data stream instead.

There is no more host check command other than the two we have added support for other commands are for services so I guess we don't need to change the data_stream name.

Could we split the "services" into a few different data streams? I appreciate the idea of having a multi-select box, but we'd rather go with multiple data streams if possible. Feel free to propose any way of dividing them.

So as we discussed over the same that we decided to have a single data_stream.
The suggestion was to use toggle buttons with fix 8 types mentioned and have all check commands based on selected toggles.
But the problem is we can not pass multiple values in the query. Check commands can also be dynamic (check_xi_service_ping!3000.0!80%!5000.0!100%). and Current API doesn't support this kind of query (query: check_command=lk:check_xi_service_ping).
we tried this : check_command=in:check_xi_service_ping but it also doesn't support multiple queries in a single call.
So we have to use service names in queries. (display_name=in:HTTP,Current Users,etc.....)

mtojek · 2022-03-17T12:34:32Z

Thanks for the update, @kush-elastic.

So as we discussed over the same that we decided to have a single data_stream.
The suggestion was to use toggle buttons with fix 8 types mentioned and have all check commands based on selected toggles.
But the problem is we can not pass multiple values in the query. Check commands can also be dynamic (check_xi_service_ping!3000.0!80%!5000.0!100%). and Current API doesn't support this kind of query (query: check_command=lk:check_xi_service_ping).
we tried this : check_command=in:check_xi_service_ping but it also doesn't support multiple queries in a single call.
So we have to use service names in queries. (display_name=in:HTTP,Current Users,etc.....)

I'm wondering if there is an option to perform multiple queries, one per selected item. Having it defined as the query isn't the best user experience as @akshay-saraswat raised yesterday, it's error prone. Maybe we should split it into different data streams?

Let's collect some feedback around this - @ruflin, @jsoriano WDYT? You can take a look at the "query" on the screenshot above.

ruflin · 2022-03-17T15:44:16Z

Is there a limit to the nagios services metrics? How many are there? Do you have a link to the query language by chance?

jsoriano · 2022-03-17T16:24:43Z

You can take a look at the "query" on the screenshot above.

What are these queries? Are these names of checks defined in Nagios?

kush-elastic · 2022-03-21T06:21:41Z

Hey @ruflin,
You can try the demo version which the official Nagios XI provides for a query language reference. Object Reference
Here's link.

kush-elastic · 2022-03-21T06:31:16Z

Hey @jsoriano,
We are using display_name from the below response.
https://nagiosxi.demos.nagios.com/nagiosxi/api/v1/objects/servicestatus?apikey=7I4CZHGQSbbE79Isaop9LG4IRkpFOC6bAaMP7oVYC7TZvBrJYT80PcIWvnCEgX56&pretty=1&display_name=in:SSH,Current Users

{
    "recordcount": 1,
    "servicestatus": [
        {
            "host_name": "localhost",
            "service_description": "Current Load",
            "display_name": "Current Load",
            "host_object_id": "145",
            "host_address": "127.0.0.1",
            "host_alias": "localhost",
            "icon_image": "",
            "icon_image_alt": "",
            "notes": "Testing 123",
            "notes_url": "",
            "action_url": "",
            "servicestatus_id": "21",
            "instance_id": "1",
            "service_object_id": "157",
            "status_update_time": "2020-04-06 14:16:30",
            "output": "OK - load average: 0.26, 0.32, 0.30",
            "long_output": "",
            "perfdata": "load1=0.260;5.000;10.000;0; load5=0.320;4.000;6.000;0; load15=0.300;3.000;4.000;0;",
            "current_state": "0",
            "has_been_checked": "1",
            "should_be_scheduled": "1",
            "current_check_attempt": "1",
            "max_check_attempts": "4",
            "last_check": "2020-04-06 14:16:30",
            "next_check": "2020-04-06 14:21:30",
            "check_type": "0",
            "check_options": "0",
            "last_state_change": "2020-02-01 16:10:38",
            "last_hard_state_change": "2020-02-01 16:10:38",
            "last_hard_state": "0",
            "last_time_ok": "2020-04-06 14:16:30",
            "last_time_warning": "1969-12-31 18:00:00",
            "last_time_unknown": "1969-12-31 18:00:00",
            "last_time_critical": "1969-12-31 18:00:00",
            "state_type": "1",
            "last_notification": "1969-12-31 18:00:00",
            "next_notification": "1969-12-31 18:00:00",
            "no_more_notifications": "0",
            "notifications_enabled": "1",
            "problem_has_been_acknowledged": "0",
            "acknowledgement_type": "0",
            "current_notification_number": "0",
            "passive_checks_enabled": "1",
            "active_checks_enabled": "1",
            "event_handler_enabled": "1",
            "flap_detection_enabled": "1",
            "is_flapping": "0",
            "percent_state_change": "0",
            "latency": "0.00219999998807907",
            "execution_time": "0.00254",
            "scheduled_downtime_depth": "0",
            "failure_prediction_enabled": "0",
            "process_performance_data": "1",
            "obsess_over_service": "1",
            "modified_service_attributes": "0",
            "event_handler": "",
            "check_command": "check_local_load!5.0,4.0,3.0!10.0,6.0,4.0\\!test!!!!!!",
            "normal_check_interval": "5",
            "retry_check_interval": "1",
            "check_timeperiod_object_id": "129"
        }
    ]
}

Users have to specify the Services Names that they want to monitor.

kush-elastic · 2022-03-21T07:03:08Z

Hey @mtojek,
If we split data_streams with commands, then we have this many commands. and we will also have to create as many data_streams. and in the future, if we want to support more data_streams. we will have to add as many data_streams.

check_local_users
check_local_load
check_ssh
check_xi_service_ssh
check_ping
check_xi_service_ping
check_local_swap
check_local_procs
check_http
check_xi_service_http
check_local_disk

What do you think we should do??

mtojek · 2022-03-21T08:53:26Z

Don't worry about the number of data streams, it's still fine. Take a look at the zeek package :)

jsoriano · 2022-03-21T10:51:27Z

Current approach of filtering by display name can be error prone, and it can be also seen as arbitrary to some users that may prefer to filter by checks or hosts.

Having datastreams per check may not fit so well for something like nagios, where it is very likely that the customer has custom checks. Nagios checks are basically scripts, and someone relying on nagios for monitoring probably has custom scripts for their custom applications. We wouldn't be supporting this common use case. Or we would need something different for these use cases, what can be a bad user experience.

Have we considered to start by collecting everything by default? This is what we use to do when collecting data from other monitoring systems (see prometheus, statsd or munin).

Then we can add specific fields for specific filters. For example it seems possible to filter per host using the host_address field (example). We could have an optional field to filter per host. Display name could be another optional filter.
I haven't found a reliable way to filter per command when they have parameters, but there may be one.

With this kind of filters we could also leverage autodiscover at some point. There could be a nagios autodiscover provider that queries the inventory of hosts there, and generates one config per monitored host, with the possibility of using conditions, variables and all the available features.

Btw, another question I have is how are we handling very big API responses? I guess that the size of the responses may easily grow for users with hundreds or thousands of hosts.

mtojek · 2022-03-29T09:10:30Z

packages/nagios_xi/_dev/build/docs/README.md

+
+## Compatibility
+
+This module has been tested against `Nagios-XI Version: 5.8.7`


nit: This integration

packages/nagios_xi/_dev/deploy/docker/docker-compose.yml

mtojek · 2022-03-29T09:14:59Z

packages/nagios_xi/data_stream/events/fields/fields.yml

@@ -0,0 +1,14 @@
+- name: nagios_xi.event
+  type: group
+  release: beta


I think you can drop this property.

Sure, Thanks.

mtojek · 2022-03-29T09:16:42Z

packages/nagios_xi/data_stream/host/elasticsearch/ingest_pipeline/default.yml

+    field: json.last_check
+    formats:
+    - yyyy-MM-dd HH:mm:ss
+    target_field: nagios_xi.metrics.host.last_check


Something is fishy here. Why do we need the . metrics. group in the middle? It should be nagios_xi.host.....

Sorry for confusion, we updated logs to events. to avoid conflicts we added metrics before host and services. as we have events now we will update this with your suggestion.
Thanks

mtojek · 2022-03-29T09:17:40Z

packages/nagios_xi/data_stream/host/fields/fields.yml

+  type: group
+  release: beta
+  fields:
+    - name: host


Could you please re-check these fields against ECS? I guess you can map some of them.

Sure, I will check it out.
Thanks

mtojek · 2022-03-29T09:19:19Z

packages/nagios_xi/data_stream/service/fields/fields.yml

+  type: group
+  release: beta
+  fields:
+    - name: service


Same story here: please check these fields against ECS.

mtojek · 2022-03-29T09:20:45Z

packages/nagios_xi/manifest.yml

+  - monitoring
+release: beta
+conditions:
+  kibana.version: ^8.0.0


I guess that you can pair it with next release of Kibana, ^8.2.0.

Hey @akshay-saraswat ,
What are your thoughts on this? Should we just keep the next Kibana release i.e. 8.2.0. If so, we would have to test it on 8.2.0 image since as we discussed, we have just tested it on 8.0.0 and 8.1.0.

8.2.0 sounds okay to me. If we release it with 8.2.0, we anyways would have to test it with that version. If we keep the minimum Kibana version 8.2.0, we will have fewer versions to consider for backward compatibility and support.

mtojek · 2022-03-29T09:21:37Z

packages/nagios_xi/manifest.yml

+icons:
+  - src: /img/nagios_xi-logo.svg
+    title: Nagios XI logo
+    size: 279x187


Hmmm.. could you please use a standard-sized logo :) like 256x256 or 512x512.

mtojek · 2022-03-29T09:21:49Z

packages/nagios_xi/manifest.yml

+          - name: api_key
+            type: text
+            title: API Key
+            description: "API key for the Nagios XI API. (Location of API_KEY in Nagios XI: Click on your username on the top right corner of Nagios XI Panel -> My Account -> Account Information -> General Account Settings -> API Key)"


mtojek · 2022-03-29T09:23:52Z

packages/nagios_xi/kibana/dashboard/nagios_xi-3a61d470-a0fe-11ec-8479-e1ade3344260.json

@@ -0,0 +1,434 @@
+{


What do you think about splitting it into following dashboards:

Overview

Service dashboard

Events dashboard

Hosts dashboard
?

We don't have much visualizations in Nagios XI as most of them are metrics only.
What do you think of current dashboard as it has more compact and full view of current status? (I have Updated SS in PR description)

What do you think of current dashboard as it has more compact a

I would say it's packed with too many information at once. For example, I'd like to see a screen with logged events and the percentage of incoming ones, to see which are the most frequent.

What does the HTTP response size mean? and response time? is it related to a particular endpoint?

What kind of hosts does it present?

akshay-saraswat · 2022-03-30T06:08:57Z

For dashboards, I have the same suggestion that Marcin said during the demo. Let's move the Event and Host visualizations somewhere after the service visualizations because those are lists of logs and IP addresses that may not be apt for an overview dashboard that is constantly displayed on a TV in an office.

ruflin · 2022-03-30T07:07:35Z

Can we please follow the same approach as recommended in #2811 (comment) here and split up the work to data streams?

kush-elastic · 2022-03-31T06:53:38Z

Hey,
This are the new split PRs for Nagios Integration.

ruflin · 2022-03-31T07:55:17Z

Is my understanding correct that the one we should start with reviewing is #2940 Lets set the other 2 to draft until we have this one in to make sure we don't review things twice. As soon as the base skeleton is in, the PR's can be opened in parallel.

kush-elastic · 2022-03-31T08:57:28Z

Yes @ruflin, Right you can start reviewing from #2940.
I have set other two PRs as Draft PR.
Thanks

kush-elastic · 2022-05-09T13:07:06Z

Closing this PR as it was split up into multiple PRs as mentioned in the comment #2824 (comment). All the parts are now merged and the linked issue (#2626) has been closed.

Thanks a lot @mtojek, @ruflin, @jsoriano, @akshay-saraswat and @lalit-satapathy for taking out time to review the PRs and providing valuable feedback!

initial commit for Nagios package

281d6d3

kush-elastic requested a review from a team as a code owner March 15, 2022 12:43

mtojek requested a review from a team March 15, 2022 12:45

mtojek reviewed Mar 15, 2022

View reviewed changes

kush-elastic self-assigned this Mar 15, 2022

kush-elastic added enhancement New feature or request Team:Integrations Label for the Integrations team New Integration Issue or pull request for creating a new integration package. labels Mar 15, 2022

kush-elastic added 3 commits March 17, 2022 15:19

Requested changes

c35f5be

Kibana dashboards version 8.1.0->8.0.0

397e2c9

CODEOWNERS nagios->nagios_xi

0a1003b

kush-elastic requested a review from mtojek March 17, 2022 12:18

Update with host_address and display_name

0c7aee2

mtojek requested a review from ruflin March 29, 2022 08:42

mtojek reviewed Mar 29, 2022

View reviewed changes

kush-elastic marked this pull request as draft March 30, 2022 18:22

kush-elastic closed this May 9, 2022

ShourieG mentioned this pull request Dec 13, 2022

[Cisco Meraki] Error handling improvement for timezone field #4811

Merged

4 tasks


		## Compatibility

		This module has been tested against `Nagios-XI Version: 5.8.7`

[nagios] Add Nagios package #2824

[nagios] Add Nagios package #2824

Conversation

kush-elastic commented Mar 15, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Mar 15, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

mtojek left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 15, 2022

kush-elastic commented Mar 17, 2022

mtojek commented Mar 17, 2022

ruflin commented Mar 17, 2022

jsoriano commented Mar 17, 2022

kush-elastic commented Mar 21, 2022

kush-elastic commented Mar 21, 2022

kush-elastic commented Mar 21, 2022

mtojek commented Mar 21, 2022

jsoriano commented Mar 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshay-saraswat Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshay-saraswat commented Mar 30, 2022

ruflin commented Mar 30, 2022

kush-elastic commented Mar 31, 2022 • edited Loading

ruflin commented Mar 31, 2022

kush-elastic commented Mar 31, 2022

kush-elastic commented May 9, 2022 • edited Loading

kush-elastic commented Mar 15, 2022 •

edited

Loading

elasticmachine commented Mar 15, 2022 •

edited

Loading

akshay-saraswat Mar 30, 2022 •

edited

Loading

kush-elastic commented Mar 31, 2022 •

edited

Loading

kush-elastic commented May 9, 2022 •

edited

Loading