EventCommand execution location should be more flexible (not depend on CheckCommand execution location)

[Tontonitch]

Hello,
According to the documentation, an event handler is always launched on the location where the check command has been launched:

https://www.icinga.com/docs/icinga2/latest/doc/03-monitoring-basics/#event-commands

If the host/service is located on a client as command endpoint the event command will be executed on the client itself (similar to the check command).

It may be okay for lots of cases, but in my case it is problematic.

Indeed, I use the Icinga2 client on our Windows servers, so I specify the command_endpoint in the service definition so the check commands are executed on the targets. With such a configuration, the event handler (EventCommand) is also launched on the target itself.
In my case, such a behavior is problematic, as the service result would indicates a major server issue and the eventhandler should launch an action on another server (on a System Center Orchestrator). It would be better to not rely on the target for the execution of the eventhandler by on the icinga2 master node.

Also, the execution of the evenhandler from the icinga2 master node(s) would be more secure as it simplify firewall rules.

Please note that this limitation has been reported by some other users:

• https://monitoring-portal.org/index.php?thread/41226-event-handlers-with-icinga2-and-windows-endpoint/&postID=253542&highlight=Event%2BCommand%2Bnot%2Bcommand%2Bendpoint#post253542 (see last post)
• https://monitoring-portal.org/index.php?thread/41573-event-commands-vs-notifications/&postID=254261&highlight=Event%2BCommand%2Bnot%2Bcommand%2Bendpoint#post254261

For your information, as a workaround, I use for this specific case a snmp check, which is launched by the icinga2 master node, so in that case the eventcommand is also launched from there.

Expected Behavior

Allow an eventcommand to be executed on icinga2 master nodes / satellites even if the command_enpoint variable is set to the taget node.

Current Behavior

Currently, if the host/service is located on a client as command endpoint the event command will be executed on the client itself (similar to the check command).

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

• Version used (icinga2 --version): r2.7.0-1
• Operating System and version: Red Hat Enterprise Linux Server release 7.4 (Maipo)
• Enabled features (icinga2 feature list): api checker command ido-mysql livestatus mainlog notification perfdata
• Icinga Web 2 version and modules (System - About):
• Config validation (icinga2 daemon -C):
• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

[dnsmichi]

Unfortunately there were no recommendations made in #2895 which would allow a non-breaking change for command endpoint checks. This is the same for this issue being a duplicate - propose a configuration attribute and technical implementation and sponsor a patch.

[Tontonitch]

Ok, it missed this issue when I searched.

The proposed implementation would be okay for me:

a variable like "event_command_endpoint" which is by default set to the command_endpoint, but can be set to i.e. "local" (1, simplier) or even anothers icinga-remote-instances's endpoint-name which executes the command (2, more complex)

Just to explain my situation, we somehow integrate Icinga2 with System Center Orchestrator.

A check command executed on a monitored host can produces a CRITICAL hard state which would trigger an event command. This event command could be a call to the System Center Orchestrator web services (to launch a runbook). In such a case, for configuration and security reasons, it would be much better to launch this event command from the icinga2 master/satellite servers, even if the check command has been executed on the monitored host via the icinga2 client.

[b33rcity]

Would also like to have this feature as it would enable immediate testing of alternate routes to a node should it become unresponsive from the master.

[lxf1992521]

Instead of doing more with EventCommand, using master node notification to achieve this is better and elegant.

I treat icinga USER object as "output channel" , created some users: email, sms, alertmanager, rpccallback, graylog...., send check output to them, and send notidication or run script by these third party programs.

[sysadminsha]

Hi, did you ever retrack this issue? At the moment we have a request to monitor the service A on Server X, Y and Z; And if service A is failing on one of these Servers, a script "restart_A.sh" on server X should be executed. This restart request is not possible until command_endpoint and event_command endpoint cannot be separated.

[dnsmichi]

@sysadminsha This is a known problem, with no clear vision how to proceed. That's why it is left opened as TODO.

Here's some notes I just made for a concept.

Description

The culprit is the naming of the config attribute, event_command_endpoint could work, but does not cleanly align with command_endpoint itself. We could also introduce 2 new attributes, which overrule the command_endpoint:

check_command_endpoint
event_command_endpoint

Considering this clutter in the configuration objects, it is worthwhile to think about a general overhaul of the agent and commend_endpoint components and provide a better solution for this, and the other feature requests.