Bringing back host metadata as observer

[andrewvc]

We removed the add_host_metadata processor from heartbeat for 7.0, thinking that it was an abuse of the host.* fields, since users might expect host.* to correspond with the instance being monitored, not the instance doing the monitoring. Indeed this is what the ECS spec says

ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken.

After discussing this with @webmat and it seems that we can simply put this data under observer.*. To do this we'll need to create an add_observer_metadata processor, which could be implemented as a sort of alias to add_host_metadata with the parent field changed.

[elasticmachine]

Pinging @elastic/uptime

[webmat]

Thanks for opening this, @andrewvc. A few thoughts on this.

ECS currently doesn't allow for nesting of host at observer.host. I'm open to having the discussion to add this. But for compliance with ECS 1.0.0, this processor should at least fill observer.hostname, observer.ip and potentially observer.mac.

This new processor will help solve agent / observer issues we've been having in many situations (Metricbeat monitoring other hosts, Filebeat listening on syslog port, etc). It will then be straightforward for people to apply either add_host_metadata or add_observer_metadata per event source.

cc @ruflin

[ruflin]

+1 on having this processor. For now I would stick to what @webmat proposed above with observer.* having many shared fields with host.*. This is similar to what we have in agent.* too.

If I remember correctly, @elastic/apm-server already populates observer.*? Perhaps one of them can chime in?

[jalvz]

Yes, this is what we have at the moment:

https://github.com/elastic/apm-server/blob/7807b0a8c5dfd9e5e89e4ce8873d1d26443edf48/publish/pub.go#L72

[andrewvc]

PR is up here: #11394