Unified Pipeline/output metrics

[urso]

This PR adds metrics support to the publisher pipeline + unifies the metrics used by outputs.

The metrics are registered dynamically and can be removed later on (e.g. pipeline already removes metrics after on close).

Metrics support is somewhat standardised and decoupled from outputs/publisher pipeline, by having some kind of event listener/observer/... object defining a set of common events. On every event from the outputs, the metrics (potentially multiple metrics) are updated accordingly.

The original per output type metrics have been removed (no more output.elasticsearch... and so on), in favor of a standardized set of metrics.

The observer is passed to the outputs and publisher pipeline. This is used to collect metrics for different pipeline instances (xpack and libbeat namespace).

pipeline metrics:

pipeline.clients: number of beat.Client instances (internal connections to pipeline)
pipeline.events.total: total number of events processed by a client
pipeline.events.filtered: total number of events removed by processors
pipeline.events.published: total number of events pushed to the queue/broker
pipeline.events.failed: total number of events failed to be pushed to queue (e.g. disconnect)
pipeline.events.dropped: total number of events dropped
pipeline.events.retry: total number of events retried
pipeline.queue.acked: total number of events ACKed by the event queue/buffer
pipeline.events.active: (gauge) number of active events in pipeline

output metrics:

output.type: configured output type (logstash, elasticsearch, ...)
output.events.batches: total number of batches processed by output
output.events.total: total number of events processed by output
output.events.acked: total number of events ACKed by output
output.events.failed: total number of events failed in output
output.events.active: (gauge) events sent and waiting for ACK/fail from output
output.write.bytes: total amount of bytes written by output
output.write.errors: total number of I/O errors on write
output.read.bytes: total amount of bytes read
output.read.errors: total number of I/O errors while waiting for response on output

[ruflin]

This is a great change. It heavily simplifies the internal and external handling of the metrics. Makes it very easy to filter by one output type etc.

Makes we wonder, if there are other places where we should do the same ;-)

[ruflin]

tricky abbreviation. only know what it means because of the diff

[urso]

oh... to me CAS on atomics is pretty well known/common :)

[ruflin]

@urso The wait_shutdown_ok test seems to fail, which worries me a bit. But not related to this PR I think.

[urso]

@ruflin checked the test. The test is completely unrelated to this PR. Although the test is failing (cause the shutdown timer times out), I verified the state in the registry still matches expectations.