Telemetry

[findkim]

This PR adds OpenTelemetry as the metrics engine and hashicorp/hcl-opentelemetry boilerplate library for loading configuration and initializing sinks. These changes aim to report the same metrics proposed in #67

Metrics

Metric Name	Description
consul-esm.coord.txn	A counter of node check updates using Consul txn API
consul-esm.check.txn	A counter of check updates using Consul txn API
consul-esm.checks.update.ms	The duration (milliseconds) to update checks

Example output

Stdout

{"time":"2020-06-26T16:19:40.233463-05:00","updates":[{"name":"consul-esm.checks.update.ms","min":0,"max":0,"sum":0,"count":1,"quantiles":[{"q":0.5,"v":0},{"q":0.9,"v":0},{"q":0.99,"v":0}]}]}

Dogstatsd

2020-06-26 16:30:30.435691 consul-esm.checks.update.ms:0|h
2020-06-26 16:30:45.437495 consul-esm.coord.txn:1|c

Prometheus

$ curl localhost:8888/metrics
# HELP consul_esm_check_txn A counter of node check updates using Consul txn API
# TYPE consul_esm_check_txn counter
consul_esm_check_txn 1
# HELP consul_esm_checks_update_ms The duration (milliseconds) to update checks
# TYPE consul_esm_checks_update_ms histogram
consul_esm_checks_update_ms_bucket{le="+Inf"} 2
consul_esm_checks_update_ms_sum 0
consul_esm_checks_update_ms_count 2

[findkim]

A lot of work has been put into experimenting with a new telemetry library, but sitting on this PR for a while I am not happy with how timers are currently reported. The measurement type is denoted by manually added suffix to the metric name, and puts the responsibility of type conversion on the user rather than systematically by the metric platform.

I'm going to close this in preference for the work @mikeyyuen and @lornasong have prepared in #67