Handlers: Add metrics

[lukedirtwalker]

A handler that is registered at the messenger will now have metrics by default.
There are three metrics:
in_calls_total: counts the total received requests.
in_results_total: counts the results of the received requests.
in_calls_latency: a histogram of the latency of the requests.
Analogous to in_* the current messenger metrics (for outgoing requests)
have been renamed to out_*.

Also fixes the registration of messenger metrics (init() doesn't work because
we don't want to register on the default Prometheus registerer).

Note that not all handlers always return the correct result/status,
that will be done in separate commits.

Contributes #2329, #2330, #1819

---

This change is 

[sustrik]

Reviewable status: 0 of 6 files reviewed, 1 unresolved discussion (waiting on @kormat, @lukedirtwalker, and @sustrik)

---

go/lib/infra/modules/trust/metrics.go, line 37 at r1 (raw file):

)

func initMetrics() {

I would strongly prefer to have these metrics reported by the messanger itself (i.e. in go/lib/infra/messenger/metrics.go) So instead of current calls_total we would have incoming_calls_total and outgoing_calls_total etc. That would mean having a uniform metrics for all RPC calls, irrespective of which service does the RPC.

As for labelSrc we can log that as a separate metric.

[lukedirtwalker]

Reviewable status: 0 of 16 files reviewed, 1 unresolved discussion (waiting on @kormat and @sustrik)

---

go/lib/infra/modules/trust/metrics.go, line 37 at r1 (raw file):

Previously, sustrik (Martin Sustrik) wrote…

I would strongly prefer to have these metrics reported by the messanger itself (i.e. in go/lib/infra/messenger/metrics.go) So instead of current calls_total we would have incoming_calls_total and outgoing_calls_total etc. That would mean having a uniform metrics for all RPC calls, irrespective of which service does the RPC.

As for labelSrc we can log that as a separate metric.

Done.

[scrye]

Reviewed 1 of 6 files at r1, 14 of 15 files at r2.
Reviewable status: 14 of 15 files reviewed, 2 unresolved discussions (waiting on @kormat, @lukedirtwalker, and @sustrik)

---

go/lib/infra/metrics.go, line 30 at r2 (raw file):

// HandlerResult contains a result label and a status label.
type HandlerResult struct {

Please add godocs for the fields. It's not entirely clear when writing a handler what each field should contain, and why they are different.

[scrye]

Reviewed 1 of 15 files at r2.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @kormat, @lukedirtwalker, and @sustrik)

[lukedirtwalker]

Reviewable status: 14 of 15 files reviewed, 2 unresolved discussions (waiting on @kormat, @scrye, and @sustrik)

---

go/lib/infra/metrics.go, line 30 at r2 (raw file):

Previously, scrye (Sergiu Costea) wrote…

Please add godocs for the fields. It's not entirely clear when writing a handler what each field should contain, and why they are different.

Done.

[sustrik]

Reviewed 1 of 1 files at r3.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @kormat and @scrye)

[sustrik]

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @kormat and @scrye)

[scrye]

Reviewed 1 of 1 files at r3.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @kormat)