Flush the LS tracer, since Close() doesn't seem to.

[cory-stripe]

Summary

Flush spans before we close the tracer.

Motivation

Based on my read of the LS tracer code, it doesn't actually flush on close. This means that if you stuff a bunch of spans in really fast and call Close() you're dropping things.

I based this on this code that calls this code none of which calls Flush anywhere.

It's my theory that our recent deploy of Veneur to prod has actually broken tracing becuase we're not flushing before close.

While were here, this adds a metric that tracks the duration of each flush so we can see how much we're impacting things.

Test plan

Validating in QA and testing on a single prod apibox, since this is where the problem was manifested.

Rollout/monitoring/revert plan

Merge, make PR for this SHA in puppet-land and test a single box.

r? @stripe/observability

[stripe-ci]

Gerald Rule: Copy Observability on Veneur and Unilog pull requests

cc @stripe/observability
cc @stripe/observability-stripe

[joshu-stripe]

Looks good to me!

I’m curious why lightstep isn’t flushing these themselves, and may do a little digging to see if I can find out.

[cory-stripe]

re-approving for CL entry and rebase