improve ingest lock

[trinity-1686a]

• add metric for how long a lock was held
• when a log is held for too long, log which purpose it had
• when waiting for too long, log why we wanted to acquire the lock
• await inside the macro to force awaiting at the right time (currently, we sometime await outside of the macro, meaning we record an instant access to the lock, and then actually proceed to acquire the lock). I'm not a fan of awaiting inside a macro as it's hidden control flow, but i consider it worth the easy-to-miss mistake

[github-actions]

On SSD:

Average search latency is 1.0x that of the reference (lower is better).
Ref run id: 3649, ref commit: 26d983d
Link

On GCS:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 3650, ref commit: 26d983d
Link

[fulmicoton]

one of the sad things with relying on dynamic labels like this, is that you end up doing one hashmap lookup very time you need to access the counter (here once per lock).

It is probably not catastrophic in this case, but here is the alternative.

have a histogram: directly on the TimedMutexGuard

histogram: &'static Histogram,

You can then cache the Histogram counter in different ways. In ingest v2 you can see the pattern done here. The value are stored in the metrics.rs directly. https://github.com/quickwit-oss/quickwit/blob/main/quickwit/quickwit-ingest/src/ingest_v2/metrics.rs#L59-L74.

You could also cache it closer to the usage site if you prefer.

I kind of like to keep the MetricsVec definition in metrics.rs, because it really helps to maintain some consistency in the naming/help etc. and it makes it easier to search for the relevant metrics name...

But it is less critical for metrics labels...

[trinity-1686a]

i like this pattern of statically defining things, however it's a kind of a pain, and a large typo hazard to implement. Having some sort of macro to derive "Build a static (histogram|gauge|counter) vec from a dynamic vec" would be nice, and would likely make us use this more