feat!: OpenTelemetry metrics

[Joibel]

Note to reviewers, this is a draft to check CI at the moment.

Led by #12589.

Motivation

As discussed in #12589 OpenTelemetry is the future of observability. This PR changes the underlying codebase to collect metrics using the otel libraries, whilst attempting to retain compatibility with prometheus scraping where this makes sense.

It helps lay the groundwork for adding workflow tracing using otel.

This PR amends and extends the built in metrics to be more useful and correctly named.

This PR does not attempt do do anything else with otel, nor does it attempt to change custom metrics.

Modifications

Mostly removed prometheus libraries, and replaced with otel libraries.

Allows use of both prometheus /metrics scraping and opentelemetry protocol transmission.

See the migration guide for a quick summary of the changes.

See the metrics documentation for more details on new metrics.

Extends the workqueue metrics to include all of them exposed by client-go/workqueue

Allows removal of metrics, removal of attributes (labels) and reconfiguration of histogram buckets for all metrics using options in the workflow-controller-configmap

Allows temporality configuration for opentelemetry counters.

Adds a number of other new metrics:

• build info
• cronworkflow triggers
• k8s request duration
• workflow template runtimes
• workflow template counts

Removed argo_workflows_workflows_processed_count as a non-functioning metric.

originName enhancements proposed in #12589 were not added.

Verification

[agilgur5]

I only briefly went through this, with particular focus on the docs and a bit on the config -- just an initial quick review. There's a good amount of docs changes needed.

As I've mentioned in other docs PRs you've had, please follow the docs style guides. In this quick look through, I particularly noticed many misses on the k8s style guide regarding "Simple and direct language" and our style guide regarding "Prefer 1 sentence per line of markdown".
The latter is explicit, while the former contains examples -- if you're struggling with that, a rough mindset you can use while reading/writing/editing is "can I say this with less words?" (while ofc still following the rest of the style guide, existing conventions, and without making things very confusing). The answer is very often "yes" as this style of writing can be quite different from the way we speak or respond to users etc (you and I in particular use more "level of confidence" or "hedging" words as well as other descriptors). For instance, in the very preceding sentence, can remove "very", "quite", etc to simplify as: "Often this writing style is different from others".

[agilgur5]

hmm, this isn't a breaking change... we typically don't describe additions in the upgrading notes

[Joibel]

It is a breaking change for anyone who doesn't explicitly set secure in their configmap.

[agilgur5]

I was commenting on the "New Metrics" section (that's an addition), not the TLS bit. It's also the longest section, so removing it would make the upgrade notes a lot shorter.

I think conventionally we just add new features to the existing docs (no specific upgrade note for them in the docs, though maybe in the release blog post) and add the "v3.6 and after" modifier where needed.

[Joibel]

New metrics will get consumed into your data collection tool by default, which may be costly. It feels important to warn people that they could be in for an increased bill by upgrading if they don't do something about it. I'd only expect people to read the upgrading guide.

[agilgur5]

Given how large this is, I wonder if we should split this into multiple PRs for readability & reviewability.

For instance:

• Prometheus -> OTel
• New metrics (could potentially be further split if necessary, e.g. workqueue vs own metrics)
• New configurations (could potentially be further split if necessary, e.g. individual metric config vs the rest)

[agilgur5]

I feel like there's a better name for this, but it is evading me right now

[Joibel]

I've changed it to "modifiers" locally, which is the best I have for now.

[agilgur5]

That does sound better, although also feels like it could be better 😅

[Joibel]

Suggestions welcome of course

[Joibel]

Thanks for all of the review comments.

I wasn't really ready for a review yet (as noted by this being in draft and the big comment at the top of the description), but I've tried to accommodate your comments.

Splitting this would be pretty non-fun given everything except for the base commit would want rebasing every time anything in the base commit changes. I'd tried to give fair warning in #12589 that I'd be doing what I've done.

The config changes are pretty small.
The new metrics are nearly all self contained.

[agilgur5]

I wasn't really ready for a review yet (as noted by this being in draft and the big comment at the top of the description), but I've tried to accommodate your comments.

Yes I noticed that of course. I didn't go through the code at all and not even the entirety of docs or config (nor as thorough as I can be), so kept it quick, high-level, and not CI impacting. I figure you might want early feedback if you put up a draft and that early feedback is always better than later feedback. You don't have to respond to the comments immediately, we can just keep a running list of feedback until it's more ready and then can address them -- usually, in my experience, drafts have an expectedly slower review cycle on both ends.

[agilgur5]

Splitting this would be pretty non-fun given everything except for the base commit would want rebasing every time anything in the base commit changes.

PR stacks a la Graphite or Sapling (etc) can help automate that (both are overlays on top of Git; they're not as useful without their respective review tools though, and dedicated review tools are better than GitHub code review 9 times out of 10). I still do it all manually myself though, but it's exactly what PR stacks are designed for.
But the PRs sequential and largely contained, as you wrote, so there wouldn't be too many conflicts and they'd still be reviewed sequentially given that GH does not support PR stacks (so a subsequent diff will contain everything until the prior one is merged).

I'd tried to give fair warning in #12589 that I'd be doing what I've done.

As the main reviewer there, I'm well aware of that. That doesn't change any readability or reviewability concerns.
The config changes in particular are new and expose a new API surface that we'd want to try to keep more stable.
There are also more new metrics than you originally wrote about.

The config changes are pretty small.
The new metrics are nearly all self contained.

At ~5k LoC changed across 64 files -- with almost none of it being generated code either -- anything can help. The new metrics in particular I would think might not need much iteration, but the OTel, config, and docs changes definitely need a careful eye given how fundamental they are as well as to reduce chance of future breakage and make as understandable to the user as possible.

[Joibel]

Replaced by #13265