roachprod: add prometheus/grafana monitoring

[msbutler]

Previously, only roachtests could spin up prom/grafana servers that lasted the
lifetime of the roachtest. This PR introduces new roachprod cmds that allow
a roachprod user to easily spin up/down their own prom/grafana instances. The PR
also hooks up roachtests that rely on prom/grafana into this new infrastructure.

Release note: none

[cockroach-teamcity]

This change is 

[irfansharif]

This is pretty slick! I have one medium-sized suggestion and a few minor ones.

Medium: We ought to avoid de-duplicating prometheus.go as we've done so here. Looking at diff --unified pkg/cmd/roachtest/prometheus/prometheus.go pkg/roachprod/prometheus/prometheus.go, the differences look very minor, so I'd recommend making whatever changes are needed here to work for both use cases (roachtests, roachprod). It would make it (a) easier to review and (b) reduce likelihood of introducing minor differences in the two versions, reducing maintenance overhead.

Minor suggestions are in the comments below. When reviewing this PR and trying it out, I found it helpful to type out some changes myself (reflecting the comments below so perhaps this is helpful to you too): https://gist.github.com/irfansharif/c4d6c2e5873fcbeb40cdcba535bfe123.

[tbg]

@msbutler meta question, are you planning on pushing this over the finish line? There is a bit of dependency on this PR now, with #81516 and #80724, so this shouldn't linger for too long. As per our comments above, roachtest should be able to do what #81516 does even once roachprod learns its new tricks, so I would be inclined to merge that first (or rather, split out the prometheus-roachtest part into a PR and merge that) and then refactor as necessary to let roachprod do its thing (i.e. rebase this PR on top and take it from there). Is that okay for you?

[msbutler]

@tbg happy to push this over the finish line, hopefully by the end of the week (barring a crazy L2 rotation this week). Just to clarify the order of operations:

1. Toby will open a PR that moves roachtest/prometheus to roachprod/prometheus, incorporating his changes in roachtest: add admission/follower-overload #81516
2. I will rebase on that PR and address all feedback above, including teaching all roachtests that use roachtest/prometheus semantics to use my new roachprod/prometheus semantics.
3. Merge this PR

[msbutler]

Thanks for all the feedback! Before rebasing on #83148 and refactoring the roachtests, I've tried to address the roachprod internals and UX feedback. Could y'all take a look at my diff and give the following commands a spin before I rebase? prom-start,grafana-url, and prom-stop

[irfansharif]

I think we're pretty close. I imagine we're planning on rebasing on top of #83148 now that it merged? Looking forward to seeing prometheus.go deduplicated from this PR in this current form.

[msbutler]

tpcc is now hooked into the the new roachprod package! Currently, no preconfigured dashboard is passed to tpcc.

Also, note that I simplified prometheus config: a scrapdeNode now represents a single node. Take a look at how that changed all the With...() helper functions.

[tbg]

Going to leave this review to Irfan as one cook in the kitchen is enough. Looking forward to trying this out myself when it has landed!

[irfansharif]

Thanks for working on this Michael, I look forward to not having to look at admin UI metrics ever again.

[otan]

hope you don't mind my random chime but this is niccceee :D

[irfansharif]

LGTM, do squash the commits and maybe add a bit of text to the commit message + PR body itself for future readers.

[msbutler]

bors r=irfansharif

[craig]

Build succeeded:

• GitHub CI (Cockroach)