Prometheus metric for memory/time used per task prefix

[mrocklin]

We have some metrics around number of tasks completed over time. I think that we could use a couple of others:

1. Amount of time spent in each prefix. I think that we already store this data in all_durations (I brought this up when @ntabris did his work before as probably what we actually wanted, rather than number of tasks completed).
2. Amount of memory currently used by each prefix. This is probably not a counter, but instead an instantaneous measurement of current state

This would be useful for after-the-fact debugging of clusters. @dchudz was curious about this. @fjetter is this easy for someone on your team to do?

[mrocklin]

Currently queued/processing per prefix might also be interesting.

[ntabris]

For (1) if we can get that as a cumulative total, that would be great.

For the memory gauge, what's the time resolution at which that's useful data?

[mrocklin]

For (1) if we can get that as a cumulative total, that would be great.

I think that this is already stored in all_durations on the TaskGroup/Prefix objects

For the memory gauge, what's the time resolution at which that's useful data

I think instantaneous is fine. For example when looking at a cluster that appears to be globally paused (what we saw earlier) I'm curious, which task prefixes are responsible for all of this data? Which task prefixes have yet to run?

[dchudz]

@fjetter any sense of where this fits into your next few days/weeks/months?

[crusaderky]

Do we already have number of completed tasks per prefix? We need it in order to calculate mean task runtime grouped by prefix

[mrocklin]

Do we already have number of completed tasks per prefix?

Yes

We need it in order to calculate mean task runtime grouped by prefix

We actually already record an exponentially weighted moving average of task runtime by prefix. This was actually the very first motivation for task prefixes. It's pretty core to the scheduling heuristics. This functionality actually pre-dates the TaskPrefix class itself 🙂

[crusaderky]

@mrocklin is your answer specific to Prometheus, or are you talking about internal metrics in the scheduler?

[mrocklin]

Internal metrics on the scheduler.

[crusaderky]

We need to export the count of completed tasks per prefix to Prometheus, so that grafana/whatever can calculate mean runtimes.