ui: timeseries: add per-node CPU and memory usage charts #23733
Conversation
|
This change is |
|
Why did you remove the aggregate CPU time? Both are useful but at different times. We either need to keep both or have an option to switch between them. Ideally we'd show the aggregated view by default with some sort of alert to call attention to the per-node view if there is a significant discrepancy. |
Release note (admin ui change): add per-node CPU time graph
Release note (admin ui change): add per-node resident set size graph
|
Removed it because I didn't really see the utility of aggregated CPU (it doesn't help you debug which nodes are overloaded), but just added it back in and updated the screenshots — I guess it's helpful to see how much CPU you're using overall. |
|
Toggling between them or alerting if there's a discrepancy in per-node usage would be nice. Would probably be hard to choose a good discrepancy threshold though. Anecdotally, looking at Jordan's cluster the usage between nodes is pretty different, with a bunch of nodes near zero and some above one: (As an aside, having a bar chart in these tooltips to compare values would be great) |
| <LineGraph | ||
| title="Memory Usage (per-node RSS)" | ||
| sources={nodeSources} | ||
| tooltip={`Per-node Resident Set Size.`} |
There was a problem hiding this comment.
I don't think we've explained what RSS means anywhere else, including the aggregated Memory Usage chart where RSS is just referred to as "Total Memory".
I think (per-node Total Memory) is fine for the title, while the tooltip could be Total Memory used by the CockroachDB process on each node."
| tooltip={`CPU usage per node, in CPU seconds per second. E.g. if a node has four | ||
| cores all at 100%, the value will be 4.`} | ||
| > | ||
| <Axis units={AxisUnits.Count} label="cpu seconds / second"> |
There was a problem hiding this comment.
I'd recommend that we keep the units consistent between the two CPU charts.
| @@ -17,7 +17,62 @@ export default function (props: GraphDashboardProps) { | |||
| </LineGraph>, | |||
|
|
|||
| <LineGraph | |||
| title="Memory Usage" | |||
| title="CPU Time (per-node)" | |||
There was a problem hiding this comment.
Why'd you swap the charts?
| title="Memory Usage" | ||
| title="CPU Time (per-node)" | ||
| sources={nodeSources} | ||
| tooltip={`CPU usage per node, in CPU seconds per second. E.g. if a node has four |
There was a problem hiding this comment.
It seems like we should either sum the user and system time or specifically call out that this is just user time. I think I prefer that we sum them, since system time is nontrivial.
|
Superseded by work under #27085 |
In #23379, @petermattis asked for the ability to switch from aggregated to per-node charts, to see what nodes are overloaded. That would be a fairly general change to our TS code; in the meantime it seems useful to at least be able to see per-node CPU and memory to get a basic sense of what nodes are overloaded.
Before:
CPU time aggregated across the cluster:
Memory usage by type, but not by node:
After: