[UI] Compare Node Metrics with Average

[ChengYanJin]

Component: ui, nodes

Why this is needed:
We want to provide the System Admin a way to compare the select node with the average metric value.

What should be done:
The design:

1. toggle OFF
   

2. toggle ON
   

3. toggle ON with hover popup
   

Implementation proposal (strongly recommended):

• https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators
  Please consider using avg (calculate the average over dimensions) to get the average value of the nodes instead of implementing JS Code.
• Set argument tooltip to true in LineChart in order to enable the tooltip in the charts.
• When toggling on average, the URL should change as well.
• (Additional) Please update the color of the lines with the new design(Hardcode the color for now)

Test plan:
TBD

[thomasdanan]

Actually, I am thinking about 3 distinct use cases where this feature could help:

1. Being able to understand how a particular node is behaving compared to the rest of the nodes
2. Compare one node to specific other nodes
3. Quickly identify some nodes having strange behavior

[gdemonet]

Actually, I am thinking about 3 distinct use cases where this feature could help:

1. Being able to understand how a particular node is behaving compared to the rest of the nodes

Could be useful to compare with the average (and maybe percentiles), not just any node(s) / all of them.

2. Compare one node to specific other nodes

3. Quickly identify some nodes having strange behavior

Again, comparing with aggregate values could be both easier for a user to read, and for a developer to handle.

[thomasdanan]

Actually, I am thinking about 3 distinct use cases where this feature could help:

1. Being able to understand how a particular node is behaving compared to the rest of the nodes

Could be useful to compare with the average (and maybe percentiles), not just any node(s) / all of them.

2. Compare one node to specific other nodes

3. Quickly identify some nodes having strange behavior

Again, comparing with aggregate values could be both easier for a user to read, and for a developer to handle.

Yes, I think it is a very good idea @gdemonet (especially for use case 1).
For use case 3, I think the feature we are talking about will not help, because one would have to go through all the nodes, one by one to identify the one having some strange behavior. For this specific use, I think we are more looking for a Top10 kind of representation

[ChengYanJin]

Very good point!! @gdemonet
Maybe we should always display the average value in the graph to give the users an idea of how this node behaviors than others.

So I guess we should limit the number of nodes/volumes the user can select @thomasdanan?

[Cuervino]

This "use case 3" is specific > the user doesn't have to start from a specific node, but want to identify the outliers from the full total.
In this case, a top5/top10 could work.

As for the moment, the logic is still to start from a selected node, what we can do is to have 4 ways of displaying the values

1. -Raw (only the selected node)
2. -Versus average values
3. -Comparison / multiselection
4. -Versus top5 & bottom5

I don't know how to define the top 5 / bottom 5 > for example, to take the nodes with the current "top latency" values, or to take the one with the "top latency" values over the period?

Anyway, here could be the next iteration.

[gdemonet]

Thanks @Cuervino for the fast proposal 👌

I agree that the four proposed scenarios are likely covering most, if not all, our use cases.
However, I think having the user choose between any of them is an extra step which may not be required. Here's my suggestion:

• keep the toggle from the first design (either looking at a single node, or comparing it with others)
• in the list of nodes to select for comparison, add a "fake" node for averages
• top 3 / bottom 3 can be achieved if we have sorting in the list (not for now, and not on the metrics, so I'm not sure that's enough), or using other "fake" nodes

[ChengYanJin]

Yes. I would agree with @gdemonet, with so many choices this metrics tab will become extremely crowded. Maybe we should build the top 3/5/10 in the Grafana dashboard or overview page.

At the same time, I am not sure to have a "fake" node appearing in the list. Where this node should locate when we add sorting or filtering?

I am thinking more to have an average/base line displayed in the graph when we active the multi-node comparison mode.
Just to give users a general overview. For more details, they should go to the Grafana node detailed dashboard which is accessible by clicking on "Advanced metrics".

[ChengYanJin]

Before starting the implementation, I would like to summary a few details, so we can finalize the design.

1. The URL when the multi-node comparsion mode is enabled:

/nodes/<node-name>/metrics?multi=true&selected=<node1>,<node2>,<node3>...

2. The query I should use to get the average metrics of all the nodes?

"avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"system\"}[1m])) by (instance)",

Let's take CPU Usage as an example:

For a specific node:

(((count(count(node_cpu_seconds_total{instance=~"${instanceIP}:${PORT_NODE_EXPORTER}"}) by (cpu))) - avg(sum by (mode)(irate(node_cpu_seconds_total{mode='idle',instance=~"${instanceIP}:${PORT_NODE_EXPORTER}"}[5m])))) * 100) / count(count(node_cpu_seconds_total{instance=~"${instanceIP}:${PORT_NODE_EXPORTER}"}) by (cpu))

Cluster-wide query from Grafana:

CPU Usage

sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster=""}) by (cluster)

3. Need to figure out how to dynamically add the line(s) in the chart

4. Need to have at least 6 colors (including variant color) ready for the line chart.

[JBWatenbergScality]

I'm not sure how pertinent would it be to enable node to node comparison. Also the purposed screen design attached let me feel that all the tabs should contain aggregated values corresponding to the selected nodes in the table, which is not what we want to represent as the selection should only impact the metrics tab.

The use cases that I would see is comparing each single node with the average of others as Guillaume purposed to enable detection of nodes deviating from classical metrics level.

Additionally we might want to offer the possibility to compare average of metrics by groups of nodes based on their labels (for eg in multisite deployment or segregated network zones we will probably have nodes labelled with those availability information) so it might be interesting to see how the charge is spreaded across those different areas.

[thomasdanan]

@ChengYanJin @gdemonet @Cuervino @JBWatenbergScality
I suggest that we start with a first iteration where one can compare the current node metric with the average of all other nodes, without implementing any kind of node selection to compare with in a first step.

[Cuervino]

Here is a new design for comparing the Node against "Cluster Avg"

1. toggle OFF
   

2. toggle ON
   

3. toggle ON with hover popup
   

[gdemonet]

@Cuervino TBH, I find the dotted lines to be almost invisible (maybe I'm blind 😅). Is there any way we can increase contrast so it's easier to spot?

[ChengYanJin]

@gdemonet @Cuervino
We can increase the opacity of the dotted lines.
Or we can choose more visiable stroke dash on vega-lite(the chart library) side. Example