Running transform before rollups (eg how to do abs_sum_over_time)

[lihmc]

Is your question request related to a specific component?

metricsql

Describe the question in detail

Hi, I have a metric that has both positive and negative values. Say it is Metric_shares_traded (negative for sells). I would like to retrieve a rolling sum of the absolute value. However, when I wrap the metric I query inside abs before doing sum_over_time, I drop results as now the Metric_shares_traded is auto-wapped by default_rollup and values are dropped.

eg sum_over_time(abs(Metric_shares_traded)) will drop randomly, especially over longer query time ranges because default_rollup does a last rollup.

Similarly, trying to do sum_over_time(Metric_shares_traded > 0) - sum_over_time(Metric_shares_traded < 0) gives the same issue because Metric_shares_traded is wrapped by default_rollup before the comparison.

I think this is similar to #2020

Is there a workaround to getting what I want to work?

Troubleshooting docs

• General - https://docs.victoriametrics.com/Troubleshooting.html
• vmagent - https://docs.victoriametrics.com/vmagent.html#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert.html#troubleshooting

[valyala]

This functionality isn't supported by MetricsQL at the moment, but I think we can add sum_gt_over_time(m[d], threshold) and sum_le_over_time(m[d], threshold) functions, which would calculate the sum of raw samples on the given lookbehind window d if they are bigger (for sum_gt_over_time) or smaller or equal (for sum_le_over_time) to the given threshold. MetricsQL already provides similar functions - count_gt_over_time and count_le_over_time,

[valyala]

FYI, the commit d6e22f2 adds support for sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions. This commit will be included in the next release. In the mean time it is possible to test these functions by building VictoriaMetrics from this commit according to these docs. If you run cluster version of VictoriaMetrics, then you can build cluster components from the commit 5f2905d according to these instructions.

[valyala]

The sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions are available in VictoriaMetrics starting from v1.98.0.

Now the sum_over_time(abs(Metric_shares_traded)) query can be properly processed when re-written to the following one:

sum_gt_over_time(Metric_shares_traded[duration], 0)

Closing the feature request as done.