Time based and interval queries for integration with grafana

[olizilla]

I've been playing with the graphql api in https://observablehq.com/d/4e09dcb851b5ed05 – it's a really helpful tool for developing an understanding of the filecoin data model, and for creating useful visualisation of how the state changes over time...

We already have the Heights query which allows us to get a contiguous range of epochs... but for longer range queries, you can end up asking for with waaay more data than you need. Something like an interval parameter would be nice:

Heights(from: Int, to: Int, interval: Int) - get epochs between from amd to every interval apart.

Going further.... heights are a proxy for time that is specific to Filecoin. What if we supported the more universal concept of TIME! This would let us integrate statediff directly with grafana, via https://grafana.com/grafana/plugins/fifemon-graphql-datasource?src=grafana_add_ds

e.g.

TimeRange(fromUnixEpoch: Int!, toUnixEpoch: Int!): [LotusBlockHeader!]!

query {
  TimeRange(fromUnixEpoch:"$__from", toUnixEpoch:"$__to"){
    Time # Int! calculated unixEpoch in seconds
    Height 
    ParentStateRoot {
      Actors{
        At(key:"f099") {
          Balance
        }
      }
    }
  }
}

basically, can we have the grapql equivelant of these SQL functions: https://github.com/filecoin-project/sentinel-visor/blob/5bef7da1e9a6aead50d6585fc3e08ba582d7c354/storage/migrations/21_height_functions.go#L11-L17

Of note... Once we point grafana at something, it tends to get a lot of queries, and humans love to press the "show me all the date for last year" button... which prompted the suggestion about the interval parameter.

[willscott]

would grafana be good at automatically setting the interval parameter? (or would it be preferable to have an argument instead like "samples" so you can just say "i want 30 sample spread over this time range"?)

[olizilla]

grafana has some chops here

Interval - The interval is a time span that you can use when aggregating or grouping data points by time.

Grafana automatically calculates an appropriate interval and it can be used as a variable in templated queries. The variable is either in seconds: $__interval or in milliseconds: $__interval_ms. It is typically used in aggregation functions like sum or average. For example, a Prometheus query using the interval variable: rate(http_requests_total[$__interval]).

This automatic interval is calculated based on the width of the graph. If the user zooms out a lot then the interval becomes greater, resulting in a more coarse grained aggregation whereas if the user zooms in then the interval decreases resulting in a more fine grained aggregation.

AND/OR

Max data points - If the data source supports it, sets the maximum numbers of data points for each series returned. If the query returns more data points than the max data points setting, then the data source consolidates them (reduces the number of points returned by aggregating them together by average or max or other function).

There are two main reasons for limiting the number of points, performance and smoothing the line. The default value is the width (or number of pixels) of the graph as there is no point in having more data points than the graph panel can display.

– https://grafana.com/docs/grafana/latest/panels/queries/

[willscott]

do you need the derived Time # Int! calculated unixEpoch in seconds that you propose in the block header?

[olizilla]

Yes; adding a timestamp/unix epoch per data point will allow grafana and other time-series assuming tools to graph it. Calling it Time means it will work automatically with the grafana-graphql plugin:

Timeseries data requires a field named Time containing the timestamp in a format that can be parsed by moment() (e.g. ISO8601).
– https://grafana.com/grafana/plugins/fifemon-graphql-datasource?src=grafana_add_ds

[olizilla]

I realise now that:

1. The Time property in the response could be called anything, as the query can alias it to Time as required
2. The readme for the graphql datasource for grafana suggests the Time field should be:

in a format that can be parsed by moment() (e.g. ISO8601)

and digging around it looks like a unix epoch would eventually get passed to moment.utc() which would return an incorrect datetime rather than erroring.

https://github.com/grafana/grafana/blob/61bd33c24187184b058571c42f0e5f5129d22320/packages/grafana-data/src/datetime/parser.ts#L71-L76

$ node      
> var moment = require('moment')
undefined
> moment.utc(1606304490)
Moment<1970-01-19T14:11:44Z>
> moment.unix(1606304490)
Moment<2020-11-25T11:41:30+00:00>

so, for this experiement to work, I think we need Time to be an ISO8601 Timestamp String

[olizilla]

For consistency, the from and to parameters could be ISO8601 timestamps as well, as grafana can be coerced into providing them in either timestamp or unix epoch flavour: https://grafana.com/docs/grafana/latest/variables/variable-types/global-variables/#__from-and-__to

[olizilla]

Noting, as I stumbled across this on my explorations: the graphql plugin wants to migrate to a backend, go-based plugin: fifemon/graphql-datasource#34 - it is currently a ui side typescript one.