[Bug] [regression] grafana2 -> 3 influx 0.9 -> 0.11 templating doesn't show all the values

[mb-m]

• What grafana version are you using? 2.6.0 -> 3.0.0-beta2
• What datasource are you using? influxdb 0.9 -> 0.11.1

We are using grafana for dashboarding an hbase and hdfs database, and using templating to make sure we can differentiate between the various instances of the hbase database.

As a distributed system, it's useful to see metrics for the cluster as a whole (rather than individual servers within it), so we template with:

SHOW TAG VALUES WITH KEY = cluster

In influx 0.9 in the CLI (and similarly in the JSON), this results in:

> SHOW TAG VALUES WITH KEY = cluster
name: clusterTagValues
----------------------
cluster
hadoop-dr
hadoop-live
hbase-dr
hbase-live
live
live-hadoop
live-hbase
live_resources0
live_zk

However in Influx 0.11.1 in the CLI (and again similarly in the JSON), this results in a set split by table:

> SHOW TAG VALUES WITH KEY = cluster
name: =regionserver
-------------------
key value
cluster stage_resources0


name: _stats
------------
key value
cluster stage_resources0
[...]
name: net
---------
key value
cluster stage_mentions0
cluster stage_resources0
cluster zk_test
[...]

Grafana's behaviour in this situation is just to take the tags from the first tabular result returned by influx, where, without a specific measurement (table) and because there is no one measurement that covers all the results, in order to see other clusters, I resort to typing them in. Obviously the behaviour I suggest here and that I'd expect (principle of least surprise etc) is the union of these results across all the tables, generating something closer to the list that we had on our old installation.

I did have a look at the code, but trying to dip in when I'm very unclear about how it's structured turned out to not be as trivial as I'd hoped.

AFAICT it's either possible to fix it: https://github.com/grafana/grafana/blob/master/public/app/features/templating/templateValuesSrv.js#L228 or presumably better: https://github.com/grafana/grafana/blob/master/public/app/plugins/datasource/influxdb/datasource.ts#L120

I may try and have a go at fixing it on Monday, as it's a work thing, but if this is a trivial-ish fix, then your (probably better) fix would be much appreciated by me. :-)

[torkelo]

This is not a bug / regression, but a change in InfluxDB. Not yet sure what the behaviour should be now that InfluxDB returns the resulting tag value for each measurement.

[mb-m]

It's a regression in the display of dashboards. It's a bug in the handling of what you say should work in your influxdb connector.

Closing it like this makes it clear that we shouldn't be recommending use of grafana in future.

Disappointed :-(

[torkelo]

@mb-m I am sorry, it's just a big change in InfluxDB has changed how this feature works in Grafana.

Before SHOW TAG VALUES WITH KEY = cluster returned all tag values, now an InfluxDB change has a different format and groups the values by measurement so now you need to include a measurement filter as well to get the same result. This is not a change that was Introduced in Grafana.

But I agree that the behavior here should be different, @bergquist why does grafana only use tag values from the first measurement, was this done deliberately? Should not all measurement's tags be returned? (deduped first). Do you know if we still have an open issue for that?

[mb-m]

@torkelo yes, I appreciate this - and thank you for your reply - I've even explained the difference above. I can't use a measurement filter for reasons I've explained, because it doesn't make sense in my situation - we're getting cluster measurements from several different places, and I'd just like to draw blank graphs if the relevant cluster doesn't have data to deal with that graph. (to give you a bit of context, my large HBase dashboard currently contains 49 graphs - and the JSON is auto-genned to make this work).

The reality, though, is that upstream APIs do change, and downstream products have to keep up (this is one of the reasons why I've generally been frustrated with products that claim "It has an API!!!" (meaning it has a kind of vaguely-defined web interface that generates XML or JSON)).

The thing is, while I appreciate it's not something introduced in grafana, grafana claims to support InfluxDB as part of its core connectors. If this is not true, then this is not something you get to say about your product. As to solving this problem, I was wrong about the locations, and it should be handled around: https://github.com/grafana/grafana/blob/master/public/app/plugins/datasource/influxdb/response_parser.ts#L15, in which I note there is explicit parsing for the responses to 'show tag values'...

The reason it just uses the first one is the assumption in the line I've referenced, as far as I can tell.

[torkelo]

@mb-m Yes, I agree, the more I see this issue the more I think @bergquist missed to look at all measurements when upgrading Grafana to handle the new response format.

[mb-m]

Thank you, sir!

[mb-m]

Interestingly, also, I think this response format was possible in the previous version but never used as, AFAICT, it appears to be the same array depth.

[bergquist]

@mb-m sorry for this and thank you for your excellent bug report! I will fix this asap.

[mb-m]

No need for apologies - I think the right thing happened in the end :-) Thank you all!

[mb-m]

Just want to say thank you both (@torkelo, @bergquist) here - we've just installed the nightly and it now works perfectly for us!