"Label filters" in Variables Settings not working at all

[Aracki]

What happened?

Whatever I put in "Label filters" the "Preview of values" is showing the same list.

What did you expect to happen?

I'd expect filter to be applied and to show only the part of the list.

Did this work before?

Yes!

I'm cocmparing Grafana v10.1.1 (0cfa76b22d) and v9.3.6 (978237e7cb)

In version 9, we had old Query Options query field (no query builder like in v10), and filters were working all the time.

How do we reproduce it?

Try to add any filter by any label and it won't work.

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana v10.1.1 (0cfa76b)
OS: MacOS
Browser: Chrome

Grafana platform?

Kubernetes

Datasource(s)?

Thanos

[usmangt]

Hi @Aracki,

Thanks for opening this issue.

Is this problem related to a specific Data source? or can be reproducible with any other Data source than Thanos?

If yes, then please describe the step-by-step instructions in more detail so that we can also able to reproduce it.

[Aracki]

Only Thanos/Prometheus datasource.

[usmangt]

Can you please share a screenshot so that it is easier to understand. Thanks!!

[Aracki]

Thanos/Prometheus

Here is the simple one. Whatever the $network is, the same variable comes up from the dropdown menu.

[natellium]

Thanks for your contribution :) As you mention this is isolated to the prometheus datasource - @grafana/observability-metrics would this be something you could look into? Thanks!

[gfarcas]

We have the same problem also, grafana 10.2.0, Thanos datasource

[gtk-grafana]

@bohandley can you take a look at this when you've got some time?

[gfarcas]

This seems to be ok according to https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names

match[]: kube_pod_container_resource_requests{namespace="kyverno"}
start: 1699320457
end: 1699342057

[gfarcas]

I have tested in thanos directly and it seems grafana respects the api specification but for some reason thanos ignores it

[gfarcas]

thanos-io/thanos#6338

[gfarcas]

seems to be fixed in thanos-io/thanos#6816

[bohandley]

Updating this as waiting for release to be fixed in Thanos. @gfarcas @Aracki can we close this issue?

[Aracki]

I've updated all Thanos components to latest v0.32.5, but still experiencing the issue unfortunately.

These two queries by Grafana returns the same result:

/api/datasources/uid/<ID>/resources/api/v1/label/cluster_name/values?match%5B%5D=eth_block_number%7Bjob%3D%22node-fantom%22%7D&start=1699430054&end=1699433654

/api/datasources/uid/<ID>/resources/api/v1/label/cluster_name/values?match%5B%5D=eth_block_number&start=1699430182&end=1699433782

although it should return less entries when job=node-fantom.

Update: Just checked; all the labels can be returned, but not when the external label is selected

[gfarcas]

We have also tested with latest and it didn't work

[bohandley]

Are you both seeing that Grafana is sending the label filters in the request to Thanos but Thanos is not respecting that?

Is it the same when you curl directly to Thanos with that endpoint with label filters?

[bohandley]

I am closing this as it is not a Grafana Prometheus data source issue but an issue with Thanos and external labels being returned in the resource.