Panels: Numeric filters throwing panel errors

[gtk-grafana]

When adding numeric filters: sometimes I see a bunch of errors:

https://ops.grafana-ops.net/a/grafana-lokiexplore-app/explore/service/grafana-grafana/labels?patterns=%5B%5D&from=2024-12-10T20:04:53.975Z&to=2024-12-10T20:19:53.975Z&var-ds=OP27Xzxnk&var-filters=service_name%7C%3D%7Cgrafana%2Fgrafana&var-fields=duration%7C%3E%7C%7B%22value%22:%227s%22__gfc__%22parser%22:%22logfmt%22%7D,7s&var-levels=&var-metadata=&var-patterns=&var-lineFilter=&urlColumns=%5B%5D&visualizationType=%22table%22&displayedFields=%5B%5D&var-fieldBy=$__all&var-labelBy=$__all

Has queries like:
https://ops.grafana-ops.net/explore?schemaVersion=1&panes=%7B%229kl%22%3A%7B%22datasource%22%3A%22OP27Xzxnk%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22logs%22%2C%22expr%22%3A%22sum%28count_over_time%28%7Bservice_name%3D%60grafana%2Fgrafana%60%7D+%7C+detected_level+%21%3D+%5C%22%5C%22+%7C+logfmt+%7C+duration%3E7s+%5B%24__auto%5D%29%29+by+%28detected_level%29%22%2C%22datasource%22%3A%7B%22type%22%3A%22loki%22%2C%22uid%22%3A%22OP27Xzxnk%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%221733861093975%22%2C%22to%22%3A%221733861993975%22%7D%2C%22panelsState%22%3A%7B%22logs%22%3A%7B%22visualisationType%22%3A%22table%22%7D%7D%7D%7D&orgId=1

Query string:

sum(count_over_time({service_name=`grafana/grafana`} | detected_level != "" | logfmt | duration>7s [$__auto])) by (detected_level)

Error:

pipeline error: 'LabelFilterErr' for series: 
'{__error__="LabelFilterErr", __error_details__="time: missing unit in duration \"0.336667575\"", app="grafana", cluster="dev-eu-west-2", [...REDACTED]. 
Use a label filter to intentionally skip this error. (e.g | __error__!="LabelFilterErr"). 
To skip all potential errors you can match empty errors.(e.g __error__="") 
The label filter can also be specified after unwrap. (e.g | unwrap latency | __error__="" )

Should we always be ending in __error__!="LabelFilterErr"?

[gtk-grafana]

Chatted with Trevor a bit about this:

The problem here is that there is a "duration" field that is missing a unit, so our data is bad.
__error__ != "LabelFilterError" will fix, without hiding other errors, but it will remove any lines that have the bad "duration" field that isn't a duration.

Easy/bad: We can either always add __error__ != "LabelFilterError" to everything, which will hide logs with bad data

Hard/good: We can conditionally add __error__ != "LabelFilterError" whenever we use a duration/bytes/float filter, which will hide logs with bad data (this would also require saving the field type as part of the ad-hoc variable value, like we currently do for the parser, which would take a bit of work, and is hacky).

medium/medium: Or we can intercept this error, show a modal that tells users that field X conflicts with field Y, and present a button which will inject the __error__ != "LabelFilterError" into an otherwise unused variable.
This enables us to educate the user about the conflict, while still allowing them to debug. Users would need to do this for each session until the data is fixed (unless they enter from a url that contains the hidden variable containing the error filter)