panic: exemplar labels have 85 runes, exceeding the limit of 64

[liguozhong]

Describe the bug
A clear and concise description of what the bug is.

/Users/fuling/go/pkg/mod/github.com/liguozhong/[email protected]/pkg/storage/chunk/cache/instrumented.go:95 +0x1af

here

https://github.com/prometheus/client_golang/blob/main/prometheus/value.go

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (SHA or version)
2. Started Promtail (SHA or version) to tail '...'
3. Query: {} term

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
• Deployment tool: [e.g., helm, jsonnet]

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

[bboreham]

Should be addressed by weaveworks/common#247.
Also note prometheus/client_golang#1091

[John-Lin]

Same issue here when we upgrade to distributed Loki v2.6.1. distributors have same error message.
We disable the tracing feature for workaround.

[liguozhong]

this pr has fix this issue.
#6859

update dependency weaveworks/common#249
and weaveworks/common#247 has fix

[periklis]

Isn't this address by #6957 already?

[liguozhong]

6859>6957 , so I don't think 6957 has fixed this yet

[periklis]

The issue is imho fixed by bumping the prometheus/client_golang to 1.13 with #6957. The changelog of client_golang (https://github.com/prometheus/client_golang/releases/tag/v1.13.0) mentions:

[BUGFIX] Raise exemplar labels limit from 64 to 128 bytes as specified in OpenMetrics spec. https://github.com/prometheus/client_golang/pull/1091

[bboreham]

One change removes the thing that made exemplar labels longer, the other raises the exemplar length limit.

Either way I agree it can be closed.