[Sidecar] Querying external label values with a matcher produces wrong results

[dmitrime]

Thanos, Prometheus and Golang version used:
Thanos: v0.30.2
Prometheus: 2.42.0
Go: 1.20.3

Object Storage Provider:
Minio/S3

What happened:
Querying GET /api/v1/label/<label_name>/values API produces wrong results when the <label_name> is an external label and a match[] parameter is provided.

I've narrowed it down to sidecar, the store results seem to be okay. For example, given 2 sidecars having an external label region with values europe and asia and some_metric present, I then query:

GET /api/v1/label/region/values?start=1683092000&end=1683092999&match[]=some_metric{region="asia"}

I receive the response:

{"status":"success","data":["europe","asia"]}

What you expected to happen:

{"status":"success","data":["asia"]}

How to reproduce it (as minimally and precisely as possible):

Add the following test to https://github.com/thanos-io/thanos/blob/main/pkg/store/acceptance_test.go#L60:

                               {
                                       start:    timestamp.FromTime(minTime),
                                       end:      timestamp.FromTime(maxTime),
                                       label:    "region",
                                       matchers: []storepb.LabelMatcher{{Type: storepb.LabelMatcher_EQ, Name: "unknown", Value: "unknown"}},
                               },

This will fail by returning eu-west when in fact the matcher doesn't match any labels.

The problematic code seems to be this: https://github.com/thanos-io/thanos/blob/main/pkg/store/prometheus.go#L696-L699
If the external label is the one requested, then its value is returned without any further matching.

[dmitrime]

Related to #3639 which got closed without resolution.

[GiedriusS]

Fixed in #6816.