Sidecar: failed to fetch initial external labels. unknown unit

[hanjm]

Thanos, Prometheus and Golang version used:

Object Storage Provider:
COS
What happened:
thanos sidecar can not start up.
error message is

2021-09-12T16:09:13.532144772+08:00 level=warn ts=2021-09-12T08:09:13.531921453Z caller=sidecar.go:179 msg="failed to fetch initial external labels. Is Prometheus running? Retrying" err="parse Prometheus config: global:\n  scrape_interval: 30s\n  scrape_timeout: 10s\n  evaluation_interval: 1m\n  external_labels:\n    hash_mod: \"0\"\n    replica: A\n    tenant_id: default\nalerting:\n  alertmanagers:\n  - follow_redirects: true\n    scheme: http\n    timeout: 10s\n    api_version: v2\n    static_configs:\n    - targets: []\nrule_files:\n- /prometheus/recording_rule.yaml\n- /prometheus/alert_rule.yaml\nscrape_configs:\n- job_name: prometheus\n  honor_timestamps: true\n  scrape_interval: 10s\n  scrape_timeout: 10s\n  metrics_path: /metrics\n  scheme: http\n  follow_redirects: true\n  static_configs:\n  - targets:\n    - localhost:9090\n- job_name: etcd_sd\n  honor_timestamps: true\n  scrape_interval: 30s\n  scrape_timeout: 10s\n  metrics_path: /metrics\n  scheme: http\n  body_size_limit: 10485760\n  sample_limit: 15000\n  target_limit: 3000\n  label_name_length_limit: 255\n  label_value_length_limit: 512\n  follow_redirects: true\n: units: unknown unit  in 10485760"

prometheus new version instroduce a exempriment config field body_size_limit, the type is units. Base2Bytes (https://github.com/alecthomas/units). it implement text.TextUnmarshaler but not umplement text.TextMarshaler (https://github.com/alecthomas/units/blob/master/bytes.go#L43).
so the marshalled config content can not be marshalled.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
Prometheus 2.29.2 + Thanos v0.23.0-rc.0

Full logs to relevant components:

Anything else we need to know:

[hanjm]

It could be fixed by only unmarshal global config. I am fixing it.

[yeya24]

Let's add Prometheus 2.29.2 version to our E2E test so that we can catch the error in CI.

[hanjm]

Let's add Prometheus 2.29.2 version to our E2E test so that we can catch the error in CI.

Yes, It will be great. but i see this TODO https://github.com/thanos-io/thanos/blob/main/test/e2e/e2ethanos/services.go#L44 // TODO(bwplotka): Run against multiple?, current E2E does not support multi version.

[yeya24]

Let's add Prometheus 2.29.2 version to our E2E test so that we can catch the error in CI.

Yes, It will be great. but i see this TODO https://github.com/thanos-io/thanos/blob/main/test/e2e/e2ethanos/services.go#L44 // TODO(bwplotka): Run against multiple?, current E2E does not support multi version.

OK. I see, it should be unit tests running by Circle CI. I think that supports different Prometheus versions.

As for E2E tests with multiple Prometheus versions, is it supported by the new E2E framework? cc @matej-g @bwplotka

[matej-g]

As for E2E tests with multiple Prometheus versions, is it supported by the new E2E framework? cc @matej-g @bwplotka

If I understand the question correctly, this should be already possible - have a look at signature of NewPrometheus, look at parameter promImage. Just specify the Docker image for whichever version you would like to run and instantiate the service.

[hanjm]

If I understand the question correctly, this should be already possible - have a look at signature of NewPrometheus, look at parameter promImage. Just specify the Docker image for whichever version you would like to run and instantiate the service.

This way could only run single test which use specific promImage, not all the E2E test about sidecar.

[matej-g]

Do we want to run all the E2E tests against multiple versions of Prometheus? I guess then we could just pass environment variable with Prometheus version we desire and replace the image version (similarly to what we do for Thanos - https://github.com/matej-g/thanos/blob/main/test/e2e/e2ethanos/services.go#L54). I don't think there is any such feature 'baked' into the E2E framework.