feat(mimir/rules/k8s): retry startup process on failure

[hainenber]

PR Description

Which issue(s) this PR fixes

Fixes #5650

Notes to the Reviewer

Should I allow the backoff settings configurable?

Logs when the component retries on its failed startups

d2edf8b5faaa6 node_id=tracing duration=3.75µs
ts=2023-12-02T09:55:10.492555426Z level=info msg="finished node evaluation" controller_id="" trace_id=f455aa145d4bb36bfc8d2edf8b5faaa6 node_id=logging duration=1.583µs
ts=2023-12-02T09:55:10.492571676Z level=info msg="finished node evaluation" controller_id="" trace_id=f455aa145d4bb36bfc8d2edf8b5faaa6 node_id=labelstore duration=12.625µs
ts=2023-12-02T09:55:10.492580426Z level=info msg="finished node evaluation" controller_id="" trace_id=f455aa145d4bb36bfc8d2edf8b5faaa6 node_id=otel duration=1µs
ts=2023-12-02T09:55:10.492583634Z level=info msg="finished complete graph evaluation" controller_id="" trace_id=f455aa145d4bb36bfc8d2edf8b5faaa6 duration=956.542µs
ts=2023-12-02T09:55:10.492684676Z level=info msg="scheduling loaded components and services"
ts=2023-12-02T09:55:10.492788093Z level=info msg="starting cluster node" peers="" advertise_addr=127.0.0.1:80
ts=2023-12-02T09:55:10.492997801Z level=info msg="peers changed" new_peers=grafana-agent-test-mzxsb
ts=2023-12-02T09:55:10.493016801Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:80
ts=2023-12-02T09:55:10.694624218Z level=error msg="failed to list rules from mimir" component=mimir.rules.kubernetes.promRules err="Get \"abc/prometheus/config/v1/rules\": unsupported protocol scheme \"\""
ts=2023-12-02T09:55:10.694697301Z level=error msg="starting up component failed" component=mimir.rules.kubernetes.promRules err="Get \"abc/prometheus/config/v1/rules\": unsupported protocol scheme \"\""
ts=2023-12-02T09:55:12.615748677Z level=error msg="failed to list rules from mimir" component=mimir.rules.kubernetes.promRules err="Get \"abc/prometheus/config/v1/rules\": unsupported protocol scheme \"\""
ts=2023-12-02T09:55:12.61595976Z level=error msg="starting up component failed" component=mimir.rules.kubernetes.promRules err="Get \"abc/prometheus/config/v1/rules\": unsupported protocol scheme \"\""
ts=2023-12-02T09:55:14.992224053Z level=error msg="failed to list rules from mimir" component=mimir.rules.kubernetes.promRules err="Get \"abc/prometheus/config/v1/rules\": unsupported protocol scheme \"\""

PR Checklist

• CHANGELOG.md updated

[tpaschalis]

LGTM, thanks a lot @hainenber 🙌

We can make the backoff configurable in a follow-up PR if it's deemed useful.