feat(kafka): Replay kafka from last commit before becoming ready

[benclive]

What this PR does / why we need it:

This stops a Kafka-ingester from becoming "ready" to serve requests before it has replayed all the latest data from Kafka.

• This implementation is heavily inspired / copied from Mimir's
• The criteria for becoming ready can be controlled by two parameters: target lag and max lag. If we take too long to replay the data, we will make sure to hit max lag, but most of the time we will hit target lag.
• Since we are using a WAL, we expect that all data prior to the last commit on the consumer group at the ingester's partition is safely stored & replayed by the WAL mechanism. Therefore, we only continue to consume data from the last commit.
  • Each ingester instance has it's own consumer group to ensure this behaviour.

Which issue(s) this PR fixes:
Fixes https://github.com/grafana/loki-private/issues/1119

Special notes for your reviewer:
I didn't explicitly test the replay & multiple attempts functions yet. I couldn't test it via the existing test framework without injecting time delays into the test kafka. I'll investigate adding some more tests for the function directly.

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[cyriltovena]

LGTM

I like this approach. Let's test it.