feat: Add backoff to flush op #13140
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
grobinson-grafana
changed the title
grobinson-grafana
force-pushed
the
grobinson/add-flush-backoff
branch
5 times, most recently
from
551891a
to
48bbf5c
Compare
| @@ -126,7 +128,10 @@ func (cfg *Config) RegisterFlags(f *flag.FlagSet) { | |||
|
|
|||
| f.IntVar(&cfg.ConcurrentFlushes, "ingester.concurrent-flushes", 32, "How many flushes can happen concurrently from each stream.") | |||
| f.DurationVar(&cfg.FlushCheckPeriod, "ingester.flush-check-period", 30*time.Second, "How often should the ingester see if there are any blocks to flush. The first flush check is delayed by a random time up to 0.8x the flush check period. Additionally, there is +/- 1% jitter added to the interval.") | |||
| f.DurationVar(&cfg.FlushOpTimeout, "ingester.flush-op-timeout", 10*time.Minute, "The timeout before a flush is cancelled.") | |||
| f.DurationVar(&cfg.FlushOpBackoff.MinBackoff, "ingester.flush-op-backoff-min-period", 10*time.Second, "Minimum backoff period when a flush fails. Each concurrent flush has its own backoff, see `ingester.concurrent-flushes`.") | |||
There was a problem hiding this comment.
I chose to declare these here instead of using cfg.FlushOpBackoff.RegisterFlagsWithPrefix() for two reasons:
- I wanted to use
-separator instead of.to be consistent withingester.flush-op-timeout. - I wanted to write a more specific help message for each of these options that explains how they are used.
grobinson-grafana
force-pushed
the
grobinson/add-flush-backoff
branch
2 times, most recently
from
5bb49f5
to
837d5c8
Compare
github-actions
bot
added
the
type/docs
grobinson-grafana
force-pushed
the
grobinson/add-flush-backoff
branch
2 times, most recently
from
1ee50bd
to
127099b
Compare
grobinson-grafana
force-pushed
the
grobinson/add-flush-backoff
branch
5 times, most recently
from
e661dcf
to
d26e496
Compare
grobinson-grafana
force-pushed
the
grobinson/add-flush-backoff
branch
2 times, most recently
from
6ca1cb2
to
1994e1a
Compare
grobinson-grafana
dismissed
cyriltovena’s stale review
I made some changes to how the timeout works (per flush instead of per op), would love another review.
grobinson-grafana
force-pushed
the
grobinson/add-flush-backoff
branch
2 times, most recently
from
1c23a0f
to
c1bf4d6
Compare
This commit adds a configurable backoff to flush ops in the ingester. This is to prevent situations where the store put operation fails fast (i.e. 401 Unauthorized) and can cause ingesters to be rate limited.
grobinson-grafana
force-pushed
the
grobinson/add-flush-backoff
branch
from
c1bf4d6
to
ae77be1
Compare
| @@ -135,8 +137,9 @@ func (i *Ingester) sweepStream(instance *instance, stream *stream, immediate boo | |||
| } | |||
|
|
|||
| func (i *Ingester) flushLoop(j int) { | |||
| l := log.With(i.logger, "loop", j) | |||
There was a problem hiding this comment.
I added the loop to the logger so when a flush fails, or is canceled after max retries, we can see which loop it was.
This was referenced Jun 10, 2024
grobinson-grafana
added a commit
that referenced
this pull request
Jun 20, 2024
7 tasks
grobinson-grafana
added a commit
that referenced
this pull request
Jul 2, 2024
grobinson-grafana
added a commit
that referenced
this pull request
Jul 2, 2024
grobinson-grafana
added a commit
that referenced
this pull request
Jul 2, 2024
grobinson-grafana
added a commit
that referenced
this pull request
Jul 2, 2024
grobinson-grafana
added a commit
that referenced
this pull request
Jul 2, 2024
grobinson-grafana
added a commit
that referenced
this pull request
Jul 2, 2024
This was referenced Jul 3, 2024
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This commit adds a configurable backoff to flush ops in the ingester. This is to prevent situations where the store put operation fails fast (i.e. 401 Unauthorized) and can cause ingesters to be rate limited.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mdproduction/helm/loki/Chart.yamland updateproduction/helm/loki/CHANGELOG.mdandproduction/helm/loki/README.md. Example PRdeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR