Producer fails faster, Retry publish after Auth error #1437
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erikvanoosten
force-pushed
the
auth-error-retry-produce
branch
from
0b2a326 to
13fdaa1
Compare
4 tasks
Introduce a `authErrorRetrySchedule` on which publishing after auth errors (`AuthorizationException` and `AuthenticationException`) can be retried. Auth error occur on some slow brokers. This change is not binary compatible due to changes in `ProducerSettings`. TODO: unit test
erikvanoosten
force-pushed
the
auth-error-retry-produce
branch
from
13fdaa1 to
2f1d122
Compare
Also improved names and documentation.
erikvanoosten
changed the title
svroonland
reviewed
Jan 19, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the producer fails to send a message in a batch of messages, it stops sending after the first error. This behavior is extended by also stopping after a failure from Kafka's callback.
Introduce producer setting
authErrorRetryScheduleon which sending after auth errors (AuthorizationExceptionandAuthenticationException) can be retried. Auth error occur on some slow brokers. Note that this feature is disabled by default because it re-orders sent messages.This is a follow-up of Allow for some auth exceptions during poll #1270 where we retry polling after an auth error.
Both changes have been made possible by a new test framework in which we have precise control over the order of callbacks.
This change is not binary compatible due to changes in
ProducerSettings.Also:
Producerscaladocs