Recreate ordered pull on heartbeats #1178
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jarema
force-pushed
the
recreate-ordered-pull-on-heartbeats
branch
from
dda151f
to
dc5fec4
Compare
piotrpio
reviewed
Jan 17, 2024
Jarema
force-pushed
the
recreate-ordered-pull-on-heartbeats
branch
3 times, most recently
from
09b2442
to
87ad89b
Compare
Signed-off-by: Tomasz Pietrek <[email protected]>
After running extensive chaos testing, we found that there is possible to improve client code resilience for ordered consumer ability to self heal and recreate itself. The main change is adding retires for all operations related to recreating the consumer. This PR also improved tracing. No tests were added, as those are long running ones and now well suited to library tests. They will become part of chaos testing. Signed-off-by: Tomasz Pietrek <[email protected]>
Signed-off-by: Tomasz Pietrek <[email protected]>
We found out that tokio_retry does not compute exponential backoff properly. This commit switches to a better supported crate maintained by a well known org. Signed-off-by: Tomasz Pietrek <[email protected]>
Jarema
force-pushed
the
recreate-ordered-pull-on-heartbeats
branch
from
87ad89b
to
1c234e9
Compare
Signed-off-by: Tomasz Pietrek <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds few improvements to ordered consumers:
heartbeatsare missed twiceAbove provide smoother experience for working with ordered consumers in unstable network/consensus/extreme load conditions.
This PR also replaces
tokio_retrywith (tryhard)[https://github.com/EmbarkStudios/tryhard], ask I found bug in howtokio_retrycounts backoffs. Alsotryhardis actively maintained by a well know entity.Signed-off-by: Tomasz Pietrek [email protected]