Retry S3 get request on 500 Internal Server Error #510
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xianwill
reviewed
Nov 29, 2021
xianwill
previously approved these changes
Nov 29, 2021
fvaleye
reviewed
Nov 29, 2021
|
@mosyp I was really surprised that the S3 retry was not managed by rusoto for these internal errors: rusoto/rusoto#234 If you want another point of view for this workaround: https://github.com/abetterinternet/prio-server/pull/597/files |
|
@fvaleye |
|
@fvaleye the solution in the links you've shared seems huge to implement within this PR and current needs, so I've opened #514. Also, since rusoto is not maintained anymore, we can give some hope for https://github.com/awslabs/aws-sdk-rust to do the job in a future |
xianwill
approved these changes
Nov 30, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We're getting 500 errors in kafka-delta-ingest on get requests (especially with several workers), which is expected and such requests have to be retried
We've already tested this in high load environments and this fix helped us avoid unnecessary restarts https://github.com/delta-io/delta-rs/tree/kdi-partition-values-quick-fix.
I've tried to find some universal approach to retry every s3 requests, but couldn't figure out one due to rusoto restrictions