Support retrying S3 connection resets on GetObject body

[hallmaxw]

Version of AWS SDK for Go?

1.8.19 (latest version)

Version of Go (go version)?

1.8

What issue did you see?

Connection reset by peer when reading the body of a s3.GetObjectOutput. I'd expect the SDK to be capable of retrying connection resets.

The problem is the REST unmarshal handler just sets GetObjectOutput.Body's to HTTPResponse.Body without reading the data. Request.Send() will then return successfully, but an error may still occur when the caller tries to read GetObjectOutput.Body. This error occurs outside of the AWS SDK's logic. Therefore, consumers can't take advantage of Request's retry logic.

Steps to reproduce

I don't have a code snippet, but this is easily reproducible if you put a decent load on an S3 bucket. It's common for S3 to reset the connection while reading the GetObjectOutput.Body body. See AWS support's response to this thread https://forums.aws.amazon.com/thread.jspa?threadID=111195.

[xibz]

Hello @mbh621, thank you for reaching out to us. I would suggest using the s3manager package. This will retry on connection resets. Please let us know if you have any issues there. Meanwhile I'll make this as a feature request for the service/s3 package.

[jasdel]

One way i think the SDK could implement this functionality is to add support for an intermediate reader that would keep track of the bytes read by the user. When a error occurs internally it could do another GetObject request, but using byte range to start where it had left off. I imagine that the retry could would be configurable.

I think this would be the best strategy for implementing this feature.

[alex]

@xibz (Several year late to this, my apologies!) one of the downsides of s3manager is that it has a significantly different API, since you give it an io.Write which it writes to, rather than returning an io.Read.

If s3manager offered an API that gave you an io.Read, I think it'd be drop-in enough that you could close this in favor of telling people to always use s3manager. (If you'd be prefer that be treated as a separate feature request, please let me know and I'll file another issue)

[github-actions]

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

[alex]

I don't think anything has changed that obviates or resolves this issue.

[skmcgrail]

Closing this issue, per the messaging here we do not intend to add support for automatically retrying connection reset errors when reading response bodies. Further details can be found in this issue thread: #2926 (comment)

We would be welcome to contributions that add an configurable mechanism to the retryer implementation that allows customers to override this particular behavior, that they could then opt-in to.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.