S3: PutObject operation with empty body fails with 501 "NotImplemented" error

[gdavison]

Documentation

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug

When the PutObject operation is called with an empty body, the request fails with the following error:

operation error S3: PutObject, https response error StatusCode: 501, RequestID: ********, HostID: *********, api error NotImplemented: A header you provided implies functionality that is not implemented

Expected behavior

The request should have succeeded

Current behavior

The request fails

2022-02-07T17:02:05.140-0800 [DEBUG] [aws-sdk-go-v2] Request
PUT /testState?x-id=PutObject HTTP/1.1
Host: ****.s3.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.13.0 os/macos lang/go/1.17.2 md/GOOS/darwin md/GOARCH/amd64 api/s3/1.24.1
Content-Length: 0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d216ba6d-9c6b-44c9-94ff-7047063fe9a9
Amz-Sdk-Request: attempt=1; max=5
Authorization: AWS4-HMAC-SHA256 Credential=********/20220208/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=9ae6f636bb39f3e475890ab1f26994020de91dcbba6105c11a4d1c7413936fb8
Content-Type: application/json
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20220208T010205Z
X-Amz-Security-Token: ********

2022-02-07T17:02:05.512-0800 [DEBUG] [aws-sdk-go-v2] Response
HTTP/1.1 501 Not Implemented
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 08 Feb 2022 01:02:04 GMT
Server: AmazonS3
X-Amz-Id-2: fhvvo1UE+UoJzAovIbsaH0cJ5N1CN/Cgy04330ogHcbGuOOVfSP5DhT7VhqEaNMNkrRZaJgrcmI=
X-Amz-Request-Id: D8Y4RJ014M60HTPE

14d
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>D8Y4RJ014M60HTPE</RequestId><HostId>fhvvo1UE+UoJzAovIbsaH0cJ5N1CN/Cgy04330ogHcbGuOOVfSP5DhT7VhqEaNMNkrRZaJgrcmI=</HostId></Error>
0

Steps to Reproduce

data := []byte{}

i := &s3.PutObjectInput{
	ContentType:          aws.String("application/json"),
	Body:                 bytes.NewReader(data),
	Bucket:               aws.String("test"),
	Key:                  aws.String("path"),
	ServerSideEncryption: "AES256",
}

ctx := context.Background()
_, err := c.s3Client.PutObject(ctx, i)

Possible Solution

No response

AWS Go SDK version used

github.com/aws/aws-sdk-go-v2 v1.13.0, github.com/aws/aws-sdk-go-v2/service/s3 v1.24.1

Compiler and Version used

go version go1.17.2 darwin/amd64

Operating System and version

macOS 10.15.7

[jasdel]

Thanks for reporting this issue @gdavison I've been able to reproduce this issue locally with the v2 SDK, and have begun investigating why this is happening. It doesn't look like this issue occurs with the v1 SDK. Even though the SDK is able to derive the Content-Length: 0 of the request, something is preventing that from being serialized in the HTTP request message sent, causing the HTTP client to use Transfer-encoding: chunked.

Did you discover this as a new bug when updating the SDK to a new version?

[jasdel]

I've created PR aws/smithy-go#356 in the smithy-go repo that addresses this issue. You can work around this issue until this changed is merged by running the following command to update your go.mod to point to refer to the PR change:

go get github.com/aws/[email protected])

[jasdel]

This issue has been resolved in the SDK's smithy-go dependency. The SDK's next release will include this fix.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[lestephane]

@jasdel I added

replace github.com/aws/smithy-go v1.11.1 => github.com/aws/smithy-go v1.11.1-0.20220225220832-6630cb6f6e84

to my .mod file, and the problem is still there.

In my case:

• the ContentLength in the PutObjectInput is set by me explcitly, to 0.
• the Request log shows that no Content-Length header is sent to the s3 endpoint.

[jasdel]

@lestephane are you still experiencing this issue after updating to the latest version of S3, github.com/aws/aws-sdk-go-v2/service/[email protected] This should also update your application to depend on github.com/aws/[email protected].

Setting ContentLength to 0 should be OK. What type of value are you using for the Body member?

[lestephane]

@jasdel I was using 1.26.0 and 1.11.1 when I got the problem and landed upon this issue in GitHub.
Then replaced smithy 1.11.1 with 1.11.1-..., and still got the issue.

I'm using a *Reader as Body.

[lestephane]

Would passing no Body when the ContentLength is 0 be an workaround? I was trying to avoid conditionals, as I'm getting the length and the reader from another library...

[jasdel]

Thanks for the update. You should be able to provide an reader that matches the content length. Though its important to note, if the ContentLength is 0 and the value of Body is an io.Reader that is unseekable the SDK won't know that you intended to send no payload. In this case it probably is best to leave Body nil if ContentLength is 0, or use a value for Body that is seekable or implements a Len() int method.

The reason the SDK is not using your provided ContentLength of 0, is because ContentLength member is not apointer. The SDK has no way to know that you set ContentLength to 0 vs that is the member's initialized zero value. If ContentLength is a non-zero value then the SDK will use it. Otherwise the SDK will attempt to determine the length of the payload from the Body value.

[lestephane]

I will try to not specify any Body when I have time, and report. Hopefully that does the trick.