Error when try to upload to presigned URL generated with parameter ACL

[slapec93]

Version of AWS SDK for Go?

v1.12.7, but I tried to update to the latest one and the issue still exists

Version of Go (go version)?

go1.9 darwin/amd64

What issue did you see?

Just like in the issue #1252, I get the error The request signature we calculated does not match the signature you provided. Check your key and signing method. when I try to upload a file to AWS with presigned URL.

Steps to reproduce

I create a new S3 object and then I create a request with the method named PutObjectRequest, just like this:

req, _ := svc.PutObjectRequest(&s3.PutObjectInput{
	ACL:           aws.String("private"),
	Bucket:      aws.String(p.Config.AWSBucket),
	Key:            aws.String(key),
	ContentLength: aws.Int64(fileSize),
})

After this call the Presign method of the request with the expire time, I don't get any error, bbut when I try to upload a file with a curl command, it retrieves the error mentioned above. My curl call looks like this:

curl -T <filename> -X PUT "<presigned url>"

In another project, which is a Rails one, we use the AWS gem, which works fine, so I compared the two generated presigned URL and found the differences below:

Rails:
<the first part of the URL>&X-Amz-SignedHeaders=host&x-amz-acl=private

Go:
<the first part of the URL>&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-acl

When I remove the ACL parameter, the upload works fine.

The use of PresignRequest method isn't an options, because I need one compact URL.

Thanks in advance 🙂

[jasdel]

Thanks for reaching out to us @slapec93 It looks like the issue you're running into is how the AWS SDK for Ruby marshals the ACL field into the query string. The AWS SDK for Go puts the ACL field in the header.

I'll look into why there is this difference.

As a workaround you can use the PresignRequest method on Request type that will return both the URL and a http.Header for all header values that must be included in the HTTP request that was presigned. The headers returned must be sent with the request or the signature will be invalid. Check out our PresignURL for a runnable example.

[viktorbenei]

@jasdel thanks for the quick reply! I'm a team mate of @slapec93 :)

PresignRequest is not an option unfortunately as we have to be able to share a single URL, without additional Header requirements. At least that's what we did so far, e.g. with the Ruby SDK (as mentioned by @slapec93 )

[jasdel]

Thanks for the update. I've investigated this issue and it looks like the URL that is being generated by the AWS SDK for Ruby is actually not restricting the presigned URL to the parameters that is being specified in the SDK's a input parameters. Specifically the Content-Length header isn't being signed allowing users of the presigned URL to upload any content length of their choosing.

In addition for ACL, if the x-amz-acl field is not in the header the ACL canned policy is ignored. So if the policy is in the query string it will be ignored and the object will have the bucket's default policy applied. Since private is generally the default policy having x-amz-acl=private in the query string is hiding the fact its not actually working. S3 only reads the ACL if it is included as a header. You can validated this by changing private to public-read and inspect the object's permissions in the AWS S3 Console.

Here is a reference to where this issue was fixed in the AWS SDK for Ruby v2.1.31 and related Github issue, aws/aws-sdk-ruby#874. I think nearly all of the S3 header values need to be preserved as headers for presigned URLs.

[viktorbenei]

Got it, thank you @jasdel ! :)

[jasdel]

Let us know if you run into any addition issues with the SDK or have feedback.

[nicozhang]

Thanks for the update. I've investigated this issue and it looks like the URL that is being generated by the AWS SDK for Ruby is actually not restricting the presigned URL to the parameters that is being specified in the SDK's a input parameters. Specifically the Content-Length header isn't being signed allowing users of the presigned URL to upload any content length of their choosing.

In addition for ACL, if the x-amz-acl field is not in the header the ACL canned policy is ignored. So if the policy is in the query string it will be ignored and the object will have the bucket's default policy applied. Since private is generally the default policy having x-amz-acl=private in the query string is hiding the fact its not actually working. S3 only reads the ACL if it is included as a header. You can validated this by changing private to public-read and inspect the object's permissions in the AWS S3 Console.

Here is a reference to where this issue was fixed in the AWS SDK for Ruby v2.1.31 and related Github issue, aws/aws-sdk-ruby#874. I think nearly all of the S3 header values need to be preserved as headers for presigned URLs.

I searched many solution, they never mention "x-amz-acl" should be putted in header. Frustrated for almost a half day util see your answer which writed " if the x-amz-acl field is not in the header the ACL canned policy is ignored." ):
Thanks a lot