Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3.GetObject returns a SignatureDoesNotMatch error #883

Closed
2 tasks done
jriquelme opened this issue Nov 9, 2020 · 12 comments · Fixed by #913
Closed
2 tasks done

s3.GetObject returns a SignatureDoesNotMatch error #883

jriquelme opened this issue Nov 9, 2020 · 12 comments · Fixed by #913
Labels
guidance Question that needs advice or information.

Comments

@jriquelme
Copy link

Confirm by changing [ ] to [x] below:

Describe the question

I'm not sure if this is a question or indeed a bug, because the API has changed considerably since v0.24. I'm getting a SignatureDoesNotMatch error when running the example program below:

package main

import (
	"context"
	"io/ioutil"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	cfg, err := config.LoadDefaultConfig()
	if err != nil {
		log.Fatal(err)
	}
	client := s3.NewFromConfig(cfg)
	output, err := client.GetObject(context.Background(), &s3.GetObjectInput{
		Bucket: aws.String("my-bucket"),
		Key:    aws.String("my-file.csv"),
	})
	if err != nil {
		log.Fatal(err)
	}
	b, err := ioutil.ReadAll(output.Body)
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("%s", b)
}

The go.mod file:

module github.com/InstaGIS/s3test

go 1.15

require (
	github.com/aws/aws-sdk-go-v2 v0.29.0
	github.com/aws/aws-sdk-go-v2/config v0.2.2
	github.com/aws/aws-sdk-go-v2/service/s3 v0.29.0
)

Output error: api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I'm missing something? Any help would be appreciated :)

@jriquelme jriquelme added the guidance Question that needs advice or information. label Nov 9, 2020
@skotambkar
Copy link
Contributor

Hi @jriquelme ,

Thanks for reaching out. I tried reproducing the issue, and was unable to do so on my end.

Could you try executing go clean -modcache to clean any existing module cache and then retry the request.
Let us know your findings.

@skotambkar skotambkar added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 9, 2020
@jriquelme
Copy link
Author

Hi @skotambkar:

I tried that before. Checking the module dependencies, the only thing odd to me is the presence of two different versions of smithy-go (but I don't know if it is an issue). Here's the mod graph I get from the sample program:

go mod graph
github.com/InstaGIS/s3test github.com/aws/[email protected]
github.com/InstaGIS/s3test github.com/aws/aws-sdk-go-v2/[email protected]
github.com/InstaGIS/s3test github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/[email protected] github.com/awslabs/[email protected]
github.com/aws/[email protected] github.com/google/[email protected]
github.com/aws/[email protected] github.com/jmespath/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/awslabs/[email protected]
github.com/jmespath/[email protected] github.com/jmespath/go-jmespath/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/awslabs/[email protected]
github.com/awslabs/[email protected] github.com/google/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/awslabs/[email protected]
github.com/jmespath/go-jmespath/internal/[email protected] github.com/davecgh/[email protected]
github.com/jmespath/go-jmespath/internal/[email protected] github.com/pmezard/[email protected]
github.com/jmespath/go-jmespath/internal/[email protected] github.com/stretchr/[email protected]
github.com/jmespath/go-jmespath/internal/[email protected] gopkg.in/[email protected]
github.com/google/[email protected] golang.org/x/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/awslabs/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/awslabs/[email protected]
github.com/awslabs/[email protected] github.com/google/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/awslabs/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/google/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/awslabs/[email protected]
gopkg.in/[email protected] gopkg.in/[email protected]

I see github.com/aws/aws-sdk-go-v2/service/internal/[email protected] depending on github.com/awslabs/[email protected].

Would the go.sum of any help to diagnose the problem?

@skotambkar
Copy link
Contributor

skotambkar commented Nov 9, 2020

Thanks for getting back with us. Can you let us know what region the request is configured for? Also do you own the bucket you are trying to query? Or have valid permissions with the credentials you are using?

jriquelme added a commit to jriquelme/s3test that referenced this issue Nov 9, 2020
@jriquelme
Copy link
Author

jriquelme commented Nov 9, 2020

The bucket is in us-east-1 and the permissions are fine. Before posting here (along with cleaning caches and everything), I verified I can copy the file with awscli (aws s3 cp ...), also the current version using the v0.24.0 doesn't have any issue with the bucket. To be sure, I compiled and ran the sample program in a ec2 machine and I got the same error :D

The sample program is available at https://github.com/jriquelme/s3test

Thank you for your help!

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 10, 2020
@skotambkar
Copy link
Contributor

skotambkar commented Nov 12, 2020

Hi @jriquelme

Unfortunately I couldn't replicate the error behavior from the example you provided above. It would help if we can log the request that is being sent. The logger utils are yet to be released, but are merged in GitHub master branch.

For now, we can use the logger using this go.mod :

module foo

go 1.15

require (
	github.com/aws/aws-sdk-go-v2 v0.29.1-0.20201110202303-65b91cc8a475
	github.com/aws/aws-sdk-go-v2/config v0.2.3-0.20201108032120-2274aef7cb52
	github.com/aws/aws-sdk-go-v2/ec2imds v0.1.5-0.20201108032120-2274aef7cb52 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v0.3.1-0.20201108032120-2274aef7cb52 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v0.3.2-0.20201109182518-8e108cc6e922 // indirect
	github.com/aws/aws-sdk-go-v2/service/s3 v0.29.1-0.20201109182518-8e108cc6e922
	github.com/aws/aws-sdk-go-v2/service/sts v0.29.1-0.20201108032120-2274aef7cb52 // indirect
	github.com/google/go-cmp v0.5.2 // indirect
)

Here's the code sample usage of setting ClientLogMode on config:

package main

import (
	"context"
	"io/ioutil"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	cfg, err := config.LoadDefaultConfig()
	if err != nil {
		log.Fatal(err)
	}

        // log the request with body along with received response
	cfg.ClientLogMode = aws.LogRequestWithBody | aws.LogResponseWithBody

	client := s3.NewFromConfig(cfg)
	output, err := client.GetObject(context.Background(), &s3.GetObjectInput{
		Bucket: aws.String("mockbucket"),
		Key:    aws.String("my-file.csv"),
	})
	if err != nil {
		log.Fatal(err)
	}
	b, err := ioutil.ReadAll(output.Body)
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("%s", b)
}

Please let us know your findings to help us investigate the issue. Appreciate your patience and follow-ups.

@skotambkar skotambkar added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 12, 2020
@jriquelme
Copy link
Author

@skotambkar, the problem is related to the key. I've updated the repo https://github.com/jriquelme/s3test; it's basically the same but with a test showing a sucessful and a failing case. There are three branches:

  • main: with the v0.29.0, one case failing
  • v0.24.0: with the v0.24.0. In this branch, both cases work as expected.
  • v0.29.1-x: with the dependencies of your previous comment, also failing with one case.

The problematic key is private/us-east-1:bc324f78-8b99-4359-a9a7-b1df7ecea360/customers_2.csv.

This is the output of the failing test case in the branch v0.29.1-x:

=== RUN   TestCopyFileFromS3/doesnt_work
SDK 2020/11/12 14:53:43 DEBUG Request
GET /private/us-east-1:bc324f78-8b99-4359-a9a7-b1df7ecea360/customers_2.csv?x-id=GetObject HTTP/1.1
Host: aws-go-sdk-v2-883.s3.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/0.29.0 GOOS/linux GOARCH/amd64 GO/go1.15.2 s3
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 6bc4aaf9-4c9c-4074-a9a5-90eb35c97eec
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIAQTKI46UPJOMHY4XO/20201112/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=52cf31253c34607dff2f06bdd40ff6cd5dc58d39674dfe0e98121bb5f228ac7a
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20201112T175343Z

SDK 2020/11/12 14:53:44 DEBUG Response
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 12 Nov 2020 17:53:44 GMT
Server: AmazonS3
X-Amz-Id-2: yPrM34tL4HeQD/6SJ8SU62g7P9HoQWQXZBLizpXM9eUfO1qpkFkm7xoCu7BH7abioXRAU8NFvS4=
X-Amz-Request-Id: CEE4A3AC8F5E2825

cee
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAQTKI46UPJOMHY4XO</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20201112T175343Z
20201112/us-east-1/s3/aws4_request
1ece8c8ff3a862937df1c8e6f301a2c198f822a4be45c6acac7b20b2c59911b3</StringToSign><SignatureProvided>52cf31253c34607dff2f06bdd40ff6cd5dc58d39674dfe0e98121bb5f228ac7a</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 30 31 31 31 32 54 31 37 35 33 34 33 5a 0a 32 30 32 30 31 31 31 32 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 31 65 63 65 38 63 38 66 66 33 61 38 36 32 39 33 37 64 66 31 63 38 65 36 66 33 30 31 61 32 63 31 39 38 66 38 32 32 61 34 62 65 34 35 63 36 61 63 61 63 37 62 32 30 62 32 63 35 39 39 31 31 62 33</StringToSignBytes><CanonicalRequest>GET
/private/us-east-1%3Abc324f78-8b99-4359-a9a7-b1df7ecea360/customers_2.csv
x-id=GetObject
accept-encoding:identity
amz-sdk-invocation-id:6bc4aaf9-4c9c-4074-a9a5-90eb35c97eec
amz-sdk-request:attempt=1; max=3
host:aws-go-sdk-v2-883.s3.us-east-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20201112T175343Z

accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 70 72 69 76 61 74 65 2f 75 73 2d 65 61 73 74 2d 31 25 33 41 62 63 33 32 34 66 37 38 2d 38 62 39 39 2d 34 33 35 39 2d 61 39 61 37 2d 62 31 64 66 37 65 63 65 61 33 36 30 2f 63 75 73 74 6f 6d 65 72 73 5f 32 2e 63 73 76 0a 78 2d 69 64 3d 47 65 74 4f 62 6a 65 63 74 0a 61 63 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 3a 69 64 65 6e 74 69 74 79 0a 61 6d 7a 2d 73 64 6b 2d 69 6e 76 6f 63 61 74 69 6f 6e 2d 69 64 3a 36 62 63 34 61 61 66 39 2d 34 63 39 63 2d 34 30 37 34 2d 61 39 61 35 2d 39 30 65 62 33 35 63 39 37 65 65 63 0a 61 6d 7a 2d 73 64 6b 2d 72 65 71 75 65 73 74 3a 61 74 74 65 6d 70 74 3d 31 3b 20 6d 61 78 3d 33 0a 68 6f 73 74 3a 61 77 73 2d 67 6f 2d 73 64 6b 2d 76 32 2d 38 38 33 2e 73 33 2e 75 73 2d 65 61 73 74 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 30 31 31 31 32 54 31 37 35 33 34 33 5a 0a 0a 61 63 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 3b 61 6d 7a 2d 73 64 6b 2d 69 6e 76 6f 63 61 74 69 6f 6e 2d 69 64 3b 61 6d 7a 2d 73 64 6b 2d 72 65 71 75 65 73 74 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>CEE4A3AC8F5E2825</RequestId><HostId>yPrM34tL4HeQD/6SJ8SU62g7P9HoQWQXZBLizpXM9eUfO1qpkFkm7xoCu7BH7abioXRAU8NFvS4=</HostId></Error>
0

    s3_test.go:14: oh no: operation error S3: GetObject, https response error StatusCode: 403, RequestID: CEE4A3AC8F5E2825, HostID: yPrM34tL4HeQD/6SJ8SU62g7P9HoQWQXZBLizpXM9eUfO1qpkFkm7xoCu7BH7abioXRAU8NFvS4=, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
    --- FAIL: TestCopyFileFromS3/doesnt_work (1.08s)

I hope it helps. And I'm sorry for the inconvinencie, I didn't put the real key in the example program because I never thought it was relevant :|

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 13, 2020
@jarreds
Copy link

jarreds commented Nov 15, 2020

Additional data point for this issue. Seeing a similar signature mismatch behavior calling out to STS. Previously working code -- seeing the following on v0.29.0:

operation error CloudFormation: CreateStack, failed to sign request: failed to retrieve credentials: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: f1ce8cd3-63d0-45e0-96eb-7cb81825b070, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

Edit:

Also noting that this issues seems data dependent, as well. The same code path fails or succeeds intermittently. The following step function invocations all use the same code, and all failures were signature mismatches to STS.

image

@jarreds
Copy link

jarreds commented Nov 16, 2020

Noting that v0.28.0 doesn't exhibit this behavior. Just confirmed.

@skotambkar
Copy link
Contributor

Hi @jriquelme Thanks for the detailed information. We identified the issue being with our serializer not correctly escaping few characters. We are currently working on debugging and fixing the issue.

Thanks for reaching out @jarreds. Could you provide us with sample example/data that we can use to reproduce the success or error cases. This would help us identify the root cause and fix the behavior.

@jarreds
Copy link

jarreds commented Nov 16, 2020

This may help. Can't do a full repro for you, but here is an extract of the code in question:

func CreateStack(ctx context.Context, event Event) (*string, error) {
	cfg, err := config.LoadDefaultConfig(config.WithRegion(event.RegionID))
	if err != nil {
		return nil, err
	}

	role := fmt.Sprintf("arn:aws:iam::%s:role/ci", event.AccountID)
	stsclient := sts.NewFromConfig(cfg)
	stsassume := stscreds.NewAssumeRoleProvider(stsclient, role)

	cfg.Credentials = &aws.CredentialsCache{Provider: stsassume}

	cfnclient := cloudformation.NewFromConfig(cfg)
	cfnrole := fmt.Sprintf("arn:aws:iam::%s:role/cloudformation", event.AccountID)

	input := cloudformation.CreateStackInput{
		Capabilities: []types.Capability{
			types.CapabilityCapability_named_iam,
			types.CapabilityCapability_auto_expand,
		},
		StackName:   &event.Name,
		TemplateURL: &event.TemplateURL,
		RoleARN:     &cfnrole,
		Tags: []*types.Tag{
			{
				Key:   aws.String("release"),
				Value: aws.String(event.Release),
			},
			{
				Key:   aws.String("template"),
				Value: aws.String(path.Base(event.TemplateURL)),
			},
			{
				Key:   aws.String("stack"),
				Value: aws.String(event.Name),
			},
		},
	}

	resp, err := cfnclient.CreateStack(ctx, &input)
	if err != nil {
		return nil, err
	}

	return resp.StackId, nil
}

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@skotambkar
Copy link
Contributor

Will create a separate issue for @jarreds STS issue. As STS issue appears unrelated to this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
guidance Question that needs advice or information.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants