Reading S3 bucket fails with error URL is not a valid S3 compliant URL

[Subhajit97]

Hi! I am trying to read a non-public S3 bucket created in my AWS account with url = s3::https://s3.us-east-1.amazonaws.com/my_bucket/test_folder?aws_access_key_id=ACCESS_KEY&aws_access_key_secret=SECRET and received the following error:

URL is not a valid S3 compliant URL

I had a look at https://github.com/hashicorp/go-getter#s3-bucket-examples and https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html, but I'm not sure what syntax to use especially for buckets which are not public.

Is there anything I'm missing in the above example?

[ghost]

@Subhajit97,
Not a hashicorp member, but I think since you're passing in parameters you need doublequotes around the whole thing ie "s3::https://s3.us-east-1.amazonaws.com/my_bucket/test_folder?aws_access_key_id=ACCESS_KEY&aws_access_key_secret=SECRET"

I've currently got a PR #394 updating the s3 formats, if you want to take a look at alternate s3 uri formats

[Subhajit97]

Thanks, @twunderlich-grapl! the example URI formats are helpful.
I was able to make the URL work. I'm using the below format:

s3::https://bucket.s3.us-east-1.amazonaws.com/folder?aws_profile=my-profile

But, when I use the same URL by explicitly passing the access key and secret to the URL, for example:

s3::https://bucket.s3.us-east-1.amazonaws.com/folder?aws_access_key_id=...&aws_access_key_secret=...

It fails with the below error:

error downloading 'https://bucket.s3.us-east-1.amazonaws.com/folder?aws_access_key_id=...&aws_access_key_secret=...': NoSuchKey: The specified key does not exist.
        status code: 404, request id: ..., host id: ...

Is there anything I'm missing here?

[buzzki11]

Thanks, @twunderlich-grapl! the example URI formats are helpful. I was able to make the URL work. I'm using the below format:

s3::https://bucket.s3.us-east-1.amazonaws.com/folder?aws_profile=my-profile

But, when I use the same URL by explicitly passing the access key and secret to the URL, for example:

s3::https://bucket.s3.us-east-1.amazonaws.com/folder?aws_access_key_id=...&aws_access_key_secret=...

It fails with the below error:

error downloading 'https://bucket.s3.us-east-1.amazonaws.com/folder?aws_access_key_id=...&aws_access_key_secret=...': NoSuchKey: The specified key does not exist.
        status code: 404, request id: ..., host id: ...

Is there anything I'm missing here?

I have the same issue. Do you managed to solve it or find workaround?