Cannot get S3 private bucket artifact to download

[davidr912]

Nomad version

0.12

Operating system and Environment details

Ubuntu 18.04/Bionic

Issue

Using artifact I cannot get a file to download from a private S3 bucket

The machine is configured with an IAM profile which works via the aws CLI tool but not in Nomad

Reproduction steps

 artifact {
        source      = "https://legacy-player-resources.s3.eu-west-2.amazonaws.com/TownSnapshot/008d6a4f-8394-4d8e-802f-77e9aa7729a2.png"
      }   

• Nomad is running as a systemd service
• https:// gives a 403
• s3:: in front explicitly says invalid S3 URL
• Explicitly setting aws_access_key_id and aws_access_key_secret for a user with the same permissions still generates a 403

And yet if I use a batch job and run

      driver = "raw_exec"     
        config {
        command = "aws"
        args    = ["s3", "ls", "s3://legacy-player-resources/TownSnapshot/"]
      }
    }

I get valid results in the job log because the AWS CLI is using the IAM profile

2020-07-17 11:53:16      29366 00353f74-a296-4538-83bc-f4ca3845695c.png
2020-07-16 14:43:10      31365 008d6a4f-8394-4d8e-802f-77e9aa7729a2.png
2020-07-13 11:27:46      33267 00e5f9c2-0b20-4e88-960b-a6c26bfd6374.png
2020-07-16 11:14:19      31952 06a90c6d-faf1-4df9-bba7-aaeb7271e108.png
2020-07-13 18:33:52      32708 06baedf8-156a-4abd-9d6a-2d816f3f88c8.png
2020-07-13 09:23:33      31916 078acf2a-6089-4ea1-8c1f-8f7088c89b86.png
2020-07-16 04:18:49      29439 0895e5a8-cc97-4369-9246-2f8c0d0d848e.png
2020-07-16 17:57:43      31280 0d6b6b7e-c4fd-4df7-85f1-dc4125af62e8.png

What am I doing wrong here?

[picatz]

👋 Hello @davidr912! I've also struggled with this before.

Dots and Dashes

😅 You did actually provide a valid S3 path. According to the documentation, it's the newer version of the "virtual hosted style", with the S3 dot region endpoint structure, instead of the older dash format. This should make more sense with the following example:

Newer virtual host style with S3 dot region endpoint structure:
https://legacy-player-resources.s3.eu-west-2.amazonaws.com/...
                                                               ☝️

Older virtual host style with S3 dash region endpoint structure:
https://legacy-player-resources.s3-eu-west-2.amazonaws.com/...
                                                               ☝️

I think you could fix the error you were seeing by replacing the . with the -, which should be supported in older AWS regions. However, the older dash-style is actually discouraged in the S3 documentation:

We recommend that you do not use this endpoint structure in your requests.

Improving S3 Detection

The artifact stanza uses the go-getter library under-the-hood. The current tests don't include the newer virtual hosted style syntax, and doesn't seem to be supported, causing valid S3 paths to fail.

In the future we should add support for this newer dash style, since this is a pretty frustrating thing to debug, and is the recommended pattern by AWS.

[davidr912]

Hi @picatz, thanks for the info

I had to mess with it a little bit more until I eventually got to just:

legacy-player-resources.s3-eu-west-2.amazonaws.com/TownSnapshot/008d6a4f-8394-4d8e-802f-77e9aa7729a2.png

which works.

If I put https:// on the front of that it will 403, and S3::https:// tells me it's not a valid S3 URL - not sure why

But I have a working format now at least, thank you 👍

[davidr912]

I should say though - annoyingly that only works with explicit credentials. I cannot get it to honour IAM still.

On top of that region seems to be ignored entirely if it's specified as an artifact option (still see BucketRegionError: incorrect region, the bucket is not in 'us-east-1' region at endpoint 's3.amazonaws.com' no matter what) - which made me suspect the go-getter documentation is in fact inaccurate and this option is perhaps always inferred from the URL?

[picatz]

I had to mess with it a little bit more until I eventually got to just:

legacy-player-resources.s3-eu-west-2.amazonaws.com/TownSnapshot/008d6a4f-8394-4d8e-802f-77e9aa7729a2.png

Awesome, that helps confirm my suspicions about go-getter not supporting virtual host dot region format! 🙌

... the go-getter documentation is in fact inaccurate and this option is perhaps always inferred from the URL?

I also suspect this is the case.

[tgross]

Closed by #9349, which we expect to ship in Nomad 1.0.2

[PeterSzegedi]

Hi @picatz, thanks for the info

I had to mess with it a little bit more until I eventually got to just:

legacy-player-resources.s3-eu-west-2.amazonaws.com/TownSnapshot/008d6a4f-8394-4d8e-802f-77e9aa7729a2.png

which works.

If I put https:// on the front of that it will 403, and S3::https:// tells me it's not a valid S3 URL - not sure why

But I have a working format now at least, thank you 👍

You have restored my sanity. Thanks :)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.