Terrform custom registry generator & promotion script

[tcsc]

This script is intended to maintain a custom terraform registry that will be used to distribute the Teleport terraform provider.

The basic promotion steps are:

1. retrieve he pre-built terraform provider(s) from the staging bucket
2. repack them into a terraform-registry compatible format
3. retrieve the current repository versions index file (if present)
4. update (or create, if necessary) a local copy of the registry index, end
5. push all of the generated & updated files up the S3 bucket that backs the registry

The promotion code runs in response to two events:

1. A terraform tag build. The promotion is dairy-chained onto the terraform provider builds using Drone's depends_on pipeline config, and the resulting package is uploaded to a staging repository, and
2. A release promotion, where the resulting package is uploaded to the production repository

The Drone YAML is configured to execute the same code in both cases, with different values supplied via the Drone environment.

[wadells]

Go code LGTM.

I may have missed this in the RFD, but I think it might be good to "shift left" the zip repacking and signing from promote to tag. For instance, this is what the current workflow looks like:

• tag -> publish to dronestorage
• promote -> copy tar from dronestorage to clientbuilds, repack from tar to zip and sign, generate new index, write to release bucket

Ignoring the fact that dronestorage and clientbuilds need to die, I'd propose:

• tag -> publish tar to dronestorage, repack from tar to zip and sign, publish to internal staging repo/bucket, generate new staging index
• promote -> copy tar from dronestorage to clientbuilds, copy signed zip from internal staging repo to release bucket, generate new release index

The benefit here is we can test changes to the zip -> tar conversion, signing, and index generation without actually publishing to customers. This is pretty valuable, especially during the next month where we'll be iterating on this quite a bit.

An alternative is to have a mirror infra (e.g. buckets in teleport-dev where publishing can be tested safely), but my gut says that wiring up to Drone will be a pain.

[tcsc]

it might be good to "shift left" the zip repacking

I think it's possible in Drone to daisy-chain a new pipeline onto the existing tag build using prerequisites or something similar. Will have to do some more research. If so, that should be pretty easy to plug this into the back of the existing tag-triggered build.

If not, my alternative was (similarly to yours, I think) to use Drone's notion of promotion environments to:

1. set up separate testing and production environments, and
2. publish to "staging" during development.

Although, this may be a bit more dangerous, as we'd have to make sure to pick the right env every time we promote.

my gut says that wiring up to Drone will be a pain.

I'll listen to your gut for now, at least in matters of Drone.

[gzigzigzeo]

I think, we could parameterise the required Go version as it seems equal for the whole suite (if Drone allows it).

[gzigzigzeo]

Not sure if my comment is applicable here, but looks like. Terraform provider tests need env var TF_ENV=true to be specified (Terraform framework requirement). Terraform must be available as well.

[tcsc]

This is change is to avoid go test picking up the tooling module here. For some reason, in my testing, ./... included the tooling package and failed, where $(shell go list ./...) correctly treated it as a separate module.

Just tested it again now an all seems fine.

[r0mant]

lgtm mostly, just a few suggestions

[r0mant]

Is there anything we can do on the tool side to make sure that the index file hasn't changed between when we downloaded it above and saving it here? In case we get simultaneous builds running somehow. I don't suppose S3 has something like compare-and-swap?

[tcsc]

Oh how I tried to do this :-)

Short answer is that there is no way to do this in S3 alone, although we might be able to implement it with some other AWS services.

Here's what I looked into:

• Definitely no CAS on S3
• Creating a lockfile, even with s3 now having strong consistency, won't work as S3 has no atomic "put only if the there is nothing there already" primitive
• S3 object locking exists, but is not really applicable to this problem. It works at a whole-bucket level and is designed to protect objects from accidental or malicious erasure, rather than locking out concurrent changes and just won't do what we want.

To use an all-AWS solution we would have to look at implementing locking with a DynamoDB table or somesuch, which is certainbly do-able, but is probably not a lot better than relying on Drone to serialize these jobs at the pipeline level.

[wadells]

I like to add both production and a pipeline specific production-terraform as triggers on promotion steps. This allows rerunning only the production-terraform step if something goes wrong with it without needing to rerun the entire pipeline -- good for speed and also not re-uploading other artifacts unnecessarily.

[tcsc]

Excellent idea. Added an explicit staging build as well for this same reason