Secrets should be more tightly knit into Builders

[milescrabill]

Right now secrets are more or less an afterthought - a YAML file specifies secret locations and values, and they are dumped onto the filesystem.

The issue is that we re-use builders for multiple deployment environments. For example, level 1 builders (untrusted) in firefox-ci-tc use the same docker_worker_gcp builder as the community-tc deployment. The difference comes when baking the image, when the secrets file is specified.

This presents an opportunity for user error, if the secrets file is incorrectly specified, the wrong secrets are used for the wrong environment, etc.

We should tie secrets more closely to builders, and consider having a builder per environment / deployment. We already sort of do this with the trusted / L3 builder. Secrets could become part of the builder config, and a builder could specify a filename key with an expected secrets mapping.

This would translate fairly well to password stores like sops and pass which store files on disk, and we could support decryption options so that decrypted secrets don't have to be stored on disk to use with monopacker.

So, instead of passing SECRETS_FILE=/path/to/secrets, the secrets file to use would be configured for the builder, and a root secrets directory and decryption command would be configured on the command line, i.e. SECRETS_STORE=~/.password-store/docker-worker/yaml/

This is still an idea, but I think it would help address a pain point with monopacker as it stands.

[djmitche]

From https://github.com/taskcluster/taskcluster/projects/9:

---

Secrets should be per-builder, rather than global to all builders.

Secrets handling in tc-admin might be a good example.

Operations that require secrets should fail if they are not present (for example, I just accidentally built images without secrets)