Provide an easy way for developers to distribute pre-release builds to customers

[wadells]

What

We have developers who want to distribute semi-official builds to end users. For instance, a build with increased logging in a particular area to help a user debug some problem.

These builds are not official releases in that all customers should know about or upgrade to them. But they are official in that we want them to be verified as 1st party, signed, sealed and delivered without resorting to sharing the bits via a side channel.

How

TBD

Why

1. This is a recurring point of confusion and pain for internal developers (see this slack thread for the most recent discussion)
2. We want paying customers to get their bits from trusted sources. That means domains owned by Teleport, and artifacts signed by our official release infrastructure.

Workaround

Download bits from s3://dronestorage.gravitational.io and re-upload them to a shared space (e.g. the community slack, or any bit bucket). Give the user a link.

[wadells]

See also: #7691

[espadolini]

Any dev release that gets promoted in Drone but doesn't have a matching (pre)release entry on GitHub will not show up in the version list on https://goteleport.com/download but will be downloadable through the usual https://get.gravitational.com/teleport-version-variant.foo links, so maybe all we need is an internal searchable list of files that can be downloaded from https://get.gravitational.com/?

[wadells]

Any dev release that gets promoted in Drone but doesn't have a matching (pre)release entry on GitHub will not show up in the version list on https://goteleport.com/download but will be downloadable through the usual https://get.gravitational.com/teleport-version-variant.foo links, so maybe all we need is an internal searchable list of files that can be downloaded from https://get.gravitational.com/?

Houston and get.gravitational.com (in its present form) will be going away as we cut over to @justinas release service work in the coming months:

https://github.com/gravitational/cloud/blob/master/rfd/0004-Release-Asset-Management.md

As such, we probably won't invest in indexing get.gravitational.com further. I think we'll bet better served by making sure dev releases are something the next gen is designed to accommodate.

[justinas]

Adding notes in regards to the new release service.

The release API itself and the CDN are pretty-much functional today, and could accept such builds. The issues that we are yet to fully solve mostly concern Teleport CI integration and reliability of the pipeline.

The API is also quite unassuming (it is very close to a "dumb asset storage") and adding a new type of build ("Teleport ad-hoc debug build") among the existing (Teleport OSS, Teleport Enterprise official releases) would be effortless.

Security

Originally, we intended to use Teleport AAP in a more granular way, to give distinct sets of permissions for build engineers, CI automation, etc., for accessing this API. There is some context about this scattered in:

https://github.com/gravitational/cloud/issues/594
https://github.com/gravitational/cloud/pull/616
#6821

The Teleport issue is still open and a good way to achieve these granular permissions with AAP has not yet been established. Thus for now, the access to the API is binary (allow or deny) and very limited. We intend to further harden this using access requests.

When access requests are implemented, we could open up the permissions to temporarily request this upload access to a broader set of engineers. This should be quite convenient for distributing such debug builds (in order to upload to the "official" bucket, you would have to get an approval from a limited set of admins).

One good thing is that the API itself enforces some safeguards, e.g. existing official releases are generally immutable. So even though we do not have advanced RBAC yet, the scope of possible malicious actions is limited.

Misc

Yet, I think these ad-hoc builds shall be treated differently from the official releases in a few ways. As mentioned in the Slack thread, they should not get listed in any public location. I would also propose some kind of expiration on them, to avoid a temporary build accidentally becoming a permanent solution 🙂 . It'd probably also be good to have some auditing, e.g. keep metadata like "author of the build".

We also have a small command-line client for this API, which aims to be somewhat high-level. IMO it would be best to extend it with a command tailored to such debug builds. That way, the engineer would not have to worry about the API data model, categorizing the build correctly, etc.

---

TL;DR: it is completely viable to use the new CDN for this, with some minor security/bookkeeping/convenience improvements.

[espadolini]

Would they still be built by the same CICD pipeline that builds the official releases?

[justinas]

Would they still be built by the same CICD pipeline that builds the official releases?

Yes, currently every git tag, even the ones that do not make it to an official release, are uploaded to this bucket. So the process could be as simple as "convert this tag v9.3.0-dev.123 to a debug release".

[justinas]

Update on this: as of https://github.com/gravitational/cloud/issues/2484 , tags like vX.Y.Z-dev.1 or vX.Y.Z-dev.justinas.1 are both automatically built when pushed to Drone, and published to the CDN when tag is "promoted" to the production environment through Drone (as if doing a normal release). They are hidden from the public downloads pages, but are downloadable by a direct link, e.g. https://cdn.teleport.dev/teleport-v12.0.0-dev.justinas.12-linux-amd64-bin.tar.gz

Note that doing this currently also clobbers Teleport image tags such as vX and vX.Y in container registries, making them point to the dev tag, rather than the latest stable release. There is a pending fix for this: #18833