Migrated to hosted Apt and RPM repos

[russjones]

Investigate moving Apt and RPM repos to Google Cloud Artifact Registry.

https://cloud.google.com/artifact-registry/docs/os-packages/debian/apt-quickstart

• Investigate and perform POC with Google Cloud Artifact Registry
• Write RFD. Collaborate with @knisbet @wadells
• Migrate

[wadells]

Some quick thoughts from my research:

1. I'd recommend a 2 stage publishing system, similar to what @webvictim set up with "push to s3://dronestorage.gravitational.io (available internally)" and then later "promote to s3://clientbuilds.gravitational.io (available externally)". I recommend this because if there is only one endpoint it becomes difficult to test changes to the push logic without impacting stuff that customers see. This makes developing and testing changes to the current package release automation risky. Releasing 7.3.10 fails with gpg: cannot open '/dev/tty': No such device or address #9726 is an example of a recent escape in my work due to the difficulty of testing.

2. As discussed in Deb repo inconsistent versions available #8166, we want to provide different channels for our deb repo, corresponding to each major version. We may also want to do minor version channels too -- though I'm not sure they'll be used as much while we have the current paradigm of "no patch releases on X.Y if X.Y+1 is already out". I haven't looked at this on the RPM side, but I can say rpm/dnf defaults behavior isn't as inane as the reprepro tool we were using for debs.

The following concerns are prorbably not needed, but worth thinking about, and explicitly excluding them from scope (if you choose to) in v1:

1. Put out beta channels chat include semver prereleases Create a beta repository for https://deb.releases.teleport.dev/ and https://rpm.releases.teleport.dev/ #7691
2. Consider how the above prelelease channels will interact with master. Where would customers try out 9.0-alpha.1, and how does that relate to our branch structure (since the early releases are usually out before branch/X is cut off master).
3. Uploading packages typically isn't logically coupled to the branch, it should be common logic across all branches. However the creation of packages is tightly coupled to the branch (e.g. see Add a libc requirements to prevent apt/dpkg from updating to broken version of Teleport #9247). Do you want to put an branched:unbranched interface in your solution, or duplicate and backport every change to the unbranched portion?

[wadells]

Also, please review the artifact storage standards defined in https://github.com/gravitational/cloud/pull/896. Consider:

1. Are there ideas that deb/rpm artifact storage has to offer our other repos
2. How do we incorporate best practices regarding audit trails and minimal permissions when working with google cloud artifact registry.

[russjones]

@fheinecke As a first step, let's do the following.

At the moment, deb and RPM repos are in our own S3 bucket and we use tools like createrepo and reprepro to create the repositories respectively. Take a look at the promote-build step of the Drone pipeline to get a better understanding of what we have today. https://github.com/gravitational/teleport/blob/master/.drone.yml#L4795

We want to move off of this to a hosted solution, ideally Google Cloud. They offer hosted deb and rpm repos.

https://cloud.google.com/artifact-registry/docs/os-packages/debian/apt-quickstart

The two most important things to find are the following.

1. Hosting multiple versions of a package for deb and rpm repos. For example, you might have installed Teleport 8, but after a 6.2 patch release was rolled out, you might be accidentally rolled back to 6.2.
2. We need channels for different versions of Teleport. For example, if install Teleport 6.2 and a few months go by, you might be upgraded to Teleport 8 (breaking the compatibility guarantee).

Once we know the answer to those questions, we can see how feasible it is to move to Google Cloud. Another option is Package Cloud if Google Cloud is not suitable.

[russjones]

@knisbet @wadells Just to give you an update @fheinecke has been looking into the Deb/RPM issue for us. We have two questions for you guys.

1. It looks like we can't upload our own signing key to Google Cloud. Google Cloud does it's own key management. This actually might be a good feature, we don't have to do key management ourselves. What do you think?
2. Because the signing key will change, we'll probably want to host the new repos at a different address. This way we won't break old customers using the old repos but new users will come in with the nice updated features. The CDN will need to be updated to support two different repos and different URLs. How hard would it be for the CDN to do this?

@fheinecke Next items for you to also research are.

1. We can setup public Apt/RPM repos after project settings are updated.
2. Verify that Google Cloud RPM support has feature parity to apt support.
3. You would upload multiple *.deb and *.rpm packages with different release channels (7, 8, ...) and setup POC repos for both deb and rpm that the team can play with.

[wadells]

1. It looks like we can't upload our own signing key to Google Cloud. Google Cloud does it's own key management. This actually might be a good feature, we don't have to do key management ourselves. What do you think?

Using GCP KMS will result in better security handling for the key. The downside I see is future proofing: we're giving away control of the keys our customers will use to validate our work. If we ever move away from GCP Artifact Registry, we'll have to perform the same dance again.

2. Because the signing key will change, we'll probably want to host the new repos at a different address. This way we won't break old customers using the old repos but new users will come in with the nice updated features. The CDN will need to be updated to support two different repos and different URLs. How hard would it be for the CDN to do this?

We currently use cloudfront as a CDN in front of S3 for rpm.releases.teleport.dev / deb.releases.teleport.dev / charts.releases.teleport.dev. These bits are not distributed via Houston or the new Releases service. Spinning up a new repo on a different domain shouldn't require any changes aside from customer facing documentation.

[russjones]

Using GCP KMS will result in better security handling for the key. The downside I see is future proofing: we're giving away control of the keys our customers will use to validate our work. If we ever move away from GCP Artifact Registry, we'll have to perform the same dance again.

@fheinecke Correct me if I am wrong, but the client can supply it's own keys is coming in the future right? So I think we're good with this one, let's go with Google controlling the keys for now, then we can investigate our own keys at some point in the future.

[fheinecke]

@russjones To my knowledge GCP has not confirmed that clients can supply their own keys in the future. If we wanted to guarantee that this feature would be available in the future then we'd need to go with another solution like Package Cloud or JFrog Artifactory. That being said, I would be a bit surprised if this was not added as Artifact Registry for deb/RPM goes to general availability.

[knisbet]

One item to watch out for / consider in the scope, is what google cloud project this would live in. My guess is as a artifact host, this would need a fair bit of scrutiny and isolation, and coordination with IT around who has access to the account, how they get access, etc.

[deusxanima]

A few customers have recently raised the debian/ubuntu apt issue again (not able to list multiple versions like they can with yum).

[fheinecke]

Remaining tasks for APT repo:

• Empty apt.releases.teleport.dev bucket
• Load current releases into bucket
• Remove testing code from apt repo build too
• Merge PR
• Internal testing
  • Test promote trigger
• Create into ops repo about DR plan
  • Merge PR
• Create/merge S3/Cloudfront access log monitoring into cloud-terraform repo
• Create PR for updating docs
  • Merge PR

[russjones]

@r0mant @fheinecke Anything left here? If not can we close?

[fheinecke]

All taken care of. New repos are available as detailed here: https://goteleport.com/docs/installation/