x/build/cmd/relui: create proof of concept for release automation #40279

toothrot · 2020-07-17T22:22:59Z

As part of an effort to improve the reliability and reduce the complexity of Go releases, we should automate more of the release process.

A possible solution is a hosted release management tool (relui) that is responsible for the scheduling and operating of the release, much as releasebot handles this process on the CLI. We can improve the observability of the process by using a persistent web UI for coordinating releases.

As a proof of concept, relui acts as the coordinator (and currently the task executor) for release tasks.

Rough Architecture

For the proof of concept, there will be a single application for both starting and running workflows. However, careful API boundaries should be maintained between the scheduler process and the runner processes to allow for separation in the future.

In general, the runner process should only know how to execute specific tasks without broader context, and report back to the scheduler. Tasks should have an associated type, and the runner should decide whether or not it has an implementation for a given task type.

Tasks, or BuildableTasks, have a shared configuration. All tasks have a type, a name, an optional task name to depend on, and an artifact URL. If a task depends on another named task, it will be provided with that task's artifact URL upon starting.

Finally, workflows should be described through a configuration, in order to allow us to share steps between workflows, and separate concerns between the implementation and the definition of all steps that must be completed for a release. This is especially important for steps that may need to run on different platforms (outside of our tooling on GCP, such as the signing process).

Rough initial workflow for a local workstation release:

Clone repo @ ref
In:

Repo
Ref

Out:

Tarball of Go src
Tarball URI

Run make.bash
In:

Tarball URI of Go Src
Out:
Tarball URI of Go src after make

Build Race Detector
In:

Tarball URI of Go src after make
Out:
Tarball URI of Go src after race build

Clean (version.cache, pkg/bootstrap, race for other GOOS/GOARCH, pkg/GOOS_GOOARCH/cmd)
In:

Tarball URI of Go src after race build
Out:
Tarball URI of Go src after cleanup

Run all.bash
In:

Tarball URI of Go Src after cleanup
Out:
Tarball URI of Go src after all.bash

Finalize
In:

Tarball URI of Go Src after cleanup (not all.bash)
Out:
Tarball URI of binary release

Tasks (remaining as of 2020-10-05)

relworker

(bootstrap worker) Subscribe to a PubSub Topic

A worker can connect to PubSub and subscribe to the configured topic. Messages should be subscribed to using the Receive API, which handles spawning goroutines for handling the message, as well as auto-extending the Ack deadline while a message is being processed.

For now, we can just log that we got the message,can handle it, and Ack it.

See: https://pkg.go.dev/cloud.google.com/go/pubsub#Subscription.Receive

Publish status on accepting a task

When the worker picks up a task, it should update the status of the task in relui as started.

Run FetchGitSource task

The FetchGitSource task should fetch the specified Git repo at the configured ref. The source should be tarred, record the artifact URL to relui, and mark the task as complete.

On gitiles, there is an +archive URL for this task:

Web: https://go.googlesource.com/go/+archive/refs/heads/master

Archive: https://go.googlesource.com/go/+archive/refs/heads/master.tar.gz

Handle non-transient errors on FetchGitSource

If a permanent error occurs when executing a task, the message should be ACK’d to prevent retries, and a terminal status for the task should be reported back to relui.

relui

Create Status API (gRPC server)

A worker can communicate the status of a task back to the coordinator as it progresses on a task. The initial API should at least be able to mark a task as started.

Host gRPC and HTTPS on the same port (with some caveats)
1. See caveats here: document what the server's ServeHTTP is missing compared to the Serve method grpc/grpc-go#2662 (comment)
Host gRPC on same service but different port as HTTPS relui web
Use separate instances of relui for gRPC

Datastore integration

Currently, relui commits state to disk. It should have a persistent database for handling multiple instances, like Datastore.

/cc @dmitshur @cagedmantis @andybons

The text was updated successfully, but these errors were encountered:

gopherbot · 2020-07-17T22:25:44Z

Change https://golang.org/cl/243340 mentions this issue: cmd/relui: add proto definition for workflows

gopherbot · 2020-07-17T22:25:45Z

Change https://golang.org/cl/243339 mentions this issue: cmd/relui: enable creation of mock workflow

gopherbot · 2020-07-17T22:25:46Z

Change https://golang.org/cl/243338 mentions this issue: cmd/relui: add relui webserver for releases

gopherbot · 2020-07-28T16:22:32Z

Change https://golang.org/cl/245277 mentions this issue: maintner/maintnerd/maintapi/version: support beta and RC release tags

This commit introduces relui, a prototype for running releases in a persistent server environment. The primary goals of using a long-running service for release management is to improve observability, scheduling, and testing of our release process. This change introduces a very basic webserver, and minimal styling. For golang/go#40279 Change-Id: I5958e5dc19e62df92b36c64583058cbcef8dd75c Reviewed-on: https://go-review.googlesource.com/c/build/+/243338 Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Andrew Bonventre <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>

This change introduces mock data, multiple templates, and an in-memory store to the release automation webserver. The goal of this change is to introduce a basic UI structure. The underlying data infrastructure is only for mock purposes, and will be replaced in a future CL. This change enables creation and viewing of an in-memory workflow, with very minor data stored. List screenshot: https://storage.googleapis.com/screen.toothrot.net/pub/2020-07-17-workflow-list.png New workflow screenshot: https://storage.googleapis.com/screen.toothrot.net/pub/2020-07-01-11_23_17-workflows-new.png For golang/go#40279 Change-Id: Id9dfcc01cb2aba1df3e36d7a6301bbf8b47476da Reviewed-on: https://go-review.googlesource.com/c/build/+/243339 Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Andrew Bonventre <[email protected]>

Using configuration for our workflows will help separate concerns between implementation and workflow configuration. Eventually, similar tasks can be re-used in different workflows, such as fetching from Git, updating the VERSION file, or publishing a tag. The current configuration definition is mainly illustrative, and is expected to change as we build out a prototype. For golang/go#40279 Change-Id: I5c6f8a18571ab819de0b1d026c86050735efeed9 Reviewed-on: https://go-review.googlesource.com/c/build/+/243340 Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Andrew Bonventre <[email protected]>

gopherbot · 2020-07-29T22:22:59Z

Change https://golang.org/cl/245645 mentions this issue: cmd/relui: add package comment

Copy the description of relui to a package comment. Adjust README formatting so the header shows up as expected. For golang/go#40279. Change-Id: I74b9ee7ff67c94c885aea5231393928e8b5be647 Reviewed-on: https://go-review.googlesource.com/c/build/+/245645 Reviewed-by: Alexander Rakoczy <[email protected]>

gopherbot · 2020-07-31T22:21:21Z

Change https://golang.org/cl/246298 mentions this issue: cmd/relui: add persistence to development database

This change renames memoryStore to fileStore, and persists data created in the UI. This will help make local development less painful when testing changes. Data stored is intended to be loaded at start-up, which will be implemented in a future change. For golang/go#40279 Co-authored-by: Carlos Amedee <[email protected]> Change-Id: Id2390c35b8e1d1d368fbf7ac13b3cdef0776ad87 Reviewed-on: https://go-review.googlesource.com/c/build/+/246298 Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Andrew Bonventre <[email protected]>

gopherbot · 2020-08-26T21:01:34Z

Change https://golang.org/cl/250917 mentions this issue: cmd/relui: load development state on boot

When relui starts, it will look in dev-data-directory for data persisted from the last boot. This enables local development to continue when testing changes manually, building on CL 246298. For golang/go#40279 Change-Id: I02f8b6e1178f82425cafcd2a0544327ba84e028e Reviewed-on: https://go-review.googlesource.com/c/build/+/250917 Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Carlos Amedee <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

gopherbot · 2020-09-24T21:26:23Z

Change https://golang.org/cl/257237 mentions this issue: cmd/relui: create pubsub topic on start

gopherbot · 2020-09-24T21:26:24Z

Change https://golang.org/cl/257239 mentions this issue: cmd/relui: publish task start message to pubsub

gopherbot · 2020-09-24T21:26:24Z

Change https://golang.org/cl/257238 mentions this issue: cmd/relui: generate uuid for workflow and task instances

toothrot · 2020-09-25T18:34:10Z

The next step before moving forward needs to be guarding routes for specific HTTP methods. Capturing this from a comment in https://golang.org/cl/257239.

This will create and connect to a pubsub topic for communicating with relui workers on application start. If the topic already exists, it will just get a reference to the topic. For golang/go#40279 Co-authored-by: Carlos Amedee <[email protected]> Change-Id: Ic173212cd15562b9d1a1cc601d307d5ee1a4e811 Reviewed-on: https://go-review.googlesource.com/c/build/+/257237 Trust: Alexander Rakoczy <[email protected]> Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

Sets workflow and tasks IDs when a workflow is created in the UI. These IDs will be used in the future when operating on workflows and tasks via the UI or API. This ID will likely change to a datastore ID after datastore integration is added. For golang/go#40279 Co-authored-by: Carlos Amedee <[email protected]> Change-Id: Ib75c00c234d156cc1bbdab8c658281f04914165b Reviewed-on: https://go-review.googlesource.com/c/build/+/257238 Trust: Alexander Rakoczy <[email protected]> Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

This change adds a new handler for starting a task by sending a message to pubsub. A test pubsub server is used in unit testing. The exact message used to start a task, as well as updating a task to mark it as started, is still under design and likely to change dramatically. This change is intended to unblock worker development. For golang/go#40279 Co-authored-by: Carlos Amedee <[email protected]> Change-Id: I59bb5c5261a9a5d17e52597c1835a2a980cf91f8 Reviewed-on: https://go-review.googlesource.com/c/build/+/257239 Trust: Alexander Rakoczy <[email protected]> Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

gopherbot · 2020-12-04T02:20:23Z

Change https://golang.org/cl/275236 mentions this issue: cmd/relworker: add relworker command

This adds a basic relworker command. relworker subscribes to the relui queue, and prints messages. Eventually, it will be responsible for executing the release workflows. For golang/go#40279 Change-Id: I6a2fd228e20b7074daa05df56470a8c95c1a8462 Reviewed-on: https://go-review.googlesource.com/c/build/+/275236 Trust: Alexander Rakoczy <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>

gopherbot · 2021-02-02T22:36:08Z

Change https://golang.org/cl/275237 mentions this issue: cmd/relui: add datastore support

gopherbot · 2021-10-08T21:34:27Z

Change https://golang.org/cl/354758 mentions this issue: cmd/releasebot: create GitHub milestones for next release

The current release process relies on humans to create GitHub milestones for future Go releases. releasebot includes automated checks that detect when that step is forgotten. CL 294249 included one of those checks, and its commit message mentioned: (Future release process improvements may include automatically making the milestone. That is better suited to be in scope of golang/go#40279.) This is the release process improvement that automates making milestones. For golang/go#40279. Change-Id: I8e02aff6714a5cf2de4ee4bcfe98aaf68abb0cd4 Reviewed-on: https://go-review.googlesource.com/c/build/+/354758 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]> Trust: Dmitri Shuralyov <[email protected]>

gopherbot · 2021-10-26T20:22:46Z

Change https://golang.org/cl/358899 mentions this issue: cmd/releasebot: add modes for making a release tweet programmatically

gopherbot · 2021-10-26T20:22:48Z

Change https://golang.org/cl/358897 mentions this issue: internal/task: add task to tweet about Go releases

gopherbot · 2021-10-26T20:22:50Z

Change https://golang.org/cl/358898 mentions this issue: internal/task: post tweets via Twitter API

This CL adds an internal API for making release tweets of various kinds, including the templates used to generate the tweet text and tweet image. It does not have code for using the Twitter API. CL 358898 will do that. For golang/go#40279. For golang/go#47403. Change-Id: Ic0cb2f5f1e59c0fd932aed7c5c4a88decc5a3d8d Reviewed-on: https://go-review.googlesource.com/c/build/+/358897 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Trust: Dmitri Shuralyov <[email protected]> Trust: Alexander Rakoczy <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>

Use the Twitter API to post tweets with the generated text and image. Credentials are fetched from secret manager, and OAuth 1.0a is used for authentication during Twitter API calls¹. Tested with the real Twitter API (to be sure it would work). ¹ https://developer.twitter.com/en/docs/authentication/oauth-1-0a For golang/go#40279. Fixes golang/go#47403. Change-Id: I69d0798a69c8e5a0ae8b150dd9d7673c86bdf849 Reviewed-on: https://go-review.googlesource.com/c/build/+/358898 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Trust: Dmitri Shuralyov <[email protected]> Trust: Alexander Rakoczy <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>

This allows exercising (and benefiting from improvements offered by) the programmatic tweet creation tasks as soon as the next minor release. The same task will be added and used by relui when possible. For golang/go#40279. Updates golang/go#47403. Change-Id: I8b47f86489e753bea6a767990ff98dd525e668df Reviewed-on: https://go-review.googlesource.com/c/build/+/358899 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]> Trust: Dmitri Shuralyov <[email protected]> Trust: Alexander Rakoczy <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]>

gopherbot · 2022-03-15T20:34:58Z

Change https://go.dev/cl/393018 mentions this issue: internal/task: remove repetition in major Go release tweets

There was accidentally one too many "go" in the tweet text for the major Go release type. Updates golang/go#47403. Updates golang/go#40279. Change-Id: I7983bdb78f22c258121348a6cfaa2ed66592db54 Reviewed-on: https://go-review.googlesource.com/c/build/+/393018 Trust: Dmitri Shuralyov <[email protected]> Run-TryBot: Dmitri Shuralyov <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]>

gopherbot · 2022-05-06T20:41:18Z

Change https://go.dev/cl/404535 mentions this issue: cmd/relui: add mail-dl-cl workflow

Previously, this task was exposed only via the cmd/releasebot CLI. Now that relui workflows support slice of string parameters, it's easy to expose this task via a relui workflow too. This is another step in having relui do more release work. For golang/go#40279. Change-Id: Icf94e49ac8124ffb77a78b80c65c477d6b43bf05 Reviewed-on: https://go-review.googlesource.com/c/build/+/404535 Reviewed-by: Dmitri Shuralyov <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Dmitri Shuralyov <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]>

gopherbot · 2022-06-14T23:27:44Z

Change https://go.dev/cl/412176 mentions this issue: internal/relui: simplify rendering of homepage

This refactors the homepage rendering to use a nested template for each task row. This will help simplify the template as we add more complex layout to the outputs of workflows and tasks. Updates golang/go#51797 Updates golang/go#40279 For golang/go#53382 Change-Id: I85a86b82bdc79c7fb4e837d884af922c7028295d Reviewed-on: https://go-review.googlesource.com/c/build/+/412176 Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Alex Rakoczy <[email protected]>

gopherbot · 2022-07-28T00:39:21Z

Change https://go.dev/cl/419431 mentions this issue: cmd/releasebot: delete

By now we've started using relui for all Go releases, and don't expect to need releasebot anymore. It's not being maintained to keep up with changes to the release mechanics, and some of the early communication tasks integrated into it are getting in the way of API simplifications. So, delete it. We thank releasebot for its service over the years and many Go releases it helped us successfully issue. For golang/go#40279. Change-Id: I405820f95315f1270534f5c188e3e6aa68ebd361 Reviewed-on: https://go-review.googlesource.com/c/build/+/419431 Auto-Submit: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Run-TryBot: Dmitri Shuralyov <[email protected]> Reviewed-by: Jenny Rakoczy <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

gopherbot · 2023-02-23T16:16:24Z

Change https://go.dev/cl/470755 mentions this issue: cmd/release: deprecate

gopherbot · 2023-02-23T16:16:26Z

Change https://go.dev/cl/470756 mentions this issue: cmd/release: delete

Relui replaced release since Go 1.18 or so. We kept it around a bit longer while there were remaining Go+BoringCrypto releases that still used the release command. Those are no more, so to avoid confusion, remove cmd/release code given it's no longer maintained or supported. Updates golang/go#40279. Updates golang/go#58659. Fixes golang/go#45893. Change-Id: Id0d641bee49c9584c52e5616322f0656b89cd851 Reviewed-on: https://go-review.googlesource.com/c/build/+/470755 Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]> Reviewed-by: Carlos Amedee <[email protected]> Run-TryBot: Dmitri Shuralyov <[email protected]>

Delete the empty command after some time passes. Updates golang/go#40279. Change-Id: I4064e4ba329d9a3637459816420603813652ebe2 Reviewed-on: https://go-review.googlesource.com/c/build/+/470756 Run-TryBot: Dmitri Shuralyov <[email protected]> Reviewed-by: Carlos Amedee <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

toothrot added the Builders x/build issues (builders, bots, dashboards) label Jul 17, 2020

toothrot added this to the Unreleased milestone Jul 17, 2020

toothrot self-assigned this Jul 17, 2020

dmitshur added the NeedsFix The path to resolution is known, but the work has not been done. label Aug 26, 2020

dmitshur mentioned this issue Sep 10, 2020

x/build: consider pushing open backport issue milestones automatically at start of each month #41323

Open

dmitshur mentioned this issue Sep 21, 2020

x/build/cmd/relui: consider patch version updates to dependencies that become available before minor Go releases #41538

Open

dmitshur mentioned this issue May 24, 2021

x/build/cmd/release: darwin-amd64 release tests aren't passing, cmd/go's TestScript/list_std_stale fails #46292

Closed

dmitshur mentioned this issue Apr 13, 2022

all: provide macOS downloads for Go 1.18.1 #52317

Closed

dmitshur mentioned this issue Jun 13, 2022

x/build/cmd/releasebot: build releases regularly, automatically #29205

Closed

heschi moved this to Planned in Go Release Sep 27, 2022

heschi added this to Go Release Sep 27, 2022

toothrot closed this as completed Nov 30, 2022

Repository owner moved this from Planned to Done in Go Release Nov 30, 2022

golang locked and limited conversation to collaborators Feb 23, 2024

gopherbot added the FrozenDueToAge label Feb 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build/cmd/relui: create proof of concept for release automation #40279

x/build/cmd/relui: create proof of concept for release automation #40279

toothrot commented Jul 17, 2020 •

edited

Loading

gopherbot commented Jul 17, 2020

gopherbot commented Jul 17, 2020

gopherbot commented Jul 17, 2020

gopherbot commented Jul 28, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 31, 2020

gopherbot commented Aug 26, 2020

gopherbot commented Sep 24, 2020

gopherbot commented Sep 24, 2020

gopherbot commented Sep 24, 2020

toothrot commented Sep 25, 2020 •

edited by dmitshur

Loading

gopherbot commented Dec 4, 2020

gopherbot commented Feb 2, 2021

gopherbot commented Oct 8, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Mar 15, 2022

gopherbot commented May 6, 2022

gopherbot commented Jun 14, 2022

gopherbot commented Jul 28, 2022

gopherbot commented Feb 23, 2023

gopherbot commented Feb 23, 2023

x/build/cmd/relui: create proof of concept for release automation #40279

x/build/cmd/relui: create proof of concept for release automation #40279

Comments

toothrot commented Jul 17, 2020 • edited Loading

Rough Architecture

Rough initial workflow for a local workstation release:

Tasks (remaining as of 2020-10-05)

relworker

(bootstrap worker) Subscribe to a PubSub Topic

Publish status on accepting a task

Run FetchGitSource task

Handle non-transient errors on FetchGitSource

relui

Create Status API (gRPC server)

Datastore integration

gopherbot commented Jul 17, 2020

gopherbot commented Jul 17, 2020

gopherbot commented Jul 17, 2020

gopherbot commented Jul 28, 2020

gopherbot commented Jul 29, 2020

gopherbot commented Jul 31, 2020

gopherbot commented Aug 26, 2020

gopherbot commented Sep 24, 2020

gopherbot commented Sep 24, 2020

gopherbot commented Sep 24, 2020

toothrot commented Sep 25, 2020 • edited by dmitshur Loading

gopherbot commented Dec 4, 2020

gopherbot commented Feb 2, 2021

gopherbot commented Oct 8, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Oct 26, 2021

gopherbot commented Mar 15, 2022

gopherbot commented May 6, 2022

gopherbot commented Jun 14, 2022

gopherbot commented Jul 28, 2022

gopherbot commented Feb 23, 2023

gopherbot commented Feb 23, 2023

toothrot commented Jul 17, 2020 •

edited

Loading

toothrot commented Sep 25, 2020 •

edited by dmitshur

Loading