Automate release procedure #104
Comments
hannobraun
added
help wanted
|
@jamwaffles suggested |
|
Hey @hannobraun, I'd like to support you with your release process (including #57). I have built https://github.com/hendrikmaus/rust-workflows as a reference implementation to copy and hack on for one's own repositories. It also includes a CI workflow that might be of interest. I am also looking to adopt rust-workflows includes:
As mentioned before, it is made to copy the parts one wants and hack on them. It should make it way faster and easier to setup a complex Rust release process on GitHub Actions. I'd propose to make multiple smaller change-sets that slowly build up the release process. |
|
Hey @hendrikmaus, that sounds awesome! I'm really busy with work on the CAD kernel, and this whole release business is not really my specialty anyway, so I'm really happy to receive help here. I've taken a quick look at your repository, and it does look really promising. Where do you propose we start? Maybe address #57 first, then automate the release procedure starting with creating the release on GitHub?
Yes, I'm all for that 👍 |
|
Yes, let's tackle #57 first. We should be able to store the binaries on each workflow run, using the artifacts feature. That way, no release is needed to host them. |
|
Sounds good! |
|
@jamwaffles made me aware of release-plz: https://github.com/MarcoIeni/release-plz I haven't had the chance yet to check it out, but it seems worth looking into. |
|
Nice, I was researching in the same direction in the last days. I adopted the "original" release please in this crate https://github.com/hendrikmaus/kube-leader-election but, at a glance, the new proposal is even better. Are you also ok with adopting conventional commits? I am trying right now in every project and so far it is beneficial. |
For the following comment, I'm assuming we're talking about this specification. I wasn't aware that existed, but I tried a similar technique a few years back, to generate changelogs automatically. Based on that experience, and a short reading of the specification, I'm against enforcing it (although I have no problem merging commits that follow that format). The specification has a section titled Why Use Conventional Commits. I'm going to go through each of the items to explain my position.
I strongly believe that this is fundamentally the wrong approach. The resulting changelogs are too detailed and not readable. Making them even remotely readable puts too much of a burden on contributors (for creating compliant commits) and maintainers (for policing those commits, to make sure they're compliant). Enforcing the commit format in CI takes care of some of the maintenance burden, but doesn't help make the changelog readable. I see changelogs as documentation, and that's an art, not a science. I summarize and editorialize when writing them. (The current changelog is not the best example of what I'm aiming for, as the last release still covers a lot of ground from the time when I just pushed directly to
That would be great, but that automation is only going to be as reliable as the humans writing the commit messages. And breaking changes can be subtle, easy to miss, and recognizing them reliably would again put too much of a burden on contributors and maintainers. I don't know, if the technology to do this is available, but I'd feel better about an automated scan of the API at release time, to determine if a release is breaking or not.
Worthwhile goals, but there's a trade-off to be made here: Do we optimize for ease of writing, or ease of reading? Personally, I put a lot of care into my commits and pull requests. I make sure my commits are not too big and not too small. I regularly split off commits that are self-contained and centered around a common topic from my WIP branch and submit them separately. While I do that, I modify my history all the time, to make it more clear. (Although I could put more effort into my commit messages. I'm working on it, but sometimes bad habits slip in). If I were paying a highly skilled team to work on Fornjot, I would expect the same level of care from them. I do not expect that level of care from contributors. I don't even expect contributors to realize that putting in that level of care is a possibility. Encouraging people to get involved is essential. Even (and especially!) inexperienced people, who are looking for a project to learn with. In that context, optimizing for ease of writing (within reason) is the right call, in my opinion.
Sounds like a good idea, especially in the context of this issue. We wouldn't need to adopt the whole spec to do this, though. This has turned into a long comment, but I appreciated the opportunity to write down my thoughts on the matter. It's always nice to have something to refer back to, if similar suggestions pop up. Maybe I'll even turn this into a blog post for the Fornjot website 😁 If you think I'm wrong, I'd be happy to be convinced otherwise. But I do feel strongly about this topic, and I think the arguments I've made are sound. |
|
Thank you for explaining your position, many other maintainers would've simply posted "no" and be done with it. I appreciate you took the time. Also, I agree with you on a lot of points.
Yes, that was also my thought after reading through your comment. I am currently challenged by getting an overview of all the tools and their opinions. Composing them to create one's own workflow is not a trivial undertaking. Each tool makes certain assumptions about the overall process which might not be matching every single time. I will keep experimenting a bit on other projects as well. This is a highly iterative process for me. Back to Fornjot, I think we should be able to build a neat pull-request based process. I will keep working on it. |
|
Thanks, @hendrikmaus! I appreciate you taking the time to figure this stuff out. This is the kind of thing I simply wouldn't have the capacity to do. Just a reminder, this doesn't need to be perfect. If one tool provides 80% of the solution, that might be worth adopting. Same for a homegrown alternative. I'm happy to merge anything that's an improvement over the current state. Further improvement later is always possible. |
|
So I have been thinking and experimenting a bit. I'd like to offer my current conclusion as a proposal before starting to make any changes. All holistic tools I've tried are too opinionated about the process and impose restrictions that do not fit Fornjot. I did a small case study to get a feel for writing a custom solution in the GitHub Actions context; I am quite pleased with the idea: I want to write a small Rust command-line application that lives in
If all of the above can be asserted, a release shall happen. E.g. Now, since we're in the context of GitHub and GitHub Actions, we can fully embrace it on the the implementation details:
This keeps the manual steps in place, like maintaining the changelog and release announcements, while allowing the workflow to take over more and more steps in an automated fashion. Looking at the current Release Procedure, here is a patch to it: 1. [manual] Create release branch: git switch -c v0.1.2
2. [manual] Update changelog
3. [manual] Write release announcement
4. [manual] Update versions
- 5. [manual] Publish the release
+ 5. [manual] Open a release pull-request
+ label it as `release`, merge it with a message like `release v0.1.2`
- 6. [manual] Tag the release: git tag v0.1.2
- 7. [manual] Create release on GitHub
+ 6. [auto] Create release on GitHub (includes a new `tag`)
+ 6.1 [auto] Add relevant portion of the changelog to the release body
+ 7. [auto] Compile binaries and attach them to release incl. checksums
+ 7.1 [auto] Publish to crates.io
+ 7.2 [auto] Build and push Linux container image
8. [manual] Finish publishing release announcement
9. [manual] Promote release announcement
10. [manual] Improve release procedureSteps 8. and 9. also call for automation potential at a later stage. |
|
Thank you @hendrikmaus, this looks really good. I'm not surprised that the existing tools wouldn't work for us. Your assessment that they are too opinionated matches my own impression (which, to be honest, is based on years-old experience and little up-to-date research). The concept you present here is fully in line with my own (rather superficial) thinking on the topic. Consider the concept approved, as far as I'm concerned 😁 I've left some comments below. I don't feel strongly about either of them. Use or ignore them, as you prefer.
Is there a specific reason that it should live in (And
Can we skip that step and just check for pull requests? Unless there's a reason to keep the matching of commits to pull requests to a minimum (API rate limiting, or whatever), this step seems redundant.
Plus, we'll have downloads on the website at some point, so there will definitely be more opportunities for automation. One additional note: #141 is going to start happening in the near-ish future, which would mean we'd have more crates to release. In contrast to the two crates we have currently, I wouldn't expect all of them to follow the same release lifecycle. There's always a tradeoff between monorepos and splitting things up. It has been my experience that monorepos become cumbersome once you have crates in them that release at different cadences, and have different version numbers. I expect that the same problem would apply to this system. Ideally, we'd release all crates in this repository at the same time, with one shared version/changelog/tag/release/announcement/etc. Any crates I'd like to move to separate repositories, are probably going to be lower-level dependencies that are on a slower release cadence than the main body of Fornjot. I don't think this is going to be a problem, as this repository (including the automated release system) can just treat them like any other dependency. Even though I don't think this will affect the automated release system, I wanted to give you a heads up, in case you have thoughts (especially if they contradict my own). |
|
Thanks for your feedback, I'll start to submit pull-requests now.
Couldn't have gone better then :)
No, I was sitting on the fence between a dir in the root and in .github; root it is.
Yes, there is no hard requirement for it. In regards to #141: Either way, we can leverage the new release-operator. I would lean to a poly-repo approach as well. |
Great, thank you! 👍
"Great minds think alike", or maybe "zwei Dumme, ein Gedanke". Choose the one you prefer 😁
Sounds great! |
|
@hannobraun which part should we automate next? Publishing releases to crates.io could also be done from within the workflow. It requires a token to be put in the repository secrets though. Should we containerize anything? |
I think publishing to crates.io would be the logical next step, as it's adjacent to the part that's already automated, and should be relatively easy to do. After that, maybe updating all versions and path dependencies? The token doesn't sound like a problem? Just tell me what you need and where you need it.
I don't know what that means, outside the context of deploying backend services.
I have little experience with Homebrew, and no experience publishing to it. If it's easy, then I guess why not. Long-term, I'd like the Fornjot app to be available anywhere people would expect to install their software from. I don't think there's any pressing need right now though. It might even be better to wait for another 2-3 releases. By that time, Fornjot has hopefully approached a minimally useful state for more use cases. |
|
Ok, let's go with publishing to crates.io next then. When you log in to crates.io, you can create API tokens in your profile. Generate one and put it into the Actions Secrets of the Fornjot repository. The name of the secret, usually all uppercase, could be something like I would make the workflow idempotent on the version - i.e. I'd query crates.io before publishing a version to determine if it is already there. Otherwise, the call to publish a crate will fail. Could you provide a list of crates to publish? I would then build the workflow part to iterate over the list. Would you like more bash or should we start to extend the release-operator? I would lean for the latter. We have the base in-place and could simply start adding sub-commands using clap. The first sub-command would be the one for detecting the release (i.e. what a simple cargo run does right now) and the second one would be the publish routine. I reckon we can skip Linux containers and Homebrew for now. To cover everyone's preferred package manager and operating system will require community effort. I do not see that as a responsibility of the upstream project. |
I created a token, and added it as the repository secret There are also "Environments" in the settings, which seem to be another way to add secrets. I don't quite grasp the difference, but it seemed like a repository secret would be the way to go.
Sounds good 👍
This is subject to change, as I'm still working on #141. For now, this is the list:
For now, a static list is good enough, but eventually, I'd like a more maintainable solution. One option would be to use the Cargo workspace to generate the list (slap a A better option might be to move all public crates into a
I would also prefer to extend release-operator. It's already there, so I see no reason to build some of the infrastructure in Bash.
Well, since I'm trying to make a living off this project, my perspective is a bit different. If a broad packaging effort can help promote Fornjot in a meaningful way, then I might make it my responsibility. But in any case, it's premature. For now, I wouldn't discourage anyone from packaging Fornjot, but I don't see a reason to encourage it either. |
That's perfect. Aside: The "environments" feature is supposed to aid projects that have different release tiers, e.g. edge, stage, prod. And each of those have different secrets.
We could start with a repeatable cli option, provided to the release-operator call, e.g. Another way would be to use the [package.metadata.release-operator]
# ...One could place any number of fields in each
Makes sense. I would then propose to start a list in a separate issue. A good start would be:
|
Ah, makes sense.
All of that makes sense. I just worry that we'll be adding small crates somewhat regularly (as functionality grows, and it makes sense to split something out), and then it'll be a pain doing a release each time, because I forgot to add something or other to the list. But that's probably bullshit. Starting with an explicit list is definitely fine, and if problems show up, we can adjust.
I think opening an issue would probably be premature, but I'll add it to the feature wishlist and link to your comment. |
|
@hendrikmaus I'm starting the process of publishing the next release now. Looking forward to see how it goes! |
|
Update: Still working on the release. First the changelog took forever. Then writing the announcement took forever (not completely done yet). Then other stuff got in the way (spent a bunch of time reviewing a pull request today). Haven't gotten to the automated portion yet. |
|
I attempted the release in #524. Didn't work, unfortunately. |
|
Ah, I think I messed up the commit message. The version number I included was missing the 'v' prefix. |
|
Another attempt in #526. Same result: https://github.com/hannobraun/Fornjot/runs/6309044619?check_suite_focus=true#step:5:129 I think we need more logging in |
|
You can use RUST_LOG for more details.
…On 5. May 2022, 17:31 +0200, Hanno Braun ***@***.***>, wrote:
Another attempt in #526. Same result: https://github.com/hannobraun/Fornjot/runs/6309044619?check_suite_focus=true#step:5:129
I think we need more logging in release-operator.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
How so? It doesn't seem to emit any log messages during the release detection procedure. I've added log messages with more information here: #527 |
|
Alright, I didn’t have the code in mind any more. I am actually sick at the moment and cannot be of more help 😐
…On 5. May 2022, 17:39 +0200, Hanno Braun ***@***.***>, wrote:
> You can use RUST_LOG for more details.
How so? It doesn't seem to emit any log messages during the release detection procedure. I've added log messages with more information here: #527
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
No problem, hope you get well soon! Okay, forgot to add the |
|
Another attempt in #528. It's not finding the version number in the commit message: https://github.com/hannobraun/Fornjot/runs/6309497951?check_suite_focus=true#step:5:129 Seems like it's only looking at the first line, but the merge commit has the version in line 3. Side note: Something's not working correctly with the caching of the Release job. It's re-downloading and re-compiling a bunch of dependencies each time, even though all I'm doing is pushing empty commits. |
|
IIRC the version has to be in the first line per docs.
On 5. May 2022, 18:02 +0200, Hanno Braun q ***@***.***>, wrote:
Another attempt in #528. It's not finding the version number in the commit message: https://github.com/hannobraun/Fornjot/runs/6309497951?check_suite_focus=true#step:5:129
Seems like it's only looking at the first line, but the merge commit has the version in line 3.
Side note: Something's not working correctly with the caching of the Release job. It's re-downloading and re-compiling a bunch of dependencies each time, even though all I'm doing is pushing empty commits.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
The docs don't specify that actually. But can I even influence the first line of the merge commit? I'm not sure, as I always click that text field away so quickly. I'll have to pay more attention next time. But it doesn't matter, I think. I've modified |
|
Did you use the correct label on the release pull-request? Every release commit needs to belong to a pull-request labeled as defined in the GitHub actions step.
…On 5. May 2022, 18:22 +0200, Hanno Braun ***@***.***>, wrote:
The docs don't specify that actually. But can I even influence the first line of the merge commit? I'm not sure, as I always click that text field away so quickly. I'll have to pay more attention next time.
But it doesn't matter, I think. I've modified release-operator to search the whole commit. Still no luck: https://github.com/hannobraun/Fornjot/runs/6309787693?check_suite_focus=true#step:5:129
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Yes. It's getting past that point in the Turns out you can control the whole commit message. I just didn't notice because I've trained myself for years to click it away immediately. Added more logging to |
|
Nice! This did it: 0d847eb Also, this: cfc9743 That second change wasn't critical, I think, but it made a difference (before, after). This got me a few steps further. New failure here: https://github.com/hannobraun/Fornjot/runs/6310273848?check_suite_focus=true#step:9:24 |
|
For the record, extracting the release announcement from the website, like we talked about, works. Here's what I did:
It's getting late, so I'm doing the |
|
I just got this while publishing locally: Not sure, if this is due to the crates I actually published, or if it takes into account the recent failed attempts. If the former, this is something we'll have to deal with to get the automation working. |
|
I've opened issues for the problems I encountered: |
|
I've published a new release! Overall, the experience was still mixed, but we're making progress! Previously we discussed further automating the release procedure, by downloading the release announcement from the website. The approach we discussed is to download the HTML and processing it. I've run into new problems with doing that manually:
That whole approach of downloading and processing the HTML is probably flawed. It would be better to somehow download the original Markdown instead, which I don't think Zola can do without additional manual steps. And additional manual step render the automation moot. Unless someone can figure out how to do this with Zola, it's probably best to shelve this part of the automation until we have something more flexible to generate the website (which I don't have concrete plans for right now). Problems encountered this time:
Plus a few more that were unrelated to the release automation (repository settings I messed up, I plan to switch to a weekly release cadence starting the week after next, which should help flush out the remaining problems. |
|
Notes from the latest release, which is the first weekly one:
#788 seems to be the last problem that prevents a smooth release process. I have an idea for how to automate the "Update versions" step. I think this is a prime candidate for automation, as it's right next to the already automated part. How about this:
|
|
In Github Actions, a pipeline cannot trigger other jobs1. Hence, steps 5. and 6. wouldn't run. I think it is a good idea. It is similar to what the Rust feature in Google's release-please does. Footnotes
|
|
I just published the latest release. No problems this time! Just some good old-fashioned robotic efficiency. This leaves the lack of further automation as the most annoying aspect of the process 😄 I personally plan to create some tooling over in the website repository to assist writing the release announcement (generating the announcement from the template, getting the list of pull request from GitHub, getting the list of sponsors, stuff like that). Meanwhile, it would be great to see further automation of the release procedure itself. I've had some ideas for how to automatically provide the Markdown source of the release announcement from the website. If that pans out, this would also enable further automation here. Thought: Should we run |
|
I've published two more releases since last posting. I can definitely say that this has become routine now, and as far as I can tell, all the problems of the release automation have been flushed out and fixed. I've started attacking the larger area of release automation from another direction, and have started automating the first steps of the release procedure, writing the release announcement. Not only is this the biggest chunk of manual work, where I see the most low-hanging fruit, but it also paves the way for extending the release automation in this repository. By providing more information about the release through the website, for example, which can then be used on this side. |
|
Thanks for the update 🙏 glad to hear that everything continues to fall into place. As to your question at the end of the previous comment: I guess it is OK to skip the verification. You correct in that the code has been compiled enough times by the time the publishing runs. |
|
Thanks for the comment, @hendrikmaus! I already made the change to skip verification after last week's release (#903). This has accelerated this week's release nicely! |
|
It's been a while since I posted here, so I figured it's time for an update. The release automation has been trucking along without issues. I'm very happy with how everything works! I've also been automating more steps, using a command-line tool that primarily helps me with the release announcement, by creating it from a template, determining the release version, automatically adding all relevant pull requests and generating the list of sponsors. Writing the release announcement is still the most labor-intensive step, but I think it's as automatic as it's going to get. The next logical step is to bridge the gap between the release announcement and the rest of the automation. Right now, there are three steps here:
All of those should be mostly trivial to automate, in principle:
There are a few practical problems, however:
I'm fully confident that I can bend the website and the CLI tool to my will somehow, but I lack knowledge on the GitHub Actions side. I don't know if it can be triggered in a way we'd need it to, and what exactly it can do in regards to updating files in the repository. |
|
I've decided to close this issue. Release automation has long been in place, and this issue has gotten stale. While the release process could be improved and automated further, I think we should track this in more specific issues. In addition, with the end of the weekly release schedule and the new reduced scope (see A New Direction), I think it makes sense to take a step back and re-evaluate what is necessary. I expect to open those more specific issues in the future, as I get a better understanding of what's needed. |
It would be great to largely automate the release procedure.
There are basically just three parts that need to be manual:
It would be great, if somehow we could provide these ingredients in a commit (maybe by adding an entry to a
releases.tomlfile that specifies the version number and links the release announcement), then have an automated action be triggered by that, that takes care of everything else.I'm not sure how practical this is. For example, I don't know if such a trigger can be specified. But it would be great to have.
Labeling as https://github.com/hannobraun/Fornjot/labels/help%20wanted, as I'm no expert in GitHub Actions. If someone who knows more happens to drop by and could leave some insights about the feasibility of this, that would be very appreciated!
The text was updated successfully, but these errors were encountered: