Init Change Log fragments

[buberdds]

Opening Draft PR, because I am not sure about requirements. During planning session we were talking about enforcing usage of Change Log fragments for now without other tools we usually combine with it (gitlint/punch etc).
Release process will require running changelog and release-tag with an optional version number.

[mitjat]

Thank you, Michal! For posterity, here's a transcript of a sync on Slack (minus the pleasantries).

Michal

We are aming to use Change Log fragments in our main repos.
I will need to refresh PR and my memory tomorrow, but:

• PR enables use of our fork of towncrier. Basically make changelog has to be run manually to create "assemble PRs" like (https://github.com/oasisprotocol/oasis-core/pulls?q=is%3Apr+assemble+is%3Aclosed) which have to be reviewed. Once merged release process can start
  -PR does not include any other functionality (like punch tool for versioning in python used by other backend repos)
• at some point of a release we cut a release and we trigger workflow like https://github.com/oasisprotocol/explorer/blob/master/.github/workflows/release.yml. It will create a github release based on tag and entries from CHANGELOG.md file. I can see that you have release.yaml which I will have to update.
  My main questions are:
• What needs to be done to replace current changelog in goreleaser with towncrier?
• How your release flow looks like in terms of commands (make commands/git tag push etc)
• Any potential issues we can hit with swapping change log flow?

Mitja

What needs to be done to replace current changelog in goreleaser with towncrier?

I'm not 100% sure; but my impression is that we don't actually use the changelog-generating part of goreleaser anywhere, and it's only used to define how to build the binaries. This was set up before I joined nexus (by Nikhil, long since gone), so I'm no authority there, unfortunately.

How your release flow looks like in terms of commands (make commands/git tag push etc)

We use the GitHub GUI (plus release.yaml in CI/CD). To create a release, we go to https://github.com/oasisprotocol/nexus/releases, click "Draft a new release" in the upper right corner, manually type in the new tag (= the next version), and use GitHub's "generate release notes" GUI button to achieve something similar to towncrier (but based on PR titles, rather than changelog fragments). Then GitHub automatically creates the new git tag at HEAD and triggers release.yaml.

Any potential issues we can hit with swapping change log flow?

In terms of tech: I can't think of any. Our current solution is not particularly customized, so hopefully it will be a smooth and easy transition. But also, let's try and see :), the stakes are low.
In terms of users (= nexus devs): We might request a little handholding (and/or a README) with the first release or two. The towncrier setup is nice in that it's uniformly used across the OPF and in that it differentiates between PR messages and changelog messages; but it's also more complex and higher friction. The good news is that we've all encountered it at least a little when dipping into oasis-core.

Some other questions from my end:

• How do you cut a release in the new flow? At a low level, "cutting a release" means pushing a tag, right? Does a script do that, or do you do it manually?
• Those "assemble PRs": What happens if the "assemble PR" is merged, and then another PR is merged before we push the tag? (Maybe it can't happen because of automation? Maybe we need to be careful when pushing tags to make sure they match the "assemble PR" commit?)
• You say that the "PR does not include any other functionality". Are there functionalities that you expect nexus will want/need in the foreseeable future, or is that just because nexus is a simpler project than oasis-core (and doesn't e.g. use python)?
• Do you expect changes will be needed to our release.yaml? In my mind, that's a step that happens after the release has been cut, whereas we're trying to change HOW the release is cut (and even that mostly just in terms of managing changelog fragments). Ah I guess we do ideally want to get rid of goreleaser, for simplicity.
• What does your extract-changelog-action do? It's called here and I can see here that it extracts the current version's changes from the (presumably recently-updated) CHANGELOG.md, and re-outputs them. But I don't see at a glance what happens with that output.

[buberdds]

How do you cut a release in the new flow? At a low level, "cutting a release" means pushing a tag, right? Does a script do that, or do you do it manually?

In frontend repos we run scripts yarn changelog, yarn release-tag. In backend repos you can find equivalent make targets.

Those "assemble PRs": What happens if the "assemble PR" is merged, and then another PR is merged before we push the tag? (Maybe it can't happen because of automation? Maybe we need to be careful when pushing tags to make sure they match the "assemble PR" commit?)

Release tag validates if there are no change log fragments. Now I can see that Nexus is missing such make target (sample). I think we should add it, but it will require version in CLI as we don't use Punch tool here), so we may end up with something like
make changelog VERSION=0.1.25 to assemble changes and make releae-tag VERSION=0.1.25

You say that the "PR does not include any other functionality". Are there functionalities that you expect nexus will want/need in the foreseeable future, or is that just because nexus is a simpler project than oasis-core (and doesn't e.g. use python)?

Requirement for this task was to force usage of Change Log fragments, but if you look at stack, other repos have some versioning tools like Punch tool. There are some additional linters (markdown or gitlint), make files targets etc. I guess ideal flow is when you work with all repos using more or less the same flow. So yeah, aligning linters and versioning tools would be a next step (Tadej has detailed plan for sure). At the same time such tasks have low prior so it's hard for me to predict when we will invest more time.

Do you expect changes will be needed to our release.yaml? In my mind, that's a step that happens after the release has been cut, whereas we're trying to change HOW the release is cut (and even that mostly just in terms of managing changelog fragments). Ah I guess we do ideally want to get rid of goreleaser, for simplicity.

I think we will only add creating GitHub release with release notes task to the current workflow. goreleaser will stay, but without changelog which is not used from what I understood.

What does your extract-changelog-action do? It's called here and I can see here that it extracts the current version's changes from the (presumably recently-updated) CHANGELOG.md, and re-outputs them. But I don't see at a glance what happens with that output.

You can have simple release notes like in core or you can include all details from CHANGELOG.md (explorer). The problem with detailed option is that md file in repo and release notes parse white spaces differently and that action is cleaning stuff for GitHub release notes. Not sure that GitHub behavior was fixed lately, but you can copy any CHANGELOG.md to GitHub releases editor to see differences.

[mitjat]

we may end up with something like
make changelog VERSION=0.1.25 to assemble changes and make releae-tag VERSION=0.1.25

It would be nice to have a little automation support here. When VERSION is absent, the script could either bump just the patch version (which will be the most common), or at least print the most recent version. We're probably fine to just use server-side git tags, rather than punch? Sth like

git fetch --tags
latest_version="$(git tag --list --merged 'v[0-9]\.[0-9]\.[0-9]' | sort --version-sort | tail -n1)"
next_version="$(echo "$latest_version" | awk 'BEGIN{FS=OFS="."} {print $1, $2, $3+1}')"  # optional
echo "Latest published version is $latest_version. You are about to publish $next_version."
echo "If this is not what you want, re-run this command with VERSION=..."

The rest LGTM, thank you!

[buberdds]

I was wrong about Github release part. Will follow other backend repos here.

[buberdds]

@mitjat (actually anyone who is interested) let's lock some time next sprint for testing this.

I went through "auto bumping patch version" flow on my fork and commands seem Ok.
make changelog
make release-tag

[pro-wh]

Will track this separately.

[mitjat]

Looks nice, thank you!

Yeah let's merge it and test it out before or after one of our team syncs, if that works for you? If it's before, I just need advance warning :)

[buberdds]

let's merge it and test it out before or after one of our team syncs, if that works for you?

let's merge it after a team sync and early next sprint 🖖