Revising the release process

[ajjackson]

Discussion with @oerc0122

The existing release process for Euphonic is at https://github.com/pace-neutrons/pace-developers/blob/master/euphonic/development/02_releases.md

• There are some manual steps that could be removed/consolidated
• The test_release action always fails at first while waiting for Conda. It can be a bit unclear what this is for and perhaps should be split into test_PyPI and test_conda workflows

As we move to a meson-based build system and rework the corresponding actions, it might be wise to have a single action which:

• Updates version numbers in euphonic.version, CITATION.cff
• Renames the "Unreleased" CHANGELOG section to reflect new version number and creates a new "Unreleased" section
• Commit these changes to master
• Builds wheels with new version number (which cannot be read from git tag at this point) and tests from wheels
• IF builds and tests are successful, create new tag, push to GitHub
• Create GitHub release from CHANGELOG (Github is quite forgiving about editing this later)
• Push wheels to PyPI

[mducle]

I think the "Push wheels to PyPI" should be a separate manual action because you can only do it once (PyPI will not allow you to modify a release), so there's a high chance something will go wrong even if you make it through all the stages of the action successfully... I'd stop the automatic release action at the GitHub release stage, then manually test the wheels created in the GitHub release and if they were good then upload it to PyPI...

The system I used with libpymcr is to have an action run if there is RELEASE in the PR title and the PR is merged. This creates a GitHub release, then it runs the normal pipeline (which runs the test and creates the wheels) but the normal pipeline then detects it is a release and uploads the wheels to the release when it succeeds. The main probably with this workflow is that you have to merge the PR to trigger the release so if the release action has a bug / fails somehow you have to create a new PR. So, you could do something like this but manually triggered using a workflow_dispatch which takes the version number as an input.

[ajjackson]

Our initial thought was also to be wary of the push to PyPI step, but...

Once we've run a test suite on the already-built wheels, what other reassurance are we waiting for? What is the likely failure scenario that doesn't kill the Action first?

Currently cibuildwheel is used for the manylinux builds; this has a feature to automatically run isolated tests from the wheel after it is built. It could be a good idea to use this for the other platforms as well, and simplify things a bit. (As pointed out by @oerc0122 )

[mducle]

Yeah, I think using cibuildwheel for everything is a good idea. The one time I had to overwrite a PyPI release was due to missing docs so as you say it's quite an edge case. Since I put in the two-step thing there was one time where it caught that there was a missing file needed for a function which was not tested in the CI but which new users would run only on the first run.

[ajjackson]

Here's an idea for a 2-step process:

• set_version

  • workflow dispatch takes target branch and version number as input
  • updates files
  • also sets tag

• release

  • workflow dispatch takes target branch
  • if last commit on branch is not tagged, abort
  • if the tag is not a valid version number, abort
  • build wheels and test them
  • if tests succesful, create githib release and push to PyPI

The key tweaks/improvements here are:

• Simplify releasing from a branch. This makes it easy to backport bugfixes to an older version
• Makes it clear what to do if the build/push step fails: make fixes, move the tag and try again.

We could have the release automatically run on new tags, so that the first attempt at a release is still "one click".

[ajjackson]

If the process is more streamlined, we would be more likely to do "release candidate" versions which are another tool for catching packaging problems.

One issue with the current build setup is that conda-forge builds from the PyPI sdist (a good thing for consistency) and so doesn't readily have access to the test suite that sits outside the source folder. Currently the conda-forge recipe merely checks that the the command line tools can be run with --help (i.e. that some Euphonic modules can be imported). The post-release tests are then the only time a full test suite is typically run on a conda-forge build.

For high-risk releases, a release candidate could help. On conda-forge this is achieved by pushing to a separate feedstock branch, and it is not too easy for users to install a release candidate accidentally: https://conda-forge.org/docs/maintainer/knowledge_base/#creating-a-pre-release-build . It might be a good idea to have the test_conda_forge action able to access/run pre-releases as well as full releases.

[ajjackson]

Another interesting thought for release tests (and maybe regular tests): uv is faster than Pip and its pip-like interface includes a --resolution lowest option that will automatically find the lowest allowed versions of dependencies. If we used that, we wouldn't need to keep a minimum_requirements.txt file in sync with pyproject.toml.

[ajjackson]

One complication: Github deliberately makes it slightly tricky to have actions run in response to an automated push.

peter-evans/create-pull-request#48

It can be done by adding a PAT to secrets, but let's start out with manual triggers and see how it goes.

[ajjackson]

A bit of reflection on pre/post releases and CHANGELOG manipulation with @oerc0122 led to this proposal:

• post-releases are releases that make some helpful tweak to packaging or distribution but do not change the main code. By default pip will not "upgrade" a release to a corresponding post-release, but will prefer to install a post-release over the base release.

  • these should have their own CHANGELOG entry, explaining why they were needed.
  • Release logic should expect to find a consistent release-tag, version numbers and changelog references for the post-release

• pre-releases are unsupported builds for development/testing before an official release

  • they are not routinely used in Euphonic development but should be considered when making changes to build systems etc.
  • they should not have a CHANGELOG entry: changes remain "Unreleased" until they are offically released.
  • Release logic should expect to find notes in the "Unreleased" section of CHANGELOG and leave this intact if the release tag is a pre-release. Other version numbers should be updated.

• If a release attempt has been cancelled, fixed up and re-attempted, there may already be an appropriate CHANGELOG section.

  • In this case, release logic should make sure nothing is in the "Unreleased" section and continue without making changes. If there is something in "Unreleased", the release pipeline should fail and require manual cleanup.

This keeps the logic for checking and manipulating CHANGELOG fairly simple, and means that as a developer the rule is:

• If fixing a release branch after an unsuccessful release attempt, release notes should go in the (already-generated) section that shares a name with the upcoming release tag
• Otherwise, release notes should always be written under "Unreleased"

[ajjackson]

The updated proposal for a "1-step" release action is:

1. Manual dispatch action, providing the source branch and new version number
2. Validate/normalize the proposed version number
3. update euphonic/version.py and CITATION.cff
4. update CHANGELOG according to principles above
5. create local commit/tag but do not push yet
6. build wheel packages and test with cibuildwheel
7. (in parallel with 6) build sdist and test
8. Push commit and tag to Github
9. Create release payload from CHANGELOG and make Github release
10. Push packages to PyPI

It would make sense to provide a boolean "dry-run" option that quits the workflow after step 7.

Overall then this combines to functions of the existing "build wheels and push to pypi" workflow and the (soon to be merged) "set version" workflow.

I had initially thought that the "set version" workflow would remain separate and be called as a step by "the big one"; but that doesn't make it so easy to defer pushing the commit/tag until tests have run. So maybe a single mega-workflow is the way to go.