Improve testability of release workflows

[mx-psi]

During the v0.96.0 release we had some last minute issues that made us have to re-tag. The root cause was the lack of testing when implementing #429: because of it we ran into the issue in the first place and fixing it was also difficult, requiring a trial-and-error process over multiple PRs (see #493, #494 and #496).

To avoid this, it would be helpful to restructure the release process so that we can run more of it on PRs for testing. For example, we could run the prepare job on PRs but skip the release job.

cc @TylerHelmuth @jpkrohling any thoughts?

[TylerHelmuth]

Ya changing this workflow is hard because we use goreleaser-pro, which makes testing in forks/local harder. In the future getting access to a goreleaser-pro key would allow testing the actual release job in a fork.

We could test the prepare job of the release workflow on PRs, but if I recall correctly it is the same as the CI test job.

[mx-psi]

We could test the prepare job of the release workflow on PRs, but if I recall correctly it is the same as the CI test job.

There must be some difference, because CI was passing on PRs but the prepare job was failing on release

[jpkrohling]

I have access to the key and can share the privately for testing purposes.

[TylerHelmuth]

@mx-psi you're right, the release workflow has an extra step in the prepare job to upload the artifacts and thats specifically the step that was failing.

I'd still prefer to enforce testing in forks/locally with the key before merging any more PRs that change the release workflow. I think this is the most realistic testing solution. I can updated some process docs to make fork/local testing a requirement.

[mx-psi]

I'd still prefer to enforce testing in forks/locally with the key before merging any more PRs that change the release workflow.

Makes sense to also do that, though I don't see the harm in also running prepare on PRs if we can do so easily

[TylerHelmuth]

Would we want to run actions/upload-artifact@v4 on each PR tho?

[mx-psi]

Would we want to run actions/upload-artifact@v4 on each PR tho?

I am fine running it if it has a very short lifespan (say, 24 hours). We don't run that many PRs and 24 hours is more than enough for an actual release

[jpkrohling]

How about a nightly build for those ones, perhaps with a "dispatch workflow" trigger? They take a long time, and they are risky only if changed. Ensuring on a daily basis that they run, and providing a way to run them on demand would keep the risks low, while not increasing the CI time for PRs.

[TylerHelmuth]

I like that idea. We could tag the image with -nightly or something like that.

[mx-psi]

That works for me as well, it's a reasonable compromise

[Enzujp]

Hello everyone @mx-psi @jpkrohling @TylerHelmuth, I was assigned to issue #498 above, but it's dependent on #497 as it can only be done with GoReleaser Pro and the key is not readily available. Regardless, I'd love to see the implementation of it when it's done, and if I can help out with it - I'd love to. Thank you .