Verify release action #977
Conversation
danielrbradley
force-pushed
the
release-verification
branch
from
d4031c4 to
6a1a1f5
Compare
|
About how many CI minutes of macOS runner time would be incurred weekly from running verify release? I imagine we can estimate # of releases weekly * length of verification. I ask because we have a maximum concurrency of 5 macOS runners on OSS repositories and this could result in CI brownouts for pulumi/pulumi. Cc @mjeffryes |
|
This looks like it adds verification after prerelease, so we'd be looking at num_provider_commits * verification_time. We closed 866 PRs in providers last week, most probably were merged and ran prerelease. Even if verification time is < 2min, it's going to burn a fair amount. We could potentially only test windows and mac on a full release to relive some pressure there. |
|
I'll disable verifying pre-releases on Mac for the time being - only verify them on Mac after stable releases. This should suffice but still provide fast feedback on any issues. |
Make release verification opt-in as it needs configuration for the programs to run. Add example configuration as documentation.
danielrbradley
force-pushed
the
release-verification
branch
from
6a1a1f5 to
d844a37
Compare
- Rename tag_sdk to publish_go_sdk to match pre-release (and what we're actually doing). - Add explicit enableMacosRunner input for prerelease for explicit documentation of behaviour.
danielrbradley
force-pushed
the
release-verification
branch
from
d844a37 to
03a93b5
Compare
- Remove unused version action.
There was a problem hiding this comment.
What's the rollout plan here? @danielrbradley will you roll this out to all providers? To tier 1 providers?
Do we intend to roll this out to all providers eventually? If so, we will need to budget for that (not all providers have tests at all right now). If not, then we probably shouldn't warn when a test is missing:
- run: echo "::warning file=.ci-mgmt.yaml,title=Python release verification missing::Add the key releaseVerification.python to .ci-mgmt.yaml pointing to the directory containing a Pulumi python project to preview."
| verify-release: | ||
| name: verify-release | ||
| strategy: | ||
| matrix: |
There was a problem hiding this comment.
It feels like we could expand this matrix to also include language. That would give us parallelism and reduce repeated code.
You might need to build a simple lookup table to map #{{ .Config.releaseVerification.nodejs }}# to nodejs, perhaps using GH action outputs.
There was a problem hiding this comment.
I had considered putting the runtime in the matrix, but given we're only doing a preview, this is a very fast run - most of the time is spent just installing dependencies. We could tweak the way it runs so we still report all the tests even if earlier steps fail if we like.
Initially, this is just rolling out the machinery to be able to start implementing this in each provider. Tier 1 are obviously the priority, but this should really be implemented for all providers given we're not actually deploying any real cloud resources - only validating the SDK and provider setup. If the warnings are too messy then we could tidy these up a bit into a single warning perhaps, but it would make for more messy ci-mgmt code. These warnings are also only visible in the actual release workflow run view, so they really don't actually get in the way too much. |
|
I'd prefer to not introduce a class of warnings unless we plan to address them (immediately), since it builds the habit of ignoring warnings. That said, we already ignore plenty of build warnings, so it's probably not that much worse here. We can always disable them later if we need to. |
Remove printing of warnings and instead just mark the whole job as skipped if not configured. Move the setup instructions into a workflow comment.
|
@iwahbe I've updated the code to completely skip the verify-release job if it's not implemented instead of the warnings |
Add new job in our release workflows to verify that our released SDKs and provider are usable.
This will have to be configured into each provider as we will need a program specified in the local repo for each runtime we want to verify. Ideally this will include: nodejs, python, dotnet and Go. Without configuration, the job will not do any verification. There's an example of the available configuration as documentation in the
bridged-provider.config.yaml.The verify-release workflow is also runnable directly if we want to do a one-off verification. See the pulumi/verify-provider-release action for the underlying implementation.
Fixes #972
Example run without configuration skips the job:
Example prerelease with nodejs configuration runs on Ubuntu and Windows: