Skip to content

release process

Jump to bottom
Daniel Cleyrat edited this page Mar 28, 2022 · 5 revisions

Release Processes

Table of Contents

The Red Hat Marketplace operator uses a branch model called git-flow for release management.

You can read more about git flow here and there is a handy cheat sheet here.

Prerequisites

To perform all the actions for automatic install you will need to have the git flow plugin installed locally.

  1. gitflow

How to set up git flow plugin?

Once the plugin is installed, run this command:

git flow init -d

This will set up the plugin with the default rules we use.

Branches

branch use
master Master branch is current stable version.
develop Develop branch is where all future work is branched from and merged to.
feature/* Feature branches for future work. Start off develop.
bugfix/* Bug fixes for the next release. Lower priority. Branch from develop.
hotfix/* Hot fix for production code. High priority. Branch from master.
release/* Release branches for the next release. Hotfix is also a release branch. Can only be one. Branch from develop.

When enough features are ready, a release branch is created off of develop. Release branch is worked until all testing is completed as a beta release. Once the beta release is stable, the release is merged into master. Merge to master creates the next official stable release.

Hotfixes are started off of master. Bugfixes off of stable.

Automatic Releases

Here is a convenient flow chart to use for releasing:

Automatic Release Flowchart

Release

  1. Sync your master branch.

    git fetch
git checkout master
git pull

  2. Start from 'develop' branch.

    git checkout develop
git pull

  3. Run

    git flow release start $(make current-version)
git flow release publish

    A new branch called release/x.x.x will be made for you and pushed to the repository.

  4. Once the branch is pushed, a PR will be generated for you.

  5. The build will build and push images. Different workflows will generate assets for you to use as well.

    • test-catalog-source.yaml is generated to allow local testing before publishing
    • bundles for submission to Partner Connect are also generated
    • results from operator-sdk scorecard tests

  6. Once the PR checks finish, verify the checks for the PR are passing.

  7. Test the newly generated bundles using the test-catalog-source. For more info see testing latest build.

  8. Test the build against the certification pipeline

    1. Login with kubeadmin context to a certification cluster
    2. Check all nodes are up and healthy oc get nodes
    3. Set the TAG to the build of the release branch
    4. Set the PYXIS_API_KEY
    5. Run the certification pipeline using the script. Accept all defaults when prompted. 
      export TAG=2.5.0-839
export IMAGE_REGISTRY=quay.io/rh-marketplace
export PYXIS_API_KEY=YOURKEY
make operator/test-certify

  9. Resolve any issues reported by the certification pipeline. Troubleshooting

  10. When the release PR is approved and testing complete, publish the bundle by submitting it using the certification pipeline

    1. Authorizing a GITHUB_TOKEN 
      export TAG=2.5.0-839
export IMAGE_REGISTRY=quay.io/rh-marketplace
export PYXIS_API_KEY=YOURKEY
export SUBMIT=true
export GITHUB_TOKEN=YOURTOKEN
make operator/test-certify

  11. The pipeline submit opens a pull request

    1. If the pull request checks pass and is merged, the operator bundle is now published
    2. If the pull request checks fail, inspect for a reason, or open a ticket as appropriate

  12. Merge the release PR.

  13. A new tag with the release version will be created. Create a release in Github for this tag by selecting the tag and creating a release. See 0.1.2 for an example of the content.

  14. A PR to merge master to develop will be gnerated. Merge this PR.

  15. You're done. Spread the good news about a successful release.

Hotfix

Same as release, but change git flow commands to git flow hotfix and start from master branch.

Manual Releases

Prerequisites

To do all the manual actions. You will need to have these additional local commands.

  1. operator-sdk
  2. operator-courier

Release or Bugfix

Warning: to do these steps you need pull/push access to master/develop branch. And admin access to the docker registry. Only to be done if github actions are down.

  1. Start from 'develop' branch.

    git checkout develop
git pull

  2. Run

    git flow release start $(make current-version)

    A new branch called release/x.x.x will be made for you and pushed to the repository.

  3. Generate the csv files and commit them. The release branch should be used to create manifests for the beta channel. Updates to the bundle in Partner connect will only impact beta.

    make generate-csv
git add ./deploy/olm-catalog

  4. Now you need to bundle the images for the release

    export IMAGE_REGISTRY=quay.io/rh-marketplace
export OPERATOR_IMAGE_TAG=$(make current-version)

make build push publish-image bundle

  5. Upload your bundle to partner connect. And publish it when it passes. The bundle command will leave a zip file for upload in the ./bundle

  6. Publish your images in Partner Connect and wait for the Operator scans to pass.

  7. If the operator scan fails, see the troubleshooting section. Otherwise finish the release.

  8. Once the release is finished. Run these commands:

    git flow release finish $(make current-version)

  9. Create the release using the published tag. Notify #rhm-general channel.

Hotfix

Same as release, but change git flow commands to git flow hotfix and start command from master branch.

Updating Partner Connect

Before releasing, we need to do a few steps in partner connect.

  1. Go into Partner Connect

  2. Navigate to the "Red Hat Marketplace Operator Image" project.

  3. Find the tag associated to the release and publish it. Publish Image

    • To get to this UI, you'll need to select "Try our new interface link" new ui link

  4. Download the bundle from the Generate Bundle action. generate bundle

    1. Go to the checks > select "Generate Bundle" workflow > Select artifacts download bundle 1
    2. Download latest artifact with the prefix stable- download bundle 2
    • The bundle with stable- is for updating the stable channel. This is the one you will primarily use.
    • The bundle starting with beta- is for updating the beta channel. This not working on the Partner Connect side.
    • Also notice the bundle describes what it's deploying bundle-s0.1.2-b0.1.2 says stable is version 0.1.2 and beta is 0.1.2. This helps verify you're not making mistake.
    • The zip file for the bundle has a zip file in it, you'll want to use it to deploy. bundle zip file layout

  5. Then navigate to the "Red Hat Marketplace Operator" project.

  6. Once inside the operator project, select Operator Config.

  7. Now upload the zip file included in the bundle you downloaded from the action download. Publish Image

  8. You'll need to make sure the image is published and all the scorecard tests pass.

  9. If all the tests are passed and you're read, select publish and your new version of the operator will be released.

Troubleshooting

If your upload fails its scan, here are some common problems:

Image verification step fails

This is the last step in the process, normally it'll say quay.io/xx/xx@sha256... cannot be found. This normally means that the image you used was published in Partner Connect. Go back to the Updating Partner Connect and retry the first few steps. Investigating the image's sha may also be required. You can use the skopeo tool to inspect images for their digests running skopeo inspect docker://${YOUR_IMAGE}.

Scorecard step fails

We run the scorecard as well in our Deploy workflow. You'll want to verify that the results of our scorecards tests are valid as well. There can be some false-positive results so a particular test failing does not always mean it's a bad manifest. If the same test is failing, fixing it for our runs will likely fix it for the manifest scanning on Partner Connect.

A CR test isn't running properly

Sometimes if the example CRs in 'deploy/crds' is updated to the same name as other resources we deploy then the test will fail. This failure normally comes through as xx resource already exists. Double check the names are example- or otherwise unique.

"Owned CRDs have resources listed failed"

You can ignore this one, will not stop certification.

Testing Latest Builds

On any branch build of Deploy, you can download an bundle index with just that release in it.

  1. Go to checks > Select drop down for "Deploy OPM Index" under the "Deploy Image" workflow > Select Artifacts catalog source image one

  2. Download the "test-catalog-source.yaml" artifact, unzip and apply the YAML to your cluster catalog source image two

  3. Test source creates a new provider in Operator Hub. On Openshift <= 4.3, it will be under the "Custom" type. On Openshift 4.4 >= it will be "RHM Test".

  4. To install latest, you will need to install the beta channel. It's best to install stable first then upgrade to beta at least once to verify upgrade.

Workflows

Each workflow perform specific tasks on different branches. Together they help build and deliver software.

Workflow Git Action Branches Description
Bump Version Push develop, hotfix/** Bumps version when it detects a tag on develop that's already been created in git
Create Manifest Push release/**, hotfix/** Creates the new versions manifest and commits it to the branch
Deploy Push master, develop, release/**, hotfix/**, feature/**, bugfix/**
Deploy/Deploy OPM* Push Same as deploy Creates the bundle image for testing
Deploy/Scorecard* Push Same as deploy Runs the operator-sdk scorecard as is available on the Partner connect
Generate Bundle PR master Generate the bundle data as zip files for releasing to Partner Connect
License Push All Verifies all files have the appropriate license
Release Push master Tags new releases on master branch and creates sync from master to develop when new commits are present.
Test Push All Runs unit tests

Home

Links

Usage

Contributing

Learning

Development

Release

Testing

Clone this wiki locally