SPIKE | Automatic deployment from main to staging

[cdanfon]

This spike is to investigate options for automatic deployments from main to staging.

These are a few initial thoughts/questions:

• What problem/pain point are we trying to solve?
• What are the technical options?
• What things do we have to consider around internal stakeholders?
• How does the workflow look like?

[Time box to 2 hours]

[tbrlpld]

@cdanfon @mtdenton @danielfmiranda @fessehaye

I have been looking into this now for 2h and here are my notes:

• What problem/pain point are we trying to solve?
  • Current process:
    • A feature or bug is developed on a dedicated branch. The work receives code review. Once the work is approved and passes all tests on CI, it is merged into the main branch.
    • After the merge to main a developer needs to check that CI on main still passes (maybe we are missing merge migrations etc.)?
    • Once CI passes on main the developer needs to pull the latest changes from GitHub to their local machine.
    • Once the developer has the latest changes on their machine, they can push the changes to staging.
  • Pain points:
    • We often have to wait for CI to pass twice, first on the feature branch before the merge and then on main. The CI on the feature branch is needed because often the branch needs to be updated before the merge which triggers CI. Each CI run takes like 10-15min to pass.
    • Did we check that the merged result on main passes This needs to be checked manually. I could accidentally fetch and deploy without waiting for CI to pass.
    • Did something else get merged into main between checking CI to pass and pulling the changes locally. This could lead to accidentally pushing unknown changes.
    • The manual pushing could deploy a wrong branch or wrong changes on accident. The local branch is not protected and could differ from what other devs have or be broken.
    • We sometimes forget to deploy our changes to staging right after merging. This leads to a single deployment releasing multiple possibly unrelated changes to staging.
    • If merged work is not deployed to staging right after it is merged and CI passes we are unnecessarily delaying QA, which delays deliver to stakeholders / users.
• What are the technical options?
  • Activate the Heroku GitHub integration: https://devcenter.heroku.com/articles/github-integration
    • This is what we have used in the past until the Heroku-Github security breach: https://status.heroku.com/incidents/2413
    • But the security team of the Mozilla GitHub org has concerns about this integration. If we wanted to reactivate this, we would need to move our repositories (including foundation.mozilla.org) to a new GitHub organization that is different from the from https://github.com/mozilla
    • The Heroku-Github integration would also re-enable the use of review app which we also used to use until the incident. https://devcenter.heroku.com/articles/github-integration#review-apps
    • The Heroku-Github integration would also re-enable the option to promote changes from staging to production via the Heroku dashboard. This is how deployments to production used to be triggered.
  • Use GitHub Action to deploy to Heroku
    • We could make use of a the “Git Deployment” that we are currently performing manually but automate this as a GitHub Action that is automatically triggered after a merge to main happens and the CI pipeline passes (https://devcenter.heroku.com/articles/git).
      • https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#running-a-workflow-based-on-the-conclusion-of-another-workflow
    • This GitHub Actions workflow should be pretty simple as it only has to checkout the code (our other workflows already do this) and use Git to push it to Heroku (that is what we are currently doing manually) → git push heroku main.
    • I guess we need to setup Heroku as a Git remote to for this to work. Hopefully also just a single command (git remote add [<options>] <name> <url>).
    • We will need a (dedicated) user account’s API key to perform the Git push (https://devcenter.heroku.com/articles/git#http-git-authentication). This API will need to be set as an Actions secret on GitHub. There already is a [email protected] that has access to the repo. So maybe we can use that.
      • https://devcenter.heroku.com/articles/authentication
    • We could use the same approach (action) to deploy to production. The only thing that we would need to change would be the deployment target.
    • We can use environments and require reviews to ensure that QA has passed before we deploy from staging to production:
      • https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#required-reviewers
      • https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments
• What things do we have to consider around internal stakeholders?
  • Activating the Heroku-Github integration goes against the security teams recommendations.
  • Moving the repositories to a new GitHub organization will require DevOps work. It has been done for the donate stack (https://github.com/MozillaFoundation/donate-wagtail), but I am not sure about the complexity and what a possible timeline could look like.
  • GitHub Action to deploy to Heroku with Git should be pretty simple to setup. It’s only a few commands, so we could be able to do this quite independently.
  • Using GitHub Actions for deployment would allow us to create some custom logic according to our internal workflows.
  • Design is used to reviewing PR on review apps and not staging. I might be more complex to setup review apps with GitHub Actions. Or, Design could review on staging?
• How does the workflow look like?
  • Review the above options and discuss.
  • Get feedback from other stakeholders (Design, DevOps, SecOps).
  • Decide which path we want to take:
    • Integrations path: Move repositories to a new organization to enable the Heroku-GitHub integration.
    • Actions path: Create custom GitHub Actions workflows for deployment to staging (and maybe production)
  • Implement necessary changes.
    • Integrations path:
      • Related ticket: Move foundation.mozilla.org to different organization #9189
      • Move repos to new organization
      • Activate Heroku-GitHub integration
      • …
    • Actions path
      • Create a deployment user and retrieve API token for this user.
      • Define the credentials as secrets as secrets in GitHub.
      • Create workflow file (I have attempted one in this PR: Custom GitHub Actions workflow to continuously deploy main to the Heroku staging app #9494 -- I was not able to test because I can not set secrets in the repo)
      • Test the workflow (e.g. by triggering without the branch requirement, but only push the main branch instead of the current).
      • Change test config back to something that only runs on main branch.
      • Review and merge the workflow into main.

[cdanfon]

Hi @tbrlpld Thanks for doing this, you've done great work breaking everything down so that it's pretty easy to understand.

@mtdenton Feels like the next action is for the engineering team to review the options and decide which one to go for. Are you OK to organise a meeting for this with Jen, Simon, Daniel and Tibor?

[mtdenton]

Agreed, looking to have discussions surrounding this sometime after PNI launch.

[fessehaye]

#9221 Just want to document that I attempted some solution to this around august.

[tbrlpld]

@fessehaye What was the result of your attempt? Was there something blocking you from getting further with it?

I could not finish my attempt because I can not set secrets on the repo. See #9494

[tbrlpld]

It appears no one is actively working on this. Moving this out of the sprint and back into sprint ready.

[tbrlpld]

@mtdenton Just wanted to raise awareness of this one, now that the PNI integration branch has been merged.

[tbrlpld]

Pulling this into the sprint because I have some time after finishing my tickets and this will make all future work less painful.

[tbrlpld]

Since we are not going to have DevOps support for a while, we can go with the custom GitHub action for now. I had already explored what that could look like in #9494.

I will push that further to make it fully functioning.

If we want to switch the approach in the future, switch to a different org and fully activate the Heroku-Github integration we can do that later on. Creating the custom Action is not going to block that. So this decision is easily reversible.

[tbrlpld]

#9494 is now cleaned up and ready for review.

[tbrlpld]

@fessehaye @danielfmiranda @mmmavis Just tagging you all for review of PR #9494

[tbrlpld]

Looks like I forgot to fetch before push 😓

https://github.com/mozilla/foundation.mozilla.org/actions/runs/3527479856/jobs/5916589418#step:6:9

[tbrlpld]

This is still not working 😓

[tbrlpld]

Continuous deployments are now working: https://github.com/mozilla/foundation.mozilla.org/actions/runs/3605800399/jobs/6076534615