Flesh out the Lotus release process

[arajasek]

Based on feedback received from the 1.5.1 release, we need to be doing more formal testing & validation, and slowing down our overall release process. The first step is to build a release checklist and process.

• Announce a public initiative to improve our release process, asking for feedback and support
• Create an issue template analogous to https://github.com/ipfs/go-ipfs/blob/master/docs/RELEASE_ISSUE_TEMPLATE.md
• Create a Lotus Release Philosophy document #5948

[BigLep]

This needs to get moved publicly, but some notes from the retrospective about this are in https://www.notion.so/protocollabs/Develop-network-upgrade-runbook-checklist-b6c1328025b448f79a148ae05d54a0e1

[dd45e640b42e6da7da96faee3996ef7c]

i can add some:

• when a new version is tagged: specify what binaries need to be updated (lotus, lotus-miner, worker) in order for the "ensemble" to still work together.

bigger setups might want to have "emergency" fixes for node/miner but do not want to update 100s of workers out side planned maintenance windows - for example

had this open in october: #4244

[arajasek]

The template in #5826 gets the ball rolling, but there's tons of open questions / TODOs:

• Metrics section needs to be fleshed out

  • are they easy to get for infra?
  • can they put on a dashboard somewhere?
  • what will we be doing with them? eyeballing?

• Community Dev Testing

  • what exactly are we asking of MinerX and beta users? just informing them?
  • what exactly are we asking community partners to do? test their own product? test our release? do we block on them?
  • @eshon and @dineshshenoy suggest categorizing releases when possible, and notifying appropriate subsets of partners

• Current template only works for minor versions. I think it'll be unwieldy to combine them all into one, but separate templates for major and patch versions might be excessive?

• Flagged by biglep:

  • I like the opennes in GitHub. Potential things I don't think it captures though are:

    Who is the owner for a certain step.
    What are the expected dates for a certain operation.
    Binary checkboxes don't capture if something is in-progress or blocked.
    How easy will it be for posting in graphs or other comments while executing the release process?
    I'm game for whatever, but a Notion public document is another option.

  • I'm not saying it belongs in this document, but I assume we need to also capture our branching strategy and how Lotus versions relate to network versions.

    • yes, that belongs in the "philosophy" document.

• Scripting

  • version bumps
  • running the mkreleaselog script, adding it to CHANGELOG
  • tagging the RC
  • cutting the release on Github (could be ~1 click by feeding in inputs using Prefill new "Releases" with query parameters isaacs/github#1410 (comment))

[coryschwartz]

The 1-click images are a compoent of what eventually gets deployed during a release. Currently they are deployed simultaneously into AWS and the digitalocean account for every tag.

AWS AMIs are automatically copied to every region and there is nothing further required. They are public AMIs that anyone can use.

The infra team currently has a few of these AMIs we built out for internal teams to dev against. These are terraform-provisioned aws instancees that sit on the butterfly, calibration, nerpanet or mainnet networks, but they aren't primary infrastructure.

Images are pushed into the DigitalOcean account as well, but there is no automation for updating the "marketplace" app. Digitalocean has a manual approval process wherein we submit the image through the "marketplace portal", which can be accessed by anyone who has access to the digitalocean account.

the non-automated way for updating the marketplace listing:

[ ] Go to https://marketplace.digitalocean.com/vendorportal/
[ ] Edit filecoin lotus
[ ] Change the version information
[ ] pick the new image.

Once submitted, a jira ticket will be created in which someone from digitalOcean might have questions or comments, but the listing will go live after they approve it.