lotus Release Process

[jennijuju]

Hey everyone! We wanted to let you know that we are currently revising the lotus release process(leave a comment if you have any suggestion). As many of you have shared, we haven't been doing a good enough job of testing our release candidates before shipping them. We intend to discipline the pace at which we put out optional releases and slow down the pace for the mandatory releases. We are also developing a comprehensive release checklist that incorporates a long testing window.

Some changes that you may notice as a result of this work:

• Minor version bumps will no longer indicate "mandatory" releases that include network upgrades. Moving forward minor version bumps (1.6.0, 1.7.0, etc.) will be the norm for any release, including those that introduce network upgrades.
• Patch version bumps (1.6.1, 1.6.2, etc.) will only be used for small releases that make fixes.
• Lotus releases will no longer be cut off themaster branch. Instead, there will be a dedicated releases branch from which releases will be cut. The master branch will generally be ahead of releases, and will be called a next-release-dev version of a future release. We will aim to keep master as a generally reliable, but slightly more experimental branch with the latest features and fixes.

Thank you for you all your feedback about our release process. We're paying close attention to the problems you've reported, and are determined to make things better! We will talk to you again when the revised release process is final!

[stuberman]

Is there a convention for version releases that are mandatory network upgrades? 2.x, 3.x, etc?

[ribasushi]

Want to chime in as one exactly this kind of user: I run a number of specific-purpose instances, where there is no benefit to any type of upgrade once the operation is underway ( e.g. chain exports ). It is rather problematic losing the ability to know with certainty when an upgrade must be carried out because "there's just no other way".

An alternative alert-space where imminent and upcoming network consensus upgrades are detailed would be a sufficient/better replacement for the version indicator.

[arajasek]

I agree with @dineshshenoy and @ribasushi here. I'm not opposed to using major versions for this, it's just that we'd blow through them somewhat quickly, which is an odd look. I'd also like to reserve the major version for genuinely exciting releases (and not "minor consensus-critical tweaks").

@Stebalien is of the opinion that we shouldn't have a concept of "mandatory" releases. With a long enough window, many releases can be produced before a network upgrade, and the messaging becomes "you must be running X.Y.Z or later before ".

I'll let him explain further.

[jennijuju]

I’m seeing dramatically increasing amount of confusion wrt the new version convention in the past two weeks.
For example- as we are planning ahead and created a release issue for v1.7.0, some community member now think that there is another mandatory release-rc on apr 12, even worse scenario is some of them were thinking they may skip v1.6.0.

we can’t really get rid of the idea of “mandatory release” since we still have to say that in release notes, announcements and so on to ensure folks don’t miss a network upgrade. Tho I think we may also bump the version when there is critical bug fixes even tho a network upgrade is not required.

[whyrusleeping]

We currently use minor versions to mean network upgrades. I think it would be fine to use major versions for this instead. I dont really care about the vanity argument, or 'save the major bumps for something really big!', as network upgrades are really big on their own.

we should absolutely stick to the convention that some versioning change implies a network upgrade, whether keeping minor as it is today, or switching up to major.

[Stebalien]

So:

• We need to be able to cut real patch releases. That means we need minor releases for features.
• We could use major releases for network upgrades, but that's going to be really annoying:
  • All internal lotus imports would need to be renamed due to go modules. This would cause rebase/merge hell for all outstanding PRs.
  • All users importing lotus will need to upgrade their import paths.

I believe there is some tooling to make the second part easier, but it's still a massive pain.

Can we use some secondary version? I.e., can we announce the version as "$NV:$VERSION"? The go version doesn't have to match the version we announce to users.

[Meatball13]

Ah, this is good to hear. Be nice for there to be a set time window for consensus breaking patches to be available before the network moves on? Such as an epoch always set 5 or 7 days in the future, something static so people can run the new patch for a while and make sure everything is good.

[jennijuju]

We have been thinking about network upgrade maintainence window for a while now, it’s a little bit different than what you are describing and we will communicate that with the community as the plan fleshes out more.
IMO, what you are describing is what staging branch and release candidates serve. And if you take a look at the release process, we definitely want to put RCs out earlier. As for the mandatory network upgrade, I think we have been putting out the final release at least 5-7 days ahead upgrade epoch for the past few upgrades, unless it’s a urgent hot fix. So my suggestion here is to update your node and miner when the release is out sooner than later.

[dineshshenoy]

With the new branching strategy changes (which I'm VERY supportive of), I think the following would be helpful:

1. Rename master to "develop", "dev", or "upcoming". Master often means stable or production in other projects so this would make it less ambiguous
2. Add a tag called "stable" which is updated every release and points to the latest released branch. This is mostly helpful for a new user who doesn't know that we're on 1.5.3 (and they don't want to run something that hasn't been fully and robustly tested)
3. Update this page [https://docs.filecoin.io/get-started/lotus/installation/#systemd-service-files] for new users to use "stable" instead of master during installations.
4. Change the "default" branch in github to point to the rename in item 1 (develop, or whatever)
5. Add the long-living terms (develop/stable) to the README so new users know which tags to use.

[jennijuju]

Thanks for your feedback! We will review the options and keep the community posted!

[raulk]

Personally I think that the semver policy is underspecified and poorly applicable for distributed systems. Distributed systems have many API surfaces that can break (binary API, local JSON-RPC API, application protocols, networking stack, etc.). Bumping the major version is highly unspecific. Moreover, Golang tightly couples source code with versioning (major versions beyond v1 leak into import paths) -- this poses logistical difficulties to using major versions here.

Concretely, if we were to pick a policy that bumped the major version on every network upgrade, we would disrupt every single downstream library/application that consumed the native Lotus API, e.g. libraries depending on the JSON-RPC client, testground tests, etc. They would need to update their code every single time that we released a network breaking change, even if it brought on zero expectation of breakage for the Golang APIs that they depend on. (i.e. we are signalling breakage on the wrong API surface! signalling breakage on the Go level, when what breaks is the network protocol)

I personally think we should establish a clear, understandable versioning policy, with stable and consistent heuristics for both Lotus developers and the community to follow and reason about.

Concretely, I propose the following strategy:

1. All network upgrades are introduced with x.y.0 with releases.
   • These releases are made on top of what's considered the latest stable patch version from the previous network upgrade.
   • .0 releases introduce the minimum indispensable diff for miners to upgrade to the protocol fork, nothing more.
   • No extra functionality is delivered.
   • The risk surface is therefore contained to the network upgrade, nothing else.
   • Examples:
     • ✅ v1.6.0 was exactly v1.5.3 (latest patch release from previous network version) with the minimum indispensable diff to upgrade to nv10, no more, no less. I think this time we got it right.
     • ❌ It would be a breach of policy if v1.6.0 was based off master (and master contained features on top of v1.5.3), as these would leak into the "minimal indispensable upgrade".
2. x.y.1 is the first release containing unreleased features and bugfixes (e.g. master) on top of the "safe" x.y.0.
3. It is perfectly acceptable to release x.y.0 (minimum indispensable network upgrade diff) and x.y.1 (first feature release atop the network upgrade) at the same time.
   • Miners can choose which release to ride the fork with, depending on their risk appetite/aversion.
4. It's perfectly acceptable for miners to ride the upgrade with x.y.0, and upgrade to x.y.1 the day after.
   • In fact, this should be the preferred path, as it is the safest one because it doesn't leak potential bugs in new features into the network upgrade ceremony.
5. All subsequent features/bug fixes are released atop x.y.1 and children.
6. Lotus developers guarantee a 10-day or 2 week freeze in patch versions prior to releasing a new x.y.0. This allows miners to track a stable baseline before they are required to upgrade to stay in sync wih the network.

[raulk]

TL;DR:

1. x.y.0 should NOT introduce any features, only the minimal indispensable diff to upgrade the network off the last stable patch (x.{y-1}.z).
2. x.y.1 is the first release to deliver features and bugfixes pending in master.
3. x.y.0 and x.y.1 can be released at the same time, and miners can choose which one to ride the upgrade with.

[jennijuju]

@Stebalien you will summarize our discussion & conclusion today, yes?🤔

[Stebalien]

Conclusion:

1. Use major versions for major changes to lotus (e.g., if we re-architect). Unfortunately, changing major versions in go is non-trivial.
2. Use minor versions to indicate network upgrades.
   1. Use odd minor versions for the feature/mainline version. "Patch" (z in x.y.z) releases may contain features.
   2. Use even minor versions for the "stable" (need a better name) version. Patch releases will contain no features, only critical fixes.

A similar even/odd versioning scheme is also used in Linux, GNOME, etc.

Network Upgrades

When a network upgrade approaches:

1. An RC for the next "stable" (even) release will be cut from the latest "feature" (odd) release.
2. The next odd release will be cut from the lotus master branch.

For example, if 1.18 is the latest "stable" release:

1. 1.19.X will become 1.20.0-RC1
2. master will become 1.21.0-RC1
3. One week later (TBD), 1.21.0-RCN will become 1.21.0.
4. ~4 weeks later (TBD), 1.20.0-RCN will become 1.20.0 and the network upgrade height will be finalized.

Questions/Notes:

• Timelines for RCs.
• Synchronization between Filecoin implementations. In practice, 1.20.0 would be cut once all implementations are ready for the network upgrade.
• I'm assuming all the network upgrade logic will actually be "done" by 1.19.X so 1.19.X can turn into 1.20.0-RC1 without adding any new features.

Analysis

• Advantages:
  • Clear communication of network upgrades.
  • No need for major version bumps.
  • A clear "stable" versus "feature" split.
• Drawbacks:
  • No way to distinguish between a bug fix release and a feature release on the "feature" version. Both cases will use the "patch" version number.
  • Not semver (potential issues with go modules, confusion, etc).

[BigLep]

@Stebalien : thanks for writing this up. This captures the conversation from my point of view.

A couple of other things:

1. I think we should state what the requirements are that were guiding the conversation. These included:

• Not bumping the major version given the pain that causes in Go (as @raulk described in lotus Release Process #5889 (comment) )
• Have a clear signal for network upgrade
• Need ability to do patch releases after a network upgrade
  The proposal above satisfies these.

2. I'm up for what others think, I'm wondering if we should the avoid the word "stable" as that can imply that the other line isn't stable (we don't want a "stable" vs. "unstable" motif). The other term we threw out was "network line" (for even minor release) vs the feature/main line for for odd minor releases.

[f8-ptrk]

can someone visualize that?