chore(build): Derive application versions from git

[sfoster1]

Overview

Until this PR, the way we did versioning was:

• All the subdirectories that contain projects have a package.json, which is something that's used in the js ecosystem. The python packages also had this.
• That file contains a "version" key, which contains the version
• When you want to change the version, you use a tool called lerna (which we wrapped in a makefile script) that lets you change the version in all those package.json files at the same time, as well as all the entries in js project package.jsons that set co-versioned dependencies on other projects
• You check that in and now it's the new version
• This also used to manage changelogs but doesn't anymore

This has a couple problems that hurt the way we do things:

• Things in subdirectories of our monorepo are packages. Those packages are then used in applications. Most applications use more than one package; many packages are used in more than one application.
• That means that either every application has to be at the same version, or we have to escape from the above workflow for some applications, which is what we did
• So we ignore the stuff in package.json for our webstack applications like PD and labware library
• But we use package.json and only package.json for our python applications, so each python package can only have one version no matter what application it's used in
• To change the version you need a git commit, so you can never look at the current state of the git repository and say "this is good I'd like to release this" - you can't, you always have to have another commit

One way to fix this (there are others) is to pull the source of truth for the "version" of something out of the content of the git repo and into the metadata of the git repo - as git tags. We already tag versions of different applications, but these tags are only descriptive - regardless of the state of the tag, a build of a given commit will always have the same version.

Instead, we can make the tag determine the version, when you build an application. We can do this by hooking into git repo metadata during the build process of each package - the versions injected into js applications by webpack can come from git tags, the versions injected into python packages by setup.py can come from git tags. This lets us do a couple very useful things.

The first new capability, and really the point of this PR, is that all packages are now capable of being in different applications that have different versions. The opentrons python package in api/ can be built such that it uses the latest robot stack version (the latest tag starting with v), and it will be when we build the OT-2 software - but it could also be built such that it uses the latest ot3 development version (the latest tag starting with ot3), and will be when we build the OT-3 software. If we wanted to stop coversioning the desktop app and robot software, it could even be built with the latest app version (call it the latest tag starting with app@) when it's intended for use in offline analysis. It could even be built with its own special version for deploying to pypi by using a different tag. This allows us to set up a separate release stream for the OT-3, with its own alpha/beta/release channels, so that we can issue releases without bothering customers.

The second new capability, and maybe it's even better, is that you don't need a commit to change the version - you only need a tag. Think about the release process for the robot stack: that means there are no more bump PRs, each of which required two robot stack builds - the PR content, then the release branch after the merged PR - before you push a release tag.

The third new capability is a capability of understanding: all of our stacks are now versioned in the same way. Protocol designer and robot stack now work exactly the same with different tag prefixes. You can trigger anything you want on tags, you can push tags, the versions are separate, it's all the same. A commit of components can be at v6.7.8 as part of a robot stack release and at [email protected] as part of a protocol designer release and at [email protected] as part of a labware library release. There's no more conflict, no more little snippets in webpack config that dodge the way everything's configured by default - the process is all the same.

This PR should both give us new build system capabilities and lessen the toil of making a robot stack release.

Changelog and Review Requests

This is a very broad PR that touches quite a few different files. It's organized into roughly standalone commits, and that's the way I'd recommend approaching it - JS folks, please take a look at the JS parts; python folks, please take a look at the python parts (or anybody look at anything, but if there's some part you're not interested in you can skip it pretty easily).

The commits are these:

Aspirationally update releasing.md

This commit updates the releasing docs to match the new workflow. It happened before the rest of the work, and is a good way to think about the goals of the rest of the work. Not much to look at here in particular, but copyedits always appreciated.

some changelogs crept back in

Another small one, fixes a merge issue from 6.2.0 - a previous pr removed the checked-in CHANGELOGS.md, and then the release merge brought them back, and this gets rid of them again.

do not use lerna for versions

The first step of the actual work: remove lerna as a dependency and remove make bump. Leaves the contents of the rest of the package.jsons untouched, but after the following commits we won't need this anymore.

find py versions from git

The first major commit in the PR. This

• Removes the now-unused package.jsons from py projects and removes them as yarn workspaces
• Updates the python_build_utils to be able to get a version from git
• use python_build_utils to provide the versions for packages
• Find the current version at runtime based on the installed package metadata
• Make the buildroot recipes work with the new python_build_utils
• there also need to be changes in the openembedded builds; see opentrons-robot-server: version from git oe-core#37 if you're interested.

Reviewers here, please check out

• the way project versions are defined
• whether the metadata introspection works well

remove versions from js package json

The first of the two commits that implement the above for js projects. It got split up because we needed more fixes for the js side. The js intra-monorepo dependencies were coversioned with the latest release; this won't work when the versions are all dynamically determined, and it won't work to make all those versions 0.0.0-dev because we've previously released some packages to npm and yarn really wants to prioritize them. This means we now use yarn path dependencies with a link: prefix which produces the same node_modules structure as before.

Reviewers please leave some opinions on whether this is fine going forward or if we have to find another way

find js app versions from git

The second of the two commits that implement git based versioning for js projects. Here, we're now building in the actual versions to the bundled packages.

One thing to note is that right now, those versions are only present in the webpack output bundles. That means they won't be present in tests, lint, or mock.

In general these changes take their cues from how versions were injected into the web stack projects with webpack DefineVersion plugins - we just need to get the payload of those plugins from git instead of package.json. This works for all bundles, so dev servers as well as actual release bundles. The one odd-one-out here is of course the app.

For the app, you can set the version in webpack config and the settings page will happily display the right thing... but you'll still get update popups as if the app was version 0.0.0-dev. The electron-updater pulls the "current version" of the app from electron.app.getVersion which is hardcoded to pull its data from the package.json of the package that bundles electron (in our case, app-shell). We solve this by injecting the version in the electron-builder config, but of course that only works when you're using electron-builder - when you make distributable build of the app (including those on CI, happily). When you run make -C app dev, we run yarn electron in app-shell, and it pulls directly from package.json, and dev builds will therefore always think they're version 0.0.0-dev for the purposes of update, though the settings will show that they're the correct version.

Reviewers, please let me know if I'm doing something... suboptimal... in any of these webpack plugins. I'd like to keep changes minimal here, but if there's something that hurts to look at please let me know.

Risk assessment

Medium. This changes the fundamentals of how we version things, but that's really not a problem until we do a new edge-sourced release, and the next couple of those we'll probably do are for internal distribution anyway so it's ok.

Closes RCORE-429

[codecov]

Codecov Report

Merging #11863 (dd81a7e) into edge (fa7ffb6) will increase coverage by 0.08%.
The diff coverage is 75.00%.

@@            Coverage Diff             @@
##             edge   #11863      +/-   ##
==========================================
+ Coverage   74.25%   74.34%   +0.08%     
==========================================
  Files        2125     2150      +25     
  Lines       58877    59415     +538     
  Branches     6233     6238       +5     
==========================================
+ Hits        43720    44171     +451     
- Misses      13686    13773      +87     
  Partials     1471     1471              

Flag	Coverage Δ
app	72.98% <ø> (+0.02%)	⬆️
components	65.28% <ø> (ø)
g-code-testing	97.19% <ø> (-0.49%)	⬇️
hardware	58.28% <ø> (+0.11%)	⬆️
labware-library	49.74% <ø> (ø)
notify-server	89.13% <ø> (-0.05%)	⬇️
protocol-designer	45.85% <ø> (ø)
react-api-client	83.05% <ø> (ø)
shared-data	86.02% <75.00%> (+1.21%)	⬆️
step-generation	88.46% <ø> (ø)
update-server	65.47% <ø> (+4.49%)	⬆️
usb-bridge	81.35% <ø> (-13.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ared-data/python/opentrons_shared_data/_version.py	69.23% <69.23%> (ø)
api/src/opentrons/__init__.py	45.54% <75.00%> (ø)
...ared-data/python/opentrons_shared_data/__init__.py	100.00% <100.00%> (ø)
...e/hardware_control/motion_planning/move_manager.py	89.36% <0.00%> (-4.26%)	⬇️
...are/hardware_control/motion_planning/move_utils.py	91.54% <0.00%> (-1.50%)	⬇️
shared-data/js/constants.ts	100.00% <0.00%> (ø)
app/src/App/OnDeviceDisplayApp.tsx	28.57% <0.00%> (ø)
...isms/ProtocolDetails/RobotConfigurationDetails.tsx	82.14% <0.00%> (ø)
...vices/ProtocolRun/SetupLabware/LabwareListItem.tsx	85.41% <0.00%> (ø)
...ices/ProtocolRun/SetupLabware/SetupLabwareList.tsx	100.00% <0.00%> (ø)
... and 30 more

[koji]

maybe

const pkg = require("./package.json");
const electronVersion = pkg.devDependencies["electron"]);

[sfoster1]

I think I don't want to do this because it would limit the version constraint in package.json to whatever electron-builder.config.js can accept (exact-version constraint) and also to the semantics we want. @mcous @b-cooper @shlokamin do yall know why we didn't single-source the electron version before?

[mcous]

Prior to #11537, electron was not listed in app-shell/package.json, and was instead, a development dependency in the root package.json. Keeping them separate was more of a philosophical thing than a technical thing, though we have had issues with electron-builder selecting improper dependency versions for inclusion in the asar.

I'm down to make this change, but let's give it its own PR. From reading the electron-builder docs, I believe omitting the field is the suggested way of using the installed version from package.json

[mcous]

Looks really good! Did not see any obvious problems with the code, but offered some readability suggestions. I will run some sanity checks for local builds but otherwise figure we'll rely pretty heavily on CI to tell us if this is working properly

[mcous]

Can this move above HERE and ditch the E402 now that this file is more simple?

[mcous]

Same question as above

[mcous]

Not sure how helpful this is; down to avoid preserving it in this new system

[mcous]

Prior to #11537, electron was not listed in app-shell/package.json, and was instead, a development dependency in the root package.json. Keeping them separate was more of a philosophical thing than a technical thing, though we have had issues with electron-builder selecting improper dependency versions for inclusion in the asar.

I'm down to make this change, but let's give it its own PR. From reading the electron-builder docs, I believe omitting the field is the suggested way of using the installed version from package.json

[mcous]

Can we split up the require and the config call just to make it a little more clear what's going on here?

const webpackPreprocessor = require('@cypress/webpack-preprocessor')
const createWebpackConfig = require('../../webpack.config')

module.exports = async (on, config) => {
  const webpackOptions = await createWebpackConfig()
  on('file:preprocessor', webpackPreprocessor({ webpackOptions }))
}

[mcous]

Same comment as above about template strings

[mcous]

Can omit all : value if variable name matches desired key

Suggested change

	module.exports = {
	detailsFromTag: detailsFromTag,
	tagFromDetails: tagFromDetails,
	prefixForProject: prefixForProject,
	latestTagForProject: latestTagForProject,
	versionForProject: versionForProject,
	monorepoGit: monorepoGit,
	}
	module.exports = {
	detailsFromTag,
	tagFromDetails,
	prefixForProject,
	latestTagForProject,
	versionForProject,
	monorepoGit,
	}

[mcous]

Same question about the noqa

[mcous]

Same comment about not moving this forward to avoid side effects on import