Use rushjs instead of lerna for monorepo management

[rogerhutchings]

So far we've used lerna to manage the monorepo, and that's caused its share of problems, mostly because we're not using it the right way - it's for publishing multiple packages, not enabling local development across multiple packages. This has caused us some headaches in the past.

Recently, I've come across Rush.js, which is a monorepo manager built by MS, and I tried out a quick test, which went pretty well:

• There are a bunch of packages that don’t work with the ‘preferred’ package manager (pnpm), but npm works fine
• Hoisting just magically works ✨
• It uses hard links, so I was able to remove the hacks we needed for styled components ❤️
• It has a global build command, but no other global commands (like lerna run / exec) - the apps would need some package scripts renamed (otherwise they’d build a production version every time)
• It works out the dependency build order and has a purge command, so we could largely trash the existing scripts
• It supports multiple directories, so we can create a libs dir, an apps dir, etc, so we can move away from the slightly clunky naming pattern we have currently
• we can experiment with not transpiling libraries (until we have a use case for them) using next-transpile-modules, which should help in tree-shaking

So I'm going to propose we switch over to using Rush after TESS is deployed, since this will affect the build process.

To do:

• Write up ADR explaining the change
• Move over packages
• Update documentation
• Update build processes
• Communicate changes on Slack (with quick update on how to do stuff now), ideally with a few days' lead time
• Merge update PR

Comments / suggestions welcome!

cc #545

[eatyourgreens]

How are we going to test Rush before we switch over? If it turns out that it makes life difficult for us in production, is there a fallback?

[rogerhutchings]

1. I'll push up a branch once I've got a working version ready, and I'll require several approving reviews before merge - I'd like you and @srallen to review at a bare minimum
2. We can always switch back to lerna, everything will be in the git history

[rogerhutchings]

Hang on - when you say production, do you mean in terms of ops or day-to-day development?

[eatyourgreens]

I was thinking day-to-day development. I'm sure the initial PR will be fine but, based on our experience with Lerna, problems might surface once we've tried to use it as a team and maybe come across things we need to do that it doesn't handle. It's a worst-case scenario, I know, but should we hold off on using new tools in production until we've had a chance to get comfortable with them as a team?

[rogerhutchings]

I get your point, but how would we do that without just making the switch?

One way we could try doing it is to fork the repo, make the changes, let people try out rush for a while, submit some non-critical PRs; at the end of a fixed period, we either merge the changes only back into the main repo, or changes plus new tooling. It's a recipe for merge conflicts, mind.

Actually, a better way would be to add a rush branch, let people submit PRs to it as a base branch, and then I can revert the commits adding rush if we decide it's not suitable.

My gut reaction is:

1. We're using Lerna in a way it's not really intended for, and I can very easily see its API changing to the point where we can't use it anyway (even if it's a major version);
2. I want to get rid of those require-control hacks
3. We need a tool to reliably handle package management and linking local dependencies; anything else shouldn't be more than a shell script away

Thoughts / feels?

[eatyourgreens]

Sorry, my point is we can do this by making the switch, but what's the plan if it turns out that Rush doesn't work for us? ie. how do we avoid doing what we did with Lerna?

[rogerhutchings]

Oh, got you. Well, we know exactly how to get Lerna working for us, and everything's still in the git history, so if rush turned out to be a disaster we could roll this change back. We'd have to:

1. Get our old lerna.json and scripts out of the git history
2. Move the packages back to their previous locations
3. Add require-control back in
4. Remove the rush-related stuff

There's nothing inherently different about any of the packages in the repo, other than the use of require-control to make symlinking work the way we have it; if we have the fallback strategy above documented in the ADR, it should be as frictionless as possible.

[srallen]

The symlinking that lerna does is preventing us from successfully running npm audit fix commands too. I'd like us to switch even though there's risk in quickly switching underlying tech like this just for the ease of quickly making security patches. Dependabot since the github take over hasn't reliably been opening PRs for us, so I'd like npm audit fix to be a reliable back up to that.

[rogerhutchings]

Closing as we're going with yarn - rush currently gets a bit flaky with npm versions above 4.5.0 :(

microsoft/rushstack#886