Minimal node and npm upgrade

[youknowriad]

This PR includes the minimal changes to the repository to upgrade NodeJS and NPM. There are other things we could do (adopt workspaces, update the packages "engine" config... but let's leave that for a separate PR #48950)

• Upgrade Node.js to v18
• Upgrade npm to v9

Why

It is becoming very urgent to update node and npm. Without mentioning the benefits of the upgrade in terms of features and performance, we're now starting to be blocked as a lot of packages have updated their minimum supported versions for node and npm. (I've been trying to use yjs in a PR and got stuck because of this).

Jobs

I've fixed all the CI jobs, except one (Compressed size) but we can ignore this one, it's failing because it's running npm install in trunk with the new npm version. It will fix itself after merge.

Testing instructions

• Clone the PR
• Update npm and node locally
• Just try npm run dev or any of the commands you use daily to work in Gutenberg.

[github-actions]

Flaky tests detected in 8c91d9c.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/5512940997
📝 Reported issues:

• [Flaky Test] should toggle "lightbox" in saved attributes #52350 in /test/e2e/specs/editor/blocks/image.spec.js

[noahtallen]

@youknowriad Since we now have access to npm dedupe and need to regenerate the lockfile anyways, we could run that to simplify the dependency tree. I tried it on this branch, and it looks the result is 36,229 - 20,998 = ~15k deletions from the lock file

[youknowriad]

@noahtallen Sure, assuming it doesn't break anything (tests or whatever), I think it's important that we ship this PR quickly to avoid the constant rebases.

[noahtallen]

If you like, I can push a commit to try it today, and then revert if it causes issues :)

[youknowriad]

I've pushed the dedupe, let's see how it goes

[youknowriad]

@noahtallen there are some very minor failures in the unit tests and one failure in the test that checks the validity of the package-lock.json. I'm certain that it's fixable and that it's probably due to the job itself but for the sake of not blocking this I'm considering reverting the dedupe to leave it to its own PR. WDYT.

[noahtallen]

I was able to update the unit test snapshots and they look pretty minor -- let me see if I can also find a quick fix for the package-lock issue

[noahtallen]

The package-lock issue is weird; it's changing the lockfile versions for a few of the internal react-native packages like packages/react-native-editor from v1.98.1 to v1.99.0 during normal npm install (Which I can't replicate locally even after deleting node_modules.) As far as I can tell, 1.99.0 doesn't exist anywhere since these aren't published to npm.

I wonder if this issue will happen consistently

[youknowriad]

What's certain is that It didn't happen before the dedupe. I wonder if it's related to the "peers" issue in native modules.

[noahtallen]

I wonder if it's related to the "peers" issue in native modules.

Hm, I feel like it'd be something else. Since there are other peer dependency issues in the repo with other packages, I don't think the react-native packages would be unique. Another point of data, the version set before the dedupe was also 1.98.0. So it feels related to something internal to npm somehow

[youknowriad]

yeah maybe an npm bug.

[jsnajdr]

Regarding regenerating the package-lock.json file, I don't think deleting the file and running npm install will suffice

I think the info about Lerna and updating dependencies is outdated. It applies to the old version of npm (6.x) which didn't support workspaces (packages) and had many bugs when it comes to maintaining the package tree and deduplicating it.

With the latest npm version, after the upgrade, npm install should safely regenerate the lockfile. The only caveat you need to be aware of is that installing from scratch will install the latest matching versions of all packages. So, if there is a dependency declared as "foo": "^1.2.3", and until now the lockfile contained [email protected], after the reinstall you can get version [email protected]. In other words, the old lockfile made sure that foo version is kept stable unless you're doing some operation (install, upgrade) that directly affects it. But when regenerating the lockfile, you're getting potentially hundreds of package updates at once. That's hard to control, and can be risky because even minor upgrades can introduce bugs and unintentional breaking changes.

[youknowriad]

@noahtallen Great catch on the rnmobile update, it seems like all is good now. Thanks for the update here.

[jsnajdr]

I was curious why a Node upgrade would cause change of production JS assets. That shouldn't happen. And tracked it down to caniuse-lite upgrade:

• what actually changed in the bundle is that globalThis.webpackChunk references changed to self.webpackChunk.
• they changed because our supported browsers no longer all support globalThis.
• the browser that doesn's support globalThis is and_uc -- Android UC browser.
• we support this browser because it has more than 1% usage.
• it's only in the latest caniuse-lite data where this browser went over 1% -- it's 1.044%. It was under 1% in the old version.

A standalone PR that updates caniuse-lite (and maybe also pins it in package.json?) would be nice, but not necessary.

[jsnajdr]

I merged the caniuse-lite/browserslist/core-js upgrade separately in #52420. The newly supported browser (and_uc) caused different output for webpack runtime code (the hashes in test snapshots agree!), and also new CSS autoprefixes like -webkit-margin-before for margin-block-start.

[jsnajdr]

you're getting potentially hundreds of package updates at once. That's hard to control, and can be risky because even minor upgrades can introduce bugs and unintentional breaking changes.

And now we can see an example of this in this very PR: #52363 (comment)

The caniuse-lite package was upgraded, it has the latest data about browser usage, and the and_uc browser just reached 1% (1.044%), so Gutenberg started supporting it, and this browser doesn't support the globalThis keyword, so the webpack output has changed, and therefore asset hashes in unit test snapshots have changed 🙂

[youknowriad]

Interesting. Thanks for digging there @jsnajdr

[youknowriad]

So it doesn't seem like there's any blockers left. After some discussions with @kevin940726 and @desrosj We're planning on landing the upgrade by 6.3 RC1 (so in 10 days approx) to avoid too much disruptions for developers and most of us will switch to 6.4 work at that time.

Core and Gutenberg don't necessarily need to be updated at the same time but it's a good thing to try to synchronize.

[noahtallen]

But when regenerating the lockfile, you're getting potentially hundreds of package updates at once. That's hard to control, and can be risky because even minor upgrades can introduce bugs and unintentional breaking changes.

Indeed -- I tried regenerating the lockfile from scratch yesterday out of curiosity, and ran into a ton of CI issues due to the updates.

[noahtallen]

I rebased to resolve conflicts and to incorporate @jsnajdr's changes.

[jsnajdr]

I rebased to resolve conflicts and to incorporate @jsnajdr's changes.

Hope it didn't cause too much trouble with updating the package-lock.json.

[noahtallen]

Hope it didn't cause too much trouble with updating the package-lock.json.

Not a problem! Turns out there's another lockfile conflicts anyways. Not sure if there's a better way to do this, but I just dropped all the package-lock changes that existed already, and ran npm install && npm dedupe again. (Thus leaving us with a single commit we could drop again if needed.)

I often rely on yarn auto-fixing the lock file in other repos... though I do see this: npm/npm#18007

[desrosj]

Sorry if I'm missing something obvious, but what exactly does this PR unblock that #48950 doesn't? AFAIK, npm workspaces isn't a blocker. The engine bump in #48950 also targets >=14 which shouldn't block core updates either. Copying code from #48950 without citing sources could also easily lose the discussions and context. The node/npm upgrade has been blocking contributors for months, so I'll love to see it merged, but I think we should maybe ask core folks first about their progress?

I'd just like to get some clarification on this to make sure there's nothing being missed and that the reasoning is clear.

I've updated wordpress-develop#4028 to be up to date with the current state of trunk. Unfortunately, Core will unfortunately need to remain on 16.x because of a systems level blocker. There are some "unable to resolve dependency" warnings, but everything seems to proceed without an issue. If you have any thoughts around this, please follow up on that PR (which we should merge around the same time).

I don't love that the two different code bases will be using different versions. The contributor experience of having to remember to switch versions each time you change from the Core to Gutenberg repos seems like it could cause a lot of strange scenarios.

I guess, as long as we continue to run tests on 16 in the Gutenberg repository, we can manage for now though. Testing in 16.x would ensure merging code into WordPress Core will go smoothly. I'm sure we'll encounter some strangeness between 16 and 18 at some point.

It would also be great to not enforce 18.x here and still allow contributors to use 16.x if desired. I'm not sure if there are any unforeseen consequences of that, though. This would allow anyone bouncing between Core and Gutenberg to remain on 16 without having to change.

Finally, this change will need to be communicated to the development community at large. I'll draft a post for this early next week.

[noahtallen]

The contributor experience of having to remember to switch versions each time you change from the Core to Gutenberg repos seems like it could cause a lot of strange scenarios.

I did want to mention using nvm to manage Node versions (https://github.com/nvm-sh/nvm). It's pretty easy to install and switches versions mostly automatically as you move between directories. I wouldn't be surprised if many JS devs already have this set up for any other projects too. Potentially worth mentioning in the post.

[desrosj]

I did want to mention using nvm to manage Node versions (https://github.com/nvm-sh/nvm). It's pretty easy to install and switches versions mostly automatically as you move between directories. I wouldn't be surprised if many JS devs already have this set up for any other projects too. Potentially worth mentioning in the post.

Totally agree. I think detailing this in the post is an easy win. I also have my machine set up to automatically detect an .nvm file when changing directory so I almost never need to worry about it. But I guess I'm more concerned with an additional barrier being added to getting started if you're a new contributor. Just something we should consider and weigh into any decisions.

[simison]

@youknowriad just reminding about earlier point and especially lack of security upgrades beyond 2023-09-11 is kind of a bummer:

Just heads up that Node.js v16 maintenance support ends in about 3 months (ref), i.e. end of security updates. So I'd assume that's when you need some kind of v18 upgrade plan regardless.

Edit: just saw https://make.wordpress.org/core/2023/08/09/updating-wordpress-to-use-more-modern-versions-of-node-js-npm/ posted, with explanation. :-) All good, thanks for your work folks!