misc: migrate all packages to ESM-only

[Josh-Cena]

Breaking changes

• For plugin authors or others who write server-side code that imports from Docusaurus packages: please migrate your code to ESM as well in order to continue importing our code. We will help migrate a few widely used community packages.

Motivation

Resolve #5379. Since we are now Node 14+, we can work on migrating our packages to ESM to be in sync with the JS community, especially Remark/Rehype.

The current TS tranpilation is not good enough so E2E tests are failing. From the release notes of TS 4.5, there would be prioritized support for ESM, so we can wait to upgrade TS.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

E2E tests, local dev preview

[slorber]

great 👍

isn't the problem in copyUntypedFiles.js ?

[Josh-Cena]

isn't the problem in copyUntypedFiles.js?

Man, it's far more than that😅 If it's just require => import I would've done it already

With ESM:

• You have to always specify the .js extension
• You have to explicitly say index.js when you are trying to import a directory

These are to make the behavior exactly like the browser behavior. There's a node flag --experimental-specifier-resolution=node to restore the previous behavior, but I've been unable to get it working. we may have to use the NODE_OPTIONS in the executable's hashbang. I'll look at that later, but with the prospected TS 4.5 migration, I'd wait a bit more

[Josh-Cena]

It's almost working! Currently blocked by import-fresh which uses CJS require. Seems the ESM support is stalled: sindresorhus/import-fresh#22

[slorber]

FYI we'd also need to require Node >= 14.14, not just >= 14

https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

[Josh-Cena]

When I check for features it doesn't seem 14.5 and 14.18 have a big difference besides some class fields stuff... I guess that Sindre Sorhus gist was just playing it safe by recommending the last minor version (while 14.14 is not the last minor version now)? 12.20 is also in the list, so I guess it would be fine. We have CI checks anyways

[slorber]

We have CI checks anyways

CI checks are "14", I suspect it's different than "14.0". We'll figure out later what min version to recommend ;)

[Josh-Cena]

@slorber what do we use import-fresh to achieve anyways? Is it used for watch mode?

[slorber]

importFresh(configPath) ensures notably that when file watchers trigger a site reload, imported JS modules (like config) are not cached and we actually load a fresh config, including the recent updates

[Josh-Cena]

I see... I saw somewhere that a common practice is to add unique query strings to each import call in the ESM world, but I also saw there's no proper GC in this case and risks OOM

[slorber]

Yes, the former cached modules will likely stay in the cache forever.

Isn't it possible to keep using importFresh and mixing ESM with some CJS here and there?

Or will we have to ask our users to convert their config/sidebars to ESM too?

[Josh-Cena]

Isn't it possible to keep using importFresh and mixing ESM with some CJS here and there?

Or will we have to ask our users to convert their config/sidebars to ESM too?

The interoperability of CJS and ESM is like:

• require of ESM always fails (because Node wants you to upgrade to use new features);
• import of CJS usually works (because Node has to support those ancient packages), but sometimes resolving named imports can be wonky because they have to be statically determined pre-runtime

Config files can stay as CJS as long as they don't try to require any Docusaurus package. I don't know if it would always work; worth trying. importFresh would never work because we use it to load plugins, which are now ESM-only.

[Josh-Cena]

There's an interesting suggestion of using worker threads to import fresh modules by killing the worker thread👀 I think workers can offer a new perspective of how Docusaurus works, but not sure if we want such architectural changes.

[RDIL]

Some day, this flag is may be removed or renamed and completely break everything. I feel like it's not a good idea to hard-code it.

[Josh-Cena]

Yes, this is just to make sure things work. Soon after things start working we will try to explicitly specify file names & extensions (especially when TS 4.6 has better support for this)

[RDIL]

I like the direction this is going in, but I really don't think ESM is near ready enough.

[Josh-Cena]

I like the direction this is going in, but I really don't think ESM is near ready enough.

Well we gotta make this move. MDX v2 is ESM-only, so we either have to build it as CJS and re-publish it, or we stay on v1 forever. A lot of remark/rehype plugins are ESM only as well, and ESM cascades down the require chain. Ultimately you just want the entire code base to be ESM.

[Josh-Cena]

Closing because of too many conflicts. We will revisit this in the future, and probably do it progressively (e.g. migrating tests first, then core, then plugins, then utils)