Docusaurus 2 memory leak with custom remark plugin.

[phryneas]

🐛 Bug Report

Recently our build have been exploding with a memory leak (unfortunately I can't pinpoint it exactly, CI seems to still build due to caching, but even the current master will OOM without cache).

Prerequisites

• I'm using the latest version of Docusaurus.
• I have tried the npm run clear or yarn clear command.
• I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.

Description

We are using https://github.com/phryneas/remark-typescript-tools to transpile our TS example code down to JS and typecheck it in the process.

The module exports, like every remark plugin, an attacher "Plugin function" that is called with the options declared in the docusaurus.config.js.

This attacher then creates a TypeScript compiler instance and returns a transformer function that can be called on files to do work.

Now, I'm very sure that previously, attacher was called only once, the compiler instance was reused and transformer was called multiple times, once for each file.

But what currently happens it that attacher is called 51 one times, creating 51 TypeScript compiler instances before transformer is even called for the first time.
With the default memory settings of 4GB, I don't even get to that point, setting it up to 16GB makes it work

Have you read the Contributing Guidelines on issues?

Yes

To Reproduce

git clone https://github.com/reduxjs/redux-toolkit
cd redux-toolkit
cd website
npm i
npm run netlify-build

Expected behavior

One instance of the plugin is created and reused.

Actual Behavior

[en] Creating an optimized production build...
ℹ Compiling Client
ℹ Compiling Server
Compiler 0: creating
Compiler 1: creating
Compiler 2: creating
Compiler 3: creating
Compiler 4: creating
Compiler 5: creating
Compiler 6: creating
Compiler 7: creating
Compiler 8: creating
Compiler 9: creating
Compiler 10: creating
Compiler 11: creating
Compiler 12: creating
Compiler 13: creating
Compiler 14: creating
Compiler 15: creating
Compiler 16: creating
Compiler 17: creating
Compiler 18: creating
Compiler 19: creating
Compiler 20: creating
Compiler 21: creating
Compiler 22: creating
Compiler 23: creating
Compiler 24: creating
Compiler 25: creating
Compiler 26: creating
Compiler 27: creating
Compiler 28: creating
Compiler 29: creating
Compiler 30: creating
Compiler 31: creating
Compiler 32: creating
Compiler 33: creating
Compiler 34: creating
Compiler 35: creating
Compiler 36: creating
Compiler 37: creating
Compiler 38: creating
Compiler 39: creating
Compiler 40: creating
Compiler 41: creating
Compiler 42: creating
Compiler 43: creating
Compiler 44: creating
Compiler 45: creating
Compiler 46: creating
Compiler 47: creating
Compiler 48: creating
Compiler 49: creating
Compiler 50: creating
Compiler 51: creating
Compiler 22: running /home/weber/tmp/redux-starter-kit/docs/rtk-query/usage/pagination.mdx
Compiler 22: finished
Compiler 38: running /home/weber/tmp/redux-starter-kit/docs/rtk-query/usage/code-splitting.mdx
Compiler 38: finished
Compiler 41: running /home/weber/tmp/redux-starter-kit/docs/rtk-query/api/setupListeners.mdx
Compiler 41: finished
Compiler 7: running /home/weber/tmp/redux-starter-kit/docs/api/otherExports.mdx
Compiler 7: finished
Compiler 14: running /home/weber/tmp/redux-starter-kit/docs/api/createSelector.mdx
Compiler 14: finished
Compiler 27: running /home/weber/tmp/redux-starter-kit/docs/rtk-query/usage/customizing-create-api.mdx
Compiler 27: finished
Compiler 36: running /home/weber/tmp/redux-starter-kit/docs/rtk-query/usage/code-generation.mdx
Compiler 47: running /home/weber/tmp/redux-starter-kit/docs/rtk-query/api/created-api/redux-integration.mdx
Compiler 47: finished
Compiler 0: running /home/weber/tmp/redux-starter-kit/docs/usage/usage-guide.md
Compiler 1: running /home/weber/tmp/redux-starter-kit/docs/usage/usage-with-typescript.md
Compiler 3: running /home/weber/tmp/redux-starter-kit/docs/api/createAsyncThunk.mdx
Compiler 3: finished
Compiler 4: running /home/weber/tmp/redux-starter-kit/docs/api/serializabilityMiddleware.mdx
Compiler 4: finished
Compiler 5: running /home/weber/tmp/redux-starter-kit/docs/api/createSlice.mdx
Compiler 5: finished
Compiler 6: running /home/weber/tmp/redux-starter-kit/docs/api/configureStore.mdx
Compiler 6: finished
Compiler 8: running /home/weber/tmp/redux-starter-kit/docs/api/createAction.mdx
Compiler 8: finished
Compiler 9: running /home/weber/tmp/redux-starter-kit/docs/api/immutabilityMiddleware.mdx

and a stack trace for how attacher is called:

at Function.attacher (/home/weber/tmp/redux-starter-kit/website/node_modules/remark-typescript-tools/dist/remark-typescript-tools.cjs.development.js:325:27)
    at freeze (/home/weber/tmp/redux-starter-kit/node_modules/unified/index.js:128:28)
    at Function.process (/home/weber/tmp/redux-starter-kit/node_modules/unified/index.js:362:5)
    at compile (/home/weber/tmp/redux-starter-kit/node_modules/@mdx-js/mdx/index.js:126:37)
    at Object.docusaurusMdxLoader (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/mdx-loader/src/index.js:67:20)
    at LOADER_EXECUTION (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:132:14)
    at runSyncOrAsync (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:133:4)
    at iterateNormalLoaders (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:250:2)
    at iterateNormalLoaders (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:239:10)
    at /home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:254:3
    at context.callback (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:124:13)
    at Object.markdownLoader (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/plugin-content-docs/lib/markdown/index.js:14:25)
    at LOADER_EXECUTION (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:132:14)
    at runSyncOrAsync (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:133:4)
    at iterateNormalLoaders (/home/weber/tmp/redux-starter-kit/node_modules/@docusaurus/core/node_modules/loader-runner/lib/LoaderRunner.js:250:2)
Error: initializing cache
    at cached (/home/weber/tmp/redux-starter-kit/website/node_modules/remark-typescript-tools/dist/remark-typescript-tools.cjs.development.js:285:11)

At this point and with this knowledge, of course I can try to rewrite the plugin, but I am very sure that similar "memory-explosive" problems will occur with other remark plugins as well.

[phryneas]

Having found out what Docusaurus does (spawning the plugin once for each file), I could now optimize for that case and memory consumption went back to 2.5GB. This is what I did on plugin side.

Of course this does not resolve this behaviour on Docusaurus' side and I guess a lot of Docusaurus' high memory consumption would go down by resolving it, but I thought I would share my success :)