OOMs on the Jest docusaurus site

[orta]

Maybe a leak in Shiki?

[orta]

Somewhere in the stack, there must be a str += char or something, all the new objects are concatenated strings:

E.g. this string is 3000 links which I think are just 3000 +=s of one char:

Which is this tmlanguage in shiki:

[orta]

I think it's more likely that the issue lays in the remark-shiki-twoslash somehow re-creating the highlighters, but I can't prove it.

[frencojobs]

Came across this bug today. This is a severe bug since this makes twoslash completely unusable. I think a lot more people might be using docusaurus with jest and this breaks the setup entirely.

The error is also scarier because I don't understand any of the logs and what's happening.

[frencojobs]

Could this be related to this facebook/docusaurus#4978?

[orta]

Yeah, I thought we were correctly caching the highlighters with https://github.com/shikijs/twoslash/blob/main/packages/remark-shiki-twoslash/src/index.ts#L61-L62

But maybe caching the highlighter doesn't cache the syntax JSONs

It's not a problem I've had on Gatsby with the TS website which has many hundreds of twoslash code samples, so there's a chance it's a docusaurus level but I think we can still find a way at this library level

[orta]

I think #43 might have fixed this,

[frencojobs]

I just checked it. Indeed, it is fixed. Thanks!

[frencojobs]

Probably not really the fault of twoslash. But remotion-dev/remotion@2f00e61 is still experiencing this. I'm guessing those garbage collectible weakmaps can only help a bit, probably might have to wait for facebook/docusaurus#4997.

[frencojobs]

This is reproducible with docusaurus build while docusaurus start yields no problem during development whatsoever.

[frencojobs]

Managed to reproduce the behavior with 80 (it was fine till 50) markdown files of this content on a plain docusaurus template.

```ts twoslash
type CreateMutable<Type> = {
  -readonly [Property in keyof Type]: Type[Property];
};

type LockedAccount = {
  readonly id: string;
  readonly name: string;
};

type UnlockedAccount = CreateMutable<LockedAccount>;
\```

The content is probably not important, just put it out there for reference.

[frencojobs]

When I increased the memory to 4096MB, the error is changed to something else.

Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\xml.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\xml.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\tsx.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\xml.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\xml.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\xml.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\groovy.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\xml.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\xml.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\xml.tmLanguage.json'
Error: EMFILE: too many open files, open 'D:\a\packages\docs\node_modules\shiki\languages\erlang.tmLanguage.json'

Maybe XML is related to tsx, but I don't have these groovy and erlang languages used in the code. Does shiki read all these files necessary or not??

Ref: https://github.com/remotion-dev/remotion/runs/2931935721

[orta]

Hrm, still seems a tad off - I would have thought that caching the shiki highlighter would mean those files only get set up once, but if there's dupes in there - it obviously isn't

[frencojobs]

Edit: Nope. Nevermind. It's still the same error on remotion code base with a lot more content.

Hey, I've found a fix. Use the remark-shiki-twoslash plugin as a remarkPlugin in the preset settings. The same 80 files and everything work fine with that setup. Since the hover script injection via theme is also no longer in use in our docusaurus preset, we would not even need it. Tomorrow I'll try to refactor the stuffs and probably make a theme to handle the MDXComponents overriding, adding copy to clipboard as done in the docusaurus normal preset etc.

Compiled successfully in 2.88m

The compilation time is also normal and way less longer than with the preset.

[frencojobs]

I'm really confused right now. Does the remark-shiki-twoslash code contain a logic bug or something? I set up a counter server and got this (on my 80 files docusaurus setup with remark plugin).

{
  "getHighlighter": 182, // how many times shiki.getHighlighter is called
  "setCache": 0, // how many times the data is set to weakmap
  "getFromCache": 0, // how many times the highlighters are taken from weakmap
  "fileHit": 1 // how many times the file is called
}

[frencojobs]

EDIT: I think I found the fix. Just needed to make the getHighlighter promise global and all functions await only on it so it's resolved only once. Let's go.

EDIT: Tried the fix prototype with 100+ files. Working smooth like butter. Woo.

I found out that the caches are set but every remark API call tries to load the highlighter and set the cache also. It seems like the remark's async handlers are being run parallel and the getHighlighter is taking quite some time so caching becomes useless as all of the functions are trying to load highlighters into cache at the same time.

When I add a console.log line on the start and end(before return) of the highlightersFromSettings function, I got all the before lines at once instead of getting them in pairs.

before 0 // <- length of highlightCache map
before 0
before 0
// ...
after 1 // <- length of highlightCache map
after 2 // <- why duplicated when there's only one setting
after 3

There's also the problem with duplicated keys in the map.

The correct behavior would be,

before 0
after 1 // only hit once because the function shouldn't reach to end if the cache exists
before 1 
before 1
before 1