module: rework of memory management in `vm` APIs with the `importModuleDynamically` option #48510

joyeecheung · 2023-06-21T16:01:05Z

This PR (hopefully) fixes a bunch of memory leaks & use-after-free surrounding vm.Script, vm.compileFunction, vm.SyntheticModule and vm.SourceTextModule when dynamic import is used (see the issues referenced below) and makes it possible for people to upgrade from older versions of Node.js.

This cannot land as-is on v20.x as it depends on #49491 and #49419 which break the ABI on v20x. We need a special Node.js v20-only extension to the V8 API for this to work without breaking the ABI.

module: use symbol in WeakMap to manage host defined options

Previously when managing the importModuleDynamically callback of
vm.compileFunction(), we use an ID number as the host defined option
and maintain a per-Environment ID -> CompiledFnEntry map to retain
the top-level referrer function returned by vm.compileFunction() in
order to pass it back to the callback, but it would leak because with
how we used v8::Persistent to maintain this reference, V8 would not
be able to understand the cycle and would just think that the
CompiledFnEntry was supposed to live forever. We made an attempt
to make that reference known to V8 by making the CompiledFnEntry weak
and using a private symbol to make CompiledFnEntry strongly
references the top-level referrer function in
#46785, but that turned out to be
unsound, because the there's no guarantee that the top-level function
must be alive while import() can still be initiated from that
function, since V8 could discard the top-level function and only keep
inner functions alive, so relying on the top-level function to keep
the CompiledFnEntry alive could result in use-after-free which caused
a revert of that fix.

With this patch we use a symbol in the host defined options instead of
a number, because with the stage-3 symbol-as-weakmap-keys proposal
we could directly use that symbol to keep the referrer alive using a
WeakMap. As a bonus this also keeps the other kinds of referrers
alive as long as import() can still be initiated from that
Script/Module, so this also fixes the long-standing crash caused by
vm.Script being GC'ed too early when its importModuleDynamically
callback still needs it.

module: fix leak of vm.SyntheticModule

Previously we maintain a strong persistent reference to the
ModuleWrap to retrieve the ID-to-ModuleWrap mapping from
the HostImportModuleDynamicallyCallback using the number ID
stored in the host-defined options. As a result the ModuleWrap
would be kept alive until the Environment is shut down, which
would be a leak for user code. With the new symbol-based
host-defined option we can just get the ModuleWrap from the
JS-land WeakMap so there's now no need to maintain this
strong reference. This would at least fix the leak for
vm.SyntheticModule. vm.SourceTextModule is still leaking
due to the strong persistent reference to the v8::Module.

module: fix the leak in SourceTextModule and ContextifySript

Replace the persistent handles to v8::Module and
v8::UnboundScript with an internal reference that V8's GC is
aware of to fix the leaks.

Refs: #44211
Refs: #42080
Refs: #47096
Refs: #43205
Refs: #38695

nodejs-github-bot · 2023-06-21T16:01:09Z

Review requested:

@nodejs/loaders
@nodejs/modules
@nodejs/vm

nodejs-github-bot · 2023-06-21T16:05:54Z

CI: https://ci.nodejs.org/job/node-test-pull-request/52320/

joyeecheung · 2023-06-21T16:34:12Z

Still need to do a bit more investigation to figure out it's really safe this time around, so marking it as WIP (I've done some walkthrough of the vm.compileFunction() part, but haven't really looked into the ModuleWrap part). I suppose we probably also want to put it behind flags until symbol-as-weakmap-key gets to stage 4 though. Also need to think of more test cases to exercise the new memory management.

lib/internal/vm/module.js

lib/internal/modules/esm/create_dynamic_module.js

lib/internal/modules/esm/loader.js

lib/internal/modules/esm/translators.js

lib/internal/vm.js

lib/vm.js

joyeecheung · 2023-06-22T11:11:55Z

Looks like this fixes the segfault in #43205 and #38695 as well

targos · 2023-06-22T12:21:48Z

amazing!

hildjj · 2023-06-22T13:23:49Z

It looks to me like symbols-as-weakmap-keys is now Stage 4 as of 2023-01-30, but not all of the documents have been updated.

I would guess back-porting this fix to Node 16 or 18 is going to be a challenge though, right?

joyeecheung · 2023-06-22T16:52:48Z

It looks to me like symbols-as-weakmap-keys is now Stage 4 as of 2023-01-30, but not all of the documents have been updated.

Nice. Then I think we don't have to flag this for v20. But yeah v18 or v16 didn't implement that proposal (and we probably shouldn't backport it / cannot do a full upgrade because LTS) so it cannot be backported.

I've tweaked the registry object a bit, added tests, and some analysis to the commit messages about why it fixes the leaks/segfaults. Although I think for the vm.Modules this just fixes vm.SyntheticModule, but vm.SourceTextModule would still leak until we find a way to create ModuleWrap -> v8::Module references without using the problematic v8::Persistent approach (maybe using the namespace object to transitively hold a reference to the Module would work, but that feels hacky to me, so I'll leave it for now).

nodejs-github-bot · 2023-06-22T21:06:11Z

CI: https://ci.nodejs.org/job/node-test-pull-request/52367/

Original commit message: [api] mark v8::Script and v8::UnboundScript as v8::Data v8::UnboundModuleScript and v8::Module are already v8::Data. Mark v8::Script and v8::UnboundScript as v8::Data so that they can be used in other V8 APIs that takes v8::Data. Refs: nodejs#48510 Bug: v8:14120 Change-Id: I2dd5648528c1b0030292872441758d4fb2cfcc1c Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4827307 Reviewed-by: Camillo Bruni <[email protected]> Commit-Queue: Joyee Cheung <[email protected]> Cr-Commit-Position: refs/heads/main@{#89727} Refs: v8/v8@b60a03d PR-URL: nodejs#49491 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Nitzan Uziely <[email protected]>

Previously when managing the importModuleDynamically callback of vm.compileFunction(), we use an ID number as the host defined option and maintain a per-Environment ID -> CompiledFnEntry map to retain the top-level referrer function returned by vm.compileFunction() in order to pass it back to the callback, but it would leak because with how we used v8::Persistent to maintain this reference, V8 would not be able to understand the cycle and would just think that the CompiledFnEntry was supposed to live forever. We made an attempt to make that reference known to V8 by making the CompiledFnEntry weak and using a private symbol to make CompiledFnEntry strongly references the top-level referrer function in nodejs#46785, but that turned out to be unsound, because the there's no guarantee that the top-level function must be alive while import() can still be initiated from that function, since V8 could discard the top-level function and only keep inner functions alive, so relying on the top-level function to keep the CompiledFnEntry alive could result in use-after-free which caused a revert of that fix. With this patch we use a symbol in the host defined options instead of a number, because with the stage-3 symbol-as-weakmap-keys proposal we could directly use that symbol to keep the referrer alive using a WeakMap. As a bonus this also keeps the other kinds of referrers alive as long as import() can still be initiated from that Script/Module, so this also fixes the long-standing crash caused by vm.Script being GC'ed too early when its importModuleDynamically callback still needs it. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Previously we maintain a strong persistent reference to the ModuleWrap to retrieve the ID-to-ModuleWrap mapping from the HostImportModuleDynamicallyCallback using the number ID stored in the host-defined options. As a result the ModuleWrap would be kept alive until the Environment is shut down, which would be a leak for user code. With the new symbol-based host-defined option we can just get the ModuleWrap from the JS-land WeakMap so there's now no need to maintain this strong reference. This would at least fix the leak for vm.SyntheticModule. vm.SourceTextModule is still leaking due to the strong persistent reference to the v8::Module. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Replace the persistent handles to v8::Module and v8::UnboundScript with an internal reference that V8's GC is aware of to fix the leaks. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Notable changes: deps: * add v8::Object::SetInternalFieldForNodeCore() (Joyee Cheung) nodejs#49874 doc: * deprecate `fs.F_OK`, `fs.R_OK`, `fs.W_OK`, `fs.X_OK` (Livia Medeiros) nodejs#49683 * deprecate `util.toUSVString` (Yagiz Nizipli) nodejs#49725 * deprecate calling `promisify` on a function that returns a promise (Antoine du Hamel) nodejs#49647 esm: * set all hooks as release candidate (Geoffrey Booth) nodejs#49597 module: * fix the leak in SourceTextModule and ContextifySript (Joyee Cheung) nodejs#48510 * fix leak of vm.SyntheticModule (Joyee Cheung) nodejs#48510 * use symbol in WeakMap to manage host defined options (Joyee Cheung) nodejs#48510 src: * (SEMVER-MINOR) allow embedders to override NODE_MODULE_VERSION (Cheng Zhao) nodejs#49279 stream: * use bitmap in writable state (Raz Luvaton) nodejs#49834 * use bitmap in readable state (Benjamin Gruenbaum) nodejs#49745 * improve webstream readable async iterator performance (Raz Luvaton) nodejs#49662 test_runner: * (SEMVER-MINOR) accept `testOnly` in `run` (Moshe Atlow) nodejs#49753 * (SEMVER-MINOR) add junit reporter (Moshe Atlow) nodejs#49614 PR-URL: nodejs#49932

Previously when managing the importModuleDynamically callback of vm.compileFunction(), we use an ID number as the host defined option and maintain a per-Environment ID -> CompiledFnEntry map to retain the top-level referrer function returned by vm.compileFunction() in order to pass it back to the callback, but it would leak because with how we used v8::Persistent to maintain this reference, V8 would not be able to understand the cycle and would just think that the CompiledFnEntry was supposed to live forever. We made an attempt to make that reference known to V8 by making the CompiledFnEntry weak and using a private symbol to make CompiledFnEntry strongly references the top-level referrer function in nodejs#46785, but that turned out to be unsound, because the there's no guarantee that the top-level function must be alive while import() can still be initiated from that function, since V8 could discard the top-level function and only keep inner functions alive, so relying on the top-level function to keep the CompiledFnEntry alive could result in use-after-free which caused a revert of that fix. With this patch we use a symbol in the host defined options instead of a number, because with the stage-3 symbol-as-weakmap-keys proposal we could directly use that symbol to keep the referrer alive using a WeakMap. As a bonus this also keeps the other kinds of referrers alive as long as import() can still be initiated from that Script/Module, so this also fixes the long-standing crash caused by vm.Script being GC'ed too early when its importModuleDynamically callback still needs it. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Previously we maintain a strong persistent reference to the ModuleWrap to retrieve the ID-to-ModuleWrap mapping from the HostImportModuleDynamicallyCallback using the number ID stored in the host-defined options. As a result the ModuleWrap would be kept alive until the Environment is shut down, which would be a leak for user code. With the new symbol-based host-defined option we can just get the ModuleWrap from the JS-land WeakMap so there's now no need to maintain this strong reference. This would at least fix the leak for vm.SyntheticModule. vm.SourceTextModule is still leaking due to the strong persistent reference to the v8::Module. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Replace the persistent handles to v8::Module and v8::UnboundScript with an internal reference that V8's GC is aware of to fix the leaks. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Previously when managing the importModuleDynamically callback of vm.compileFunction(), we use an ID number as the host defined option and maintain a per-Environment ID -> CompiledFnEntry map to retain the top-level referrer function returned by vm.compileFunction() in order to pass it back to the callback, but it would leak because with how we used v8::Persistent to maintain this reference, V8 would not be able to understand the cycle and would just think that the CompiledFnEntry was supposed to live forever. We made an attempt to make that reference known to V8 by making the CompiledFnEntry weak and using a private symbol to make CompiledFnEntry strongly references the top-level referrer function in nodejs#46785, but that turned out to be unsound, because the there's no guarantee that the top-level function must be alive while import() can still be initiated from that function, since V8 could discard the top-level function and only keep inner functions alive, so relying on the top-level function to keep the CompiledFnEntry alive could result in use-after-free which caused a revert of that fix. With this patch we use a symbol in the host defined options instead of a number, because with the stage-3 symbol-as-weakmap-keys proposal we could directly use that symbol to keep the referrer alive using a WeakMap. As a bonus this also keeps the other kinds of referrers alive as long as import() can still be initiated from that Script/Module, so this also fixes the long-standing crash caused by vm.Script being GC'ed too early when its importModuleDynamically callback still needs it. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Previously we maintain a strong persistent reference to the ModuleWrap to retrieve the ID-to-ModuleWrap mapping from the HostImportModuleDynamicallyCallback using the number ID stored in the host-defined options. As a result the ModuleWrap would be kept alive until the Environment is shut down, which would be a leak for user code. With the new symbol-based host-defined option we can just get the ModuleWrap from the JS-land WeakMap so there's now no need to maintain this strong reference. This would at least fix the leak for vm.SyntheticModule. vm.SourceTextModule is still leaking due to the strong persistent reference to the v8::Module. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Replace the persistent handles to v8::Module and v8::UnboundScript with an internal reference that V8's GC is aware of to fix the leaks. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Previously when managing the importModuleDynamically callback of vm.compileFunction(), we use an ID number as the host defined option and maintain a per-Environment ID -> CompiledFnEntry map to retain the top-level referrer function returned by vm.compileFunction() in order to pass it back to the callback, but it would leak because with how we used v8::Persistent to maintain this reference, V8 would not be able to understand the cycle and would just think that the CompiledFnEntry was supposed to live forever. We made an attempt to make that reference known to V8 by making the CompiledFnEntry weak and using a private symbol to make CompiledFnEntry strongly references the top-level referrer function in nodejs#46785, but that turned out to be unsound, because the there's no guarantee that the top-level function must be alive while import() can still be initiated from that function, since V8 could discard the top-level function and only keep inner functions alive, so relying on the top-level function to keep the CompiledFnEntry alive could result in use-after-free which caused a revert of that fix. With this patch we use a symbol in the host defined options instead of a number, because with the stage-3 symbol-as-weakmap-keys proposal we could directly use that symbol to keep the referrer alive using a WeakMap. As a bonus this also keeps the other kinds of referrers alive as long as import() can still be initiated from that Script/Module, so this also fixes the long-standing crash caused by vm.Script being GC'ed too early when its importModuleDynamically callback still needs it. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Previously we maintain a strong persistent reference to the ModuleWrap to retrieve the ID-to-ModuleWrap mapping from the HostImportModuleDynamicallyCallback using the number ID stored in the host-defined options. As a result the ModuleWrap would be kept alive until the Environment is shut down, which would be a leak for user code. With the new symbol-based host-defined option we can just get the ModuleWrap from the JS-land WeakMap so there's now no need to maintain this strong reference. This would at least fix the leak for vm.SyntheticModule. vm.SourceTextModule is still leaking due to the strong persistent reference to the v8::Module. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Replace the persistent handles to v8::Module and v8::UnboundScript with an internal reference that V8's GC is aware of to fix the leaks. PR-URL: nodejs#48510 Refs: nodejs#44211 Refs: nodejs#42080 Refs: nodejs#47096 Refs: nodejs#43205 Refs: nodejs#38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

hildjj · 2024-01-28T20:06:24Z

@joyeecheung Is there a way to detect this fix from unprivileged code that is better than checking a bunch of version numbers?

Previously when managing the importModuleDynamically callback of vm.compileFunction(), we use an ID number as the host defined option and maintain a per-Environment ID -> CompiledFnEntry map to retain the top-level referrer function returned by vm.compileFunction() in order to pass it back to the callback, but it would leak because with how we used v8::Persistent to maintain this reference, V8 would not be able to understand the cycle and would just think that the CompiledFnEntry was supposed to live forever. We made an attempt to make that reference known to V8 by making the CompiledFnEntry weak and using a private symbol to make CompiledFnEntry strongly references the top-level referrer function in #46785, but that turned out to be unsound, because the there's no guarantee that the top-level function must be alive while import() can still be initiated from that function, since V8 could discard the top-level function and only keep inner functions alive, so relying on the top-level function to keep the CompiledFnEntry alive could result in use-after-free which caused a revert of that fix. With this patch we use a symbol in the host defined options instead of a number, because with the stage-3 symbol-as-weakmap-keys proposal we could directly use that symbol to keep the referrer alive using a WeakMap. As a bonus this also keeps the other kinds of referrers alive as long as import() can still be initiated from that Script/Module, so this also fixes the long-standing crash caused by vm.Script being GC'ed too early when its importModuleDynamically callback still needs it. PR-URL: #48510 Backport-PR-URL: #51004 Refs: #44211 Refs: #42080 Refs: #47096 Refs: #43205 Refs: #38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Previously we maintain a strong persistent reference to the ModuleWrap to retrieve the ID-to-ModuleWrap mapping from the HostImportModuleDynamicallyCallback using the number ID stored in the host-defined options. As a result the ModuleWrap would be kept alive until the Environment is shut down, which would be a leak for user code. With the new symbol-based host-defined option we can just get the ModuleWrap from the JS-land WeakMap so there's now no need to maintain this strong reference. This would at least fix the leak for vm.SyntheticModule. vm.SourceTextModule is still leaking due to the strong persistent reference to the v8::Module. PR-URL: #48510 Backport-PR-URL: #51004 Refs: #44211 Refs: #42080 Refs: #47096 Refs: #43205 Refs: #38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Replace the persistent handles to v8::Module and v8::UnboundScript with an internal reference that V8's GC is aware of to fix the leaks. PR-URL: #48510 Backport-PR-URL: #51004 Refs: #44211 Refs: #42080 Refs: #47096 Refs: #43205 Refs: #38695 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [node](https://nodejs.org) ([source](https://github.com/nodejs/node)) | minor | `18.19.1` -> `18.20.0` | --- ### Release Notes <details> <summary>nodejs/node (node)</summary> ### [`v18.20.0`](https://github.com/nodejs/node/releases/tag/v18.20.0): 2024-03-26, Version 18.20.0 'Hydrogen' (LTS), @richardlau [Compare Source](nodejs/node@v18.19.1...v18.20.0) ##### Notable Changes ##### Added support for import attributes Support has been added for import attributes, to replace the old import assertions syntax. This will aid migration by making the new syntax available across all currently supported Node.js release lines. This adds the `with` keyword which should be used in place of the previous `assert` keyword, which will be removed in a future semver-major Node.js release. For example, ```console import "foo" assert { ... } ``` should be replaced with ```console import "foo" with { ... } ``` For more details, see - [#50134](nodejs/node#50134) - [#51622](nodejs/node#51622) Contributed by Nicolò Ribaudo in [#51136](nodejs/node#51136) and Antoine du Hamel in [#50140](nodejs/node#50140). ##### Doc deprecation for `dirent.path` Please use newly added `dirent.parentPath` instead. Contributed by Antoine du Hamel in [#50976](nodejs/node#50976) and [#51020](nodejs/node#51020). ##### Experimental node-api feature flags Introduces an experimental feature to segregate finalizers that affect GC state. A new type called `node_api_nogc_env` has been introduced as the const version of `napi_env` and `node_api_nogc_finalize` as a variant of `napi_finalize` that accepts a `node_api_nogc_env` as its first argument. This feature can be turned off by defining `NODE_API_EXPERIMENTAL_NOGC_ENV_OPT_OUT`. Contributed by Gabriel Schulhof in [#50060](nodejs/node#50060). ##### Root certificates updated to NSS 3.98 Certificates added: - Telekom Security TLS ECC Root 2020 - Telekom Security TLS RSA Root 2023 Certificates removed: - Security Communication Root CA ##### Updated dependencies - ada updated to 2.7.6. - base64 updated to 0.5.2. - c-ares updated to 1.27.0. - corepack updated to 0.25.2. - ICU updated to 74.2. Includes CLDR 44.1 and Unicode 15.1. - npm updated to 10.5.0. Fixes a regression in signals not being passed onto child processes. - simdutf8 updated to 4.0.8. - Timezone updated to 2024a. - zlib updated to 1.3.0.1-motley-40e35a7. ##### vm: fix V8 compilation cache support for vm.Script Previously repeated compilation of the same source code using `vm.Script` stopped hitting the V8 compilation cache after v16.x when support for `importModuleDynamically` was added to `vm.Script`, resulting in a performance regression that blocked users (in particular Jest users) from upgrading from v16.x. The recent fixes allow the compilation cache to be hit again for `vm.Script` when `--experimental-vm-modules` is not used even in the presence of the `importModuleDynamically` option, so that users affected by the performance regression can now upgrade. Ongoing work is also being done to enable compilation cache support for `vm.CompileFunction`. Contributed by Joyee Cheung in [#49950](nodejs/node#49950) and [#50137](nodejs/node#50137). ##### Commits - \[[`c70383b8d4`](nodejs/node@c70383b8d4)] - **build**: support Python 3.12 (Shi Pujin) [#50209](nodejs/node#50209) - \[[`4b960c3a4a`](nodejs/node@4b960c3a4a)] - **build**: fix incorrect g++ warning message (Richard Lau) [#51695](nodejs/node#51695) - \[[`8fdea67694`](nodejs/node@8fdea67694)] - **crypto**: update root certificates to NSS 3.98 (Node.js GitHub Bot) [#51794](nodejs/node#51794) - \[[`812b126dd9`](nodejs/node@812b126dd9)] - **deps**: V8: cherry-pick [`d90d453`](nodejs/node@d90d4533b053) (Michaël Zasso) [#50077](nodejs/node#50077) - \[[`9ab8c3db87`](nodejs/node@9ab8c3db87)] - **deps**: update c-ares to 1.27.0 (Node.js GitHub Bot) [#51846](nodejs/node#51846) - \[[`c688680387`](nodejs/node@c688680387)] - **deps**: update c-ares to 1.26.0 (Node.js GitHub Bot) [#51582](nodejs/node#51582) - \[[`9498ac8a47`](nodejs/node@9498ac8a47)] - **deps**: compile c-ares with C11 support (Michaël Zasso) [#51410](nodejs/node#51410) - \[[`8fb743642f`](nodejs/node@8fb743642f)] - **deps**: update c-ares to 1.25.0 (Node.js GitHub Bot) [#51385](nodejs/node#51385) - \[[`7bea2d7c12`](nodejs/node@7bea2d7c12)] - **deps**: update zlib to 1.3.0.1-motley-40e35a7 (Node.js GitHub Bot) [#51274](nodejs/node#51274) - \[[`57a38c8f75`](nodejs/node@57a38c8f75)] - **deps**: update zlib to 1.3.0.1-motley-dd5fc13 (Node.js GitHub Bot) [#51105](nodejs/node#51105) - \[[`b0ca084a6b`](nodejs/node@b0ca084a6b)] - **deps**: update zlib to 1.3-22124f5 (Node.js GitHub Bot) [#50910](nodejs/node#50910) - \[[`4b43823f37`](nodejs/node@4b43823f37)] - **deps**: update zlib to 1.2.13.1-motley-5daffc7 (Node.js GitHub Bot) [#50803](nodejs/node#50803) - \[[`f0da591812`](nodejs/node@f0da591812)] - **deps**: update zlib to 1.2.13.1-motley-dfc48fc (Node.js GitHub Bot) [#50456](nodejs/node#50456) - \[[`16d28a883a`](nodejs/node@16d28a883a)] - **deps**: update base64 to 0.5.2 (Node.js GitHub Bot) [#51455](nodejs/node#51455) - \[[`13a9e81cb6`](nodejs/node@13a9e81cb6)] - **deps**: update base64 to 0.5.1 (Node.js GitHub Bot) [#50629](nodejs/node#50629) - \[[`b4502d3ac5`](nodejs/node@b4502d3ac5)] - **deps**: update simdutf to 4.0.8 (Node.js GitHub Bot) [#51000](nodejs/node#51000) - \[[`183cf8a74a`](nodejs/node@183cf8a74a)] - **deps**: update simdutf to 4.0.4 (Node.js GitHub Bot) [#50772](nodejs/node#50772) - \[[`11ba8593ea`](nodejs/node@11ba8593ea)] - **deps**: update ada to 2.7.6 (Node.js GitHub Bot) [#51542](nodejs/node#51542) - \[[`73a946d55c`](nodejs/node@73a946d55c)] - **deps**: update ada to 2.7.5 (Node.js GitHub Bot) [#51542](nodejs/node#51542) - \[[`cc434c1a39`](nodejs/node@cc434c1a39)] - **deps**: update ada to 2.7.4 (Node.js GitHub Bot) [#50815](nodejs/node#50815) - \[[`3a3808a6ae`](nodejs/node@3a3808a6ae)] - **deps**: upgrade npm to 10.5.0 (npm team) [#51913](nodejs/node#51913) - \[[`c8876d765c`](nodejs/node@c8876d765c)] - **deps**: upgrade npm to 10.3.0 (npm team) [#51431](nodejs/node#51431) - \[[`5aec3af460`](nodejs/node@5aec3af460)] - **deps**: update corepack to 0.25.2 (Node.js GitHub Bot) [#51810](nodejs/node#51810) - \[[`a593985326`](nodejs/node@a593985326)] - **deps**: update corepack to 0.24.1 (Node.js GitHub Bot) [#51459](nodejs/node#51459) - \[[`d1a9237bf5`](nodejs/node@d1a9237bf5)] - **deps**: update corepack to 0.24.0 (Node.js GitHub Bot) [#51318](nodejs/node#51318) - \[[`adac0c7a63`](nodejs/node@adac0c7a63)] - **deps**: update corepack to 0.23.0 (Node.js GitHub Bot) [#50563](nodejs/node#50563) - \[[`4a6f83e32a`](nodejs/node@4a6f83e32a)] - **deps**: escape Python strings correctly (Michaël Zasso) [#50695](nodejs/node#50695) - \[[`c13969e52a`](nodejs/node@c13969e52a)] - **deps**: V8: cherry-pick [`ea996ad`](nodejs/node@ea996ad04a68) (Nicolò Ribaudo) [#51136](nodejs/node#51136) - \[[`6fbf0ba5c3`](nodejs/node@6fbf0ba5c3)] - **deps**: V8: cherry-pick [`a0fd320`](nodejs/node@a0fd3209dda8) (Nicolò Ribaudo) [#51136](nodejs/node#51136) - \[[`68fd7516e1`](nodejs/node@68fd7516e1)] - **deps**: update timezone to 2024a (Michaël Zasso) [#51723](nodejs/node#51723) - \[[`f9b229ebe1`](nodejs/node@f9b229ebe1)] - **deps**: update icu to 74.2 (Michaël Zasso) [#51723](nodejs/node#51723) - \[[`90c73d2eb4`](nodejs/node@90c73d2eb4)] - **deps**: update timezone to 2023d (Node.js GitHub Bot) [#51461](nodejs/node#51461) - \[[`2a2bf57028`](nodejs/node@2a2bf57028)] - **deps**: update icu to 74.1 (Node.js GitHub Bot) [#50515](nodejs/node#50515) - \[[`425e011e52`](nodejs/node@425e011e52)] - **deps**: add v8::Object::SetInternalFieldForNodeCore() (Joyee Cheung) [#49874](nodejs/node#49874) - \[[`58c70344a2`](nodejs/node@58c70344a2)] - **deps**: V8: cherry-pick [`705e374`](nodejs/node@705e374124ae) (Joyee Cheung) [#51004](nodejs/node#51004) - \[[`b0e88899e1`](nodejs/node@b0e88899e1)] - **deps**: V8: cherry-pick [`1fada6b`](nodejs/node@1fada6b36f8d) (Joyee Cheung) [#51004](nodejs/node#51004) - \[[`d87a810b81`](nodejs/node@d87a810b81)] - **deps**: V8: cherry-pick [`3dd9576`](nodejs/node@3dd9576ce336) (Joyee Cheung) [#51004](nodejs/node#51004) - \[[`6d50966876`](nodejs/node@6d50966876)] - **deps**: V8: cherry-pick [`94e8282`](nodejs/node@94e8282325a1) (Joyee Cheung) [#51004](nodejs/node#51004) - \[[`fafbacdfec`](nodejs/node@fafbacdfec)] - **deps**: V8: cherry-pick [`9a98f96`](nodejs/node@9a98f96b6d68) (Joyee Cheung) [#51004](nodejs/node#51004) - \[[`d4a530ed8d`](nodejs/node@d4a530ed8d)] - **deps**: V8: cherry-pick [`7f5daed`](nodejs/node@7f5daed62d47) (Joyee Cheung) [#51004](nodejs/node#51004) - \[[`1ce901b164`](nodejs/node@1ce901b164)] - **deps**: V8: cherry-pick [`c400af4`](nodejs/node@c400af48b5ef) (Joyee Cheung) [#51004](nodejs/node#51004) - \[[`f232064f35`](nodejs/node@f232064f35)] - **doc**: fix historical experimental fetch flag (Kenrick) [#51506](nodejs/node#51506) - \[[`194ff6a40f`](nodejs/node@194ff6a40f)] - **(SEMVER-MINOR)** **doc**: add deprecation notice to `dirent.path` (Antoine du Hamel) [#50976](nodejs/node#50976) - \[[`0f09267dc6`](nodejs/node@0f09267dc6)] - **(SEMVER-MINOR)** **doc**: deprecate `dirent.path` (Antoine du Hamel) [#50976](nodejs/node#50976) - \[[`8bfb8f5b2f`](nodejs/node@8bfb8f5b2f)] - **doc,crypto**: further clarify RSA_PKCS1\_PADDING support (Tobias Nießen) [#51799](nodejs/node#51799) - \[[`c7baf7b274`](nodejs/node@c7baf7b274)] - **doc,crypto**: add changelog and note about disabled RSA_PKCS1\_PADDING (Filip Skokan) [#51782](nodejs/node#51782) - \[[`a193be3dc2`](nodejs/node@a193be3dc2)] - **esm**: use import attributes instead of import assertions (Antoine du Hamel) [#50140](nodejs/node#50140) - \[[`26e8f7793e`](nodejs/node@26e8f7793e)] - **(SEMVER-MINOR)** **fs**: introduce `dirent.parentPath` (Antoine du Hamel) [#50976](nodejs/node#50976) - \[[`5b5e5192f7`](nodejs/node@5b5e5192f7)] - **lib**: fix compileFunction throws range error for negative numbers (Jithil P Ponnan) [#49855](nodejs/node#49855) - \[[`7552de6806`](nodejs/node@7552de6806)] - **module**: fix the leak in SourceTextModule and ContextifySript (Joyee Cheung) [#48510](nodejs/node#48510) - \[[`2e05cf1c60`](nodejs/node@2e05cf1c60)] - **module**: fix leak of vm.SyntheticModule (Joyee Cheung) [#48510](nodejs/node#48510) - \[[`a86a2e14a3`](nodejs/node@a86a2e14a3)] - **module**: use symbol in WeakMap to manage host defined options (Joyee Cheung) [#48510](nodejs/node#48510) - \[[`32906ddcac`](nodejs/node@32906ddcac)] - **node-api**: segregate nogc APIs from rest via type system (Gabriel Schulhof) [#50060](nodejs/node#50060) - \[[`1aa71c26ff`](nodejs/node@1aa71c26ff)] - **node-api**: factor out common code into macros (Gabriel Schulhof) [#50664](nodejs/node#50664) - \[[`3d0b233f52`](nodejs/node@3d0b233f52)] - **node-api**: introduce experimental feature flags (Gabriel Schulhof) [#50991](nodejs/node#50991) - \[[`96514a8b9f`](nodejs/node@96514a8b9f)] - **src**: iterate on import attributes array correctly (Michaël Zasso) [#50703](nodejs/node#50703) - \[[`2c2892bf88`](nodejs/node@2c2892bf88)] - **src**: set ModuleWrap internal fields only once (Joyee Cheung) [#49391](nodejs/node#49391) - \[[`ff334cb774`](nodejs/node@ff334cb774)] - **src**: cast v8::Object::GetInternalField() return value to v8::Value (Joyee Cheung) [#48943](nodejs/node#48943) - \[[`270b519971`](nodejs/node@270b519971)] - **stream**: do not defer construction by one microtick (Matteo Collina) [#52005](nodejs/node#52005) - \[[`95d7a75084`](nodejs/node@95d7a75084)] - **test**: fix dns test case failures after c-ares update to 1.21.0+ (Brad House) [#50743](nodejs/node#50743) - \[[`cd613e5167`](nodejs/node@cd613e5167)] - **test**: handle relative https redirect (Richard Lau) [#51121](nodejs/node#51121) - \[[`40f10eafcf`](nodejs/node@40f10eafcf)] - **test**: fix `internet/test-inspector-help-page` (Richard Lau) [#51693](nodejs/node#51693) - \[[`5e426511b1`](nodejs/node@5e426511b1)] - **test**: deflake test-vm-contextified-script-leak (Joyee Cheung) [#49710](nodejs/node#49710) - \[[`0b156c6d28`](nodejs/node@0b156c6d28)] - **test**: use checkIfCollectable in vm leak tests (Joyee Cheung) [#49671](nodejs/node#49671) - \[[`1586c11b3c`](nodejs/node@1586c11b3c)] - **test**: add checkIfCollectable to test/common/gc.js (Joyee Cheung) [#49671](nodejs/node#49671) - \[[`902d8b3d4b`](nodejs/node@902d8b3d4b)] - **test**: fix flaky http-chunk-extensions-limit test (Ethan Arrowood) [#51943](nodejs/node#51943) - \[[`1743d2bdc1`](nodejs/node@1743d2bdc1)] - **test**: test surrogate pair filenames on windows (Mert Can Altın) [#51800](nodejs/node#51800) - \[[`1c1a7ec22d`](nodejs/node@1c1a7ec22d)] - **test**: increase platform timeout zlib-brotli-16gb (Rafael Gonzaga) [#51792](nodejs/node#51792) - \[[`931d02fe3e`](nodejs/node@931d02fe3e)] - **test, v8**: fix wrong import attributes test (Nicolò Ribaudo) [#52184](nodejs/node#52184) - \[[`d9ea6c1f8d`](nodejs/node@d9ea6c1f8d)] - **tls**: fix order of setting cipher before setting cert and key (Kumar Rishav) [#50186](nodejs/node#50186) - \[[`3184befa2e`](nodejs/node@3184befa2e)] - **tools**: fix update-icu.sh (Michaël Zasso) [#51723](nodejs/node#51723) - \[[`06646e11be`](nodejs/node@06646e11be)] - **(SEMVER-MINOR)** **vm**: use import attributes instead of import assertions (Antoine du Hamel) [#50141](nodejs/node#50141) - \[[`fe66e9d06e`](nodejs/node@fe66e9d06e)] - **vm**: reject in importModuleDynamically without --experimental-vm-modules (Joyee Cheung) [#50137](nodejs/node#50137) - \[[`052e095c6b`](nodejs/node@052e095c6b)] - **vm**: use internal versions of compileFunction and Script (Joyee Cheung) [#50137](nodejs/node#50137) - \[[`9f7899ed0a`](nodejs/node@9f7899ed0a)] - **vm**: unify host-defined option generation in vm.compileFunction (Joyee Cheung) [#50137](nodejs/node#50137) - \[[`6291c107d0`](nodejs/node@6291c107d0)] - **vm**: use default HDO when importModuleDynamically is not set (Joyee Cheung) [#49950](nodejs/node#49950) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://harton.dev/cinder/cinder-space/pulls/25 Co-authored-by: Renovate Bot <[email protected]> Co-committed-by: Renovate Bot <[email protected]>

Notable changes: deps: * add v8::Object::SetInternalFieldForNodeCore() (Joyee Cheung) nodejs#49874 doc: * deprecate `fs.F_OK`, `fs.R_OK`, `fs.W_OK`, `fs.X_OK` (Livia Medeiros) nodejs#49683 * deprecate `util.toUSVString` (Yagiz Nizipli) nodejs#49725 * deprecate calling `promisify` on a function that returns a promise (Antoine du Hamel) nodejs#49647 esm: * set all hooks as release candidate (Geoffrey Booth) nodejs#49597 module: * fix the leak in SourceTextModule and ContextifySript (Joyee Cheung) nodejs#48510 * fix leak of vm.SyntheticModule (Joyee Cheung) nodejs#48510 * use symbol in WeakMap to manage host defined options (Joyee Cheung) nodejs#48510 src: * (SEMVER-MINOR) allow embedders to override NODE_MODULE_VERSION (Cheng Zhao) nodejs#49279 stream: * use bitmap in writable state (Raz Luvaton) nodejs#49834 * use bitmap in readable state (Benjamin Gruenbaum) nodejs#49745 * improve webstream readable async iterator performance (Raz Luvaton) nodejs#49662 test_runner: * (SEMVER-MINOR) accept `testOnly` in `run` (Moshe Atlow) nodejs#49753 * (SEMVER-MINOR) add junit reporter (Moshe Atlow) nodejs#49614 PR-URL: nodejs#49932

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jun 21, 2023

joyeecheung marked this pull request as draft June 21, 2023 16:01

joyeecheung changed the title ~~WIP: fix vm.compileFunction() dynamic import callback leak~~ WIP: fix leak in vm.compileFunction() dynamic import callback and the ModuleWrap Jun 21, 2023

This was referenced Jun 21, 2023

vm: fix leak in vm.compileFunction when importModuleDynamically is used #46785

Closed

vm: memory leak on vm.compileFunction with importModuleDynamically #44211

Closed

aduh95 reviewed Jun 21, 2023

View reviewed changes

ljharb reviewed Jun 21, 2023

View reviewed changes

lib/internal/vm.js Show resolved Hide resolved

lib/vm.js Show resolved Hide resolved

joyeecheung mentioned this pull request Jun 22, 2023

Segmentation fault when executing worker with eval: true containing dynamic import after await #43205

Closed

joyeecheung changed the title ~~WIP: fix leak in vm.compileFunction() dynamic import callback and the ModuleWrap~~ WIP: use symbol in weakmap to maintain lifetime of modules' host defined options Jun 22, 2023

joyeecheung force-pushed the fix-compile branch 2 times, most recently from 1751348 to 76f5ce9 Compare June 22, 2023 16:47

This comment was marked as outdated.

Sign in to view

joyeecheung force-pushed the fix-compile branch from 76f5ce9 to 9b07d0c Compare June 22, 2023 16:56

This comment was marked as outdated.

Sign in to view

joyeecheung force-pushed the fix-compile branch from 9b07d0c to 3fca427 Compare June 22, 2023 17:18

This comment was marked as outdated.

Sign in to view

This was referenced Jun 23, 2023

CI Reliability 2023-06-23 nodejs/reliability#596

Open

CI Reliability 2023-06-24 nodejs/reliability#597

Open

CI Reliability 2023-06-25 nodejs/reliability#598

Open

sutt0n mentioned this pull request Jun 26, 2023

[Bug]: Memory consumption issues on Node JS 16.11.0+ jestjs/jest#11956

Closed

joyeecheung force-pushed the fix-compile branch from aace036 to 59a6ca5 Compare June 27, 2023 18:46

joberstein mentioned this pull request Oct 29, 2023

fix: update dependency cosmiconfig to v8.3.6 - autoclosed conventional-changelog/commitlint#3654

Closed

1 task

SimenB mentioned this pull request Oct 30, 2023

[Bug]: False positive memory leak when running with --experimental-vm-modules and using --detectLeaks jestjs/jest#14649

Closed

joyeecheung mentioned this pull request Nov 30, 2023

VM's context when evaluating ESM script not garbage-collected (works with CJS) #50964

Closed

joyeecheung mentioned this pull request Dec 1, 2023

[18.x] vm: backport vm-related fixes #51004

Closed

davidmehren mentioned this pull request Jan 18, 2024

replace flexsearch-ts with orama and upgrade to next 14 hedgedoc/hedgedoc#5354

Merged

3 tasks

codebytere mentioned this pull request Mar 1, 2024

fix: node::loader::ImportModuleDynamically crash electron/electron#41491

Merged

4 tasks

alexvy86 mentioned this pull request Mar 8, 2024

fix(ci): Disable jest cache for app-insights-logger tests microsoft/FluidFramework#20037

Merged

robhogan mentioned this pull request Mar 13, 2024

Support ESM react-native.config files react-native-community/cli#2167

Closed

richardlau mentioned this pull request Mar 20, 2024

v18.20.0 proposal #52165

Merged

joyeecheung mentioned this pull request May 22, 2024

cli: add --expose-gc flag available to NODE_OPTIONS #53078

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

module: rework of memory management in `vm` APIs with the `importModuleDynamically` option #48510

module: rework of memory management in `vm` APIs with the `importModuleDynamically` option #48510

joyeecheung commented Jun 21, 2023 •

edited

Loading

nodejs-github-bot commented Jun 21, 2023

nodejs-github-bot commented Jun 21, 2023

joyeecheung commented Jun 21, 2023 •

edited

Loading

joyeecheung commented Jun 22, 2023 •

edited

Loading

targos commented Jun 22, 2023

hildjj commented Jun 22, 2023 •

edited

Loading

joyeecheung commented Jun 22, 2023 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

nodejs-github-bot commented Jun 22, 2023

hildjj commented Jan 28, 2024

module: rework of memory management in vm APIs with the importModuleDynamically option #48510

module: rework of memory management in vm APIs with the importModuleDynamically option #48510

Conversation

joyeecheung commented Jun 21, 2023 • edited Loading

module: use symbol in WeakMap to manage host defined options

module: fix leak of vm.SyntheticModule

module: fix the leak in SourceTextModule and ContextifySript

nodejs-github-bot commented Jun 21, 2023

nodejs-github-bot commented Jun 21, 2023

joyeecheung commented Jun 21, 2023 • edited Loading

joyeecheung commented Jun 22, 2023 • edited Loading

targos commented Jun 22, 2023

hildjj commented Jun 22, 2023 • edited Loading

joyeecheung commented Jun 22, 2023 • edited Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

nodejs-github-bot commented Jun 22, 2023

hildjj commented Jan 28, 2024

module: rework of memory management in `vm` APIs with the `importModuleDynamically` option #48510

module: rework of memory management in `vm` APIs with the `importModuleDynamically` option #48510

joyeecheung commented Jun 21, 2023 •

edited

Loading

joyeecheung commented Jun 21, 2023 •

edited

Loading

joyeecheung commented Jun 22, 2023 •

edited

Loading

hildjj commented Jun 22, 2023 •

edited

Loading

joyeecheung commented Jun 22, 2023 •

edited

Loading