[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

joyeecheung · 2024-10-01T16:42:48Z

This backports the following PRs that are related to the unflagging of --experimental-require-module:

Note that I don't think this backport PR is strictly required - the PRs seem to land cleanly on v22.x except the unflagging commit has a small conflict in the YAML changelog description. This PR is mostly opened to look for regressions in v22 with CITGM.

Refs: #52697

nodejs-github-bot · 2024-10-01T16:42:54Z

Review requested:

@nodejs/loaders

joyeecheung · 2024-10-01T16:46:41Z

CITGM for v22.x-staging: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3481/
CITGM for this PR: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3482/

joyeecheung · 2024-10-02T13:52:39Z

Some regressions found:

yarn is expecting ERR_REQUIRE_ESM in its tests, probably to check that they didn't do anything to alter Node.js module loading https://github.com/yarnpkg/berry/blob/33f249b6cc06db1803f03eb5f17306a23d11115c/packages/acceptance-tests/pkg-tests-specs/sources/pnp-esm.test.ts#L617
Gulp tests are breaking because they don't expect the experimental warning coming out of Node.js https://github.com/gulpjs/gulp/blob/2fa4981a910d7bdedb758bd09868620c9bb21d54/test/index.test.js#L90 (I am not sure how they load the .mjs configs, though it seems they are also doing a try-catch of require(config) somewhere)
2 also happened to Jest.
In local testing, I also noticed that the debug package (used by npm, etc.) has a try { require(esm) } catch { /* fallback */ } pattern here, which only gets improvements out of require(esm) support after the unflagging, but then the experimental warning would also break the output.

I think for backporting to v22.x we should probably do some changes:

a. Provides process.features.require_module for feature detection
b. Do not emit the experimental warning if it's coming out of node_modules

b would not fix the test regressions 2-4, the packages still need to update their tests which is to be expected. But at least it would not break the CLIs for end users.

joyeecheung · 2024-10-02T14:58:59Z

As for esm, from what I can tell in standard-things/esm#883 it seems to have been broken in different ways since many major releases of Node.js ago. The regression that predated unflagging of require(esm) I found in #55085 (comment) also seem to come from some TDZ issue in the way the package is written (it relies on access to Node.js internals and seem to expect certain internals to be invoked in certain order). It doesn't seem to be very cost-effective to do anything about it on v22.

richardlau · 2024-10-02T15:27:11Z

I think for backporting to v22.x we should probably do some changes:

a. Provides process.features.require_module for feature detection
b. Do not emit the experimental warning if it's coming out of node_modules

b would not fix the test regressions 2-4, the packages still need to update their tests which is to be expected. But at least it would not break the CLIs for end users.

These SGTM

joyeecheung · 2024-10-02T16:33:21Z

Opened #55241 for feature detection.

joyeecheung · 2024-10-04T10:39:20Z

Added a commit from #55243 and a commit for v22.x specifically (and maybe v20.x in the future) to disable the warning when require() comes from node_modules.

targos · 2024-10-04T11:58:50Z

I just updated v22.x-staging so this now has conflicts because of duplicated commits I guess.

joyeecheung · 2024-10-07T15:49:07Z

Rebased to drop the duplicate commits. Also cherry-picked #55243 and #55241 from the main branch. I think for backport to v22.x it is ready, note that the last commit (silence the experimental warning when it comes from node_modules) is only targeting v22.x, as proposed in #55217 (comment) and agreed in the last TSC meeting.

nodejs-github-bot · 2024-10-07T15:56:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62972/

aduh95 · 2024-10-08T10:47:39Z

Can you rebase please?

mcollina

lgtm

This is faster and more consistent with other places using the regular expression to detect node_modules. PR-URL: nodejs#55243 Backport-PR-URL: nodejs#55217 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Refs: nodejs#52697

Previously we assumed if `--experimental-detect-module` is true, then `--experimental-require-module` is true, which isn't the case, as the two can be enabled/disabled separately. This patch fixes the checks so `--no-experimental-require-module` is still effective when `--experimental-detect-module` is enabled. Drive-by: make the assertion messages more informative and remove obsolete TODO about allowing TLA in entrypoints handled by require(esm). PR-URL: nodejs#55250 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]>

This previously compiles a script and run it in a new context to avoid global pollution, which is more complex than necessary and can be too slow for it to be reused in other cases. The new implementation just checks the frames in C++ which is safe from global pollution, faster and simpler. The previous implementation also had a bug when the call site is in a ESM, because ESM have URLs as their script names, which don't start with '/' or '\' and will be skipped. The new implementation removes the skipping to fix it for ESM. PR-URL: nodejs#55286 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>

When emitting the experimental warning for `require(esm)`, include information about the parent module and the module being require()-d to help users locate and update them. PR-URL: nodejs#55397 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]>

When a ESM module cannot be loaded by require due to the presence of TLA, its module status would be stopped at kInstantiated. In this case, when it's imported again, we should allow it to be evaluated asynchronously, as it's also a common pattern for users to retry with dynamic import when require fails. PR-URL: nodejs#55502 Fixes: nodejs#55500 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Chemi Atlow <[email protected]>

Trim off irrelevant internal stack frames for require(esm) warnings so it's easier to locate where the call comes from when --trace-warnings is used. PR-URL: nodejs#55496 Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Paolo Insogna <[email protected]>

As part of the standard experimental feature graduation policy, when we unflagged require(esm) we moved the experimental warning to be emitted when require() is actually used to load ESM, which previously was an error. However, some packages in the ecosystem have already being using try-catch to load require(esm) to e.g. resolve optional dependency, and emitting warning from there instead of throwing directly could break the CLI output. To reduce the disruption for releases, as a compromise, this patch skips the warning if require(esm) comes from node_modules, where users typically don't have much control over the code. This warning will be eventually removed when require(esm) becomes stable. This patch was originally intended for the LTS releases, though it seems there's appetite for it on v23.x as well so it's re-targeted to the main branch. PR-URL: nodejs#55960 Refs: nodejs#55217 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]>

joyeecheung · 2024-11-23T22:42:01Z

Updated the branch and reordered the commits a bit to match the main branch.

nodejs-github-bot · 2024-11-23T22:46:43Z

CI: https://ci.nodejs.org/job/node-test-pull-request/63677/

nodejs-github-bot · 2024-11-24T06:45:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/63679/

ruyadorno · 2024-11-25T01:39:55Z

Thank you @joyeecheung!

As part of the standard experimental feature graduation policy, when we unflagged require(esm) we moved the experimental warning to be emitted when require() is actually used to load ESM, which previously was an error. However, some packages in the ecosystem have already being using try-catch to load require(esm) to e.g. resolve optional dependency, and emitting warning from there instead of throwing directly could break the CLI output. To reduce the disruption for releases, as a compromise, this patch skips the warning if require(esm) comes from node_modules, where users typically don't have much control over the code. This warning will be eventually removed when require(esm) becomes stable. This patch was originally intended for the LTS releases, though it seems there's appetite for it on v23.x as well so it's re-targeted to the main branch. PR-URL: nodejs#55960 Refs: nodejs#55217 Refs: nodejs#52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]>

As part of the standard experimental feature graduation policy, when we unflagged require(esm) we moved the experimental warning to be emitted when require() is actually used to load ESM, which previously was an error. However, some packages in the ecosystem have already being using try-catch to load require(esm) to e.g. resolve optional dependency, and emitting warning from there instead of throwing directly could break the CLI output. To reduce the disruption for releases, as a compromise, this patch skips the warning if require(esm) comes from node_modules, where users typically don't have much control over the code. This warning will be eventually removed when require(esm) becomes stable. This patch was originally intended for the LTS releases, though it seems there's appetite for it on v23.x as well so it's re-targeted to the main branch. PR-URL: #55960 Refs: #55217 Refs: #52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]>

This unflags --experimental-require-module so require(esm) can be used without the flag. For now, when require() actually encounters an ESM, it will still emit an experimental warning. To opt out of the feature, --no-experimental-require-module can be used. There are some tests specifically testing ERR_REQUIRE_ESM. Some of them are repurposed to test --no-experimental-require-module. Some of them are modified to just expect loading require(esm) to work, when it's appropriate. PR-URL: #55085 Backport-PR-URL: #55217 Refs: #52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Filip Skokan <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #54563 Backport-PR-URL: #55217 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Refs: #52697

PR-URL: #55199 Backport-PR-URL: #55217 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Refs: #52697

This is faster and more consistent with other places using the regular expression to detect node_modules. PR-URL: #55243 Backport-PR-URL: #55217 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Marco Ippolito <[email protected]> Refs: #52697

Previously we assumed if `--experimental-detect-module` is true, then `--experimental-require-module` is true, which isn't the case, as the two can be enabled/disabled separately. This patch fixes the checks so `--no-experimental-require-module` is still effective when `--experimental-detect-module` is enabled. Drive-by: make the assertion messages more informative and remove obsolete TODO about allowing TLA in entrypoints handled by require(esm). PR-URL: #55250 Backport-PR-URL: #55217 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]> Reviewed-By: Rafael Gonzaga <[email protected]> Refs: #52697

This previously compiles a script and run it in a new context to avoid global pollution, which is more complex than necessary and can be too slow for it to be reused in other cases. The new implementation just checks the frames in C++ which is safe from global pollution, faster and simpler. The previous implementation also had a bug when the call site is in a ESM, because ESM have URLs as their script names, which don't start with '/' or '\' and will be skipped. The new implementation removes the skipping to fix it for ESM. PR-URL: #55286 Backport-PR-URL: #55217 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Refs: #52697

When emitting the experimental warning for `require(esm)`, include information about the parent module and the module being require()-d to help users locate and update them. PR-URL: #55397 Backport-PR-URL: #55217 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Stephen Belanger <[email protected]> Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Chengzhong Wu <[email protected]> Refs: #52697

When a ESM module cannot be loaded by require due to the presence of TLA, its module status would be stopped at kInstantiated. In this case, when it's imported again, we should allow it to be evaluated asynchronously, as it's also a common pattern for users to retry with dynamic import when require fails. PR-URL: #55502 Backport-PR-URL: #55217 Fixes: #55500 Refs: #52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Chemi Atlow <[email protected]>

Trim off irrelevant internal stack frames for require(esm) warnings so it's easier to locate where the call comes from when --trace-warnings is used. PR-URL: #55496 Backport-PR-URL: #55217 Reviewed-By: Marco Ippolito <[email protected]> Reviewed-By: Paolo Insogna <[email protected]> Refs: #52697

As part of the standard experimental feature graduation policy, when we unflagged require(esm) we moved the experimental warning to be emitted when require() is actually used to load ESM, which previously was an error. However, some packages in the ecosystem have already being using try-catch to load require(esm) to e.g. resolve optional dependency, and emitting warning from there instead of throwing directly could break the CLI output. To reduce the disruption for releases, as a compromise, this patch skips the warning if require(esm) comes from node_modules, where users typically don't have much control over the code. This warning will be eventually removed when require(esm) becomes stable. This patch was originally intended for the LTS releases, though it seems there's appetite for it on v23.x as well so it's re-targeted to the main branch. PR-URL: #55960 Backport-PR-URL: #55217 Refs: #55217 Refs: #52697 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Jacob Smith <[email protected]>

ruyadorno · 2024-11-27T05:47:16Z

Landed in 9ffd2dd...cb527a9

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. v22.x v22.x Issues that can be reproduced on v22.x or PRs targeting the v22.x-staging branch. labels Oct 1, 2024

RedYetiDev added module Issues and PRs related to the module subsystem. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. labels Oct 1, 2024

RedYetiDev added the needs-citgm PRs that need a CITGM CI run. label Oct 1, 2024

tats-u mentioned this pull request Oct 4, 2024

Enable consuming of ES modules in extensions microsoft/vscode#130367

Open

targos mentioned this pull request Oct 5, 2024

doc: fix typos #55066

Merged

This was referenced Oct 5, 2024

src: implement IsInsideNodeModules() in C++ #55286

Merged

Tracking issue: require(esm) #52697

Open

joyeecheung force-pushed the backport-require-esm-2 branch from dda6fc3 to 9c282c3 Compare October 7, 2024 15:45

joyeecheung marked this pull request as ready for review October 7, 2024 15:47

joyeecheung force-pushed the backport-require-esm-2 branch from 9c282c3 to 3d0afe0 Compare October 7, 2024 15:52

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Oct 7, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Oct 7, 2024

joyeecheung force-pushed the backport-require-esm-2 branch 2 times, most recently from 5ed6462 to 898396a Compare October 8, 2024 19:00

mcollina approved these changes Oct 9, 2024

View reviewed changes

joyeecheung force-pushed the backport-require-esm-2 branch from 898396a to 5ea60dc Compare October 9, 2024 17:08

joyeecheung added 6 commits November 23, 2024 23:11

joyeecheung force-pushed the backport-require-esm-2 branch from f7bdcee to a09b58c Compare November 23, 2024 22:31

joyeecheung force-pushed the backport-require-esm-2 branch from a09b58c to 4af41f0 Compare November 23, 2024 22:41

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Nov 23, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 23, 2024

ruyadorno closed this Nov 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

joyeecheung commented Oct 1, 2024 •

edited

Loading

nodejs-github-bot commented Oct 1, 2024

joyeecheung commented Oct 1, 2024

joyeecheung commented Oct 2, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading

richardlau commented Oct 2, 2024

joyeecheung commented Oct 2, 2024

joyeecheung commented Oct 4, 2024

targos commented Oct 4, 2024

joyeecheung commented Oct 7, 2024

nodejs-github-bot commented Oct 7, 2024

aduh95 commented Oct 8, 2024

mcollina left a comment

joyeecheung commented Nov 23, 2024

nodejs-github-bot commented Nov 23, 2024

nodejs-github-bot commented Nov 24, 2024

ruyadorno commented Nov 25, 2024

ruyadorno commented Nov 27, 2024

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

Conversation

joyeecheung commented Oct 1, 2024 • edited Loading

nodejs-github-bot commented Oct 1, 2024

joyeecheung commented Oct 1, 2024

joyeecheung commented Oct 2, 2024 • edited Loading

joyeecheung commented Oct 2, 2024 • edited Loading

richardlau commented Oct 2, 2024

joyeecheung commented Oct 2, 2024

joyeecheung commented Oct 4, 2024

targos commented Oct 4, 2024

joyeecheung commented Oct 7, 2024

nodejs-github-bot commented Oct 7, 2024

aduh95 commented Oct 8, 2024

mcollina left a comment

Choose a reason for hiding this comment

joyeecheung commented Nov 23, 2024

nodejs-github-bot commented Nov 23, 2024

nodejs-github-bot commented Nov 24, 2024

ruyadorno commented Nov 25, 2024

ruyadorno commented Nov 27, 2024

joyeecheung commented Oct 1, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading