[v12.x backport] src: improve embedder API #35241

addaleax · 2020-09-17T15:05:02Z

It’s been a while since these have landed on v14.x, so we should have had enough baking time to backport these:

src: improve embedder API #30467 – src: improve embedder API
test: wait for message from parent in embedding cctest #32563 – test: wait for message from parent in embedding cctest
embedding: provide hook for custom process.exit() behaviour #32531 – embedding: provide hook for custom process.exit() behaviour
test: use InitializeNodeWithArgs in cctest #32406 – test: use InitializeNodeWithArgs in cctest
test: use common.buildType in embedding test #32422 – test: use common.buildType in embedding test
src: initialize inspector before RunBootstrapping() #32672 – src: initialize inspector before RunBootstrapping()
worker: unify custom error creation #33084 – worker: unify custom error creation
src: use env’s instead of isolate’s RequestInterrupt() + v8::Platform #32523 – src: use env’s instead of isolate’s RequestInterrupt() + v8::Platform
src: add callback scope for native immediates #34366 – src: add callback scope for native immediates
test,doc: add missing uv_setup_args() calls #34751 – test,doc: add missing uv_setup_args() calls

There were no major conflicts to be resolved, mostly neighbouring-line conflicts. (Non-conflict changes that were necessary for backporting are in the fixup! commits.)

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2020-09-17T15:21:18Z

CI: https://ci.nodejs.org/job/node-test-pull-request/33156/

addaleax · 2020-09-17T20:08:20Z

The CI failed on debian-8 because its GCC is too old, and that warning is printed on the command line. Is it intentional that we have GCC 4.9.2 in our CI here?

nodejs-github-bot · 2020-09-17T20:48:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/33158/ (:green_heart:)

richardlau · 2020-09-18T16:22:08Z

The CI failed on debian-8 because its GCC is too old, and that warning is printed on the command line. Is it intentional that we have GCC 4.9.2 in our CI here?

We dropped Debian 8 for Node.js 13 and above in nodejs/build#1974 based on the reasoning in nodejs/build#1970 (comment). Debian 8 went out of support in June 2020: https://www.debian.org/releases/jessie/ and we state in https://github.com/nodejs/node/blob/v12.x/BUILDING.md that we do not support running Node.js on End-of-Life platforms. I'd be open to dropping Debian 8 from testing on Node.js 12 -- maybe dropping it altogether?

cc @nodejs/build @nodejs/release

mhdawson · 2020-09-18T20:01:41Z

looks like the minimum for 12.x is GCC 6.x https://github.com/nodejs/node/blob/v12.x/BUILDING.md#official-binary-platforms-and-toolchains, even the kernel level seems to say Debian 9 is the minimum https://github.com/nodejs/node/blob/v12.x/BUILDING.md#platform-list

Debian 8 also went EOL on June 30 2020 , so I think dropping it from 12.x testing should be ok? @rvagg any concern?

rvagg · 2020-09-21T05:18:15Z

no, debian is always hard to maintain over time, no objections to removing it

Make the calls `stop_sub_worker_contexts()`, `RunCleanup()` part of the public API for easier embedding. (Note that calling `RunAtExit()` is idempotent because the at-exit callback queue is cleared after each call.) PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

Workers are fully in control of their Isolates, and this helps avoid a problem with later changes to `CreateEnvironment()` because now we can run the boostrapping code inside the latter. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

In our codebase, the assumption generally is that `!is_main_thread()` means that the current Environment belongs to a Node.js Worker thread. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This addresses some long-standing TODOs by Joyee and me about making the embedder API more powerful and us less reliant on internal APIs for creating the main thread and Workers. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This allows embedders to flexibly control how they start JS code rather than using `third_party_main`. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

Allow passing a string as the main module rather than using the callback variant. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

Refs: nodejs#31910 PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This makes this bit of the embedder situation a bit easier to use. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This is a decent replacement for the to-be-deprecated Init() API. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This should eventually remove any necessity to use the global-state `GetMainThreadMultiIsolatePlatform()`. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This addresses some long-standing TODOs by Joyee and me about making the embedder API more powerful and us less reliant on internal APIs for creating the main thread and Workers. Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This allows embedders to flexibly control how they start JS code rather than using `third_party_main`. Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

Allow passing a string as the main module rather than using the callback variant. Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

Refs: #31910 Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This makes this bit of the embedder situation a bit easier to use. Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This is a decent replacement for the to-be-deprecated Init() API. Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

This should eventually remove any necessity to use the global-state `GetMainThreadMultiIsolatePlatform()`. Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

We do not need a Node.js-provided `v8::TracingController`, generally. Loosen that restriction in order to make it easier for embedders to provide their own subclass of `v8::TracingController`, or none at all. Refs: electron/electron@9c36576 Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

If created platform with CreatePlatform, the crash occurs because it does not check if it was initialized to v8_platform when DisposePlatform was called. Backport-PR-URL: #35241 Refs: #31260 Co-authored-by: Anna Henningsen <[email protected]> PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

There is currently no way to properly do this. Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

Add an embedder cctest that also covers a multi-Environment situation, including worker_threads-style inspector support. Co-authored-by: Joyee Cheung <[email protected]> Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

I’ve seen this fail a few times in CI, presumably because the inspector commmand did not reach the child thread in time. Explicitly waiting seems to solve that. Refs: #30467 Backport-PR-URL: #35241 PR-URL: #32563 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]>

Embedders may not want to terminate the process when `process.exit()` is called. This provides a hook for more flexible handling of that situation. Refs: #30467 (comment) Backport-PR-URL: #35241 PR-URL: #32531 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: James M Snell <[email protected]>

This makes sense given that terminating execution of the parent thread this way likely also is supposed to stop all running Worker threads spawned by it. Backport-PR-URL: #35241 PR-URL: #32531 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: James M Snell <[email protected]>

Refs: d7f1107 Fixes: #30257 Backport-PR-URL: #35241 PR-URL: #32406 Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jiawen Geng <[email protected]> Reviewed-By: Matheus Marchini <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: James M Snell <[email protected]>

This un-breaks testing in the case of `./configure --debug-node`. Backport-PR-URL: #35241 PR-URL: #32422 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: James M Snell <[email protected]>

This is necessary for `--inspect-brk-node` to work, and for the inspector to be aware of scripts created before bootstrapping. Fixes: #32648 Refs: #30467 (comment) Backport-PR-URL: #35241 PR-URL: #32672 Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Eugene Ostroukhov <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: James M Snell <[email protected]>

Mostly, this introduces a pattern that makes sure that if a custom error is reported, `stopped_` will be set to `true` correctly in every cast, which was previously missing for the `NewContext().IsEmpty()` case (which led to a hard crash from the `Worker` destructor). This also leaves TODO comments for a few cases in which `ERR_WORKER_OUT_OF_MEMORY` was not used in accordance with the documentation for that error code (or according to its intention). Fixing that is semver-major. Backport-PR-URL: #35241 PR-URL: #33084 Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]>

Otherwise this potentially leads to race conditions when used in a multi-threaded environment (i.e. when used for its very purpose). Backport-PR-URL: #35241 PR-URL: #32523 Reviewed-By: Matheus Marchini <[email protected]> Reviewed-By: James M Snell <[email protected]>

Fix this to account for the fact that `Stop()` may already have been called from a cleanup hook when the `inspector::Agent` is deleted along with the `Environment`, at which point cleanup hooks are no longer available. Backport-PR-URL: #35241 PR-URL: #32523 Reviewed-By: Matheus Marchini <[email protected]> Reviewed-By: James M Snell <[email protected]>

This cleans up `Agent::RequestIoThreadStart()` significantly. Backport-PR-URL: #35241 PR-URL: #32523 Reviewed-By: Matheus Marchini <[email protected]> Reviewed-By: James M Snell <[email protected]>

This simplifies the code significantly, and removes the dependency of the inspector code on the availability of a `MultiIsolatePlatform` (by removing the dependency on a platform altogether). It also fixes a memory leak that occurs when `RequestInterrupt()` is used, but the interrupt handler is never called before the Isolate is destroyed. One downside is that this leads to a slight change in timing, because inspector messages are not dispatched in a re-entrant way. This means having to harden one test to account for that possibility by making sure that the stack is always clear through a `setImmediate()`. This does not affect the assertion made by the test, which is that messages will not be delivered synchronously while other code is executing. #32415 Backport-PR-URL: #35241 PR-URL: #32523 Reviewed-By: Matheus Marchini <[email protected]> Reviewed-By: James M Snell <[email protected]>

This avoids an edge-case memory leak. Refs: #32523 (comment) Backport-PR-URL: #35241 PR-URL: #32523 Reviewed-By: Matheus Marchini <[email protected]> Reviewed-By: James M Snell <[email protected]>

This ensures that microtasks scheduled by native immediates are run after the tasks are done. In particular, this affects the inspector integration since 6f9f546. Fixes: #33002 Refs: #32523 Backport-PR-URL: #35241 PR-URL: #34366 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: James M Snell <[email protected]>

libuv 1.39.0 will begin requiring uv_setup_args() to be called before attempting to access the process title. This commit adds uv_setup_args() calls that were missing in order for the test suite to pass (and updates the documentation). Backport-PR-URL: #35241 PR-URL: #34751 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: Rich Trott <[email protected]>

addaleax · 2020-09-23T18:29:38Z

Landed in 1d3638b...26ede7f

addaleax added semver-minor PRs that contain new features and should be released in the next minor version. embedding Issues and PRs related to embedding Node.js in another project. v12.x labels Sep 17, 2020

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. labels Sep 17, 2020

addaleax added the request-ci Add this label to start a Jenkins CI on a PR. label Sep 17, 2020

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 17, 2020

addaleax added the request-ci Add this label to start a Jenkins CI on a PR. label Sep 17, 2020

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 17, 2020

addaleax force-pushed the v12.x-staging branch 4 times, most recently from 55fe022 to 65b7bf4 Compare September 22, 2020 17:57

addaleax added 11 commits September 23, 2020 15:55

src: fix memory leak in CreateEnvironment when bootstrap fails

3f9fa28

PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

src: provide a variant of LoadEnvironment taking a callback

c694f08

This allows embedders to flexibly control how they start JS code rather than using `third_party_main`. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

src: add LoadEnvironment() variant taking a string

8a1255c

Allow passing a string as the main module rather than using the callback variant. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

test: re-enable cctest that was commented out

bb689cf

Refs: nodejs#31910 PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

src: add unique_ptr equivalent of CreatePlatform

ad95b85

This makes this bit of the embedder situation a bit easier to use. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

src: make InitializeNodeWithArgs() official public API

33ecd3b

This is a decent replacement for the to-be-deprecated Init() API. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

src: add ability to look up platform based on Environment*

31528f3

This should eventually remove any necessity to use the global-state `GetMainThreadMultiIsolatePlatform()`. PR-URL: nodejs#30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

addaleax added a commit that referenced this pull request Sep 23, 2020

test: re-enable cctest that was commented out

f325c95

Refs: #31910 Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

addaleax added a commit that referenced this pull request Sep 23, 2020

src,test: add full-featured embedder API test

4af336d

Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

addaleax added a commit that referenced this pull request Sep 23, 2020

doc: add basic embedding example documentation

1a70820

Backport-PR-URL: #35241 PR-URL: #30467 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

addaleax closed this Sep 23, 2020

addaleax deleted the backport-v12.x-30467 branch September 23, 2020 18:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v12.x backport] src: improve embedder API #35241

[v12.x backport] src: improve embedder API #35241

addaleax commented Sep 17, 2020 •

edited

Loading

nodejs-github-bot commented Sep 17, 2020

addaleax commented Sep 17, 2020

nodejs-github-bot commented Sep 17, 2020 •

edited by addaleax

Loading

richardlau commented Sep 18, 2020

mhdawson commented Sep 18, 2020

rvagg commented Sep 21, 2020

addaleax commented Sep 23, 2020

[v12.x backport] src: improve embedder API #35241

[v12.x backport] src: improve embedder API #35241

Conversation

addaleax commented Sep 17, 2020 • edited Loading

Checklist

nodejs-github-bot commented Sep 17, 2020

addaleax commented Sep 17, 2020

nodejs-github-bot commented Sep 17, 2020 • edited by addaleax Loading

richardlau commented Sep 18, 2020

mhdawson commented Sep 18, 2020

rvagg commented Sep 21, 2020

addaleax commented Sep 23, 2020

addaleax commented Sep 17, 2020 •

edited

Loading

nodejs-github-bot commented Sep 17, 2020 •

edited by addaleax

Loading