src: fix MakeCallback error handling #4507

trevnorris · 2016-01-01T20:21:49Z

The final goal of this PR is to allow MakeCallback (in both node:: and AsyncWrap::) to be reentrant. Reason is because the HTTPParser may enter the JS call stack initially, or be called while already in a stack (see @indutny's PR #4509). Also so native addon developers can always use node::MakeCallback without the worry of code being executed in the wrong order (specifically in relation to usage of nextTick or the MicrotaskQueue).

To do this I first made sure the two implementations of MakeCallback were the same. Previously they would return at different points in the case of caught exception.

Finally was the addition of AsyncCallbackScope (admittedly the concept was taken from Fedor's PR above) to allow MakeCallback to be called recursively without each subsequent call from processing the nextTickQueue or the MicrotaskQueue.

I am working with @misterdjules on the proper way of handling domains. So this PR is currently a WIP.

Ref: nodejs/node-v0.x-archive#9245

R=@bnoordhuis
R=@indutny

bnoordhuis · 2016-01-02T16:44:16Z

/cc @bajtos

indutny · 2016-01-02T19:10:55Z

Why SetVerbose doesn't allow V8 going to native again?

trevnorris · 2016-01-04T10:36:26Z

@indutny Haven't investigated as to why that's the case. Just noticed it while testing. Could be a bug in v8? Header documentation doesn't indicate that control will never return to native.

indutny · 2016-01-04T13:02:16Z

@trevnorris indeed, it should just capture the error. What exactly do you mean by "control will never return to native"?

trevnorris · 2016-01-04T16:44:42Z

@indutny What I mean is that code execution will never return to the native API. IIRC (will have to double check) but I believe this is even the case if uncaughtException is used.

trevnorris · 2016-01-04T22:04:03Z

This PR is incorrect. The try_catch will be overridden if domain.exit() or async post callbacks are called. Guess we store the error information and manually throw it later. Will give that a shot.

trevnorris · 2016-01-04T22:32:20Z

@misterdjules /cc you on this since domains are acting all strange. will post more info and would like whatever feedback you may have.

misterdjules · 2016-01-05T00:10:53Z

src/async-wrap.cc

@@ -201,42 +203,49 @@ Local<Value> AsyncWrap::MakeCallback(const Local<Function> cb,
    Local<Value> enter_v = domain->Get(env()->enter_string());
    if (enter_v->IsFunction()) {
      enter_v.As<Function>()->Call(domain, 0, nullptr);
-      if (try_catch.HasCaught())
+      if (try_catch.HasCaught()) {
+        FatalException(env()->isolate(), try_catch);


If try_catch.SetVerbose(true) was called, wouldn't FatalException be called anyway?

trevnorris · 2016-01-05T00:18:52Z

@misterdjules Here's the diff to demonstrate the issue:

diff --git a/src/async-wrap.cc b/src/async-wrap.cc
index c9f5caa..fbff146 100644
--- a/src/async-wrap.cc
+++ b/src/async-wrap.cc
@@ -216,10 +216,6 @@ Local<Value> AsyncWrap::MakeCallback(const Local<Function> cb,

   Local<Value> ret = cb->Call(context, argc, argv);

-  if (try_catch.HasCaught()) {
-    return Undefined(env()->isolate());
-  }
-
   if (ran_init_callback() && !post_fn.IsEmpty()) {
     try_catch.SetVerbose(false);
     post_fn->Call(context, 0, nullptr);
@@ -237,6 +233,10 @@ Local<Value> AsyncWrap::MakeCallback(const Local<Function> cb,
     }
   }

+  if (try_catch.HasCaught()) {
+    return Undefined(env()->isolate());
+  }
+
   Environment::TickInfo* tick_info = env()->tick_info();

   if (tick_info->in_tick()) {

And here's the script:

'use strict';
const async_wrap = process.binding('async_wrap');
const domain = require('domain');
const crypto = require('crypto');
const d = domain.create();

function noop() { }

async_wrap.setupHooks(noop, noop, noop);
async_wrap.enable();

d.on('error', function() { });

d.run(function() {
  crypto.randomBytes(4, function() { throw new Error('poo') });
});

This causes the application to abort. This is because the try_catch around cb->Call() throws, and since we never run try_catch.Reset() it causes the async wrapp post callback to be seen as having thrown. Thus aborting.

Demonstrating that the node::MakeCallback() impl current has an issue but isn't manifested in core b/c it isn't consumed by core in a way that would trigger it. Also that there's a difference in how the two impl of MakeCallback work. This intends to make AsyncWrap::MakeCallback return if an error is caught after the asyncwrap and domain exit callbacks have had a chance to run.

misterdjules · 2016-01-05T00:23:25Z

src/async-wrap.cc

  }

  if (has_domain) {
    Local<Value> exit_v = domain->Get(env()->exit_string());
    if (exit_v->IsFunction()) {
      exit_v.As<Function>()->Call(domain, 0, nullptr);
-      if (try_catch.HasCaught())
+      try_catch.Reset();
+      if (try_catch.HasCaught()) {


If I understand correctly, try_catch.HasCaught() will always return false after a call to TryCatch::Reset, so I'm not sure I understand what the use case is.

misterdjules · 2016-01-05T01:44:18Z

@trevnorris OK, I think I understand the intent of this PR. Can you please update the original description of this PR + the commit message according to what you described in your latest comment? Otherwise understanding the intent of this PR is very difficult.

Now I also understand better one of the questions you had in #node-dev:

so SetVerbose() prevents returning to C++, which means domain->exit() wasn't running if the callback threw. is this expected?

which I interpret now as:

When a callback running within a domain throws an error, is Domain.prototype.exit for the current active domain expected to run?

I would think that the best one can do in this case is to either:

Have a domain error handler that does some cleanup and exit as soon as possible.
Have a process 'uncaughtException' event handler that does some cleanup and exit as soon as possible.
Abort and generate a core dump.
Simply exit with an error message that describes the error that was thrown.

In other words, one should not expect the node runtime to be able to continue its execution normally.

Moreover, when a domain error handler is called, all domains on the stack are cleared. So I'm not sure it's useful or even if it makes sense to call Domain.prototype.exit when an error is thrown within a domain.

I'm not familiar with the AsyncWrap subsystem, but I'd also be interested to know what the semantics are regarding post hooks and errors thrown within an async callback. Do we consider the async operation to be done, or failed? Do we make a distinction between the two?

Basically, I'm wondering if the current AsyncWrap::MakeCallback implementation (i.e not calling domain.exit() and not calling the post asyncwrap hook) should not also be reflected in node::MakeCallback and anywhere else a callback is run within a domain.

bajtos · 2016-01-05T13:53:14Z

The main purpose of adding try_catch.SetVerbose was to enable Node Inspector to break on uncaught errors at the point where/when such error was thrown - see nodejs/node-v0.x-archive#5713 and nodejs/node-v0.x-archive#6460.

I vaguely remember that "break on uncaught error" may have stopped working in later v0.11 versions, i.e. it's possible it worked only in few v0.11 versions and never else.

Just my two cents to give a bit more context, I am afraid I don't have enough time to take a deeper look at this anytime soon.

trevnorris · 2016-01-05T17:47:51Z

@misterdjules My original comment about SetVerbose() preventing return of code execution to C++ was completely off. I didn't realize that in the case of an uncaught exception the application would exit() immediately.

Moreover, when a domain error handler is called, all domains on the stack are cleared. So I'm not sure it's useful or even if it makes sense to call Domain.prototype.exit when an error is thrown within a domain.

This is important. So are you saying it's not necessary to call the domain's exit() callback from MakeCallback when an error is thrown in a domain's scope?

I'm not familiar with the AsyncWrap subsystem, but I'd also be interested to know what the semantics are regarding post hooks and errors thrown within an async callback.

If an async wrap callback throws the application aborts. I have no idea what the application of the state could be in this case, and since it's currently a "private" API wanted to make sure this never happened.

Basically, I'm wondering if the current AsyncWrap::MakeCallback implementation (i.e not calling domain.exit() and not calling the post asyncwrap hook) should not also be reflected in node::MakeCallback and anywhere else a callback is run within a domain.

If it's the case that the domain's exit() callback doesn't need to be called then this sounds about right. Though I may allow the async wrap post callback to run regardless for error reporting, but that's not part of this conversation.

@bajtos Thanks for the info. With that context now I'll know what to look for. Nevermind my concerns about SetVerbose(). It's not hurting anything as it is so not worried about it.

trevnorris · 2016-01-05T18:08:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/1147/

joaocgreis · 2016-01-05T19:05:15Z

@trevnorris the windows failure was due to work in progress in CI. Relaunched here: https://ci.nodejs.org/job/node-test-commit-windows-fanned/815/

misterdjules · 2016-01-07T01:43:48Z

@trevnorris I still don't understand what this PR intends to fix in terms of error handling. Is the purpose of this PR to fix the issue described in https://gist.github.com/misterdjules/fac76caa775b3ad63ac8?

trevnorris · 2016-01-07T16:58:55Z

@misterdjules This PR has shifted a couple times now as I've been working on it. The wording of "fix error handling" is meant to be "fix timing of return in MakeCallback on caught exception". And this commit really was only meant as cleanup for the following commit, making sure MakeCallback is reentrant. I'll reword the OP to reflect this.

After attempting to use ReThrow() and Reset() there were cases where firing the domain's error handlers was not happening. Or in some cases reentering MakeCallback would still cause the domain enter callback to abort (because the error had not been Reset yet). In order for the script to properly stop execution when a subsequent call to MakeCallback throws it must not be located within a TryCatch. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Make sure that calling MakeCallback multiple times within the same stack does not allow the nextTickQueue or MicrotaskQueue to be processed in any more than the first MakeCallback call. Check that domains enter/exit poperly with multiple MakeCallback calls and that errors are handled as expected Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Implementations of error handling between node::MakeCallback() and AsyncWrap::MakeCallback() do not return at the same point. Make both executions work the same by moving the early return if there's a caught exception just after the AsyncWrap post callback. Since the domain's call stack is cleared on a caught exception there is no reason to call its exit() callback. Remove the SetVerbose() statement in the AsyncWrap pre/post callback calls since it does not affect the callback call. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Add a scope that will allow MakeCallback to know whether or not it's currently running. This will prevent nextTickQueue and the MicrotaskQueue from being processed recursively. It is also required to wrap the bootloading stage since it doesn't run within a MakeCallback. Ref: nodejs/node-v0.x-archive#9245 Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Environment::TickInfo::last_threw() is no longer in use. Also pass Isolate to few methods and fix whitespace alignment. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

After attempting to use ReThrow() and Reset() there were cases where firing the domain's error handlers was not happening. Or in some cases reentering MakeCallback would still cause the domain enter callback to abort (because the error had not been Reset yet). In order for the script to properly stop execution when a subsequent call to MakeCallback throws it must not be located within a TryCatch. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Make sure that calling MakeCallback multiple times within the same stack does not allow the nextTickQueue or MicrotaskQueue to be processed in any more than the first MakeCallback call. Check that domains enter/exit poperly with multiple MakeCallback calls and that errors are handled as expected Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Implementations of error handling between node::MakeCallback() and AsyncWrap::MakeCallback() do not return at the same point. Make both executions work the same by moving the early return if there's a caught exception just after the AsyncWrap post callback. Since the domain's call stack is cleared on a caught exception there is no reason to call its exit() callback. Remove the SetVerbose() statement in the AsyncWrap pre/post callback calls since it does not affect the callback call. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Add a scope that will allow MakeCallback to know whether or not it's currently running. This will prevent nextTickQueue and the MicrotaskQueue from being processed recursively. It is also required to wrap the bootloading stage since it doesn't run within a MakeCallback. Ref: nodejs/node-v0.x-archive#9245 Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Environment::TickInfo::last_threw() is no longer in use. Also pass Isolate to few methods and fix whitespace alignment. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

After attempting to use ReThrow() and Reset() there were cases where firing the domain's error handlers was not happening. Or in some cases reentering MakeCallback would still cause the domain enter callback to abort (because the error had not been Reset yet). In order for the script to properly stop execution when a subsequent call to MakeCallback throws it must not be located within a TryCatch. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Make sure that calling MakeCallback multiple times within the same stack does not allow the nextTickQueue or MicrotaskQueue to be processed in any more than the first MakeCallback call. Check that domains enter/exit poperly with multiple MakeCallback calls and that errors are handled as expected Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Implementations of error handling between node::MakeCallback() and AsyncWrap::MakeCallback() do not return at the same point. Make both executions work the same by moving the early return if there's a caught exception just after the AsyncWrap post callback. Since the domain's call stack is cleared on a caught exception there is no reason to call its exit() callback. Remove the SetVerbose() statement in the AsyncWrap pre/post callback calls since it does not affect the callback call. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Add a scope that will allow MakeCallback to know whether or not it's currently running. This will prevent nextTickQueue and the MicrotaskQueue from being processed recursively. It is also required to wrap the bootloading stage since it doesn't run within a MakeCallback. Ref: nodejs/node-v0.x-archive#9245 Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Environment::TickInfo::last_threw() is no longer in use. Also pass Isolate to few methods and fix whitespace alignment. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

After attempting to use ReThrow() and Reset() there were cases where firing the domain's error handlers was not happening. Or in some cases reentering MakeCallback would still cause the domain enter callback to abort (because the error had not been Reset yet). In order for the script to properly stop execution when a subsequent call to MakeCallback throws it must not be located within a TryCatch. Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

Make sure that calling MakeCallback multiple times within the same stack does not allow the nextTickQueue or MicrotaskQueue to be processed in any more than the first MakeCallback call. Check that domains enter/exit poperly with multiple MakeCallback calls and that errors are handled as expected Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

trevnorris added the c++ Issues and PRs that require attention from people who are familiar with C++. label Jan 1, 2016

trevnorris mentioned this pull request Jan 1, 2016

Fix async wrap http #4509

Closed

trevnorris force-pushed the throw-mc-fix branch from 62f58b8 to be27b71 Compare January 4, 2016 22:22

trevnorris force-pushed the throw-mc-fix branch from be27b71 to eb72544 Compare January 4, 2016 23:46

trevnorris changed the title ~~WIP src: remove SetVerbose from MakeCallback~~ WIP src: fix MakeCallback error handling logic. Jan 4, 2016

misterdjules reviewed Jan 5, 2016
View reviewed changes

trevnorris force-pushed the throw-mc-fix branch from eb72544 to edade11 Compare January 5, 2016 00:17

misterdjules reviewed Jan 5, 2016
View reviewed changes

trevnorris changed the title ~~WIP src: fix MakeCallback error handling logic.~~ src: fix MakeCallback error handling Jan 5, 2016

trevnorris force-pushed the throw-mc-fix branch 2 times, most recently from 1a68317 to 160b3d1 Compare January 5, 2016 18:02

trevnorris force-pushed the throw-mc-fix branch 4 times, most recently from 57702ae to f9af215 Compare January 7, 2016 00:09

MylesBorins added land-on-v4.x and removed lts-watch-v4.x labels Jun 14, 2016

MylesBorins pushed a commit that referenced this pull request Jun 24, 2016

src: remove unused TickInfo::in_tick()

359f304

Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

MylesBorins pushed a commit that referenced this pull request Jun 28, 2016

src: remove unused TickInfo::in_tick()

0860758

Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

MylesBorins pushed a commit that referenced this pull request Jul 12, 2016

src: remove unused TickInfo::in_tick()

48b7b71

Ref: #7048 PR-URL: #4507 Reviewed-By: Fedor Indutny <[email protected]>

MylesBorins mentioned this pull request Jul 12, 2016

v4.5.0 proposal #7688

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

src: fix MakeCallback error handling #4507

src: fix MakeCallback error handling #4507

trevnorris commented Jan 1, 2016

bnoordhuis commented Jan 2, 2016

indutny commented Jan 2, 2016

trevnorris commented Jan 4, 2016

indutny commented Jan 4, 2016

trevnorris commented Jan 4, 2016

trevnorris commented Jan 4, 2016

trevnorris commented Jan 4, 2016

misterdjules Jan 5, 2016

trevnorris commented Jan 5, 2016

misterdjules Jan 5, 2016

misterdjules commented Jan 5, 2016

bajtos commented Jan 5, 2016

trevnorris commented Jan 5, 2016

trevnorris commented Jan 5, 2016

joaocgreis commented Jan 5, 2016

misterdjules commented Jan 7, 2016

trevnorris commented Jan 7, 2016

src: fix MakeCallback error handling #4507

src: fix MakeCallback error handling #4507

Conversation

trevnorris commented Jan 1, 2016

bnoordhuis commented Jan 2, 2016

indutny commented Jan 2, 2016

trevnorris commented Jan 4, 2016

indutny commented Jan 4, 2016

trevnorris commented Jan 4, 2016

trevnorris commented Jan 4, 2016

trevnorris commented Jan 4, 2016

misterdjules Jan 5, 2016

Choose a reason for hiding this comment

trevnorris commented Jan 5, 2016

misterdjules Jan 5, 2016

Choose a reason for hiding this comment

misterdjules commented Jan 5, 2016

bajtos commented Jan 5, 2016

trevnorris commented Jan 5, 2016

trevnorris commented Jan 5, 2016

joaocgreis commented Jan 5, 2016

misterdjules commented Jan 7, 2016

trevnorris commented Jan 7, 2016