-
Notifications
You must be signed in to change notification settings - Fork 66
Initial version of eps for ABI-Stable-Module-API #20
Conversation
FYI @nodejs/api, @nodejs/addon-api |
node::ni::SetPrototypeMethod(ni_env, recv, name, callback) | ||
node::ni::SetTemplate(ni_env,...) | ||
node::ni::SetPrototypeTemplate(ni_env,...) | ||
node::ni::SetInstanceTemplate(ni_env,...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Most of these are fluff. Two functions should suffice: a Set(primitive_or_template)
for templates and a Set(value)
for objects.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Set(primitive_or_template)
is probably still too V8-centric to work with e.g. the SM API for defining classes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not exposing templates would kill performance, would it not? I am not familiar with how SM does it, but from a pragmatic approach, should V8 not remain a first-class citizen?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be clear, I'm not saying the V8 implementation shouldn't use templates, but following the V8 API too closely might make it awkward to implement it efficiently for other engines.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In terms of the boundary between the methods exported by the node.js binary and how the module calls them, we need a strong separation that I believe will limit the use of templating at that level. We can't have engine code being incorporated into the module and still achieve ABI stability.
In terms of the api itself, however, the use of templates should still be able to be used to make the API easier to use but the code pulled in from the templating would be limited to code that is part of the API as opposed to the engine code.
There are performance implications so part of the work will be validating that for most modules the level of performance achieved is acceptable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now I am even more confused. We were discussing V8's templates which are used to build functions and other objects. They are not a matter of ease-of-use, but a necessity for building functions and other objects with V8. Somehow I get the impression that you are thinking of C++ templates, which are an entirely different matter, having no connection to V8's templates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right I was thinking of C++ templates, v8's templates need to be completely separate from the API as we cannot pull in any engine specific code into the modules. In the implementation for the API provided and exported by the node.js binary, the implementation of the API can use v8 specifics but none of that can leak out to the API itself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't everyone seem to hate C++ templates or something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Fishrock123 I think its a love it or hate it kind of thing. There are some people who argue strongly that using them and C++ will make the API easier to user. On the other side some people find the C style API easier to deal with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
imo, go with what is easiest to understand coming from JavaScript while remaining semantically good in most use-cases.
Regarding what addons to test on, don't just go for the most popular ones, pick the most complex addons, in regard to the binding part, not whatever libraries they provide. NAN's test suite, while far from comprehensive, is a good start. There is also need to decide on what to do with Isolates. The main reason that NAN does not currently do Isolates is because of 0.10-compatibility. Should Node ever be multi-isolate, from a practical point of view? I can only think of one or two addons I know of that actually use multiple Isolates. |
|
||
**WORK IN PROGRESS, NOT ANYWHERE NEAR COMPLETE.** | ||
|
||
This API will be built up from a minimal set as we explore the full set of Nan functions. To start it will only include the minimal set that we need to get started. What this means is we have not define the full API yet but instead want to give a feel of the general shape (which is not really quite there yet). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/define/defined
first reactions: doesn't seem quite as minimal as discussed? Also, |
* Native modules must be recompiled for each version of Node.js | ||
* The code within modules must often be modified for a new version of | ||
Node.js | ||
* It.s not clear which parts of the V8 API the Node.js community believes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's
How so? Seems to be exactly what was discussed. |
Perhaps it wasn't communicated fully? I was the under the impression we'd do only:
As such, things like function templates seem a little beyond that? Perhaps I am wrong. |
Additionally I don't think the EP process along will be enough to suffice for this. (i.e. we shouldn't just write up this conceptually and sign it off) We should aim to not expand the EP from here directly but rather get these working in a shim, make sure that suits modules, get the feedback from that, and iterate. We should make sure to make it without any guarantees until we pass a finalized version in core though since versions of it are unlikely to be maintainable for long and the API may change. I'm totally down to help where possible, I care about this an awful lot. |
I'll also note that the best way of actually getting this new API into use would be to enable NAN (or a replacement) to add another (transitional) shim on top of this, ensuring backwards compatibility at least down to 0.12 or 4.0, which presumably will not have access to it. No addon maintainer wants to maintain several versions of the same code base. |
+1. But we'd have to call it naan then ;)
|
One more thing, make it possible to seamlessly integrate this new (stable) API with the (unstable) VM-specific API. There is no realistic way of foreseeing every possible use case or developing a practical solution that will fit everything. What I consider one of the strong aspects of NAN is that the developer is free to pick and mix between raw V8 and NAN. Some things are not achievable in any other way, but requiring a couple of specific things here and there should not condemn the rest of the code to constant breaking. For example, promises are (currently) not handled by NAN, since they are not available in Node 0.10 and there is not realistic way of backporting all of that. However, a developer is still able to easily use promises where available, while the rest of the code can (safely) use NAN. C# has a similar concept in that mixing safe and unsafe code is quite doable. |
I think the goal will probably end up making sure at very least node core can run ontop of it. (Since that's part of VM neutrality anyways.) |
I'm a bit surprised by the direction this took. I was under the impression that we agreed on a different plan in the F2F meeting, namely to take the NAN module and replace the remaining v8 dependencies with new wrappers, i.e., v8::Localv8::Object should become something like nan::Localnan::Object. The direction this is now taking seems to be a newly designed C-style API. Deviating that much from NAN will make it unnecessarily difficult for native modules to transition over. A C-style API precludes any inlining based optimizations and puts additional cognitive burden on module authors that will have to explicitly manage lifetime of handles as opposed to relying on the Local dtor deregistering them etc.. Another thing we agreed upon was that this API should insulate native modules from the VM - using the API inside node as a means of becoming fully VM neutral was an explicit non-goal, as the latter will require fundamentally different APIs. |
+1 that the primary goal was to have an API that protects module developers As far as the relationship with nan is concerned, the idea was certainly to |
as these others like: Nan::New() were already there. The other areas would be object life cyle and exception handling, but would those be more than defining objects that the same methods could be called on ? For pushScope/popScope for lifecyle the answer is probably yes, the others I'm not so sure. In terms of using templates etc, I think that is still possible in either case, and the in-lining needs to stop before any engine code gets pulled into the module. Agreed, that this looks more C like than when we discussed at the meeting but I think that came out of the existing methods we pulled over from Nan. We can iterate to move in the right direction.
@kkoopa agreed that a Nan shim to allow this to be used transparently is one of the overall goals @Fishrock123 agreed we can't just write this up and sign off. We just want to give some visibility along this way. This eps should stay in draft while we iterate and be update frequently |
It might be helpful to take an existing module and convert it over to the proposed API (at least parts of it to demonstrate how the new API is supposed to be used, and to get a better feeling about the changes in ergonomics this would mean). |
@jeisinger that is what I think the next step is as well. Write the code for a minimal set of the APIs and then show it working with a simple module to make things more concrete. |
I'll let this settle for a day or more and then incorporate the "simple" updates (typos, clarifications, etc.) |
@kkoopa Which addons are those? Sounds like they are good to have on hand for investigation once we get the balling rolling on working prototype.
|
webworker-threads |
Yes, so that workers are a thing we can do since we've been wanting to for a while but it has been too complex to review effectively. See: nodejs/node#2133 |
The initial status shall be **"DRAFT"**. | ||
The initial status shall either be **"INCUBATING"** or **"DRAFT"**. | ||
|
||
If the status is **"INCUBATING"** that means that it is expected that the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this necessary? The incubation period is the time spent in the PR. If the EP is still so volatile that it expects to have changes or additions over time then it shouldn't land.
If this is the initial api and later there's an addition then the addition should go in a new EP. These aren't design documents that are meant to reflect a feature's current state.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@trevnorris this mistakenly got pushed as part of an update can you comment here instead: #22
Please replace |
@agnat the kinds of suggestions you are making is exactly why I'd like to get this landed as a DRAFT. You could then submit PR's to make the enhancements you are suggesting and they could be discussed in the PR. |
This LGTM, we can update with follow on PRs |
@agnat I'm going to go ahead and land the current version and then look to address your comments through PRs. Do you want to submit PRs for the suggested changes or would you like me to do that based on your comments ? |
@mhdawson that is not how it is supposed to work. People spent time and effort working through this asking very valid questions in the comments above. Not only me. Answering these questions in the comment section is not enough. Postponing them till later is not ok. These questions arise while reading the proposal. The proposal should answer them. Please respect the work reviewers put into this. Include their feedback and improve the document. I've been following this PR since pinged on day one. I postponed the review time and again because the document is no help. Diving in anyway 17some days ago it took me more than 16 hours of work to develop a mental picture of what really is about to happen. It is the documents job to describe the system. Instead I had to work it out myself. I'm ready to help with the document down the road, but as it stands it simply lacks the structure to do that.
Please provide a proper technical proposal we can work on. One we can actually discuss because it has some meat. |
@agnat, I am going to need more insight into what you are suggesting before jumping into the work you are suggesting. I think a higher bandwidth discussion between us would help me understand where/what you'd like changed. I'm thinking a 30 minute discussion were we go through and come up with the main headings in the doc together would go a long way in helping me understand the concrete changes that will address you concerns. Would that be possible ? We can do that either one on one or during the weekly get together held by the group working on the PoC (Fridays at 2EST). |
Why define the C++ interface in the same place? I would rather see the C++ wrapper defined and/or implemented somewhere else. Simpler parts working together (UNIX philosophy). As an idea: could we make a new NAN version that provides the C++ wrapper? |
I would also like to see a couple more API functions/enhancements:
1 is needed by C embedders while 2 could help enable true multithreading on Node.js. I experimented with I would be happy to submit a new issue and maybe a PR on https://github.com/nodejs/abi-stable-node to explain what I mean by 2. +1 for getting something landed, maybe something simple so people can start submitting PRs. What is the status? Any conclusions between @mhdawson / @agnat? P.S. I would be happy to raise both 1 and 2 as new issue(s). 👎 for not landing a DRAFT. |
Tagging with |
@nodejs/ctc : This EP was discussed extensively during the VM Summit today at the Node Interactive Collaborator Summit. While this is still very much a work in progress, there has been sufficient progress towards a working prototype and a well defined direction. We should make a decision about whether this EP should be accepted as a draft (and this PR landed). The recommendation of the VM Summit participants is that this should be accepted as a draft. |
@nodejs/ctc Note that the CTC is being asked to review this EP, so please review it carefully if you are able. @jasnell @Fishrock123 Any reason not to post an issue in the CTC repo and call for a vote right now? Or is there a reason the process needs to wait for the actual meeting? |
@Trott I'd like to make sure that there's at least an opportunity for the CTC members to ask any clarification questions but opening a vote is a good idea. |
|
||
struct napi_method_descriptor { | ||
napi_callback callback; | ||
char* utf8name; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const char*
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
opened nodejs/abi-stable-node#25 to update current implementation.
NODE_EXTERN napi_value napi_create_symbol(napi_env e, const char* s = NULL); | ||
NODE_EXTERN napi_value napi_create_function(napi_env e, napi_callback cbinfo); | ||
NODE_EXTERN napi_value napi_create_error(napi_env e, napi_value msg); | ||
NODE_EXTERN napi_value napi_create_type_error(napi_env e, napi_value msg); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How will these functions signal error? E.g. what if napi_create_object()
can't create an object because the VM is out of memory or in a state where it can't be entered?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding error handling is the next main milestone that the time is working on.
NODE_EXTERN int napi_get_string_length(napi_env e, napi_value v); | ||
NODE_EXTERN int napi_get_string_utf8_length(napi_env e, napi_value v); | ||
NODE_EXTERN int napi_get_string_utf8(napi_env e, napi_value v, | ||
char* buf, int bufsize); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why int instead of size_t?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
opened nodejs/abi-stable-node#25 to update current implementation.
|
||
// Methods to work with Functions | ||
NODE_EXTERN void napi_set_function_name(napi_env e, napi_value func, | ||
napi_propertyname napi_value); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How would this work with an engine like spidermonkey where you would normally declare the name upfront?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Created to nodejs/abi-stable-node#26 to investigate/figure out if we need a different API here.
napi_value* buffer, size_t bufferlength); | ||
NODE_EXTERN napi_value napi_get_cb_this(napi_env e, napi_func_cb_info cbinfo); | ||
// V8 concept; see note in .cc file | ||
NODE_EXTERN napi_value napi_get_cb_holder(napi_env e, napi_func_cb_info cbinfo); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do other engines distinguish between this and holder? (EDIT: Or is that what "V8 concept" refers to?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the note is saying its a v8 concept.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so still a work in progress to see if it makes sense across engines, so far ok for v8 and Chakra.
// Methods to support ObjectWrap | ||
NODE_EXTERN napi_value napi_create_constructor_for_wrap(napi_env e, | ||
napi_callback cb); | ||
NODE_EXTERN void napi_wrap(napi_env e, napi_value jsObject, void* nativeObj, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this work in all engines?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So far we've got it working for v8 and Chakra.
// Methods to support error handling | ||
NODE_EXTERN void napi_throw(napi_env e, napi_value error); | ||
NODE_EXTERN void napi_throw_error(napi_env e, char* msg); | ||
NODE_EXTERN void napi_throw_type_error(napi_env e, char* msg); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const char*
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
opened nodejs/abi-stable-node#25 to update current implementation.
typedef void (*napi_try_callback)(napi_env e, void* data); | ||
typedef void (*napi_catch_callback)(napi_env e, void* data); | ||
NODE_EXTERN bool napi_try_catch(napi_env e, napi_try_callback t, | ||
napi_catch_callback c, void* data); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think most JS engines have an 'is exception pending?' function; V8 is the outlier with its TryCatch construct. API strawman:
// Wrapper around a v8::TryCatch. Dummy with other engines.
typedef struct napi_trycatch { struct napi_trycatch *v; } napi_trycatch;
NODE_EXTERN void napi_trycatch_new(napi_env e, napi_trycatch *trycatch);
// Returns exception object or undefined.
NODE_EXTERN napi_value napi_trycatch_exception(napi_env e, napi_trycatch trycatch);
NODE_EXTERN void napi_trycatch_delete(napi_env e, napi_trycatch trycatch);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Create nodejs/abi-stable-node#27 to discuss as part of prototype
However, we also believe that a C++ wrapper is possible and | ||
would provide ease of use. The key element is that the C++ part must all | ||
be inlined and only use the external functions/types exported | ||
by the C API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem with inlining is that there is no way to release bug fixes without everyone needing to recompile.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is true, the sugar becomes part of the module.
@bnoordhuis has a number of good suggestions. I also know that the API will still change as the PoC continues so I opened issues in the abi-stable-node repo to address those. Earlier, before we had progressed the PoC to working code it was useful to have the functions in the eps to give people a "feel" for what it might look like. At this point I'm wondering if we should remove it and replace it with a pointer to the header in the abi-stable-node repo: https://github.com/nodejs/abi-stable-node/blob/api-prototype-6.2.0/src/node_jsvmapi.h as it will continue to evolve. |
+1 to landing this EPS as a draft and iterating on from there |
Discussed in CTC meeting last week. Agreed to land unless objects posted here. Going to land now. |
First cut of the ABI-Stable-Module-API eps. It is far from complete but is intended to allow insight into the current direction/status and will be updated regularly as we progress along the investigation and can make the API definition more complete.
Squashed commits. |
First cut of the ABI-Stable-Module-API eps. It is far from complete but is intended to allow insight into the current direction/status and will be updated regularly as we progress along the investigation and can make the API definition more complete. PR-URL: #20 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Saúl Ibarra Corretgé <[email protected]>
Landed as 735b776 |
Just noticed that it wasn't added to the index. Added it for you in 54b199c |
First cut of the ABI-Stable-Module-API eps. It is far
from complete but is intended to allow insight into the
current direction/status and will be updated regularly
as we progress along the investigation and can make
the API definition more complete.