Missing module in Init

[raffaeler]

With the legacy api, I could obtain the addon module using this code:

void Init(Local<Object> exports, Local<Object> module) {
  NODE_SET_METHOD(module, "exports", CreateObject);
}
NODE_MODULE(NODE_GYP_MODULE_NAME, Init)

I need the module to obtain the addon filename (and folder) using this code:

std::string static GetModuleFilename(v8::Handle<v8::Object> module)
{
    Isolate* isolate = Isolate::GetCurrent();
    auto v8filename = module->Get(v8::String::NewFromUtf8(isolate, "filename"));
    auto filename = NodeTypes::FromV8String(v8filename);
    return filename;
}

How can I get the module and the filename at initialization time now?

[mhdawson]

If I remember correctly there was a discussion around including the module or not and the the decision was to drop the module from the parameters in order to avoid tying N-API to the Node.js module implementation.

It sounds like the main thing you want to figure out is the path as the name itself you probably can already figure out. Right? What are you using the path for?

[raffaeler]

@mhdawson Thank you Michael.
You are right, the main reason is obtaining the path of the addon itself. How can I get that?
I have a configuration file in that directory that I have to parse to retrieve a number of parameters.

But module can give me access to the require function as well as other symbols. The problem in dropping support for Module is the inability to easily migrate the addon. I substantially stopped at the very beginning, when writing the Initialization function.... it is so frustrating...

[mhdawson]

@raffaeler sorry to hear its been a bad start. I wish I better remembered the discussion as I know the concern was raised about not having access to the module and in the end there was an answer as to why it was thought it would not be an issue. @digitalinfinity any chance you remember?

What other info do you need from the module for the addon?

[raffaeler]

My codebase currently retrieves dirname, filename and the require function.
Filename is easy to replace with a preprocessor definition __FILE__.
By recursivly inspecting the global object I am seeing that require is also available via process.mainModule.require but I didn't test it. Can I replace it with this one?
I know there are also other symbols registered there but I currently can't know if they will be useful for the improvements I planned.

FYI I need a solution that works both in a plain nodejs environment but also in Electron. I had problems in the past with Electron because it sets differently the "current directory".

[gabrielschulhof]

@raffaeler another reason we dropped support for v8::Local<v8::Object> module is that most native addons are fronted by a JavaScript module. So, you don't normally have index.node in your npm package, but you have build/Release/addon.node and index.js which basically does

const addon = require('./build/Release/addon');
...
module.exports = addon;

So, if your addon also uses the index.js/./build/Release/addon.node setup, you could do a bit of ping-pong: Your addon provides a single binding which accepts the exports and the path, and initializes the exports in that binding as you normally would. From the JS side this would look like this:

const path = require('path');
const addonPath = path.join('.', 'build', 'Release', 'addon.node');
const addon = require(addonPath);
addon.initialize(addon, path.resolve(path.join(__dirname, addonPath)));
// Now `addon` is ready for use.

On the C++ side, you would make the following change:

From:

namespace {
Napi::Value Init(Napi::Env env, Napi::Object exports) {
  exports["doSomething"] = Napi::Function::New(env, DoSomething);
  exports["doSomethingElse"] = Napi::Function::New(env, DoSomethingElse);
  return exports;
}
}  // end of anonymous namespace

NODE_API_MODULE(NODE_GYP_MODULE_NAME, Init)

To:

namespace {
Napi::Value Initialize(const Napi::CallbackInfo& info) {
  Napi::Object exports = info[0].As<Napi::Object>();
  Napi::String configPath = info[1].As<Napi::String>();
  // Do something with `configPath`.
  exports["doSomething"] = Napi::Function::New(env, DoSomething);
  exports["doSomethingElse"] = Napi::Function::New(env, DoSomethingElse);
}
Napi::Value Init(Napi::Env env, Napi::Object exports) {
  exports["initialize"] = Napi::Function::New(env, Initialize);
  return exports;
}
}  // end of anonymous namespace

NODE_API_MODULE(NODE_GYP_MODULE_NAME, Init)

[raffaeler]

@gabrielschulhof I don't get your solution.

1. I have no control over the js file that will use the addon, so I must minimize all the required javascript code to initialize the addon
2. I am not looking the module for the main module (I already can get it from process.mainModule.filename), but I need the exact path of the native addon at runtime which presumably will be something like someproject/node_modules/myaddon. I have a configuration file to read from there and several dlls to load out of the nodejs mechanism (I host the whole coreclr).

Thanks to the legacy API (and using Module), I currently have access to the runtime addon binary path. I am just trying to find a migration path for N-API.

I don't want to give for granted the positions on the hard disks as I found that they directories are treated differently from Electron and this can be true also for other apps/frameworks.

FYI Once I will get over this problem, I am still not sure to be able to migrate to the new API as I need to enqueue calls to the main node thread with libuv. I am not sure I will find all the API I need.

[legendecas]

FYI Once I will get over this problem, I am still not sure to be able to migrate to the new API as I need to enqueue calls to the main node thread with libuv. I am not sure I will find all the API I need.

If I understand your question correctly, I'd believe Thread Safe Function is what you expected.

[gabrielschulhof]

@raffaeler can you embed the configuration file into the native addon as part of the build process?

You can use the N-API thread-safe function to call a JavaScript function from any thread. Although the documentation marks it as experimental, we have PRs on the way to mark it as stable.

There's an example that illustrates its use.

[raffaeler]

@legendecas @gabrielschulhof Thank you. I currently manage my own functions queue. When an event handler of any asynchronous function on the CLR side completes, I marshal the results on the Node/C++ side and enqueue the results in my queue. Then, on the main libuv thread, I get the bucket, map it to a function and invoke it.
I have to investigate if I can use those functions to reproduce the same behavior.

@gabrielschulhof With regards to the configuration, I cannot embed it for two reasons:

• I am willing to provide a single set of binaries for all the configurations (this is the reason why I am willing to upgrade to N-API)
• For debugging purposes, the developer may need to tweak the configuration file

[gabrielschulhof]

@raffaeler from your description of the "functions queue" it sounds like exactly what the thread-safe function does.

May I ask how you provide the native addon? Do you provide it as an npm package? Given that you said you did not control the JavaScript that loads the addon it sounds like you're providing the binary only, and instructing your users to place the configuration file into the same directory as the binary.

If this is the case, would it be possible to instruct your users to henceforth place the configuration file in a different location? The reason I ask is that with libuv you are able to retrieve the user's home directory using uv_os_homedir() and so you could assemble a file name like ${HOME}/.my-module.conf by also examining process.platform to establish the path separator:

  size_t os_homedir_length = 1;
  char one_byte;
  char* os_homedir = nullptr;
  int result = uv_os_homedir(&one_byte, &os_homedir_length);
  os_homedir = new char[os_homedir_length];
  result = uv_os_homedir(os_homedir, &os_homedir_length);
  assert(result == 0);

  std::string platform = env.Global()
    .Get("process").As<Napi::Object>()
    .Get("platform").As<Napi::String>();

  char path_sep = platform.compare("win32") == 0 ? '\\' : '/';

  std::string config_file_path =
      std::string(os_homedir) + path_sep + ".my-module.conf";

  delete[] os_homedir;

  // Now you can use config_file_path

[raffaeler]

@gabrielschulhof thanks for the answer. I will investigate the threading function as soon as I solve the more basic problems.

Yes, the addon is provided via npm but since it can be used in multiple apps, it would not be wise to use the user's home directory. The configuration file must be kept app-specific.
The user may change the config file to add more "probing paths" that the CLR can load when dynamically loading the user's dlls. There are also logging settings and mappings used for the marshalling strategies. All these settings (and others) are definitely different from app to app.

It looks very strange not to have the ability to know where the addon binary is currently stored. I really don't get the reason for this decision.

[gabrielschulhof]

@raffaeler if the addon is provided via npm and consumed by the various applications as require('package-name') then it should always be possible to interject an index.js which then loads the addon using the ping-pong method I described earlier.

Each app that uses the addon would then install its own copy with its own config file. If the package is installed globally using npm install -g then it's not possible to have one config file for each app, even if module were available.

[raffaeler]

@gabrielschulhof If the user decides to install it globally, I (and the user) expect the settings are shared too (which is fine if this is what the user wants).
There are many packages that are suggested not to be installed globally, and this is one of them.

With regards to the index.js, it is an additional step just to receive the module information, which is something that existed before N-API ... isn't this a clear sign of a missing/forgot feature?
The native addons should have parity of features, or at least this is what the docs say.
I want to avoid js files which requires more explanations to the user as well as the .ts definitions.

Instead of adding a js file, I will probably end up in taking the project root folder (process.mainModule) and scan all the directories but this is a terrible and slow solution.

[mhdawson]

I wanted to understand what was needed from the module directory and why they are needed. From what I parse from the discussion the main gap is the ability to get the path for the module.

@gabrielschulhof we should be able to grab the path as part of the module creation and potentially stash in the env, then expose through something like napi_get_addon_path. @gabrielschulhof does that sound reasonable to you or are there problems/concerns that you can think of? @raffaeler even if we do that it may only be for the latest node versions so might only be a longer term solution.

[raffaeler]

@mhdawson thanks for the answer Michael, I was afraid to receive such an answer.
In addition to the addon folder module, I would suggest to add the other 'important' folders:

• the project root where the package.json of the app is found
• the node_modules folder
• of course the addon folder, which is important because, during development time, native folders are found in the build/debug or build/release folders.

With regards to my project: what are the paths in Env I can assume are absolutely and always available? (even when Electron is playing)

• process.mainModule.filename is the filename of the index.js (or index.html for Elecron). Its directory can be different than the project root.
• process.mainModule.paths. They appears to be 'guesses' of possible node_modules as most of them do not even exists in my tests. At least one matches the real node_modules
• process.cwd() result, the current working directory can be different than the project home. Not sure if this can be used as well.

Do you have any other suggestions?

P.S. I currently wrote a ton of C++ code probing all the folders and it works, but I am not sure it can work in any environment.

[gabrielschulhof]

@mhdawson adding stuff to env wouldn't work because env is shared among addons, unless we update the fields of env at all points before ceding control to the addon. That, in turn would add to the cost of each callback.

With regards to the index.js, it is an additional step just to receive the module information, which is something that existed before N-API ... isn't this a clear sign of a missing/forgot feature?

@raffaeler we intentionally didn't create a one-to-one mapping of pre-existing API to N-API. Another consequence, had we sent the Napi::Object module into the initializer, would have been that we would have expressed an opinion in favour of CommonJS modules and against ESM modules, which would have driven us to design explicit support for ESM modules later on. We did assume that most npm packages interject an index.js which processes the native bindings and provides a more JavaScript-friendly interface to its dependents. Passing through a JavaScript object and having the native addon decorate it is the lowest common denominator.

I acknowledge that this means that, in the case of your package, you would have no choice but to change the way it initializes itself. To that end I created https://gist.github.com/gabrielschulhof/df079c9a89fe05dfa740b7171e0aa760 to illustrate the way in which I would convey the addon's absolute path into the addon, formulating the whole thing as an npm package that provides the addon to its dependents.

Can you please go over the code and point out the ways in which it would fail under the circumstances in which your addon is being used?

[raffaeler]

@gabrielschulhof Thanks for the example, but I want to exclude any requirement in terms of a javascript initializer as the initial require can also be made from another native addon. I really don't understand why I can 'expand' __dirname for my module inside C++. It looke like a discordant note.

I am not asking to restore the Module object but just an addon specific object containing the information about the module itself.
A different approach would be a dictionary with all the modules. Not sure about the security implications of this choice.

Going back to my previous comment, the documentation doesn't help in telling me what are the paths that:

• are always available in any environment
• are giving me some starting point for probing the subfolders.

For example:

• what's the best way to find the folder fo the apps' package.json?
• is it true that (beyond development time) addons are always inside the node_modules folder?

[raffaeler]

An additional consideration. I remember to have seen different results (even in javascript) when the addon is packed from webpack.
I strongly believe that:

• the documentation should give more info about the possible change of the variable/objects values depending on all these situations (pure node, electron, webpack, etc.)
• add to the runtime a number of values that are always available and trustable

my 2 cents

[gabrielschulhof]

I really don't understand why I can 'expand' __dirname for my module inside C++.

@raffaeler Node.js wraps CommonJS modules in the following:

(function (exports, require, module, __filename, __dirname) {
// original contents of JS file
});

and then passes exports, require, module, __filename, and __dirname to the resulting function when executing the module.

AFAIK it does no such thing for native modules, and also AFAIK there is no way to retrieve values of variables that are in scope. I guess it's setting __filename on module and passing that into the native module loader, and that was useful.

I have to say that I'm amazed that there is code out there that loads one native addon from another native addon! Thanks for bringing up this use case! How exactly is that accomplished? Is it done by acquiring and then calling an instance of the require() function, or perhaps by using uv_dlopen() directly?

We should definitely consider formalizing some values we make available to native modules, but we should do so while keeping in mind the diverse nature of the environments in which these modules might be loaded.

[gabrielschulhof]

We also need to keep in mind that dlopen() may follow symlinks when loading a module, so the path given to dlopen() may not be the path where the module actually resides, and dlopen() doesn't report back the results of its traversal – at least not in a portable way. In this case, if we were to report that path to the module, it would be incorrect.

[raffaeler]

@gabrielschulhof got it, now I get why __dirname is not available, thanks for the explanation.

How exactly is that accomplished? Is it done by acquiring and then calling an instance of the require() function, or perhaps by using uv_dlopen() directly

I expect my addon can be loaded from other native addons. On my side I did it in other addons by getting process.mainModule.require from C++, but my addon is not currently doing it.

Said that, my addon do load other dlls shipped as part of the addon package (no need for require in this case).
The reason for this is that my addon is a generic host for .NET code (which dynamically generates the code needed to call and marshal from/to javascript and typescript). I dynamically load using dlopen() and once the .NET CLR is loaded it will load other assemblies that will be found in the folders I passed to it.

We should definitely consider formalizing some values we make available to native modules, but we should do so while keeping in mind the diverse nature of the environments in which these modules might be loaded.

That would be great. Glad to have contributed to that.

[gabrielschulhof]

Hey! Can you unmute me? I joined by phone.

[raffaeler]

@gabrielschulhof I can't ... you only can do it via web interface. Sorry

[mhdawson]

@raffaeler @gabrielschulhof we've discussed in the recent N-API team meetings and @gabrielschulhof is trying out some different options and will post an update once he's done that.

[argv-minus-one]

@gabrielschulhof The fact that the addon initializer returns a single napi_value and takes a parameter napi_value exports is already an expression of favor of CommonJS over ES6 modules. ES6 modules export zero or more individual named values, not exactly one unnamed value like CommonJS and current N-API.

[gabrielschulhof]

@argv-minus-one the intention with N-API was to get bindings into JS somehow and let JS impose its abstractions, like CommonJS or ESM, on top. The simplest way to achieve that was to transmit one napi_value into JS.

The problem in this case arises from the fact that the native addon is the package entry point, rather than a script.

[mhdawson]

We've discussed this and don't see any good options that fit with the abstractions we want to preserve in the API. Since there are alternatives which covers most of the use case, at this point unless there are more requests/cases where it blocks adoption its not in the work we are planning. Alternatives include:

• Suggestion outlined in - Missing module in Init #449 (comment)
• making the exports a function that can receive the module and return the bindings as a function of that -> const addon = require('./build/Release/addon.node')(module); there are then several options of how to create the bindings with knowledge of the module object.

[raffaeler]

Hi @mhdawson
I can think to other use-cases that will benefit from my request.
For example if you want to host javascript code from native code (or any other language), the starting point cannot be javascript.
(I want to remember that in my case I am not in control of javascript code at all)

Let's see the issue from another perspective.
Instead of the module, could you instead expose the information that the runtime environment has? (I need the addon path but others may be useful as well)
I could see that even many js libraries are struggling in bulding paths without having precise/reliable information.

Thanks

[gabrielschulhof]

@raffaeler I have an idea how you can get your add-on's path from the address of a symbol inside your add-on, however, it's platform-dependent. What kinds of platforms do you need to support?

[raffaeler]

@gabrielschulhof frankly at this stage I can't predict them. For sure Windows and Linux, but since my addon is a kind of "glue" between nodejs and .NET, it can potentially be used anywhere.

BTW by spying file activities I see a lot of addons making deep search through various folder to find themselves. This means that many others have the same need but just didn't ask for it. Making this search leads to poor performance and can potentially break very easily.
It is my belief that an official way to take the path would have beneficial effects on many addons.

[mhdawson]

@raffaeler do you have that list of other modules that you can share?

[raffaeler]

I didn't save a list @mhdawson. I installed (and later removed) a bunch of addons to try understanding how could they get the correct folders. Using Windows utility procmon I discovered they were just trying probing entire trees.

BTW if I remember well, not directly related, but node-gyp itself probes debug, release and other folders to search for the native addon just because there is no info about the directory structure

[gabrielschulhof]

@raffaeler it's not node-gyp that does that but bindings. However, since bindings itself is loaded from another JS module, it is able to compute the package root directory and it searches a hard-coded list of (historical) paths for modules descending from that package root directory.

But check out nodejs/node#37195 – let's see how it goes 🤷

[raffaeler]

Great work @gabrielschulhof
Starting from what version this change will be available?
Thank you