Resolve node modules in parent directories

[geirsagberg]

Currently, node modules like aspnet-webpack must be installed in the root directory of the web app.

When node modules (and package.json) are placed in a parent directory (e.g. a solution folder), node will still resolve packages correctly using require, but packages called via node services will fail to resolve the same packages.

If NodeServices could search up the directory tree like node does, it would be easier to share node packages between multiple web projects in a single solution.

[SteveSandersonMS]

NodeServices never searches for node_modules at any specific location on disk - it just uses the require API in Node itself, so I'm surprised that there's any difference in how the module resolution works.

Can you specify an example project layout that triggers this issue so I can repro and investigate?

[geirsagberg]

https://github.com/geirsagberg/NodeServicesFun

I have basically just done the following:

1. Generated a project using generator-aspnetcore-spa
2. Delete node_modules, moved package.json to a parent directory and do npm install there

Running webpack from NodeServicesFun.Web works, because it finds the parent node_modules, but running the website fails with the error Exception: Call to Node module failed with error: To use prerendering, you must install the 'aspnet-prerendering' NPM package..

[SteveSandersonMS]

Thanks for reporting this. I was surprised to hear that module resolution behaves unexpectedly because internally it just calls Node's require function, but now I've looked into it in some detail, it makes sense.

This happens because the bootstrapping code for "webpack dev middleware" and "prerendering" is embedded into the NuGet packages as string resources, and to execute them in Node, it writes them to temporary files on disk and then gets Node to load them from there. Since the temporary directory is not within your project, when code inside those modules tries to load other modules, it has no way to find your node_modules dir. The fact that Node scans up the dir hierarchy doesn't help, because the temp dir is not inside your project.

The packages already deal with that to some extent by automatically putting a NODE_PATH environment variable on the launched Node process, and pointing it to <yourproject>/node_modules. This is how it works in the typical case where node_modules is at the root of your project. Unfortunately, Node does not scan up the hierarchy from the NODE_PATH environment variable - it requires that to match your node_modules dir exactly.

I've been trying to think of a good way to make this better, and the best option I came up with was to change how the bootstrapping modules are loaded, so that instead of just calling require(theTemporaryPath), we instead use a more complex (and unfortunately hacky) way of hooking into Node's Module API to construct a virtual module whose code is the contents of that file, but whose disk path is configured to match your project root. This would work, but it involves relying on undocumented Node APIs, and it risks a performance impact or surprising inconsistencies, because by default Node would be forced to re-load the bootstrapping module on every request (it wouldn't know it had already been loaded), which means we'd have to implement our own caching mechanism and it's hard to guarantee the behaviour would be identical to Node's native one.

There might be some other way of changing how the paths are resolved. For example, there could be an optional NPM module called aspnet-nodeservices that our Node entrypoint code looks for, and if it finds it, it could delegate to that for module resolution.

Altogether this is quite a tricky thing to resolve cleanly in total generality. But if you just want a way of doing it for your project, you can just set the environment variable NODE_PATH pointing to the actual location of your node_modules dir. Remember to specify an absolute path.

Since there's a relatively simple solution for individual projects, and making a general built-in solution is quite complex and possibly risky, I'm going to suggest we don't try to put in a general solution but rather just document that you can set NODE_PATH.

Does that seem like a reasonable solution?

[geirsagberg]

I had a hunch it might be related to the bundled resources :)

Your solution seems like a good workaround, I agree that this problem doesn't warrant risky hacks. Thanks for looking into it so quickly :)

[geirsagberg]

Just a thought; could it be possible to write the temporary files to somewhere in the project folder instead?

[SteveSandersonMS]

Just a thought; could it be possible to write the temporary files to somewhere in the project folder instead?

That would solve this immediate problem, but risks creating worse problems. We don't know that your webserver has write permission to that directory (whereas temp directories are writable), plus writing files there could trigger an app-restart loop (if you have a file watcher that restarts the app when one of its source files are written).

Your solution seems like a good workaround, I agree that this problem doesn't warrant risky hacks. Thanks for looking into it so quickly :)

Thanks - I'll close this.

[geirsagberg]

How about using NuGet contentFiles instead of embedding the files? That way the files will be generated on package restore, and you wouldn't need the NODE_PATH at all. I might fiddle around and see if I can get a PoC working...

[geirsagberg]

Looks like specifying NuGet contentFiles in project.json is blocked by NuGet/Home#2470. I guess one could manually edit the generated nuspec file, but that seems hacky as well. Guess I'll leave this alone for now.

[buvinghausen]

@SteveSandersonMS why was this issue closed? The issue still exists a full year later. I want my config level items in the parent folder not in same directory as my web application. This is a common paradigm in the JavaScript community to nest your actual source code in a src subfolder but keep your webpack.config.js and node_modules in the root. I'm already having a hard enough time convincing my front end developers that we need .NET to serve their applications rather than node I don't need to make them cd into src to run npm install too...

[MarkPieszak]

Thats common behavior, npm runs commands in whatever directory you're currently in.
Just make a blank package.json in your root if you want and make the install script do it for you in a post hook

"postinstall": "cd src && npm install"

[buvinghausen]

Mark,

I appreciate the info and that solves the problem of npm install for the developers but just like any workaround/hack it completely ignores any parameters you pass npm so then my build server has to run into the src folder to run the production version of npm install and if the time comes and this gets fixed and I want to go back and clean up my repositories then I now have to remember to fix the build server as well.

As an aside I scanned the open issues on GitHub didn't see anything. I can't find any documentation on this but I tried every possible permutation of ../ ./../ .. & ./.. and nothing works node keeps executing under the context of the src folder and thus can't load any modules without a symbolic link.

I spent too much time on this thinking it was user error when in reality it was just a closed issue that shouldn't have been closed. I get that it's not top priority and I appreciate all the work you guys have done to get us to this point where we can hot reload and pre-render from .NET but if issues are going to get dismissed carte blanche my confidence level drops.

[geirsagberg]

I use this workaround: https://gist.github.com/geirsagberg/0f5812d5451d2b0938b15b309c23bdd4

[geirsagberg]

For convenience:

public static class HostingEnvironmentExtensions
{
    public static void UseRootNodeModules(this IHostingEnvironment hostingEnvironment)
    {
        var nodeDir = Path.Combine(hostingEnvironment.ContentRootPath, "../node_modules");
        Environment.SetEnvironmentVariable("NODE_PATH", nodeDir);
    }
}

If your web project is deeper than 1 level (e.g. in a src/ folder), use "../../node_modules" instead.

Usage:

public void Configure (IApplicationBuilder app, IHostingEnvironment env) {
    if (env.IsDevelopment()) {
        env.UseRootNodeModules();
        // Extra options are just my preferences
        app.UseWebpackDevMiddleware(new WebpackDevMiddlewareOptions {
            HotModuleReplacement = true,
            ReactHotModuleReplacement = true,
            HotModuleReplacementClientOptions = new Dictionary<string, string> {
                {"reload", "true"}
            }
        });
    }
}

[healthycola]

If people are still having this issue, you can use something like this. You don't need to update the node_modules environment like above for this.

app.UseWebpackDevMiddleware(new WebpackDevMiddlewareOptions
                {
                    HotModuleReplacement = true,
                    ConfigFile = "config/webpack.config.js",
                    ProjectPath = System.IO.Path.Combine(env.ContentRootPath, "ClientApp/")
                });

Directory structure is something like below:

Startup.cs (contains above code)
ClientApp
|- package.json
|- node_modules
|- config
---|- webpack.config.js