Support for node_modules

[mikegwhit]

Hi Ryan et al.,

Not trying to troll, I just spent the last 6-9 months building software to readily allow multi-package Node.js development. I think I'm going to release it as a CLI tool. It basically reads folders from your node_modules filesystem (with caching) like "api"/"www"/"configs"/"scripts". For instance, if you wanted to execute a script found in a node_modules package, no longer do you need to navigate to that package folder or type the absolute path. Config management works with inheritance so your top-level package overrides configs defined in node_modules.

I hear the merits of TC39-style import statements and utilizing what TypeScript has put forward. However, breaking compatability can be seen in softwares like Angular 1.x -> Angular 2+. They eventually found something workable (and in some ways more elegant than React by wrapping build tools). However, they alienated people in the process, and ultimately people like a tool like React because it's easy to understand and developer trust isn't broken. You know exactly what the render loop is doing since the developer must call render explicitly.

I could go into the argumentation about code schools literally bargaining with people for a job for them to pick up the frontend technologies (i.e. people didn't adopt these tools for their merits), but let's leave that aside.

The other ultimate direction of Angular is that people think it's very, "enterprise". I don't fully agree with this perception but that seems to be what it's become. Basically, when developers with lots of visibility have, "ah-ha" moments and shift their thinking suddenly the perception becomes that the technology is "less accessible".

Besides this, I built a component library and got upstaged by Angular. I think React ended up with something more reasonable (and as such, I coded the start of a template literal implementation in contrast with JSX).

I just don't want you to upstage my efforts just because you had a whim. There should at least be engineering discourse that is inclusive to the merits of all approaches. You have done something awesome with Node.js Microsoft did something great with TypeScript. Please, let's not compromise the principles of usability/accessibility by changing the spec and breaking backwards compatability.

Happy to offer more details on what I've got! And of course, am looking for easy ways to transition to import/export syntax but for now I greatly like not having to transpile code and simply want to get a release out there without already being a dinosaur :)

Thanks

[kitsonk]

This isn't about breaking compatibility, deno !== Node.js. Any assumption of that will lead you down a rabbit hole. There is nothing preventing anyone of tighter integration of Node.js and TypeScript, but that is not the objectives of deno, and the wonders of open source that everyone if free to explore that as they wish.

Ryan said it pretty clear in his presentation at JSConfEU that node_modules was a mistake in his opinion, and something he wants to explore trying to not enact again. It wasn't the worst mistake, but I agree, module resolution on Node.js is now a sin that will be revisited on the heads of everyone now. I personally agree with the other objectives of deno, which is why I am excited to try to contribute to it. I will never expect it to be Node.js though, or Node.js + TypeScript. The module resolution is not only the most major breaking divergence from Node.js, there is the whole Node.js API set, the whole security first. So much code written for Node.js will never be runnable on deno, and that is a good thing, in my opinion.

[chainhelen]

Regret about node_modules in http://tinyclouds.org/jsconf2018.pdf , maybe helpful for you.

[slashhuang]

@kitsonk nice point

[zeh]

There should at least be engineering discourse that is inclusive to the merits of all approaches

One of the reasons new projects are created is so they can try different things without having to deal with the burden of compatibility with other systems. Pre-death by committee is not something new projects (that are not used anywhere) should have to deal with.

To truly move forward you often need someone with a vision and no constraints rather than a debate.

[calidion]

it seems deno is trying to fix what are broken in node, especially npm and node_modules where semver is executed and proven to be wrong.
npm is good for fast fetching and simple locating. but it is bad architectured and centralized. it is maintained not by the community but a private company.

as to decentralization, neither npm or node_modules is required.

but still, i would like to hear from deno to figure out a better approach on how to arrange the packages locally and to solve the problems endured with npm and semver. and to keep dependencies stable and reliable.

[beeman]

I think Ryan is quite clear about this in his presentation, specifically this quote:

I wanna simplify the module system, so screw all this stuff about how node_modules work and that like... BTW I can't be compatible with node, otherwise you just end up building node. So there's no attempt to compatibility to existing software.

Makes total sense, it's the only way to progress. Out with the old, and take with you the lessons learned.

[ry]

@mikegwhit As someone who has spent countless hours dealing with incompatibilities between Python 2 and 3, you have my sympathy and attention. There is nothing worse than superficial breaking changes in widely used software.

I am very much in favor of the continuation of Node's module resolution algorithm inside of Node. It would be pointless to break Node for relatively minor simplifications. I fear that wasn't clear in my presentation.

As @kitsonk said, this is not Node. This is not a fork and it does not intend to replace Node. It is simply new software that operates differently. Furthermore Deno is not going to be usable by end users anytime in the near future - you don't need to support it - it has no users.

[fkirc]

It's probably too late, but let's add some cost-benefit analysis for this issue.

I fully agree with all long-term goals of Deno.
@ry is an excellent developer and I agree with all technical problems of Nodejs.

However, the cost of fixing those technical problems is dramatically different depending on the specific problem.

For example, it is important to reduce the dependency on a centrally-managed repository (npm). Nevertheless, packaging stuff in a directory isn't always bad. Especially for badly maintained packages, it is nice to have all files in a local directory. I am not saying that node_modules is good, but I argue that developers should have a free choice if they want to package traditional directories.

If this isn't comprehensible, then let's have a look at raw dollar-numbers:

A forced migration from node_modules for millions of developers could easily go into a range of hundreds of million dollars, or perhaps even billions of dollars. This is why it just won't happen in the short term (but perhaps in the long term).
In contrast, a compatible rewrite of node_modules would cost a few thousand dollars.

Just to be clear: I am not saying that you need to rewrite node_modules; you are free to do whatever you want with your personal projects.

Nevertheless, I find it sad that those project-policies prevent me from using all the great features of Deno (improved security, TypeScript-support, URL-packages,...)

[fkirc]

I am very much in favor of the continuation of Node's module resolution algorithm inside of Node. It would be pointless to break Node for relatively minor simplifications. I fear that wasn't clear in my presentation.

This argument can be also applied for new software that is supposed to replace old software: Is it worth to destroy user-adaption and forwards-compatibility for "relatively minor" improvements?
Is it worth to end up with "no improvement at all" just because we were trying to push the last 10% improvements too hard?

[tysonrm]

There are clear advantages to streaming vs installing, c.f. module federation

[mschwartz]

Bumping this.

node_modules is a big deal if you're only using Deno for its lip server!