002: ES6 module interop

[bmeck]

Issue links for subtopics:

• ES Module Detection
• Module Evaluation Ordering
• ES Module Path Resolution
• CJS <-> ES import/export Conversion

[bmeck]

cc: @trevnorris @caridy

[caridy]

import.context.*

[bmeck]

reworded

[jeffmo]

(minor typo): export default foo;

[guybedford]

Using an .es file extension seems good, but am I right in thinking this means no ES module syntax in .js files then?

If so, I really think this should go alongside a package.json property to opt-in a package as an all-ES6 module package. NodeJS already reads the package.json, so simply having a field to indicate that the package is all ES6 modules and to run .js extensions as ES6 modules will avoid a lot of potential complaints about file extension. See also - https://twitter.com/guybedford/status/679684436674396160.

[bmeck]

@guybedford my complaint for purely ES6 is we lose common idioms such as :

if (process.env.NODE_ENV === ...) require(...)

module.exports = getModulesFromDir(dir);

and the ability to partially port modules.

I am also against mucking with package.json as a switch since a .js file may not explictly show what mode it is being interpretted in such as:

require('http').createServer(...)

May be ES6, but it is unclear since it lacks import/export. So I need to check package.json for any case that is unclear. Therefore when writing, I need to be careful that I don't write for the wrong mode.

It should also be made clear that under the EcmaScript spec exports are read only to observers so core modules will be stay in CJS semantics.

[bmeck]

moved .es file extension to .jsm

[ChALkeR]

@bmeck Are there any alternatives to shifting all the (new) file extensions from .js to .jsm long-term?

[bmeck]

@ChALkeR there have been several brought up and argued against. It is best to think of things as npm packages and file modules. In particular the issues are:

1. Module/Script goals in the JS spec do have different parsing mechanics.
2. Having the same file extension for different parsing requires metadata such as package.json listing all files that are in a goal (you could force packages to be of one type, but that prevents partial porting [one of the python3 problems])
3. Having different entry points in the same package for different goals such as jsnext:main, this has a few of its own problems as it tries to avoid discussing the interop of module systems, may be out of scope of interop. It leans towards point 2 as a solution of module interop.

Proposals discussed at various points:

• Parsing for usage of import/export to determine goal
• jsnext:main as an entry point - avoids discussing interop/votes for no interop, not really discussed in this PR
  • see npm main for a dist solution and a build solution npm/npm#9459 for discussing shipping multiple versions of a codebase in the same package for now
• Different file types due to differing goals
• No interop, pure "moduleType":"ES6" in the package.json.
  • drastically different approach than node takes on .node,.json
• Startup file for node determines the module system for node.

Some important points brought up around things above:

• partial porting is important
  • ES6 does not support some common use cases like reading all modules in a directory, changing dependencies on NODE_ENV, allowing monkey patching. To some code bases this is an important feature (native modules / APM tooling).
  • We should not require people to port their entire codebase to use a new feature (tests and all)
• jsnext:main approach has problems if you cannot fully port a module
• knowing your file type without knowing system metadata is important while developing
• double parsing is pricery
• any solution must discuss module interop or at least module interop on the package level

Some important differences with currently transpiled code and the proposal.

• Due to ModuleNamespace being the data view in/out of ES6 modules modifying the underlying exports from outside that module will be prevented. This keeps consistency with JS spec.

[ChALkeR]

@bmeck

I'm a bit concerned about introducing a non-standard extension (well, unless there would some standard that would describe it). For example, someone mentioned .jsx above — there are at least three of them: one, two, three. Firefox, for example, already uses .jsm for their modules: ref (and that uses EXPORTED_SYMBOLS = ['method1', 'method2'] syntax), and there at least two more different uses for that extension.

Also, this is leading us to a world where everyone would be using a non-standard filetype in all the supported modules, and that filetype would not even have the correct mime, filemanager icon, syntax highlighting, etc.

I'm sorry, it looks like this was discussed already, so I'm probably just asking silly questions.
But still: what about specifying the module system in (priority from highest to lowest):

1. Linter-style comment at the first line of the file.
2. If package.json is present — get it from there (a separate property), default to CJS if there is no such property (but only if package.json is present).
3. Runtime flag.
4. Try-hard-to-detect or even default to CJS, not sure.

3 and 4 would be needed only for not-so-often cases where there is no package.json. I am aware that looking for package.json would probably slow down the module loading a bit, but I don't think that the difference would be large.

[bmeck]

@ChALkeR we can bikeshed off .jsm/.jsx so lets skip that side of the talk, w/e extension if any we use can be carefully chosen, though given the plethora of existing extensions we will see some collision somewhere most likely (though hopefully not in web tech). We also currently have .node which is a non-standard file type, and are we talking IANA approved types or...?

Thats a lot of complexity to have 4 steps, and complexity is not something I enjoy. Don't know if your module is ES6 or CJS? check these 4 things (in 3-4 different places)? eh.....

1. Pragma at top of file like "use module" - A pragma at the top of file has been discussed, but I have had little in the way of talking to people (see comments on outdated diff) as the parsing goal is still different.

2. Package.json - As stated in my earlier comment moving the data to package.json has problems with:

   • jsnext:main as an entry point - avoids discussing interop/votes for no interop, not really discussed in this PR
     • see npm main for a dist solution and a build solution npm/npm#9459 for discussing shipping multiple versions of a codebase in the same package for now
   • No interop, pure "moduleType":"ES6" in the package.json.
     • drastically different approach than node takes on .node,.json

3. Runtime flag - We have not discussed a runtime flag, but that would effectively blacklist all existing npm modules today since they are CJS. It could be opt in, but it is out of scope and is not related to interop (like --use_strict does not discuss strict mode semantics).

4. Detection - Detection would only be possible for bumping up to Module goal not down to CJS Script. We cannot accidentally or implicitly change existing file modes. Same problem as if we were to try to detect non-strict code and then prepend "use strict". It breaks semantics even though they may not be explicitly used by the code, they could be used by consumers. There is a large discussion of this in an outdated diff comment.

[ljharb]

Re number 4 - you couldn't possibly detect it in every case, because an ES6 module need have no import or export statements nor need have any ES6 syntax - say, for a shim, or anything that solely relies on side effects. In other words, if you relied on detection, you'd have false negatives.

[bmeck]

@ljharb only if we default to "module", if we default to "script" it is possible, but once again sometimes ambiguous

[Fishrock123]

I haven't really kept up so much, what is the reason we'd want a different extension...?

[bmeck]

@Fishrock123 different parsing goals, clearly stating which mode you are in. Talked about a "use module" pragma at top, but didn't really see much from that. Different file extensions also allow shipping 2 versions of your files and having node pick up the ES6 over the CJS transpilation boilerplate.

[Fishrock123]

having node pick up the ES6 over the CJS transpilation boilerplate.

Wait, are we talking about having ...require transpile to ES6?

(also there's not really any use in referring to it as CJS, CJS is dead and it's really just node at this point.)

[bmeck]

@Fishrock123 just talking about the fact if we have an extension for ES6 (like .jsm) it can be prioritized in require.extensions over .js. Not talking about performing runtime transformation

[jokeyrhyme]

@ljharb fair enough.

Ruling out anything that needs a parser / runtime just leaves us with filenames, file extensions and package.jsons then, I suppose. Oh, and CLI flags and other out-of-band approaches.

[ljharb]

Yes, that's exactly right.

[mikesherov]

@ljharb, I agree that other contexts are important, but I'm still not sure how they are relevant to what node does. I mean that literally, not dismissively.

Let's say Node chooses package.json approach plus fallback to double parse. How does that effect what rails does? Couldn't rails still require .mjs if that was the solve there?

I don't see how Node's decision literally effects rails.

Also, that doesnt leave us with package.json vs. file extension. As I just mentioned, Node could choose to additively fallback to double parse in the situation of executing a script without a package.json to avoid having to use CLI flags.

Just because part of the solve doesn't work in non-node contexts doesn't mean it's off the table for node.

[jokeyrhyme]

What if we implement all major proposals in a series of fallbacks?

1. Use module mode if ".mjs" extension, else
2. Use module mode if "use module" pragma is found at the top of the file, else
3. Use module mode if package.json specifies it for this file / package, else
4. Use script mode, but parse as module mode if that breaks (or vice versa?)

I don't think we'll ever have a single solution that everyone is happy with AND that doesn't break compatibility with existing modules. It sucks that the community will likely fragment into different camps preferring different solutions, but at least Node.js will support them all and will be the compatibility glue between the different camps.

Perhaps after a year or so, statistics from NPM will highlight approaches that can be culled?

[curiousdannii]

How many of the semantic differences of modules are essential to the working of modules, and how many were improvements to the language that were just bundled in with it?

Going by the few that are mentioned in UnambiguousJavaScriptGrammar, only the arguments difference would apply to a required CJS package vs an imported module; this and foo would only have global scope when they're the root file loaded by node. But most of this discussion has been about required/imported modules, not root files...

[bmeck]

@curiousdannii there are several differences not mentioned in that proposal

• strict mode code that runs differently https://gist.github.com/bmeck/e54d607a4b0874e8dfca7af2a132b9c2
• await as a reserved word https://tc39.github.io/ecma262/#sec-future-reserved-words
• scoping rules (these don't actually apply to node)
• import/export is top level only (node's boilerplate to inject things is a function T_T, so breaks)
  • can't early return

It should be noted that with makes adding keywords horrendously tough, and that new features will sometimes only be targeting the Module goal.

[curiousdannii]

@bmeck Right, so a lot of the changes are an implicit strict mode and extensions to strict mode. ISTM that it's a fundamental problem to introduce changes like that which can't be deterministically identified through parsing. But that's a problem for the ES spec team, not the node team.

Sorry if this should be obvious, but is it intended that a module will be able to be run as the main file passed to the node binary? If so, would that mean that the .mjs extension would be required, and we couldn't keep our pretty extensionless bin scripts? If the .mjs is not required, then some kind of deterministic parsing would be needed?

[bmeck]

@curiousdannii as it stands --module will be a required cli flag for various needs like shebang / stdin based entry points. Also, add an extension, relying on shebangs makes things hard if you ever want Windows support.

[bmeck]

Note: the symlink npm makes for "bin" does not need to have the same name, and can drop your extension from the command.

[mikesherov]

Could also solve extensionless bin scripts with a separate node binary that is essentially equivalent to node --module and specify that in your shebang. Food for thought.

[billti]

@martinheidegger Nice work on getting the info up! Really useful stuff.

I've been studying this for a while, and I think the "fat-packages" approach is a lot of cost and hazard for little gain. I wrote details of why in the doc at https://github.com/billti/node-es2015/blob/master/fat_packages.md .

It seems without fat packages the need for the extra file extension is reduced significantly, and with suggestions from package.json (or a Node switch), parsing could "get it right" first time. I detailed how this may work in the "algorithms.md" page on that same repo. (It also agrees with what @mikesherov wrote, in that double-parsing shouldn't be a big hit when these hints aren't given for Node scripts outside a package, and any script can be unambiguously ES2015 if at least one export (e.g. export default null) is required by Node.js)

The docs also discuss how CommonJS and ES2015 modules would appear to each other, which is critical how we model modules in code (a subject close to my heart, as I work on the TypeScript team).

I'd appreciate any feedback.

[ljharb]

The gain is "ES modules survive". Without fat packages, ES modules are DOA, and the community will simply not adopt them.

No requirement is more important than the ability to smoothly migrate to ES modules.

[billti]

Which parts of the write-up did you disagree with? On thinking through the implications from a few different angles, I came to the opposite conclusion - that fat packages would hold ES2015 modules back (for the reasons outlined).

[glen-84]

So, assuming an existing package my-package at version 2.0, if the author wanted to update it to use ES modules, they could:

1. Release a new major version, my-package at version 3.0 (with an engines requirement).
2. Continue to do minor changes and bug fixes to the 2.x CJS version.
3. Optionally release a 3.0 of the CJS version as a new package (my-package-cjs), if that's ever needed.

Is this correct? If so, I don't see how ES modules would be "DOA" – why would the community not adopt the most recent release (where most development is likely to occur)? I know I would.

Edit: I don't know if point 2 makes sense or is allowed with regard to versioning. If not, skip to point 3.

[ljharb]

@glen-84 in practice, authors will release v3 and ignore v2. The only thing that will allow for everyone to have a smooth migration is if the same version of the same package works in both. Refer to python 2 vs python 3 if it needs further explanation.

[jokeyrhyme]

My team targets the current "active" Node.js LTS (v4), and uses only natively-supported features for that Node.js version. We don't currently use babel or otherwise transpile Node.js modules that we publish to npm. That means we are in the land of short arrows and generators, but are not using destructuring.

When Node.js v6 becomes the "active" LTS later this year, that will become our new baseline.

Node.js v8 is the earliest possible LTS that would support native ES modules, and it might not land until v10. So it's going to be a long time before my team, with our current customer support policy, move up to native ES modules for Node.js. If our policy holds, we probably won't be transpiling.

We do make increasingly broad use of babel for browser / webview code.

We'll probably review our Node.js-transpile policy as the productivity gains with newer ECMA standards become too good to ignore, but I figured I'd share this as a one use case data point among many.

[billti]

If you are going to add functionality that is specific to an ES2015 format module (i.e. you couldn't transpile down to a CommonJS module with identical behavior), then you would ship a package with the ES2015 module which requires an ES2015 syntax capable runtime (i.e. { "engines" : { "node" : ">= 7.0.0" }}, and you can't have side-by-side (i.e. "fat packages"). Hopefully this isn't controversial.

If you are going to ship a package where every module can be represented as a CommonJS module (and you do so for maximum reach), then if the runtime loads the ES2015 format version of the module or not should be opaque to the consumer (i.e. the behavior should be identical). Hopefully this is a goal.

So if a package includes both formats that should behave identically, what advantage does the runtime loading the ES2015 version sometimes, depending on the Node.js capabilities, provide? It seems to me high risk that you have two different source files intended to be the same module, but may have subtle differences hard to catch before deployment. (You are always testing both module formats by running on different Node.js runtimes before shipping now, right? 😉 ).

Note even if always loading the CommonJS version, nothing precludes you from including whatever ES2015 source you built it from (if you did) for reference and debugging (via sourcemaps).

I provide several other reasons, but that to me seems to be the main risk. Aiming for identical behavior in two different source files that have different syntax/semantics is non-trivial, and when folks get this wrong, code is going to break just by upgrading Node.js.

[billti]

@bmeck and @jdalton : I see the https://github.com/bmeck/UnambiguousJavaScriptGrammar write-up is starting to align re format detection of .js files, and now recommends an unambiguous grammar - which is awesome!

As critical I believe, but I don't see this covered in depth in this write-up, or the "In defense of..." proposal from @wycats and @dherman, is the semantics of how CommonJS modules appear when imported as ES2015 modules, and vice-versa (i.e.an import of a CommonJS modules, or a require of a ES2015 modules). Getting this right has big implications on how friction-free it will be to co-exist, migrate, and continue to use existing code already authored in ES2015 modules (that is currently being transpiled to CommonJS via Babel, TypeScript, etc.).

I cover one proposal in my algorithms doc along with some samples, after working through a number of scenarios with my team. How can I get involved in helping land this?

[jmm]

@billti

but I don't see this covered in depth in this write-up, or the "In defense of..." proposal from @wycats and @dherman, is the semantics of how CommonJS modules appear when imported as ES2015 modules, and vice-versa

I haven't looked at that part in depth, but I was under the impression that those semantics are included in this proposal (this PR and followups) and that In Defense of .js is just proposing an alternative approach to detecting the type (script vs module). It seems to me it'd be out of scope for the UnambiguousJavaScriptGrammar proposal as the ES spec has no awareness of CommonJS.

[ljharb]

imo it would be a big mistake to mention "module detection in node" more than casually in https://github.com/bmeck/UnambiguousJavaScriptGrammar - i think it will distract from the simple nature of that proposal, which is to remove an ambiguity in the language.

Thus yes, it is entirely out of scope.

[billti]

Thanks @jmm. I did read that section initially, but I guess I'd forget by the time I got through the other 650 comments discussing module detection 😩

@bmeck, due to the size of the discussion on this thread already, and being that it mostly covers detecting module syntax, would you rather discuss the CommonJs <-> ES2015 interop semantics on a separate thread? (e.g. #10 ).

[jmm]

@billti No problem. I wasn't sure if you hadn't seen that part, or thought it didn't address everything. I've only skimmed it, and since, like you said, most of the discussion here was about detection I assumed the rest was probably more straightforward.

but I guess I'd forget by the time I got through the other 650 comments discussing module detection 😩

I know the feeling.

I know you weren't asking me, but FWIW I think discussing the interop in #10 makes sense, but I don't know why it's closed.

[martinheidegger]

@billti Thank you for your feedback. I commented on your repository about the use of the phrase "no value".

Also after reading the discussion about the Unambiguous JS Grammar I added a (imho quite interesting) information of what could happen if we could get a change of the specification: es6modules-nodejs#17.