ES modules in Node, and .mjs files

[GeoffreyBooth]

Node 8.5 added support for ES modules behind the --experimental-modules flag. Node 10, expected to be released in April 2018, will supposedly drop the flag. Here’s a great overview.

Adding ES modules was mostly Node finally supporting the import and export syntax from ES2015, that CoffeeScript already supports. There is at least one caveat, though: Node only supports importing files with an .mjs extension, from files with an .mjs extension.

This raises the issue of how to use the CoffeeScript compiler to generate output JavaScript files with .mjs extensions. If you’re using the coffee command to compile a single file, you can specify the output filename, including extension, explicitly:

coffee --compile --output module.mjs module.coffee

But for folders, the compiler automatically outputs all .coffee files as .js files. It doesn’t take much effort to add a post-compilation step that renames these extensions, but should this perhaps be something the compiler handles?

One way to do it would be to introduce a new .mcoffee file extension, that the compiler would output as .mjs. Straightforward, though the greater ecosystem around CoffeeScript would need to be updated. (Syntax highlighters, etc.)

Another way to do it would be with a new CLI flag, e.g. --output-extension mjs. This might be useful in its own right, to allow outputting JSX files with a .jsx extension (if for some reason you wanted to simply save them, rather than immediately transpiling them into JavaScript). But @jashkenas and others (including me) feel strongly against adding yet more flags to the CLI, except as a last resort.

Are there any other ways to handle this situation? Unfortunately we can’t simply output all files with import or export statements as .mjs, because a lot of people will want the current behavior for quite a while, as Babel’s treatment of those statements is different than Node’s and many people won’t want to refactor their code anytime soon. (A great lesson in why not to start using features before they’re both standardized and implemented!)

The other thing on my mind regarding this is that I want to rewrite the modules tests to use actual import and export statements that Node evaluates, rather than comparing strings; but I think the only way to do this would be to spawn a new Node process with an .mjs file as its entrypoint. (This is regardless of whether the --experimental-modules flag is still around.) This would add considerable complexity to the test runner, but I think would be worth it.

[GeoffreyBooth]

I’m leaning toward having the compiler output .mcoffee and .litmcoffee files as .mjs files. This would allow a project’s output to contain both types of files (.mjs and .js).

In a sense, this should be unnecessary, since Node only supports importing .mjs files from other .mjs files, and an .mjs file can’t import a .js file (and vice versa), so a project really should contain either all .mjs or all .js files. But I think in practice, in the near term people will be piping everything through Babel for awhile, which will make all final output files be .js files and obscure this limitation of Node’s. So there will be projects with both filetypes at least during a transition period, as people refactor their projects to be fully .mcoffee/.mjs.

And yes, this means adding two more canonical CoffeeScript file extensions. That’s rather unfortunate. But does anyone have any better ideas? @jashkenas @lydell @zdenko

[GeoffreyBooth]

Looks like TypeScript is debating similar issues: microsoft/TypeScript#18442

[zdenko]

How about .m.coffee and .m.litcoffee. Or, perhaps .esm.coffee and .esm.litcofee?
In this way ecosystem around CS would probably require minor updates, if any.
And, it looks like mode: esm PR will be merged, so adding CLI flag might also be an option.

[GeoffreyBooth]

Those are good ideas. From what I've learned recently, modules in Node are a lot more unsettled than they first appeared, and won't be launching unflagged in Node 10.0.0. So supporting .mjs is a lot less urgent.

There's significant pushback to the new file extension, as you see in nodejs/node#18392. To be honest, I don't know why people would prefer .mjs to some of the other solutions for declaring modules. I think we should wait until it's clearer how this shakes out on Node's side before we do anything.

[jashkenas]

Great call, Geoffrey!

[GeoffreyBooth]

In discussing with @jkrems around #5268, I think we could add support to the CLI for outputting as .mjs or .jsx or any arbitrary extension by adding the extension to --output, e.g.:

coffee --compile --output dist/**/*.mjs src

This way we avoid creating a new flag or Node API option. The Node API doesn’t return filenames so it doesn’t need updating, and this way all the ecosystem plugins are unaffected. Since those plugins weren’t using the CLI for generating new filenames anyway, whatever methods they use for determining output filenames would continue to work.

@jkrems or whoever wants to take on this enhancement, please keep in mind that CoffeeScript has no dependencies. You’ll have to handle glob or star support on your own.

[GeoffreyBooth]

Also for those wondering about Node’s support for ES module CoffeeScript files in general, regular output .js files will work if you add "type": "module" to your project’s package.json: https://nodejs.org/api/esm.html#esm_package_json_type_field.

[Zolmeister]

To clarify, this issue would allow the following to work?

# index.coffee
export default -> null

// package.json
{
  "type": "module"
}

coffee index.coffee
# SyntaxError: Unexpected token 'export'

[GeoffreyBooth]

To clarify, this issue would allow the following to work?

Currently Node doesn't provide an ESM equivalent of vm.Script, which the coffee command uses the evaluate code; there's vm.Module but it's experimental and behind a flag. Until that changes, the coffee command won't be able to execute ESM code.

However the following works already today, with your index.coffee and package.json from above:

coffee --output index.js index.coffee
node index.js  # Use Node 14+ for best results

[Zolmeister]

I found a workaround thanks to nodejs/modules#507, using https://nodejs.org/api/esm.html#esm_transpiler_loader
The only change I had to make to get it working was to comment out

coffeescript/src/index.coffee

Lines 106 to 111 in 2f82b75

	if require.extensions
	for ext in CoffeeScript.FILE_EXTENSIONS then do (ext) ->
	require.extensions[ext] ?= ->
	throw new Error """
	Use CoffeeScript.register() or require the coffeescript/register module to require #{ext} files.
	"""

(I even got mocha to work by overriding it's file extension in the loader and patching it's require/import hook)

What would be the most correct way to update coffeescript core to support this pattern?

[GeoffreyBooth]

What would be the most correct way to update coffeescript core to support this pattern?

I wrote the example at https://nodejs.org/api/esm.html#esm_transpiler_loader. That code also requires an experimental Node flag, so it's not much better than --experimental-vm-modules.

[Zolmeister]

Sure, but I figured supporting that feature requires almost no changes to coffeescript core.

Edit: Actually, looking closer it appears like no changes are required (the throw is behind ?=). Thanks for taking a look though.

[danielbayley]

@GeoffreyBooth What’s the current situation on this? (i.e coffee being able to work directly based on "type": "module"…)

[edemaine]

In case it's useful, here is an ESM loader for a related language, Civet, probably based on Geoffrey's code above. It shouldn't hard to adapt to CoffeeScript. (Apologies, it's written in Civet, but you can see a built version here.) However, you'd still need to run your code via node --loader your-esm-loader filename.coffee. (No longer an experimental flag, at least.)

I'd be happy to port this over to CoffeeScript if there's interest. (Maybe give a thumbs up if so?) I think it'd be nice to have as part of the CoffeeScript distribution, so you could use node --loader coffeescript/esm.

[GeoffreyBooth]

coffee being able to work directly based on "type": "module"

The coffee command itself relies on the Node vm module which still lacks ESM support and no one is working on. There could be an alternate flow where when we're in a type module context it runs Node in a child process with a loader; that would work. But someone needs to implement this. There would also be the matter of getting --inspect to attach to this child process.