Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bin exported without extension, breaks node's native loader picker and ts-node/esm #4645

Closed
FossPrime opened this issue Jun 3, 2021 · 32 comments

Comments

@FossPrime
Copy link

FossPrime commented Jun 3, 2021

Description

custom loaders like ts-node have a really hard time figuring out the mocha "binary"

The binary lacks.js or cjs extensions. Changing the node_modules/.bin/mocha extension isn't enough.

Steps to Reproduce

  1. Sandbox: https://codesandbox.io/s/mocha-ts-node-16-ctt5k?file=/package.json
  2. Open Terminal
  3. Run npm run broken-test-npx or run source env.sh; mocha test/test.js

OR

  1. npm install ts-node typescript @types/mocha mocha on a type: "modules" project.json
  2. Set export NODE_OPTIONS="--loader ts-node/esm/transpile-only"
  3. Run npx mocha test/any-test-at-all.js

Expected behavior:

No errors, tests run.

Actual behavior:

Setting a loader prevents node from running the mocha binary, in any way

$ npm run test # Copies mocha to mocha.js as workaround
3 passing (3ms)
$ npm run test-cjs # Copies mocha to mocha.cjs as workaround
1 passing (2ms)
$ npm run broken-test-npx # Node cant tell what loader to use and errors out
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /sandbox/node_modules/mocha/bin/mocha
$ npm run broken-test-direct # same behavior as before
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /sandbox/node_modules/mocha/bin/mocha

Reproduces how often: 100% of the time

Versions

  • The output of mocha: 8.4.0
  • The output of node --version: 14 (Sandbox) and 16 as well
  • Your operating system
    • name and version: yes
    • architecture (32 or 64-bit): 64
  • Your shell (e.g., bash, zsh, PowerShell, cmd):
  • Your browser and version (if running browser tests):
  • Any third-party Mocha-related modules (and their versions):
  • Any code transpiler being used (and its version): ts-node loader

Additional Information

Transpile only mode does not make a difference. Extension of tests do not make a difference. As long as the loader boots up, when node proceeds to guess which loader to use, node errors out.

@FossPrime FossPrime changed the title Bin exports resolve to extensionless files Bin exports break native loaders and ts-node/esm Jun 3, 2021
@juergba
Copy link
Contributor

juergba commented Jun 3, 2021

This is an working example with Mocha, ESM and ts-node/esm.
Why do you think this issue is a Mocha issue? Not a loader or configuration problem?

@FossPrime
Copy link
Author

FossPrime commented Jun 3, 2021

@juergba The error actually comes from node... internal/modules/esm/get_format.js:71:15

Again, this is just how node works - it'll use the extension loader defined or fallback on the .js loader. We can't make it use ts-node as a result.

TypeStrong/ts-node#116

Most symlinks in node_modules/.bin use extensions... I have the loader set to transpile only... it doesn't have much configuration to do.

lrwxrwxrwx   1 mind ya   21 Jun  2 13:15 mocha.js -> ../mocha/bin/mocha
lrwxrwxrwx   1 mind ya   22 Jun  2 11:45 ts-node -> ../ts-node/dist/bin.js
lrwxrwxrwx   1 mind ya   26 Jun  2 11:45 ts-node-cwd -> ../ts-node/dist/bin-cwd.js
lrwxrwxrwx   1 mind ya   29 Jun  2 11:45 ts-node-script -> ../ts-node/dist/bin-script.js
lrwxrwxrwx   1 mind ya   32 Jun  2 11:45 ts-node-transpile-only -> ../ts-node/dist/bin-transpile.js
lrwxrwxrwx   1 mind ya   40 Jun  2 11:45 ts-script -> ../ts-node/dist/bin-script-deprecated.js
lrwxrwxrwx   1 mind ya   23 May 12 09:45 eslint -> ../eslint/bin/eslint.js
lrwxrwxrwx   1 mind ya   25 May 12 09:45 esparse -> ../esprima/bin/esparse.js
lrwxrwxrwx   1 mind ya   28 May 12 09:45 esvalidate -> ../esprima/bin/esvalidate.js
lrwxrwxrwx   1 mind ya   14 May 12 09:45 flat -> ../flat/cli.js
lrwxrwxrwx   1 mind ya   19 May 12 09:45 json5 -> ../json5/lib/cli.js
lrwxrwxrwx   1 mind ya   25 May 12 09:45 js-yaml -> ../js-yaml/bin/js-yaml.js
lrwxrwxrwx   1 mind ya   15 May 12 09:45 katex -> ../katex/cli.js
lrwxrwxrwx   1 mind ya   33 May 12 09:45 markdown-it -> ../markdown-it/bin/markdown-it.js
lrwxrwxrwx   1 mind ya   14 May 12 09:45 mime -> ../mime/cli.js
lrwxrwxrwx   1 mind ya   20 May 12 09:45 mkdirp -> ../mkdirp/bin/cmd.js
lrwxrwxrwx   1 mind ya   24 May 12 09:45 nanoid -> ../nanoid/bin/nanoid.cjs
lrwxrwxrwx   1 mind ya   29 May 12 09:45 replace-in-file -> ../replace-in-file/bin/cli.js
lrwxrwxrwx   1 mind ya   16 May 12 09:45 rimraf -> ../rimraf/bin.js
lrwxrwxrwx   1 mind ya   23 May 12 09:45 semver -> ../semver/bin/semver.js
lrwxrwxrwx   1 mind ya   19 May 12 09:45 shjs -> ../shelljs/bin/shjs

nandoid goes as far as using cjs

@FossPrime FossPrime changed the title Bin exports break native loaders and ts-node/esm Bin exported without extension, breaks node's native loader picker and ts-node/esm Jun 3, 2021
@juergba
Copy link
Contributor

juergba commented Jun 4, 2021

Again, this is just how node works - it'll use the extension loader defined or fallback on the .js loader. We can't make it use ts-node as a result.

I haven't understood yet, or something is wrong in your argumentation.

  • without extension the above fallback should be triggered. Which would be correct with .js loader. Why isn't this the case?
  • why do you pass Mocha, a CJS module, to your ESM loader? For transpilation?
  • why does my sample mentioned above work? What is the difference to your case?

@juergba
Copy link
Contributor

juergba commented Jun 4, 2021

If you remove --loader ... from NODE_OPTIONS and add it to .mocharc.cjs instead , does it work then?
I guess so, because in this case Mocha starts first, then the loader is called in a subprocess.. The entry point is lib/cli/cli.js.

@juergba
Copy link
Contributor

juergba commented Jun 4, 2021

Yes, above sample does not work anymore, when I remove loader: 'ts-node/esm' from .mocharc.cjs and use NODE_OPTIONS="--loader=ts-node/esm" instead.

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for D:\temp\ts-node-repros\node_modules\mocha\bin\mocha
    at defaultGetFormat (internal/modules/esm/get_format.js:71:15)
    at defer (D:\temp\ts-node-repros\node_modules\ts-node\src\esm.ts:81:7)
    at D:\temp\ts-node-repros\node_modules\ts-node\src\esm.ts:103:12
    at Generator.next (<anonymous>)
    at D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:4:12)
    at getFormat (D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:54:16)
    at Loader.getFormat (internal/modules/esm/loader.js:102:42)
    at Loader.getModuleJob (internal/modules/esm/loader.js:231:31)

@cspotcode can you have a look at this, please?

@cspotcode
Copy link
Contributor

cspotcode commented Jun 4, 2021

This is a confirmed bug in nodejs. Please remind them that they need to fix it! I have cited mocha as one of the libraries being broken by it, but it will help if mocha's users and maintainers also remind them (politely) that they need to fix this!
nodejs/node#33226

To reproduce the bug, use a completely empty --loader. This will prove that node is responsible for the error, not ts-node nor any other loader.

$ node -v
v16.2.0

$ touch empty-loader.mjs
$ touch extensionless-entrypoint
$ node --loader empty-loader.mjs ./extensionless-entrypoint
(node:10077) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
node:internal/process/esm_loader:74
    internalBinding('errors').triggerUncaughtException(
                              ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /c/Users/cspot/extensionless-entrypoint
    at new NodeError (node:internal/errors:363:5)
    at Loader.defaultGetFormat [as _getFormat] (node:internal/modules/esm/get_format:71:15)
    at Loader.getFormat (node:internal/modules/esm/loader:105:42)
    at Loader.getModuleJob (node:internal/modules/esm/loader:243:31)
    at async Loader.import (node:internal/modules/esm/loader:177:17)
    at async Object.loadESM (node:internal/process/esm_loader:68:5) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

EDIT Fixed a typo above; the node --loader invocation was missing the ./extensionless-entrypoint

@juergba
Copy link
Contributor

juergba commented Jun 4, 2021

@cspotcode thank you for your explanation, I will remind nodejs tomorrow.

@rayfoss you have the workaround with .mocharc.cjs. I don't know yet what to do with our extensionless binary.

@cspotcode
Copy link
Contributor

I think if mocha wants to implement a simple fix, they can add a new bin entrypoint file with a file extension (make sure to keep the old one, too, for backwards-compatibility) and then point package.json "bin" field to the new entrypoint with the file extension.

I believe that update can be published as a non-breaking change.

@FossPrime
Copy link
Author

FossPrime commented Jun 4, 2021

I think if mocha wants to implement a simple fix, they can add a new bin entrypoint file with a file extension (make sure to keep the old one, too, for backwards-compatibility) and then point package.json "bin" field to the new entrypoint with the file extension.

I believe that update can be published as a non-breaking change.

... My PR is not a breaking change in any way. mocha/bin/_mocha.js is a new entry point. And NPM makes a symlink called .bin/_mocha. People who use mocha/bin/_mocha directly or via the symlink .bin/_mocha would be unaffected.

There is an argument to be made that it should be mocha/bin/mocha.js, not mocha/bin/_mocha.js, which is then symlinked to .bin/_mocha by NPM, I was just preserving the convention established by mocha/bin/_mocha.

@cspotcode
Copy link
Contributor

You mention a drawback in your PR; is that non-breaking?

We might be talking about the same thing. I'm imagining that mocha ships 4x different files: mocha, mocha.js, _mocha, and _mocha.js. package.json "bin" points to the ones with file extensions.

@FossPrime
Copy link
Author

FossPrime commented Jun 4, 2021

You mention a drawback in your PR; is that non-breaking?

We might be talking about the same thing. I'm imagining that mocha ships 4x different files: mocha, mocha.js, _mocha, and _mocha.js. package.json "bin" points to the ones with file extensions.

My PR only addresses the issue at hand, I couldn't see how you guys output bundled code... So the PR has

  • _mocha (naked vanilla)
  • _mocha.js (clothed vanilla)
  • mocha (naked bundled)

Symbolic Links made by NPM:

  • .bin/_mocha (& to clothed vanilla _mocha.js)
  • .bin/mocha (& to naked bundled mocha)

We can make a mocha/bin/mocha.js that uses the naked bundled mocha/bin/mocha without duplicating the big file by using a funky, but legal, side-effect only, require or import call to the naked bundle.

Update: I corrected the description of the PR to reflect it's contents. I updated the PR code a few hours after publishing to duplicate naked vanilla mocha/bin/_mocha once I stumped on documentation and projects that directly referenced mocha/.bin/_mocha... And remembered Deno and custom package managers exist, and they may not read our package.json for the location of our stable endpoints.

@FossPrime
Copy link
Author

no longer an issue in Mocha 9

@lachrist
Copy link

Hi again @juergba, could it be that the loader mocha option override require? Sorry, I could not find any documentation on the loader option...

@juergba
Copy link
Contributor

juergba commented Aug 14, 2021

@lachrist There is no loader option in Mocha.

Just use 'experimental-loader' : './lib/loader.mjs'. Or with the alias name: loader: './lib/loader.mjs'

Edit: Mocha recognizes Node's option and forwards them to Node by spawning a child-process.

@lachrist
Copy link

Now, it makes more sense! I got confused because node has no documented loader option, hence I assumed it was a mocha option :|

Thanks for the swift answer!

@juergba
Copy link
Contributor

juergba commented Aug 14, 2021

Welcome.
Run node -h, then: --loader, --experimental-loader=... use the specified module as a custom loader

@JakobJingleheimer
Copy link

@juergba @giltayar This issue occurs today. I'm not sure why it was fixed in mocha v9 release but occurs now exactly as OP described for exactly the reason OP described.

I spoke with npm and yarn authors, and both package managers automatically alias a non-naked bin with a naked version (but not the other way around).

There is an on-going discussion within Node.js on how to handle naked bin files, but it has become rather protracted and does not appear to be resolving any time soon.

The quickest solution to get mocha working with loaders is to:

  • add a .js file extension to the currently naked bin file
  • add "type": "commonjs" to mocha's package.json (commonjs is the default, but it saves Node.js having to look further for the info and only eventually fall back to the default value, and is future-proofing for when Node.js switches the default to module)

I'd be happy to do this if you agree.

@juergba
Copy link
Contributor

juergba commented Jan 24, 2022

@JakobJingleheimer no, I'm against this .js solution. We haven't fixed anything in v9 either.

When you set --loader with NODE_OPTIONS then it fails, because Mocha's entry point is extensionless.
When you set --loader via Mocha options (mocha --loader=... or mocha -n loader=...) then Mocha launches a child-process and the entry point lib/cli/cli.js has an extension.

@JakobJingleheimer
Copy link

Ah. Perhaps a documentation update then?

@giltayar
Copy link
Contributor

@juergba why are you against this? (or at least two bin entry points: one without an extension and one with it)

@JakobJingleheimer
Copy link

Also keeping in mind that npm and yarn both handle supplying the naked one automatically.

@juergba
Copy link
Contributor

juergba commented Jan 25, 2022

@giltayar --loader is still experimental and I don't see any reason to adapt Mocha at that floating stage. Furthermore we have this easy work-around.

In aprox. April/May we will publish Mocha v10. If Node still hasn't made up their mind until then about this extensionless binary story, we evtl. could add *.js to our binary as a breaking change.

@lachrist
Copy link

Hi guys I've discussed this issue with the node team. And this will probably never be solved on their end. Checkout nodejs/node#41465. In particular, this is my final take on it:

At the moment, monitoring ECMAScript modules requires --experimental-loader flag. But this flag makes node always use the new esm loader to load the main file. Without this flag, node will sometimes use the old cjs loader. To be clear, the new esm loader is capable of loading commonjs files as well (along with native modules, web assembly modules, ...). So most of the time this change should not be noticeable. The biggest difference I encountered so far is that the cjs loader handles files without extension whereas the esm loader would throw an exception. This is intentional: the cjs loader only has to handle one kind of module (commonjs modules) whereas the esm loader has to handle different kinds of modules and the node team wants to reserve extension-less files for web assembly.

@juergba
Copy link
Contributor

juergba commented Jan 25, 2022

What is the (historic) reason that binaries (or executables?) are extensionless?
The first line #!/usr/bin/env node means, this file is an executable, right? In Linux only? In Windows this line has no purpose.

Also keeping in mind that npm and yarn both handle supplying the naked one automatically.

For which reason does npm/yarn do that?

@JakobJingleheimer
Copy link

JakobJingleheimer commented Jan 25, 2022

Yes, I'm 99% sure the historic reason is days of yore (unix/linux).

Note that adding the .js file extension would NOT be a breaking change as npm and yarn will automatically handle the extension-less part for you.

Also, I'm completely confident that making the two tiny changes I suggested will 100% be compatible now and in future, wherever loaders lands ;)

@lachrist cough I am on the node team 🙂

There is a more recent discussion, but I'm not linking to it at the moment because I need to port it from a PR (that I need to closed) to an actual discussion.

@cspotcode
Copy link
Contributor

The first line #!/usr/bin/env node means, this file is an executable, right? In Linux only?

Technically the execute bit tells Linux it's an executable, and the two-byte sequence #! is a magic number understood by the Linux kernel, used to differentiate different binary formats. That's my understanding at least.

Package managers use https://www.npmjs.com/package/@zkochan/cmd-shim to generate .ps1 and .cmd shims that achieve the same effect on Windows.

@cspotcode
Copy link
Contributor

#4645 (comment)

^^ This is a fully backwards-compatible fix and takes little effort, right? I feel like there's some confusion with talk of this being a breaking change, but it's not. Is there anything we can explain further?

@juergba
Copy link
Contributor

juergba commented Jan 27, 2022

I spoke with npm and yarn authors, and both package managers automatically alias a non-naked bin with a naked version (but not the other way around).

If you mean: bin/mocha.js results in : .bin/mocha then I agree. Otherwise ..?

I can live with adding an additional bin/mocha.js requiring the naked one. The second binary bin/_mocha is on sneaky deprecation, we should leave it as is.

@JakobJingleheimer please go ahead with your PR, if you are still on fire. Otherwise we wait till Mocha@10.

@JakobJingleheimer
Copy link

I spoke with npm and yarn authors, and both package managers automatically alias a non-naked bin with a naked version (but not the other way around).

If you mean: bin/mocha.js results in : .bin/mocha then I agree. Otherwise ..?

Yes, exactly 🙂

I can live with adding an additional bin/mocha.js requiring the naked one. The second binary bin/_mocha is on sneaky deprecation, we should leave it as is.

You don't have to do this: npm and yarn will take care of it automatically.

@JakobJingleheimer please go ahead with your PR, if you are still on fire. Otherwise we wait till Mocha@10.

Not on fire. I've just seen quite a few bugs about it. Whatever you prefer 🙂

I've consolidated the discussions into nodejs/node#41711

@juergba
Copy link
Contributor

juergba commented Jan 27, 2022

You don't have to do this: npm and yarn will take care of it automatically.

This is not true. Take a look at e.g. nanoid after installation with npm, there is a bin/nanoid.cjs but no bin/nanoid.

@cspotcode
Copy link
Contributor

cspotcode commented Jan 27, 2022 via email

@cspotcode
Copy link
Contributor

cspotcode commented Mar 1, 2022

ts-node 10.6.0 implements a workaround that hopefully mimics the bugfix which will eventually be published in node core. This means that, for users of mocha and ts-node, upgrading to the latest ts-node should avoid this issue.

https://github.com/TypeStrong/ts-node/releases/tag/v10.6.0

crapStone pushed a commit to Calciumdibromid/CaBr2 that referenced this issue Mar 8, 2022
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [ts-node](https://typestrong.org/ts-node) ([source](https://github.com/TypeStrong/ts-node)) | devDependencies | minor | [`10.5.0` -> `10.6.0`](https://renovatebot.com/diffs/npm/ts-node/10.5.0/10.6.0) |

---

### Release Notes

<details>
<summary>TypeStrong/ts-node</summary>

### [`v10.6.0`](https://github.com/TypeStrong/ts-node/releases/v10.6.0)

[Compare Source](TypeStrong/ts-node@v10.5.0...v10.6.0)

Questions about this release? Ask in the official discussion thread: [#&#8203;1666](TypeStrong/ts-node#1666)

**Added**

-   Adds workaround for extensionless entrypoints with ESM loader ([#&#8203;1649](TypeStrong/ts-node#1649), [#&#8203;1654](TypeStrong/ts-node#1654))
    -   You can now combine tools such as `mocha` with `--loader ts-node/esm`, where previously node would throw `[ERR_UNKNOWN_FILE_EXTENSION]`
    -   node has a bug where combining `--loader` with an extensionless entrypoint causes this error [nodejs/node#&#8203;33226](nodejs/node#33226)
    -   Some tools, for example `mocha`, have an extensionless entrypoint. ([source](https://github.com/mochajs/mocha/blob/547ffd73535088322579d3d2026432112eae3d4b/package.json#L37), [source](https://github.com/mochajs/mocha/blob/547ffd73535088322579d3d2026432112eae3d4b/bin/mocha))
    -   Combining `NODE_OPTIONS=--loader ts-node/esm` with these tools causes this error.  [mochajs/mocha#&#8203;4645](mochajs/mocha#4645)
    -   node intends to fix this bug in a future release: [nodejs/node#&#8203;41711](nodejs/node#41711)
    -   In the interim, we have implemented a workaround in ts-node.
-   Adds support for target "ES2022" in `moduleTypes` overrides ([#&#8203;1650](TypeStrong/ts-node#1650))

**Fixed**

-   Fixed bug where `--swc` and other third-party transpilers did not respect `moduleTypes` overrides ([#&#8203;1651](TypeStrong/ts-node#1651), [#&#8203;1652](TypeStrong/ts-node#1652), [#&#8203;1660](TypeStrong/ts-node#1660))
-   Fixed bug where node flags were not preserved correctly in `process.execArgv` ([#&#8203;1657](TypeStrong/ts-node#1657), [#&#8203;1658](TypeStrong/ts-node#1658))
    -   This affected `child_process.fork()`, since it uses `process.execArgv` to create a similar child runtime.
    -   With this fix, `child_process.fork()` will preserve both node flags and `ts-node` hooks.
-   Fixed compatibility TypeScript 4.7's API changes ([#&#8203;1647](TypeStrong/ts-node#1647), [#&#8203;1648](TypeStrong/ts-node#1648))

https://github.com/TypeStrong/ts-node/milestone/9

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

Co-authored-by: cabr2-bot <[email protected]>
Reviewed-on: https://codeberg.org/Calciumdibromid/CaBr2/pulls/1195
Reviewed-by: Epsilon_02 <[email protected]>
Co-authored-by: Calciumdibromid Bot <[email protected]>
Co-committed-by: Calciumdibromid Bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants