[DOCS] npm install does not run scripts in directory it was ran in on npm v7+ #3123

Ssaely · 2021-04-22T15:24:53Z

Current Behavior:

npm install runs in a temp cache directory (somewhere in Users/user/.npm/_cacache/tmp/my_dep/) when installing dependencies on ~~macOS~~ npm 7.10.0. This causes errors when your pre/post-install scripts relies on checking for certain files/folders in the directory the command was ran in. However, on Windows, it does run them in the current directory.

Expected Behavior:

npm install should run the scripts in the directory that it was ran in, ~~like Windows~~ like it was on npm 6, or provide some way to retrieve this directory. Note that the INIT_CWD and other env variables do not point to this directory, but rather the cache directory.

Steps To Reproduce:

Use npm 7.10.0
Create a package with the following scripts section

  "scripts": {
    "preinstall": "node pre.js",
    "postinstall": "node post.js"
  }

Create a preinstall script, pre.js, with the contents

const fs = require('fs');
const path = require('path');

// On ~~macOS~~ npm v7+ this points to npm's temp cache directory
console.log(process.env.INIT_CWD);

// my_folder is a folder that exists in the directory of "npm install".
// This works on ~~Windows~~ npm v6, but it errors on ~~macOS~~ npm v7 because this folder does not exist in npm's cache directory
const files = fs.readdirSync(path.join(process.env.INIT_CWD, 'my_folder'));

Publish or upload this package to github
Create a test folder and a package.json. Add the dependency to the package.json
Run npm install.
Preinstall script fails

Environment:

macOS:

macOS (M1 chip) Big Sur 11.2.3
Node: 16.0.0
npm: 7.10.0

Windows:

Windows 10 Pro, version 20H2, build 19042.928
Node: 14.12.0
Npm: 6.14.13, it errors when using npm 7.10.0

The text was updated successfully, but these errors were encountered:

ljharb · 2021-04-22T15:26:28Z

You’re using npm 6 on windows, what happens if you use npm 7 on windows?

Ssaely · 2021-04-22T15:41:00Z

Aha, this seems to be an NPM issue somewhere in v7 as it also points to the temp cache directory and fails using npm 7.10.0 on Windows. I've updated the original issue to reflect this.

ljharb · 2021-04-22T16:00:44Z

In that case, it might be an unintended regression in npm 7.

Ssaely · 2021-04-22T16:17:01Z

I just downgraded npm to 6.14.13 on macOS and it worked, so it does seem to be a regression in npm.

Apollon77 · 2021-04-22T20:38:59Z

Especially bad because npm 7.10.0 now is delivered by default "with" nodejs 16

ljharb · 2021-04-23T01:28:56Z

@Apollon77 npm 7 is there by default in node 15 as well, and since node 16 isn't LTS, that means it's no worse than it was prior to 16 being released.

Apollon77 · 2021-04-23T04:37:28Z

Yea and no ;-)) nodejs 15 is a development version by definition in my eyes, so more acceptable because that Version habe a mich lower adoption rate.

In the eyes of users NodeJs 16 will "become LTS", so they have an other view on this version and start adopt to that version faster. But yes you are right, we formally have some time until nodejs 16 goes LTS official.

isaacs · 2021-04-27T18:02:48Z

How are you adding the dependency in step 5? As a git dep, tarball, published semver version/range, etc?

Ssaely · 2021-05-01T16:59:01Z

@isaacs Sorry for the delay, I am adding the dependency by using a git dep.

Apollon77 · 2021-05-01T22:05:18Z

I have a very esy example:

root@io-test5:~/test# npm i ioBroker/ioBroker.js-controller
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! code 1
npm ERR! git dep preparation failed
npm ERR! command /usr/bin/node /usr/lib/node_modules/npm/bin/npm-cli.js install --force --cache=/root/.npm --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit
npm ERR! > [email protected] preinstall
npm ERR! > node lib/preinstallCheck.js
npm ERR! 
npm ERR! NPM version: 7.10.0
npm ERR! 
npm ERR! > [email protected] install
npm ERR! > node iobroker.js setup first
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! node:internal/modules/cjs/loader:943
npm ERR!   throw err;
npm ERR!   ^
npm ERR! 
npm ERR! Error: Cannot find module '../../conf/git-clone-999c2b74-dist.json'
npm ERR! Require stack:
npm ERR! - /root/.npm/_cacache/tmp/git-clone-999c2b74/lib/setup/setupSetup.js
npm ERR! - /root/.npm/_cacache/tmp/git-clone-999c2b74/lib/setup.js
npm ERR! - /root/.npm/_cacache/tmp/git-clone-999c2b74/iobroker.js
npm ERR!     at Function.Module._resolveFilename (node:internal/modules/cjs/loader:940:15)
npm ERR!     at Function.Module._load (node:internal/modules/cjs/loader:773:27)
npm ERR!     at Module.require (node:internal/modules/cjs/loader:1012:19)
npm ERR!     at require (node:internal/modules/cjs/helpers:93:18)
npm ERR!     at Setup.setup (/root/.npm/_cacache/tmp/git-clone-999c2b74/lib/setup/setupSetup.js:844:26)
npm ERR!     at processCommand (/root/.npm/_cacache/tmp/git-clone-999c2b74/lib/setup.js:500:23)
npm ERR!     at Object.module.exports.execute (/root/.npm/_cacache/tmp/git-clone-999c2b74/lib/setup.js:2904:5)
npm ERR!     at Object.<anonymous> (/root/.npm/_cacache/tmp/git-clone-999c2b74/iobroker.js:1:24)
npm ERR!     at Module._compile (node:internal/modules/cjs/loader:1108:14)
npm ERR!     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1137:10) {
npm ERR!   code: 'MODULE_NOT_FOUND',
npm ERR!   requireStack: [
npm ERR!     '/root/.npm/_cacache/tmp/git-clone-999c2b74/lib/setup/setupSetup.js',
npm ERR!     '/root/.npm/_cacache/tmp/git-clone-999c2b74/lib/setup.js',
npm ERR!     '/root/.npm/_cacache/tmp/git-clone-999c2b74/iobroker.js'
npm ERR!   ]
npm ERR! }
npm ERR! npm ERR! code 1
npm ERR! npm ERR! path /root/.npm/_cacache/tmp/git-clone-999c2b74
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c node iobroker.js setup first
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR!     /root/.npm/_logs/2021-05-01T22_04_09_810Z-debug.log

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-05-01T22_04_10_872Z-debug.log

This is trying to execute a "setup command". This all wors file if installed from npm (npm i iobroker.js-controller), but fails when istalling the same from github

darcyclarke · 2021-06-04T18:54:45Z

Action: Investigate further into whether this is a regression

Apollon77 · 2021-07-23T11:35:02Z

is there any status updte on this one?

nmss · 2021-08-09T14:30:12Z

@darcyclarke if this is classified as "not a regression", is there another recommended way for pre/post-install scripts to know the path to the directory that it was run in ?

by "Investigate further into whether this is a regression" do you want more example logs, or are you thinking about something else ?

nmss · 2021-09-07T17:32:07Z

This regression seems to have been introduced by npm/config#12 while fixing another related regression

it went from not enough INIT_CWD reset, to too much INIT_CWD reset

Apollon77 · 2021-10-20T06:59:52Z

nodejs 16 becomes LTS very soon and this issue is still not fixed :-( Any ETA on it?

isaacs · 2021-10-20T15:28:49Z

The errors I'm seeing installing ioBroker from git currently (ie, using the latest npm v8) do not appear to be related to this issue. It still isn't working, but for different reasons, which appear to be related to the way in which its many internal link dependencies are being built.

There's still something wrong here, and it's a bug we'll fix whether node 16 is LTS or not, but it's not this bug. (I'll post a new issue for that right now.) Also, if we're handling INIT_CWD improperly, we'll fix that as well, LTS or not. It would not be a semver-major change, as far as I can tell.

If anyone has a repro case they can share that shows improper INIT_CWD handling, please share it, and we'll certainly reopen this.

Apollon77 · 2021-10-20T15:36:44Z

@isaacs thank you for checking. In fact only projects (and not only about iobroker) are affected that have a postinstall script which is executed ... because these projects with get the above error when being installed from GitHub! It is only in this case ... but this is still a "ocmmon way" to deliver dev-versions to users for certain fixes without the need to publish prereleases to npm for any single change

isaacs · 2021-10-20T15:40:26Z

Current issue affecting ioBroker: #3918 (maybe related? but seems like it isn't, since the dep gets placed in node_modules, but then breaks later, when building all its local link dependencies.)

@Apollon77 Do you have a repo that demonstrates the issue? If it is just a matter of when and how consistently we set INIT_CWD, as @nmss suggests, then it should be straightforward to comb through it and fix, once we have a repro case.

Apollon77 · 2021-10-21T16:28:42Z

try npm i https://github.com/ioBroker/ioBroker.js-controller#3.3.x

npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! code 1
npm ERR! git dep preparation failed
npm ERR! command /usr/bin/node /usr/lib/node_modules/npm/bin/npm-cli.js install --force --cache=/root/.npm --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit --include=dev --include=peer --include=optional --no-package-lock-only --no-dry-run
npm ERR! > [email protected] preinstall
npm ERR! > node lib/preinstallCheck.js
npm ERR! 
npm ERR! NPM version: 8.1.0
npm ERR! 
npm ERR! > [email protected] install
npm ERR! > node iobroker.js setup first
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN old lockfile 
npm ERR! npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm ERR! npm WARN old lockfile so supplemental metadata must be fetched from the registry.
npm ERR! npm WARN old lockfile 
npm ERR! npm WARN old lockfile This is a one-time fix-up, please be patient...
npm ERR! npm WARN old lockfile 
npm ERR! npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm ERR! npm WARN deprecated [email protected]: this library is no longer supported
npm ERR! npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm ERR! npm WARN deprecated [email protected]: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm ERR! npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm ERR! npm WARN deprecated [email protected]: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm ERR! npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! npm WARN deprecated [email protected]: This module is no longer maintained, try this instead:
npm ERR! npm WARN deprecated   npm i nyc
npm ERR! npm WARN deprecated Visit https://istanbul.js.org/integrations for other alternatives.
npm ERR! node:internal/modules/cjs/loader:936
npm ERR!   throw err;
npm ERR!   ^
npm ERR! 
npm ERR! Error: Cannot find module '../../conf/git-clonegkqecm-dist.json'
npm ERR! Require stack:
npm ERR! - /root/.npm/_cacache/tmp/git-cloneGKQeCM/lib/setup/setupSetup.js
npm ERR! - /root/.npm/_cacache/tmp/git-cloneGKQeCM/lib/setup.js
npm ERR! - /root/.npm/_cacache/tmp/git-cloneGKQeCM/iobroker.js
npm ERR!     at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
npm ERR!     at Function.Module._load (node:internal/modules/cjs/loader:778:27)
npm ERR!     at Module.require (node:internal/modules/cjs/loader:1005:19)
npm ERR!     at require (node:internal/modules/cjs/helpers:102:18)
npm ERR!     at Setup.setup (/root/.npm/_cacache/tmp/git-cloneGKQeCM/lib/setup/setupSetup.js:845:26)
npm ERR!     at processCommand (/root/.npm/_cacache/tmp/git-cloneGKQeCM/lib/setup.js:487:23)
npm ERR!     at Object.module.exports.execute (/root/.npm/_cacache/tmp/git-cloneGKQeCM/lib/setup.js:2910:5)
npm ERR!     at Object.<anonymous> (/root/.npm/_cacache/tmp/git-cloneGKQeCM/iobroker.js:1:24)
npm ERR!     at Module._compile (node:internal/modules/cjs/loader:1101:14)
npm ERR!     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) {
npm ERR!   code: 'MODULE_NOT_FOUND',
npm ERR!   requireStack: [
npm ERR!     '/root/.npm/_cacache/tmp/git-cloneGKQeCM/lib/setup/setupSetup.js',
npm ERR!     '/root/.npm/_cacache/tmp/git-cloneGKQeCM/lib/setup.js',
npm ERR!     '/root/.npm/_cacache/tmp/git-cloneGKQeCM/iobroker.js'
npm ERR!   ]
npm ERR! }
npm ERR! npm ERR! code 1
npm ERR! npm ERR! path /root/.npm/_cacache/tmp/git-cloneGKQeCM
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c node iobroker.js setup first
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR!     /root/.npm/_logs/2021-10-21T16_26_27_792Z-debug.log

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-10-21T16_26_28_677Z-debug.log

PS: This is the branch of the current stable version of the project ... In future it is a monorepo and so it is not thaat relevant, but I think it would be just a matter of time until someone else uses a "install" script

isaacs · 2021-10-21T20:06:52Z

@Apollon77 I get the same error just running npm install in the project on that branch.

> [email protected] preinstall
> node lib/preinstallCheck.js

NPM version: 8.1.0

> [email protected] install
> node iobroker.js setup first

node:internal/modules/cjs/loader:930
  throw err;
  ^

Error: Cannot find module '../../conf/iobroker-3-dist.json'
Require stack:
- /Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/lib/setup/setupSetup.js
- /Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/lib/setup.js
- /Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/iobroker.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:15)
    at Function.Module._load (node:internal/modules/cjs/loader:772:27)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:93:18)
    at Setup.setup (/Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/lib/setup/setupSetup.js:846:26)
    at processCommand (/Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/lib/setup.js:487:23)
    at Object.module.exports.execute (/Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/lib/setup.js:2910:5)
    at Object.<anonymous> (/Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/iobroker.js:1:24)
    at Module._compile (node:internal/modules/cjs/loader:1095:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1124:10) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/lib/setup/setupSetup.js',
    '/Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/lib/setup.js',
    '/Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/iobroker.js'
  ]
}

It looks like the issue is that it's inferring its module name from the project folder basename, so if that doesn't match, it's going to always fail, because the conf file isn't there. In my case, it's expecting to find /Users/isaacs/dev/npm/cli/fixtures/gh-3123/iobroker-3/conf/iobroker-3-dist.json.

I'd recommend setting tools.appName more deterministically, especially since it can only be one of two possible things.

Apollon77 · 2021-10-21T20:21:50Z

Yes thats basically true ... in fact it was working all the time till npm 7 :-) But honestly ... ok for me. I tested around other cases with install scripts and they all work with npm 8.1.0 ... so for me it is ok - also because we will change to monorepo and so this exact topic is no longer relevant. Thank you for your support.

I leave it up to you if you close here or leave open to maybe update documentations because the expectation from before that the "dirname to get is the one where the npm command was executed" is now no longer true and it might be a "tmp dir somewhere" could be worth documenting somewhere

wraithgar · 2022-02-18T00:57:32Z

PR that copies $INIT_CWD explanation from npm run-script to using npm - scripts here #4436

jtiee · 2023-03-27T16:06:17Z

I've run into this issue with NPM 9.5.1 when installing a dependency from a GitHub repo that is large enough that the pacote module fetches a tarball of the repo instead of performing a clone. When this happens, the tarball is unpacked and NPM processing continues from the unpacked folder. That processing re-initilializes the INIT_CWD env var, losing access to the project folder.

I'm not sure what the best solution would be, but it seems like paying attention to whether INIT_CWD is already defined prior to defining it could make the situation better with very little code changes.

wraithgar · 2023-03-27T16:21:35Z

@jtiee if you have a reproducible case please open a new issue

Ssaely added Bug thing that needs fixing Needs Triage needs review for next steps Release 7.x work is associated with a specific npm 7 release labels Apr 22, 2021

Ssaely changed the title ~~[BUG] npm install does not run scripts in directory it was ran in on macOS~~ [BUG] npm install does not run scripts in directory it was ran in on npm v7+ Apr 22, 2021

darcyclarke added Priority 2 secondary priority issue and removed Needs Triage needs review for next steps labels Jun 4, 2021

mxschmitt mentioned this issue Jul 29, 2021

[Bug]: Fails to run in GitLab CI / mcr.microsoft.com/playwright:focal Docker image microsoft/playwright#7867

Closed

darcyclarke added Priority 1 high priority issue and removed Priority 2 secondary priority issue labels Aug 9, 2021

darcyclarke modified the milestones: OSS - Sprint 34, OSS - Sprint 35 Aug 9, 2021

darcyclarke removed this from the OSS - Sprint 35 milestone Aug 23, 2021

Apollon77 mentioned this issue Aug 27, 2021

npm hangs by installing ioBroker adapters MQTT/MQTT-client #3689

Closed

Krinkle mentioned this issue Oct 15, 2021

is:issue is:open When you install Puppeteer, it does not download a recent version of Chromium puppeteer/puppeteer#6586

Closed

isaacs mentioned this issue Oct 20, 2021

Link dependencies being mishandled when installing from git #3918

Closed

wraithgar added Documentation documentation related issue Priority 2 secondary priority issue and removed Priority 1 high priority issue labels Feb 17, 2022

wraithgar changed the title ~~[BUG] npm install does not run scripts in directory it was ran in on npm v7+~~ [DOCS] npm install does not run scripts in directory it was ran in on npm v7+ Feb 17, 2022

wraithgar mentioned this issue Feb 18, 2022

docs: explain $INIT_CWD on using scripts page #4436

Merged

wraithgar added Priority 1 high priority issue and removed Priority 2 secondary priority issue labels Feb 18, 2022

wraithgar self-assigned this Feb 22, 2022

wraithgar closed this as completed Feb 22, 2022

jtiee mentioned this issue May 2, 2023

[BUG] npm install does not set $INIT_CWD correctly for some GitHub packages #6417

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCS] npm install does not run scripts in directory it was ran in on npm v7+ #3123

[DOCS] npm install does not run scripts in directory it was ran in on npm v7+ #3123

Ssaely commented Apr 22, 2021 •

edited by darcyclarke

Loading

ljharb commented Apr 22, 2021

Ssaely commented Apr 22, 2021

ljharb commented Apr 22, 2021

Ssaely commented Apr 22, 2021

Apollon77 commented Apr 22, 2021

ljharb commented Apr 23, 2021

Apollon77 commented Apr 23, 2021

isaacs commented Apr 27, 2021

Ssaely commented May 1, 2021

Apollon77 commented May 1, 2021

darcyclarke commented Jun 4, 2021

Apollon77 commented Jul 23, 2021

nmss commented Aug 9, 2021 •

edited

Loading

nmss commented Sep 7, 2021

Apollon77 commented Oct 20, 2021

isaacs commented Oct 20, 2021

Apollon77 commented Oct 20, 2021

isaacs commented Oct 20, 2021

Apollon77 commented Oct 21, 2021 •

edited

Loading

isaacs commented Oct 21, 2021

Apollon77 commented Oct 21, 2021

wraithgar commented Feb 18, 2022

jtiee commented Mar 27, 2023

wraithgar commented Mar 27, 2023

[DOCS] npm install does not run scripts in directory it was ran in on npm v7+ #3123

[DOCS] npm install does not run scripts in directory it was ran in on npm v7+ #3123

Comments

Ssaely commented Apr 22, 2021 • edited by darcyclarke Loading

Current Behavior:

Expected Behavior:

Steps To Reproduce:

Environment:

ljharb commented Apr 22, 2021

Ssaely commented Apr 22, 2021

ljharb commented Apr 22, 2021

Ssaely commented Apr 22, 2021

Apollon77 commented Apr 22, 2021

ljharb commented Apr 23, 2021

Apollon77 commented Apr 23, 2021

isaacs commented Apr 27, 2021

Ssaely commented May 1, 2021

Apollon77 commented May 1, 2021

darcyclarke commented Jun 4, 2021

Apollon77 commented Jul 23, 2021

nmss commented Aug 9, 2021 • edited Loading

nmss commented Sep 7, 2021

Apollon77 commented Oct 20, 2021

isaacs commented Oct 20, 2021

Apollon77 commented Oct 20, 2021

isaacs commented Oct 20, 2021

Apollon77 commented Oct 21, 2021 • edited Loading

isaacs commented Oct 21, 2021

Apollon77 commented Oct 21, 2021

wraithgar commented Feb 18, 2022

jtiee commented Mar 27, 2023

wraithgar commented Mar 27, 2023

Ssaely commented Apr 22, 2021 •

edited by darcyclarke

Loading

nmss commented Aug 9, 2021 •

edited

Loading

Apollon77 commented Oct 21, 2021 •

edited

Loading