[BUG] npm install attempts to fetch already bundled indirect dependency from registry

[salvadorj]

It appears that npm install sometimes attempts to fetch indirect dependencies from the registry even though they have already been bundled with the package being installed. This causes issues if these dependencies are only available in an internal registry during install/pack-time but not available on the registry where the package is published to and eventually installed from. My understanding is that dependencies listed in bundledDependencies also include indirect dependencies.

I can only reproduce this on Windows with npm 7. The package installs just fine with npm 6 on Windows and with npm 7 on macOS.

I've created a minimal repro from my original package and used a file: dependency to avoid the need for a custom registry. The error looks a little different than with my original package since it fails with an error saying that it cannot find the package on the registry but I think it's the same cause. This is what my repro package looks like:

package.json:

{
  "name": "foo",
  "version": "1.0.0",
  "dependencies": {
    "bar": "file:bar-1.0.0.tgz",
    "tmp": "0.0.33"
  },
  "bundledDependencies": [
    "bar",
    "tmp"
  ]
}

where bar-1.0.0.tgz contains a single file bar/package.json:

{
  "name": "bar",
  "version": "1.0.0",
  "dependencies": {
    "tmp-promise": "2.0.2"
  }
}

Here's a zip file with the above files: foo.zip

There are a couple of issues already that appear similar but I'm not entirely sure they're the same:

• [BUG] npm tries to download bundled dependencies #1817 (the issue I'm seeing only reproduced on npm 7)
• [BUG] NPM tries to download transitive bundledDependencies #2143

Current Behavior:

Install fails on [email protected] as it attempts to fetch already bundled indirect dependency rimraf from the dummy registry https://0.0.0.0 even though it should already be bundled with the package:

PS C:\Users\salva\Desktop\repro\foo> node --version
v15.9.0
PS C:\Users\salva\Desktop\repro\foo> npm --version
7.5.6
PS C:\Users\salva\Desktop\repro\foo> npm list -g
C:\Users\salva\AppData\Roaming\npm
`-- [email protected]

PS C:\Users\salva\Desktop\repro\foo> npm install

added 17 packages, and audited 18 packages in 1s

1 package is looking for funding
  run `npm fund` for details

found 0 vulnerabilities
PS C:\Users\salva\Desktop\repro\foo> npm pack
npm notice
npm notice package: [email protected]
npm notice === Tarball Contents ===
npm notice 228B bar-1.0.0.tgz
npm notice 173B package.json
npm notice === Bundled Dependencies ===
npm notice balanced-match
npm notice brace-expansion
npm notice concat-map
npm notice fs.realpath
npm notice glob
npm notice inflight
npm notice inherits
npm notice minimatch
npm notice once
npm notice os-tmpdir
npm notice path-is-absolute
npm notice rimraf
npm notice tmp-promise
npm notice tmp
npm notice wrappy
npm notice bar
npm notice === Tarball Details ===
npm notice name:          foo
npm notice version:       1.0.0
npm notice filename:      foo-1.0.0.tgz
npm notice package size:  62.2 kB
npm notice unpacked size: 238.5 kB
npm notice shasum:        f5b9dea47ae5d48a40f7079d99728e905a48e808
npm notice integrity:     sha512-P9Yat2HlAmr2b[...]1/ACmYuDlxUZw==
npm notice bundled deps:  16
npm notice bundled files: 0
npm notice own files:     81
npm notice total files:   81
npm notice
foo-1.0.0.tgz
PS C:\Users\salva\Desktop\repro\foo> npm install -g --registry https://0.0.0.0 foo-1.0.0.tgz
npm ERR! code ECONNREFUSED
npm ERR! syscall connect
npm ERR! errno ECONNREFUSED
npm ERR! FetchError: request to https://0.0.0.0/rimraf failed, reason: connect ECONNREFUSED 0.0.0.0:443
npm ERR!     at ClientRequest.<anonymous> (C:\Users\salva\AppData\Roaming\npm\node_modules\npm\node_modules\minipass-fetch\lib\index.js:97:14)
npm ERR!     at ClientRequest.emit (node:events:378:20)
npm ERR!     at TLSSocket.socketErrorListener (node:_http_client:462:9)
npm ERR!     at TLSSocket.emit (node:events:390:22)
npm ERR!     at emitErrorNT (node:internal/streams/destroy:188:8)
npm ERR!     at emitErrorCloseNT (node:internal/streams/destroy:153:3)
npm ERR!     at processTicksAndRejections (node:internal/process/task_queues:81:21)
npm ERR!  FetchError: request to https://0.0.0.0/rimraf failed, reason: connect ECONNREFUSED 0.0.0.0:443
npm ERR!     at ClientRequest.<anonymous> (C:\Users\salva\AppData\Roaming\npm\node_modules\npm\node_modules\minipass-fetch\lib\index.js:97:14)
npm ERR!     at ClientRequest.emit (node:events:378:20)
npm ERR!     at TLSSocket.socketErrorListener (node:_http_client:462:9)
npm ERR!     at TLSSocket.emit (node:events:390:22)
npm ERR!     at emitErrorNT (node:internal/streams/destroy:188:8)
npm ERR!     at emitErrorCloseNT (node:internal/streams/destroy:153:3)
npm ERR!     at processTicksAndRejections (node:internal/process/task_queues:81:21) {
npm ERR!   code: 'ECONNREFUSED',
npm ERR!   errno: 'ECONNREFUSED',
npm ERR!   syscall: 'connect',
npm ERR!   address: '0.0.0.0',
npm ERR!   port: 443,
npm ERR!   type: 'system'
npm ERR! }
npm ERR!
npm ERR! If you are behind a proxy, please make sure that the
npm ERR! 'proxy' config is set properly.  See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\salva\AppData\Local\npm-cache\_logs\2021-02-22T21_35_06_434Z-debug.log

2021-02-22T21_35_06_434Z-debug.log

Expected Behavior:

Install succeeds on [email protected]:

PS C:\Users\salva\Desktop\repro\foo> node --version
v15.9.0
PS C:\Users\salva\Desktop\repro\foo> npm --version
6.14.11
PS C:\Users\salva\Desktop\repro\foo> npm list -g --depth=0
C:\Users\salva\AppData\Roaming\npm
`-- [email protected]

PS C:\Users\salva\Desktop\repro\foo> npm install
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No description
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

added 17 packages from 6 contributors and audited 17 packages in 1.766s

1 package is looking for funding
  run `npm fund` for details

found 0 vulnerabilities

PS C:\Users\salva\Desktop\repro\foo> npm pack
npm notice
npm notice package: [email protected]
npm notice === Tarball Contents ===
npm notice 173B package.json
npm notice 228B bar-1.0.0.tgz
npm notice === Bundled Dependencies ===
npm notice tmp
npm notice bar
npm notice === Tarball Details ===
npm notice name:          foo
npm notice version:       1.0.0
npm notice filename:      foo-1.0.0.tgz
npm notice package size:  65.3 kB
npm notice unpacked size: 254.1 kB
npm notice shasum:        238cc607fa070001da971e27992fb1da963f1f63
npm notice integrity:     sha512-YYaLB9+Lv5qmo[...]NvWxcaPI3fHPQ==
npm notice bundled deps:  2
npm notice bundled files: 79
npm notice own files:     2
npm notice total files:   81
npm notice
foo-1.0.0.tgz
PS C:\Users\salva\Desktop\repro\foo> npm install -g --registry https://0.0.0.0 foo-1.0.0.tgz
+ [email protected]
added 18 packages from 6 contributors in 0.524s

Steps To Reproduce:

1. Download foo.zip
2. Extract foo.zip
3. cd foo
4. npm install
5. npm pack
6. npm install -g --registry https://0.0.0.0 foo-1.0.0.tgz
7. Observe error (after timeout) when npm attempts to fetch indirect dependency rimraf from the dummy registry https://0.0.0.0

Environment:

• OS: Windows 10.0.19042 Build 19042
• Node: 15.9.0
• npm: 7.5.6

[salvadorj]

I don't know much about the internals of npm but I did a bit of debugging and the difference in behavior between macOS and Windows lead me to this for-loop that iterates over the actual tree in order to find missing edges:

cli/node_modules/@npmcli/arborist/lib/arborist/load-actual.js

Lines 318 to 329 in 0c6bb2b

	async [_findMissingEdges] () {
	// try to resolve any missing edges by walking up the directory tree,
	// checking for the package in each node_modules folder. stop at the
	// root directory.
	// The tricky move here is that we load a "dummy" node for the folder
	// containing the node_modules folder, so that it can be assigned as
	// the fsParent. It's a bad idea to *actually* load that full node,
	// because people sometimes develop in ~/projects/node_modules/...
	// so we'd end up loading a massive tree with lots of unrelated junk.
	const nmContents = new Map()
	const tree = this[_actualTree]
	for (const node of tree.inventory.values()) {

The order of the nodes in the tree are different on macOS and Windows and replacing tree.inventory.values() with [...tree.inventory.values()] actually fixes the problem on Windows. Maybe it's just a red herring but I suspect there might be an issue where the order of the nodes affect the behavior of this _findMissingEdges function because the tree is modified while it's being iterated over.

[salvadorj]

Oh, and here's the output of npm list -g for [email protected] after a successful install:

+-- [email protected]
| +-- [email protected]
| | `-- [email protected]
| |   `-- [email protected]
| |     `-- [email protected]
| |       `-- [email protected]
| |         +-- [email protected]
| |         +-- [email protected]
| |         | +-- [email protected] deduped
| |         | `-- [email protected]
| |         +-- [email protected]
| |         +-- [email protected]
| |         | `-- [email protected]
| |         |   +-- [email protected]
| |         |   `-- [email protected]
| |         +-- [email protected]
| |         | `-- [email protected] deduped
| |         `-- [email protected]
| `-- [email protected]
|   `-- [email protected]
`-- [email protected]
...

[mihai-unity]

Any updates on this bug? It seems like there's no update since end of August and we're sitting at the edge of our chairs waiting for a bugfix since it's blocking to upgrade to a much more recent version.