[BUG] Overrides are not updating after running npm install

[craigjbass]

Same, I was able to reproduce this issue

Originally posted by @surbhirverma in #4232 (comment)

This closed issue is not resolved

[ForbiddenEra]

I'm seeing this issue on 8.5.5 currently; removing node_modules and package-lock.json and doing npm install does install the overridden dependencies, however, I'm also seeing a different, potentially related issue here.

When doing npm ls --depth=10, I get errors:
(any lines with ... have been removed for brevity)

...
├── [email protected]
...
├─┬ [email protected]
...
│ ├─┬ [email protected]
...
│ │ ├── [email protected] deduped invalid: "~0.4.1" from node_modules/engine.io
...
│ │ └── [email protected] deduped invalid: "~8.2.3" from node_modules/engine.io
...
├─┬ [email protected]
│ ├── UNMET OPTIONAL DEPENDENCY bufferutil@^4.0.1
│ └── UNMET OPTIONAL DEPENDENCY utf-8-validate@^5.0.2

npm ERR! code ELSPROBLEMS
npm ERR! invalid: [email protected] ~/example/node_modules/cookie
npm ERR! invalid: [email protected] ~/example/node_modules/ws

My package.json contains:

        "dependencies": {
...
                "cookie": "^0.5.0",
                "socket.io": "^4.5.2",
...
                "ws": "^8.8.1",
        },
        "overrides": {
                "ws": "$ws",
                "cookie": "$cookie"
        },
...

I've also tried:

        "overrides": {
                "socket.io": {
                       "ws": "$ws",
                       "cookie": "$cookie"
                }
        }

and,

        "overrides": {
                "socket.io": {
                        "engine.io": {
                                "ws": "$ws",
                                "cookie": "$cookie"
                        }
                },
        },

and even,

        "overrides": {
                "socket.io": {
                        "engine.io": {
                                "ws": "$ws",
                                "cookie": "$cookie"
                        }
                },
                "ws": "$ws",
                "cookie": "$cookie"
        },

with no difference.

Another oddity I noticed - if I edit node_modules/engine.io/package.json and make NO changes (so, basically just touching the file / updating it's last modified) the error disappears without running anything between that and npm ls --depth=10. However, if I subsequently run npm install and then do npm ls --depth=10, the error comes back.

If I change node_modules/engine.io/package.json to use ^ instead of ~ for the packages and remove package-lock.json and run install, nothing changes and I still get the error - setting ^ should allow the replacement versions (8.8.1 for ws and 0.5.0 for cookie) to be allowed for that package. I suppose if I also removed the node_modules along with package-lock.json and then did npm install, that it might work, however then the changes made to that file would obviously be gone.

Potentially even worse than this (eg. other than the error on npm ls and the pain of having to force it to install the overrides by removing node_modules and package-lock.json) and possibly more frustrating/damaging is that running a npm dedupe actually replaces the overridden versions with the versions requested by the package instead! Basically npm dedupe in this situation is un-deduping! (But then, of course, the npm ls --depth=10 error is gone!)

# npm dedupe

added 2 packages, ...
# npm ls --depth=10
├─┬ [email protected]
...
│ └── [email protected] deduped
├── [email protected]
...
├─┬ [email protected]
...
│ ├─┬ [email protected]
...
│ │ ├── [email protected]
...
│ │ └─┬ [email protected]
...
├─┬ [email protected]
...

I may try updating (however this particular dev machine is on Ubuntu 18.04.6 LTS which has npm 8.5.5 as it's latest packaged version) or attempting to reproduce on another machine with a later npm version but it's not a show-stopper right now so not a huge priority (unless someone can confirm at least the dedupe and npm ls --depth=10 issues are fixed in a particular version, the not updating/installing the overridden version after adding an override in package.json is annoying but doesn't show errors or replace with the wrong version like the other issues so it's less of a concern).

[samuliasmala]

I can confirm overrides are not working in the currently latest [email protected]. This can be reproduced easily with the following package.json:

{
  "name": "test",
  "version": "1.0.0",
  "engines": {
    "npm": ">=8.3.0"
  },
  "dependencies": {
    "json-server": "^0.17.0"
  },
  "overrides": {
    "json-server": {
      "package-json": "7.0.0"
    }
  }
}

1. npm install in the folder containing only the above package.json --> 0 vulnerabilities
2. Subsequent npm install right after the previous (so node_modules and package-lock.json exists) --> 5 vulnerabilities
3. npm update --> 0 vulnerabilities
4. rm -rf node_modules/ && npm install --> 5 vulnerabilities
5. rm package-lock.json && npm install --> 5 vulnerabilities
6. rm -rf node_modules/ && rm package-lock.json && npm install --> 0 vulnerabilities

From the above it can be concluded that the overrides property is only honored when running npm install first time (i.e. without package-lock.json and node_modules present) and when running npm update.

[thomas-p-wilson]

I was able to fully reproduce @samuliasmala's above repro.

Additionally, if I'm not mistaken, it seems like there's no hope for descendants of @-scoped packages. I've certainly been having no luck. As I'm working on proprietary stuff at the moment, I don't have a repro, but I will work to produce one shortly.

[himanshuapril1]

Folks any solution (or hack) for this? This is really blocking our development.

[thomas-p-wilson]

My only workable solution so far has been to utilize the synp package to export my package-lock.json file to yarn.lock and then use yarn resolutions to pin everything, and then use synp to port back to package-lock. It's gross, it's fraught with peril, but so far it's the only semi-working way I've found...

[atienzajazz]

Also encountered this issue. Converting my package-lock.json to yarn.lock is a bit weird, cause I need to yarn install due to synp and yarn import reporting that my lockFile is corrupted even if npm ls -a is not reporting an error.

My current workable solution is to download the tgz file of the package, then revert the version.
My package.json contains:

{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "src/index.js",
  "dependencies": {
    "sails-hook-grunt": "^5.0.0"
  },
  "overrides": {
    "@sailshq/lodash": "3.10.4",
    "grunt": "1.5.3"
  }
}

1. npm install
2. download the packed file or tgz file from npm
3. reference the dependency on package.json

...
  "dependencies": {
    "sails-hook-grunt": "file:/path/to/file/sails-hook-grunt.tgz"
  }
...

4. npm install twice or until package-lock.json does not change
5. revert the dependency path and npm install, the change will only be on the location of the dependency on the package-lock.json after installing

...
  "dependencies": {
    "sails-hook-grunt": "^5.0.0"
  }
...

If I'm not mistaken, we might be able to pinpoint the issue here.
Considering if we downloaded the dependency package and used it on the project, it is overriding properly.
Hope this helps! :)

Environment

• npm: >=9.5.1
• Node.js: 18.16.0
• OS Name: macOS 12

[Mhbuur]

Also having problems with overrides here - no matter what I try (clean install, delete package-lock, remove modules etc), the versions in the newly created package-lock file remain the same, and not the versions I choose for overrides.

On a side note: I am amazed that this is not a bigger issue for more ppl - how do you manage third party package security issues? Is there perhaps some alternative method to using overrides that allow you to override deep third party dependencies?