@nx/esbuild:esbuild with thirdParty: false fails filtering pnpm lock-file correctly in 19.x by not removing overrides

[atsjo]

Current Behavior

@nx/esbuild:esbuild with thirdParty: false and generatePackageJson: true filters the root lock file, but doesn't remove unrelated overrides section, causing pnpm install to fails when using the lock file.
filtered-pnpm-lock.yaml.19.2.0-rc.0.txt

Expected Behavior

pnpm install works with filtered (pruned) lock file, as for 18.3.4 (and 19.0.0-beta.6)
filtered-pnpm-lock.yaml.19.0.0-beta.6.txt

GitHub Repo

https://github.com/atsjo/nx-pnpm-repro

Steps to Reproduce

1. clone repo
2. pnpm install with corepack enabed, and then build backend project
3. change working dir to ./dist/backend and run: pnpm install --frozen-lockfile

You will then get this error:
ERR_PNPM_LOCKFILE_CONFIG_MISMATCH  Cannot proceed with the frozen installation. The current "overrides" configuration doesn't match the value found in the lockfile

Nx Report

Node           : 20.16.0
OS             : win32-x64
Native Target  : x86_64-windows
pnpm           : 9.6.0

nx                 : 19.5.6
@nx/js             : 19.5.6
@nx/jest           : 19.5.6
@nx/linter         : 19.5.6
@nx/eslint         : 19.5.6
@nx/workspace      : 19.5.6
@nx/angular        : 19.5.6
@nx/devkit         : 19.5.6
@nx/esbuild        : 19.5.6
@nx/eslint-plugin  : 19.5.6
@nrwl/tao          : 19.5.6
@nx/web            : 19.5.6
@nx/webpack        : 19.5.6
typescript         : 5.5.4
---------------------------------------
Registered Plugins:
@nx/eslint/plugin
@nx/jest/plugin
---------------------------------------
Community plugins:
angular-calendar : 0.31.1

Failure Logs

ERR_PNPM_LOCKFILE_CONFIG_MISMATCH  Cannot proceed with the frozen installation. The current "overrides" configuration doesn't match the value found in the lockfile

Package Manager Version

No response

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

The only difference between the working and failing lock-files uploaded above is:

settings:
  autoInstallPeers: true
  excludeLinksFromLockfile: false

overrides:
  '@invertase/firestore-stripe-payments>@firebase/app': '*'
  '@invertase/firestore-stripe-payments>@firebase/auth': '*'
  '@invertase/firestore-stripe-payments>@firebase/firestore': '*'

I would think the settings part should be kept, but the overrides part should either always be removed as a whole or filtered by included packages...

I reported a similar problem in #23256 which is fixed in #22906, but it still fails if using overrides (as I do)

The root lock file is pnpm-lock.yaml.txt and the filtered package json is filtered-package.json.txt

btw: the problems also occur when using pnpm v9, which uses a newer lock-file format...

[wewelll]

I upgraded to NX 19.2.0 and still have this issue.

[atsjo]

Since this is blocking me from deploying my firebase functions, unless downgrading to nx 18.3.4 or 19.0.0-beta.6, I worked out a fix using a firebase predeploy script for filtering the lock file. I have verified this is working with both pnpm v8 and v9 in nx 19.2.3.

I added a script to functions predeploy in firebase.json, running this (last param is path to my firebase build):

node tools/filter-overrides.mjs dist/apps/firebase

with tools/filter-overrides.mjs containing:

import fs from 'fs';
const pnpm_lock_file = `${process.argv[2]}/pnpm-lock.yaml`;
const content = fs.readFileSync(pnpm_lock_file, 'utf-8');
const newContent = content.replace(/\noverrides:\n[\s\S]*?\n\n/, '\n');
if (newContent !== content) fs.writeFileSync(pnpm_lock_file, newContent, 'utf-8');

I would really prefer this being fixed in nx, but if others have the same problem as me, this is a possible workaround, allowing you to also upgrade to pnpm v9.

[wewelll]

@Coly010 Is there is any initiative from the NX team to go forward with a fix ?

Like @atsjo This is completely blocking me from upgrading to NX 19, so I'm stuck on the version 18.

[atsjo]

The fix should be easy, as the info seems informational, and should just be removed (as it was previously)....

[atsjo]

A related/similar problem is reported in #26884

[Coly010]

Can someone provide a repo with a reproduction of the issue so that I can investigate further?

[atsjo]

I did provide the lock file showcasing the problem, and an older working version, in addition to my hack to fix this via a script... and if you try running pnpm install with the lock file you will get the error... I can try to make a stripped down repo later, but currently on vacation

[wewelll]

This error happens when we run pnpm install --frozen-lockfile in the dist folder

As a temporary fix, I've noticed that adding --fix-lockfile works. Not an ideal solution, because we have no guarantee that the lockfile will be respected

[github-actions]

This issue has been automatically marked as stale because no reproduction was provided within 7 days.
Please help us help you. Providing a repository exhibiting the issue helps us diagnose and fix the issue.
Any time that we spend reproducing this issue is time taken away from addressing this issue and other issues.
This issue will be closed in 21 days if a reproduction is not provided.
If a reproduction has been provided, please reply to keep it active.
Thanks for being a part of the Nx community! 🙏

[atsjo]

Repro:

1. clone this: https://github.com/atsjo/nx-pnpm-repro
2. pnpm install with corepack enabed, and then build backend project
3. change working dir to ./dist/backend and run: pnpm install --frozen-lockfile

You will then get this error:
ERR_PNPM_LOCKFILE_CONFIG_MISMATCH  Cannot proceed with the frozen installation. The current "overrides" configuration doesn't match the value found in the lockfile

It works if the --frozen-lockfile param is removed, but this param is used in CI environments, and when deploying firebase functions... As described earlier, I think the easy solution is to remove the overrides part of the pruned lock file....

[CaporalDead]

Still same problem here with the 19.6.5, getting a non blocking error complaining about the lockfile then fallbacking to the root lockfile to finish the build

[atsjo]

Still the same for me as well, glad I found a workaround, as this is obviously not prioritised... I provided a repro 1.5 months ago, and have also described a possible solution... still assigned to @Coly010, but seems to get no attention...

[Coly010]

@atsjo I’m sorry if you feel this issue has not received the due attention it deserves.

As you’ve mentioned, it is assigned to me and during the periods of time I dedicate to solving GitHub issues, I will be able to see it on my list.

I do my best to address and reach a solution for all the issues assigned to me in the time I have to address them.

We are all working hard to find the balance between delivering new features, handling maintenance tasks (such as supporting new versions of core dependencies) and fixing bugs.

If you feel confident that you have a solution to this issue, you’re more than welcome to open a PR.

Again, I apologise that this hasn’t been resolved as of yet.

[atsjo]

I totally understand the need for prioritizing the tasks affecting the most users, and this didn't seem to affect very many (and for my own part I found a workaround)...

But it is now fixed with the release of nx 19.7.0, as it now generates overrides in the generated package.json, matching the ones in the lock file. Maybe the fix is in this PR #27601, fixing the related issue I mentioned above #26884, although it doesn't mention the esbuild executor, and it's committed about 3 week ago (maybe it wasn't backported to the 19.6 branch?)

I am closing the issue as it now is fixed...

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.