Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generatePackageJson creates incompatible package.json and package-lock.json in certain scenarios. #17653

Closed
1 of 4 tasks
jonapgar-groupby opened this issue Jun 16, 2023 · 21 comments

Comments

@jonapgar-groupby
Copy link

jonapgar-groupby commented Jun 16, 2023

Current Behavior

When using generatePackageJson: true in the webpack plugin, nested dependencies can be missing from generated package-lock.json, causing npm ci to fail.

Consider the following package-lock.json

    "node_modules/auth0": {
      "version": "3.4.0",
      "resolved": "https://registry.npmjs.org/auth0/-/auth0-3.4.0.tgz",
      "integrity": "sha512-qaRVZfykFhROZlKa6i/G7Tlo4Df6GXdtMdvO41XuJ2XlJ/AMUAxL0u/FJJyII/HWwosVtypD92nHB2gqRwIbYg==",
      "dependencies": {
        "axios": "^0.27.2",
        "form-data": "^3.0.1", // <==== requires an older version of form-data
        "jsonwebtoken": "^9.0.0",
        "jwks-rsa": "^3.0.1",
        "lru-memoizer": "^2.1.4",
        "rest-facade": "^1.16.3",
        "retry": "^0.13.1",
        "uuid": "^9.0.0"
      },
      "engines": {
        "node": ">=14"
      }
    },
    "node_modules/auth0/node_modules/axios": {
      "version": "0.27.2",
      "resolved": "https://registry.npmjs.org/axios/-/axios-0.27.2.tgz",
      "integrity": "sha512-t+yRIyySRTp/wua5xEr+z1q60QmLq8ABsS5O9Me1AsE5dfKqgnCFzwiCZZ/cGNd1lq4/7akDWMxdhVlucjmnOQ==",
      "dependencies": {
        "follow-redirects": "^1.14.9",
        "form-data": "^4.0.0" <=== but this one requires a newer one
      }
    },
    
    //---------------- the below entry is missing (pruned) when using generatePackageJson, but will be added if you run `npm install` manually in the build folder
    
    "node_modules/auth0/node_modules/axios/node_modules/form-data": {
      "version": "4.0.0",
      "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.0.tgz",
      "integrity": "sha512-ETEklSGi5t0QMZuiXoA/Q6vcnxcLQP5vdugSpuAyi6SVGi2clPPp+xgEhuMaHC+zGgn31Kd235W35f7Hykkaww==",
      "dependencies": {
        "asynckit": "^0.4.0",
        "combined-stream": "^1.0.8",
        "mime-types": "^2.1.12"
      },
      "engines": {
        "node": ">= 6"
      }
    },
    // ---------------------------------------------------------------------------------------------------
    
    "node_modules/auth0/node_modules/form-data": {
      "version": "3.0.1",
      "resolved": "https://registry.npmjs.org/form-data/-/form-data-3.0.1.tgz",
      "integrity": "sha512-RHkBKtLWUVwd7SqRIvCZMEvAMoGUp0XU+seQiZejj0COz3RI3hWP4sCv3gZWWLjJTd7rGwcsF5eKZGii0r/hbg==",
      "dependencies": {
        "asynckit": "^0.4.0",
        "combined-stream": "^1.0.8",
        "mime-types": "^2.1.12"
      },
      "engines": {
        "node": ">= 6"
      }
    },
    // ------------------------------------- Note that [email protected] is here, but it should also be in 
    // "node_modules/auth0/node_modules/axios/node_modules/form-data" as well
     "node_modules/form-data": {
      "version": "4.0.0",
      "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.0.tgz",
      "integrity": "sha512-ETEklSGi5t0QMZuiXoA/Q6vcnxcLQP5vdugSpuAyi6SVGi2clPPp+xgEhuMaHC+zGgn31Kd235W35f7Hykkaww==",
      "dependencies": {
        "asynckit": "^0.4.0",
        "combined-stream": "^1.0.8",
        "mime-types": "^2.1.12"
      },
      "engines": {
        "node": ">= 6"
      }
    },
     "node_modules/openai": {
      "version": "3.2.1",
      "resolved": "https://registry.npmjs.org/openai/-/openai-3.2.1.tgz",
      "integrity": "sha512-762C9BNlJPbjjlWZi4WYK9iM2tAVAv0uUp1UmI34vb0CN5T2mjB/qM6RYBmNKMh/dN9fC+bxqPwWJZUTWW052A==",
      "dependencies": {
        "axios": "^0.26.0",
        "form-data": "^4.0.0" // <======== is responsible for the node_modules/[email protected] install...
      }
    },

The missing package results in the following error when running npm ci (i.e. in a Dockerfile)

npm ERR! code EUSAGE
npm ERR! 
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: [email protected] from lock file
npm ERR! 
npm ERR! Clean install a project

Expected Behavior

Generated package-lock.json is valid and npm ci succeeds.

GitHub Repo

No response

Steps to Reproduce

  1. Create an nx project with an app/lib that has a imports dependencies that create two copies of the same version of the same package. For example, auth0 and openai, which both ultimately import [email protected] but will do so in different folders in node_modules, due to auth0 having a direct dependency on an earlier version of form-data.

Nx Report

Node   : 18.15.0
   OS     : linux arm64
   npm    : 9.5.0
   Hasher : Native
   
   nx (global)        : 16.2.2
   nx                 : 16.3.2
   @nx/js             : 16.3.2
   @nx/jest           : 16.3.2
   @nx/linter         : 16.3.2
   @nx/workspace      : 16.3.2
   @nx/devkit         : 16.3.2
   @nx/eslint-plugin  : 16.3.2
   @nx/nest           : 16.3.2
   @nx/node           : 16.3.2
   @nx/plugin         : 16.3.2
   @nrwl/tao          : 16.3.2
   @nx/webpack        : 16.3.2
   typescript         : 5.0.4
   ---------------------------------------

Failure Logs

npm ERR! code EUSAGE
npm ERR! 
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: [email protected] from lock file
npm ERR! 
npm ERR! Clean install a project

Operating System

  • macOS
  • Linux
  • Windows
  • Other (Please specify)

Additional Information

The root package-lock.json for the project is not missing the dependency.

Also,

Because these have the same version, perhaps one is pruned by this code:

if (
graph.externalNodes[`npm:${packageName}`] &&
graph.externalNodes[`npm:${packageName}`].data.version === versionRange
) {
return;
}

@mikehaas763
Copy link
Contributor

mikehaas763 commented Aug 8, 2023

I just ran across this issue as well, but with the @nx/vite:build executor. It appears to happen when both generatePackageJson and includeDevDependenciesInPackageJson are set to true. If includeDevDependenciesInPackageJson is set to false, I don't have this problem. I'm not sure why I had that enabled in the first place though so it doesn't really impact me.

@jaysoo jaysoo self-assigned this Aug 31, 2023
@MarcusElevait
Copy link

I came across the same issue with the @nrwl/webpack:webpack executor. Also setting includeDevDependenciesInPackageJson to false doesn't help.

@icastillejogomez
Copy link

Same here 🥲

@Markus-Ende
Copy link

Same issue, my setup:

Node   : 18.18.2
   OS     : linux-x64
   npm    : 10.2.4
   
   nx (global)        : 17.0.3
   nx                 : 17.0.3
   @nx/js             : 17.0.3
   @nx/jest           : 17.0.3
   @nx/linter         : 17.0.3
   @nx/eslint         : 17.0.3
   @nx/workspace      : 17.0.3
   @nx/angular        : 17.0.3
   @nx/cypress        : 17.0.3
   @nx/devkit         : 17.0.3
   @nx/eslint-plugin  : 17.0.3
   @nx/nest           : 17.0.3
   @nx/node           : 17.0.3
   @nx/plugin         : 17.0.3
   @nx/storybook      : 17.0.3
   @nrwl/tao          : 17.0.3
   @nx/web            : 17.0.3
   @nx/webpack        : 17.0.3
   typescript         : 5.1.6

@roryf
Copy link

roryf commented Jan 10, 2024

Encountered this issue only after upgrading typeorm from 0.3.17 to 0.3.19. Really struggling to see what changed in typeorm to cause this.

   Node   : 18.17.1
   OS     : darwin-arm64
   npm    : 9.6.7
   
   nx (global)        : 17.0.3
   nx                 : 17.0.3
   @nx/js             : 17.0.3
   @nx/jest           : 17.0.3
   @nx/linter         : 17.0.3
   @nx/eslint         : 17.0.3
   @nx/workspace      : 17.0.3
   @nx/cypress        : 17.0.3
   @nx/devkit         : 17.0.3
   @nx/esbuild        : 17.0.3
   @nx/eslint-plugin  : 17.0.3
   @nx/nest           : 17.0.3
   @nx/next           : 17.0.3
   @nx/node           : 17.0.3
   @nx/react          : 17.0.3
   @nx/storybook      : 17.0.3
   @nrwl/tao          : 17.0.3
   @nx/vite           : 17.0.3
   @nx/web            : 17.0.3
   @nx/webpack        : 17.0.3
   typescript         : 5.3.2

@roryf
Copy link

roryf commented Jan 10, 2024

After a slightly painful nx migrate, issue remains. Error seems to be

npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file

The relevant lines from typeorm diff seem to be here, I won't pretend to know what caused these or what it actually means

12064 - "node_modules/string-width": {
12971 + "node_modules/string-width-cjs": {
12972 +      "name": "string-width",
12973        "version": "4.2.3",

It's amazing the amount of harm one individual can do to an ecosystem just because of their ego and idea of "technical correctness" 😢

@saez0pub
Copy link

saez0pub commented Feb 1, 2024

Hello,

I have also a problem with the pruned lockfile generated by nx missing some versions of a dependency.
I'm wondering what is the benefit of pruning this lockfile ? a pruned package.json isn't sufficient for reducing the size of the node_modules ?
I'm currently patching it with a copy of the lockfile in the workspace root after the build or before using npm ci.
Am I missing something ?

Thank you.

@kdawgwilk
Copy link
Contributor

I have been digging into this issue quite a bit and even have a failing test written for it. It appears that when the pruning process sees a aliased package it treats it as a duplicate and removes it? @meeroslav Any thoughts? You can repro this pretty easily from the __fixtures__/mixed-keys/package.json by creating a lock file with npm i and then attempting to prune that generated lock file and a bunch of the deps are stripped away

@SirPhemmiey
Copy link

any heads on on this with yarn? I'm still having this issue with generatePackageJson set as true 😢

@CyanoFresh
Copy link

same issue

@pbintcha
Copy link

After a slightly painful nx migrate, issue remains. Error seems to be

npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file

The relevant lines from typeorm diff seem to be here, I won't pretend to know what caused these or what it actually means

12064 - "node_modules/string-width": {
12971 + "node_modules/string-width-cjs": {
12972 +      "name": "string-width",
12973        "version": "4.2.3",

It's amazing the amount of harm one individual can do to an ecosystem just because of their ego and idea of "technical correctness" 😢

May be related to isaacs/jackspeak#5

The workaround i have found was to downgrade @fastify/static package to v6 instead of v7
You can use command npm ls @isaacs/cliui to find which package depends on this lib.

@Avivbens
Copy link

Hi team 👋

Same issue in here, see this:

Trying to npm ci over the generated package:

CleanShot 2024-04-11 at 12 14 24@2x

Different with the original package.json

CleanShot 2024-04-11 at 12 13 23

Configuration

CleanShot 2024-04-11 at 12 44 21

Summary

The esbuild coping the lock file, but filters just needed deps for package.json.
Result - lock file is out of sync.

@Avivbens
Copy link

Temp solution

As a walkaround, currently added this build target to force sync the lock file:

CleanShot 2024-04-11 at 13 25 02

@ImTemporaryHere
Copy link

same here

@leosvelperez
Copy link
Member

Hey all! 👋🏻

A few fixes have been released in recent Nx versions (the last one landing on Nx 19.3.0). Could you please update to the latest version and try again?

If you still have issues after updating, please provide a reproduction so we can have a look. It's hard to troubleshoot these issues without a proper reproduction.

@leosvelperez leosvelperez added the blocked: retry with latest Retry with latest release or head. label Jun 14, 2024
@Avivbens
Copy link

Hey all! 👋🏻

A few fixes have been released in recent Nx versions (the last one landing on Nx 19.3.0). Could you please update to the latest version and try again?

If you still have issues after updating, please provide a reproduction so we can have a look. It's hard to troubleshoot these issues without a proper reproduction.

Still having this issue

> npm ci

npm ERR! code EUSAGE
npm ERR!
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR!
npm ERR! Missing: @grpc/[email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR!
npm ERR! Clean install a project
npm ERR!
npm ERR! Usage:
npm ERR! npm ci
npm ERR!
npm ERR! Options:
npm ERR! [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
npm ERR! [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
npm ERR! [--include <prod|dev|optional|peer> [--include <prod|dev|optional|peer> ...]]
npm ERR! [--strict-peer-deps] [--foreground-scripts] [--ignore-scripts] [--no-audit]
npm ERR! [--no-bin-links] [--no-fund] [--dry-run]
npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
npm ERR! [-ws|--workspaces] [--include-workspace-root] [--install-links]
npm ERR!
npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
npm ERR!
npm ERR! Run "npm help ci" for more info

npm ERR! A complete log of this run can be found in: /Users/aviv.benshahar/.npm/_logs/2024-06-14T11_24_52_514Z-debug-0.log

@Avivbens
Copy link

Nx Dependencies

npm ls
├── @nx/[email protected]
├── @nx/[email protected]
├── @nx/[email protected]
├── @nx/[email protected]
├── @nx/[email protected]
├── @nx/[email protected]
├── @nx/[email protected]
├── @nx/[email protected]
├── @zie/[email protected]
├── [email protected]

@leosvelperez
Copy link
Member

I see. Unfortunately, we'll need more information to reproduce the issue. These issues are very difficult to reproduce with only the failure logs and the nx report output. We've unsuccessfully tried to do so. Could you please provide a repo we can use to reproduce it and troubleshoot it?

Thanks for the patience!

@leosvelperez leosvelperez added blocked: repro needed and removed blocked: retry with latest Retry with latest release or head. labels Jun 14, 2024
@Avivbens
Copy link

I see. Unfortunately, we'll need more information to reproduce the issue. These issues are very difficult to reproduce with only the failure logs and the nx report output. We've unsuccessfully tried to do so. Could you please provide a repo we can use to reproduce it and troubleshoot it?

Thanks for the patience!

Cannot create such a repo

I'd be more than happy to help, but seems like we cannot create a public repo that has the issue.
I've tried, but seems like all of the problematic packages I've posted above related to internal dependencies (for now - of my company).

I can help you debug that on a Zoom call, if you'd like to 🥷

A quick guess - the Nx engine looking into public repos dependencies, instead of getting all dependencies tree from the npm ls command (?)

Copy link

This issue has been automatically marked as stale because no reproduction was provided within 7 days.
Please help us help you. Providing a repository exhibiting the issue helps us diagnose and fix the issue.
Any time that we spend reproducing this issue is time taken away from addressing this issue and other issues.
This issue will be closed in 21 days if a reproduction is not provided.
If a reproduction has been provided, please reply to keep it active.
Thanks for being a part of the Nx community! 🙏

@github-actions github-actions bot added the stale label Jun 22, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 13, 2024
Copy link

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.