[rush] NPM 5.6.0 reports "Unhandled rejection Error: Integrity check failed" during install

[jbcpollak]

I am getting stack traces like this when I check out my repo and run rush update -p in a new environment (specifically on my CI instance):

Unhandled rejection Error: Integrity check failed:
  Wanted: sha512-9skVdDNMPODzJPi+gGnLe1YKBCLAXPGULrHTLT1LXvyK8WBBn3KFrooSIH2x6UTkuv7IV41gyBNLzI22eVG8qQ==
   Found: sha512-J10xiCVXF5FLd74qv3LcesV+N7cI/IXPjjtAgUSx96WOLjC3XEOJiLTHgIrMWdcdW9QVydY3Qa5g/p5Qxuvcrg==
    at checksumError (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/cacache/lib/content/write.js:155:13)
    at write (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/cacache/lib/content/write.js:33:22)
    at putData (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/cacache/put.js:11:10)
    at Object.x.put (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/cacache/locales/en.js:28:37)
    at readFileAsync.then.data (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/pacote/lib/fetchers/file.js:38:28)
    at tryCatcher (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/bluebird/js/release/promise.js:638:18)
    at /home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/bluebird/js/release/nodeback.js:42:21
    at /home/circleci/.rush/npm-5.6.0/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:442:3)

Environment is:

• rushjs: 5.0.0-dev.23
• node: 10.4.1
• npm: 5.6.0

[octogonz]

The repro is here:

https://circleci.com/gh/6RiverSystems/monorepo-example/2

[jbcpollak]

This appears to go away with [email protected]

[octogonz]

Unhandled rejection implies some fundamentally incorrect async logic. The failed integrity check is also pretty suspect unless you generated the shrinkwrap file using a different major release of NodeJS: In the past that's anecdotally seemed to influence how the hashes are generated.

[octogonz]

Hmm... I also see that you're using node 10.4.1. That is an unstable release. NodeJS 8.x is the current LTS version. We've been burned so many times by non-LTS releases that I'm pretty sure Rush is now supposed to print a warning saying we don't support those versions. :-)

[octogonz]

BTW after running rush install --full (which rebuilds the shrinkwrap file) the rush install --purge is now succeeding for me.

[octogonz]

This code is also bothering me:

  /**
   * This is a workaround for a bug introduced in NPM 5 (and still unfixed as of NPM 5.5.1):
   * https://github.com/npm/npm/issues/19006
   *
   * The regression is that "npm install" sets the package.json "version" field for the
   * @rush-temp projects to a value like "file:projects/example.tgz", when it should be "0.0.0".
   * This causes "rush link" to fail later, when read-package-tree tries to parse the bad version.
   * The error looks like this:
   *
   * ERROR: Failed to parse package.json for foo: Invalid version: "file:projects/example.tgz"
   *
   * Our workaround is to rewrite the package.json files for each of the @rush-temp projects
   * in the node_modules folder, after "npm install" completes.
   */
  private _fixupNpm5Regression(): void {
    const pathToDeleteWithoutStar: string = path.join(this._rushConfiguration.commonTempFolder,
      'node_modules', RushConstants.rushTempNpmScope);
    // Glob can't handle Windows paths
    const normalizedpathToDeleteWithoutStar: string = Text.replaceAll(pathToDeleteWithoutStar, '\\', '/');

    let anyChanges: boolean = false;

    // Example: "C:/MyRepo/common/temp/node_modules/@rush-temp/*/package.json"
    for (const packageJsonPath of glob.sync(globEscape(normalizedpathToDeleteWithoutStar) + '/*/package.json')) {
      // Example: "C:/MyRepo/common/temp/node_modules/@rush-temp/example/package.json"
      const packageJsonObject: IRushTempPackageJson = JsonFile.load(packageJsonPath);

      // The temp projects always use "0.0.0" as their version
      packageJsonObject.version = '0.0.0';

      if (JsonFile.save(packageJsonObject, packageJsonPath, { onlyIfChanged: true })) {
        anyChanges = true;
      }
    }

    if (anyChanges) {
      console.log(os.EOL + colors.yellow(Utilities.wrapWords(`Applied workaround for NPM 5 bug`)) + os.EOL);
    }
  }

If your inference is correct, that different versions of NPM have different bugs, then Rush 5 would be a great chance to say "We now no longer support NPM releases older than X." If 6.1.0 is stable, maybe that could be X. Then we could focus on that specific release and try to make it work reliably.

In particular it's probably not a lot of work to either (1) finally fix read-package-tree to be able to read the node_modules folder without workarounds, or (2) replace this library with something else. I feel uncomfortable tampering with NPM's node_modules folder -- that workaround was intended to be shortlived.

[MartynasZilinskas]

I am encountering the same issue:
NPM: v6.1.0
Node.js: v8.11.4

Deleting common/temp and common/config/rush/npm-shrinkwrap.json fixes this issue temporarily.

[octogonz]

Does it repro with NPM 4.5.0? More recent NPM releases seem to have a lot of bugs. It's difficult for us to support because the bugs seem to be different in each version.

We generally recommend PNPM, but if you can't use PNPM you might try Yarn. Rush 5.1.0 just introduced preliminary support for Yarn. It's not been heavily tested yet, but we're very interested in feedback and issues.

[MartynasZilinskas]

@pgonzal I have tried before PNPM and encountered problems.

I will try it again and report issues.