Increase in npm ERR! cb() never called! errors

[desrosj]

Description

There has been an increase in failures running npm install and npm ci in the last 2 months. This has mainly been visible in GitHub Action workflows, but some folks have encountered this locally.

Restarting the workflow or re-running npm install usually fixes the issue.

This is an issue to track this problem, document findings, and hopefully find a solution.

[desrosj]

Here's a knowledge dump for the actions I've taken so far, and findings.

Actions so far:

• I reached out to GH support.
• They asked that I create issues on the actions/setup-node (see Increased number of npm ERR! cb() never called! failures actions/setup-node#268) and actions/virtual-environments (Increased number of npm ERR! cb() never called! failures actions/runner-images#3572).
• The virtual environments ticket has been closed with a request to reach out to the NPM team.
• I have not yet done that because there is more debugging we can do first.

The TLDR of the cb() never called! error is that every occurrence of it is unique. It’s essentially a catch all error for a number of different issues that could be in either NPM, or a dependency.

This page in the NPM wiki exactly what is going on. There’s also a very long-term issue on the NPM GH repo (npm/cli#417). It seems that it’s pretty common in projects with a large list of dependencies.

There has been some exploration of cache keys used within GHA workflows being the cause for this because one way to potentially fix the error is to force clean the cache. I think we can rule that out. I think this is more likely a network issue, or a bug within one or more of our dependencies that happens on occasion.

For NPM caching, it’s actually handled entirely by actions/setup-node now as of #33190. The only workflow still manually using actions/cache is the project management workflow. That’s because that workflow runs npm install within a subdirectory, so only a small number of dependencies are installed. If the cache key is updated from that workflow, we lose the caching benefits on all other workflows.

[gziolo]

It's worth noting that we are still using npm 6. It's hard to tell if it will get any better when we set npm 7 on CI.

[talldan]

An observation is that I've been seeing this error in incredible frequency with this PR - #34533.

Notably, that PR contains changes to the package-lock.json because it changes the dependencies of one of the packages. I wonder if these kind of changes could be a trigger for the problem in our CI.

[gziolo]

@talldan, you linked this issue. Is it related to the fix to the mobile dependency that is a forks of a package?

[talldan]

So I did! I've edited my comment.

[gziolo]

It looks every time package-lock.json gets updated this errors strikes back. The most recent example is eb4c8e3.

All checks on the PR were green.

[desrosj]

Because this has only been happening when the package-lock.json file is updated (or so it seems), I think we can rule out caching issues.

When the lock file changes, the cache key changes and a 100% fresh npm install takes place. So I think this is definitely something inside NPM that occurs with large dependency trees.

In chatting with @gziolo just now, he mentioned that there were a few dependencies that were loaded through git+ (he pointed to the react-native-editor package. My understanding is that when this versioning format is used, an actual GIT clone is performed to retrieve the needed source.

Since most of the references to the error message seem to suggest network issues, I'm thinking we should try to eliminate all git+ format dependencies to troubleshoot and try to eliminate this problem.

As en experiment, we could create a fork, change the versions to standard x.y.z format, and try making several updates to the lock file to try and reproduce.

[talldan]

One observation recently is that it does seem to be happening more regularly with React Native jobs (check recent failures to trunk to see this trend). Those workflows don't seem to do anything different to other workflows when it comes to installing npm dependencies, but they do run on Mac OS instead of Linux.

Those workflows have repeatedly failed to do a fresh npm install. It seems like for some reason Mac OS is more likely to fail than Linux, so I wondered if that operating system results in more happening during an install, or if the hardware might be not quite as capable.

[gziolo]

I'm trying to migrate npm v6 to v7 in #33892. In the process, I discovered a few issues between React Native dependencies that are tracked in #34801. As a temporary step, I also decided to remove appium from the dev dependencies and remove all references to @wordpress/react-native-editor. It helped me to move forward locally, but I'm still trying to fix other issues. It makes me wonder if we need to investigate that area first.

[gziolo]

I think was able to narrow down the issue in #34809 to package dependencies in @wordpress/react-native-editor package. When you remove all dependencies that point to git URLs then there are no cb() errors reported even though the lock files was updated.

[youknowriad]

I justed wanted to share that these npm install issues are getting too frequent and are making our release process a lot harder (due to breakage...), would be cool if we find a way forward.

How can we improve the situation here? @fluiddot @hypest @dcalhoun

[hypest]

would be cool if we find a way forward.

Indeed @youknowriad. As we speak, @ceyhun is working on a spike to replace the npm deps defined as git-dependencies, since it looks like that helps with the cb() never called! error.

[desrosj]

Wanted to check in to see if the issue is happening less frequently after #34886. Happy to dig further and try more things if it's still an issue.

I'm not heavily involved day to day in this repo, but it does seem that the issue has improved some.

[gziolo]

I'm back to my regular work schedule. I will check this week if #34886 unblocks npm 7 migration.

[fluiddot]

Wanted to check in to see if the issue is happening less frequently after #34886. Happy to dig further and try more things if it's still an issue.

I'm not heavily involved day to day in this repo, but it does seem that the issue has improved some.

The changes introduced in #34886 should have fixed the cb() never called! entirely, actually after checking the recent commits of the trunk branch, I see it hasn't been reproduced since it was merged 🎊 . In fact, I think we could close this issue unless there's something else that we would like to double-check before considering the issue solved.

[desrosj]

Just noting for reference that I have seen the error a couple times over the last 2-3 weeks. One instance: https://github.com/WordPress/gutenberg/runs/4089434918?check_suite_focus=true#step:4:4.

They do seem to be more random, and accompanied by other failures referencing separate network issues though.