Linking dependencies is taking a long time

[sfabriece]

Do you want to request a feature or report a bug?
bug
What is the current behavior?
when installing a dependency, the third step: linking dependencies is taking a long time, even for a single package
If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior?

Please mention your node.js, yarn and operating system version.
node: 6.7.0
OS: Windows 10

[macdja38]

I'm having linking dependancies take 200+ seconds with this https://github.com/macdja38/pvpsite/blob/master/package.json on Windows 10, off an SSD with a decent i7.

[macdja38]

It seems the performance issue may be caused by windows Defender, disabling it while possibly inadvisable reduced 200s to somewhere closer to 50.

[sfabriece]

I think that there should be a better solution than reducing security.

[macdja38]

Some other users have confirmed that disabling it just in the root directory of your project works, but I can't confirm as Windows Defender broke itself a bit when I tried to do that.

[wclr]

I've got the same with issue with git repo with about 30 dependencies.
Windows 10
node v5.5.0
yarn 0.16.1

Disabling Windows Defender reduced linking time significantly

[wclr]

Probably should be "solved" by this PR?

[Daniel15]

Yeah there's not a lot we can do here unfortunately :( Virus scanners scan all files, and the npm ecosystem has a lot of small files. Small files generally have a bit more overhead on NTFS compared to other file systems such as EXT4 or ZFS, but it's exacerbated by virus scanners.

Having said that, Yarn should still be faster than npm, it just won't be as fast as on Linux or Mac.

[sharat]

I am on a Mac without having any antivirus scanners installed. But I still see the same problem, linking taking considerable amount of time even with a simple angular-js app.

[bjackson]

I have this issue too. It took 174s on Ubuntu.

[alanpog]

I started having this problem only after I upgraded from 0.17.8 to 0.17.19. Mac without antivirus.

[asolopovas]

Strange thing as well is that I need to go throw linking process each time when I delete a package as well. Npm does that faster. And linking really takes long time.

[hunterloftis]

This can be reproduced with this package.json (on Heroku):

{
  "name": "yarn-link-slowness",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "axios": "^0.15.3",
    "lodash": "^4.17.2",
    "react": "^15.4.1",
    "react-dom": "^15.4.1",
    "react-player": "^0.12.1",
    "react-redux": "^4.4.6",
    "react-router": "^3.0.0",
    "react-router-redux": "^4.0.7",
    "react-scripts": "^0.8.4",
    "redux": "^3.6.0",
    "redux-auth-wrapper": "^0.9.0",
    "redux-logger": "^2.7.4",
    "redux-promise-middleware": "^4.2.0",
    "redux-thunk": "^2.1.0"
  },
  "engines": {
    "node": "7.2.1",
    "yarn": "0.17.8"
  }
}

With yarn 0.17.8, the install takes 37s. With yarn 0.17.10, the install takes 97 seconds. No other changes (new Heroku apps each time).

[aj0strow]

✨ Done in 45.10s.

    "autoprefixer": "6.3.6",
    "babel-core": "6.7.6",
    "babel-jest": "13.0.0",
    "babel-loader": "6.2.4",
    "babel-plugin-transform-class-properties": "6.9.1",
    "babel-plugin-transform-object-rest-spread": "6.8.0",
    "babel-preset-es2015": "6.6.0",
    "babel-preset-react": "6.5.0",
    "bluebird": "3.3.5",
    "cardmask": "github:aj0strow/cardmask#v1.0.0",
    "chai": "3.5.0",
    "classnames": "2.2.5",
    "copy-webpack-plugin": "2.1.3",
    "core-js": "2.4.1",
    "css-loader": "0.23.1",
    "enzyme": "2.3.0",
    "file-loader": "0.8.5",
    "force-case-sensitivity-webpack-plugin": "0.1.1",
    "jest": "13.0.0",
    "jest-cli": "13.0.0",
    "json-loader": "0.5.4",
    "lodash": "4.11.1",
    "moment": "2.13.0",
    "ms": "0.7.1",
    "node-sass": "3.4.2",
    "postcss-loader": "0.9.1",
    "raw-loader": "0.5.1",
    "react": "15.2.0",
    "react-addons-css-transition-group": "15.2.0",
    "react-addons-test-utils": "15.2.0",
    "react-css-transition-replace": "2.0.1",
    "react-dom": "15.0.1",
    "react-redux": "4.4.5",
    "react-router": "2.3.0",
    "react-textarea-autosize": "4.0.3",
    "recompose": "0.20.2",
    "redux": "3.5.1",
    "redux-actions": "0.10.0",
    "redux-thunk": "2.0.1",
    "reselect": "2.5.3",
    "sass-loader": "3.2.0",
    "sinon": "1.17.4",
    "style-loader": "0.13.1",
    "webpack": "1.13.0",
    "webpack-dev-server": "1.14.1",
    "whatwg-fetch": "1.0.0",
    "zxcvbn": "4.3.0"

[Kreozot]

Please, can anyone explain what yarn does in "Linking dependencies" step exactly?
Because max number on this step vary from ~1000 to ~65000 for same project on different machines. What does this number mean?

[mpseidel]

Having this issue as well. Adding a dependency with yarn add seems to trigger "Linking dependencies" and it takes forever. Had to switch back to npm for now.

node: 6.9.2
OS: Windows 10

[OlegLustenko]

node: 7.3.0
OS:Windows 10 64
Same for me

[RichiCoder1]

Same here. Links 23420...something, and takes about one and a half minutes on a good day.

[echobops]

Yarn 0.19.1
NodeJS 7.3.0
Windows 10

yarn add moment takes 105 seconds. It has no dependencies. :/

EDIT: Turning off Windows Defender does reduce time to ~30 to ~50 seconds. I tried excluding the directory I'm working in, but that didn't seem to help.

[JeffBaumgardt]

Just ran a fresh copy of create-react-app and it took 876.37s. I understand you don't have much/any control over how anti-virus works but my experience with NPM and CRA was much faster on Windows.

[wclr]

On windows 10 just use Ubuntu bash shell as a general advice.

[Daniel15]

On windows 10 just use Ubuntu bash shell as a general advice.

Disk I/O is extremely slow in Windows Subsystem for Linux, it's a known limitation at the moment

but my experience with NPM and CRA was much faster on Windows.

@JeffBaumgardt - Interesting... Yarn is slower on Windows, but it should still be faster than npm!

[SleeplessByte]

@Daniel15 it probably should be, but it's not. Node installs and de-installs were smaller for me. So I would do npm add <packages> --save-dev, delete yarn.lock and run yarn and that be fast than running yarn add <packages> -D a single time. Now hat everyone is on yarn, of course I don't want to delete the lock and force everyone to upgrade their bundle. Instead the following has been great:

cc @echobnet

For anyone on windows 10 + windows defender

1. First click settings

   

2. Scroll down to exclusions

   

3. Run yarn cache dir to get the location of your cache folder

   • Add this cache folder to the exclusion list
   • Add your project node_modules folder to the exclusion list

4. Speed-up for a react project x 3-10

[kybarg]

@SleeplessByte or you can simply add yarn and node to excluded processes.

[jribe]

Not just a problem on windows. I've been seeing horrendous link times on my Mac Pro.

OS: OS X 10.11.6 (El Capitan)
Node: 7.6.0
Yarn: 0.20.3

[donaldpipowitch]

I can confirm that it is very slow on Mac 10.12.3, too. Not related to windows.

And it seems my setup is way slower than others in this thread. yarn sometimes tries to link around 600.000 files in small projects. This can take more than 30 minutes. I currently try it with a clean cache and nightly (v0.22.0-20170226.1626). I use the official registry as well as a custom private registry for certain scoped packages. However my colleagues don't suffer from this problem, so I don't think the custom private registry is the problem (and fetching packages is already finished anyway). We also have some relative files with path: protocol in our package.json.

I have a lot of problems installing https://github.com/google/material-design-icons which has a lot of small files which seem to be troublesome for other people as well (google/material-design-icons#518). I don't know if my hardware is broken handling a lot of small files or something like that or if this is related at all. Again my colleagues don't have as much problems installing https://github.com/google/material-design-icons as I do.

Update

I'm not sure... it looks like installing a package with file: puts a module in the cache containing node_modules/ and other stuff. This is really a problem if you have multiple examples all containing their own node_modules/. It seems .npmignore, etc. is ignored for file: installs. This probably boils down to #2165, if the solution is to not cache locally resolved files at all. If I open my cache ($ yarn cache dir) and look for modules why where install with file: and they contain a node_modules directory or other big directories, I can speed up the linking phase by removing these directories manually. Now everything seems to install with good speed.

[rajat1saxena]

success Saved lockfile.
Done in 1737.79s.

Ubuntu 16.04
i5, 8 GB RAM

:(

[Benno007]

Windows 10 v 1709 + SSD + PowerShell + Node 6.12.2:
Yarn install was fast until the last package, seemed to get stuck on a preinstall command.
Followed the instructions here to add exclusions for windows defender, but also I had my source checked out to %USERPROFILE%\source which slowed it down drastically. Checked out in c:\ it was heaps faster.

[besrabasant]

Any solution for Ubuntu platform? I literally have to think twice before adding a package.

[raybooysen]

Ubuntu is super fast for me, no slowness at all.

…

On Fri, 23 Feb 2018, 11:13 Basant Besra, ***@***.***> wrote: Any solution for Ubuntu platform? I literally have to think twice before adding a package. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1496 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcMheTtAYOsXcrnej_f2F8bY5D3nDT2ks5tXizngaJpZM4Kh3OZ> .

[samdenty]

This is super annoying. I literally changed one line in a module, republished it under a new version and yarn add module took over 5 minutes.

It'd be quicker just to manually update my packages using a text editor

[klis87]

I also experience this issue:

success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ @material-ui/[email protected]
info All dependencies
└─ @material-ui/[email protected]
Done in 93.43s.

My system is Linux manjaro 4.14.31-1-MANJARO #1 SMP PREEMPT Wed Mar 28 21:42:49 UTC 2018 x86_64 GNU/Linux

NodeJS: v9.9.0
Yarn: v1.5.1

[zebapy]

Also super slow for me Done in 254.32s.

node v8.10.0
npm 5.6.0

OSX 10.11.6 (15G19009)

[besrabasant]

I have switched back to [email protected] are things are working pretty great.

[lanoxx]

We are using the offline cache feature to avoid this problem most of the time, but as soon as the package.json or yarn-lockfile change, then we are back to this problem. Linking takes 10 minutes on our Linux machine. I do not think this is windows specific.

[chipit24]

This is definitely not a Windows only issue (which should be evident from all the posts from people on non-Windows machines)!

I'm on macOS High Sierra 10.13.4, using node 10.1.0 (npm 5.6.0) and yarn 1.6.0. Using yarn, installing a dependency took ~40s. I switched to npm and it took about ~10 seconds. I'll stick to npm for the time being.

[brona90]

Same for our centos 7 box. Any updates on this?
yarn: v1.7.0
npm: v5.7.1

[sibelius]

it is happening to me on 1.9.2 on mac on node 10

[btxtiger]

For me on macOS HighSierra, Avast FileShield was causing the issue. I've added the yarn executable as excluded path, using which yarn. It seems ok now, I'll give an update if it returns.

[tlaak]

it is happening to me on 1.9.2 on mac on node 10

Same here. Yarn 1.9.2, node 10.6.0 on High Sierra.

[davidgoli]

@bestander this is not a Windows issue. I can repro on my Mac with Yarn 1.9.4. This issue should be reopened.

[bestander]

@davidgoli , better open a new issue, this is a new one and should be triaged separately

[killua99]

Yarn is quite slow for any environment I ran it. Debian, Mac, Windows. The new issue was open? or the RFC that get rid of node_modules will solve this?

[freddysidauruk]

i have same issue, already switch to npm but still got bug

[kirensiva]

I have also same issue with Yarn. Any solution found?

[alepee]

This is a duplicate of #990, there is a comparison chart and seems like latest versions of Yarn made some good progress there.
If it is still a problem please file a new issue with repro steps and a comparison with latest npm

This not a duplicate, this issue is not only about Windows. Opening a new issue would cause a loss of context.

[reduardo7]

I have same issue!

yarn install
yarn install v1.16.0
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
[###############################################---------------------------------------------------------------------------------------------] 22778/67399

Done in 179.59s.

MacOS / Docker

[mkohlmyr]

Vagrant 2.2.4
Guest: Ubuntu 18.10 (GNU/Linux 4.18.0-25-generic x86_64)
Host: MacOS 10.14.5 Mojave

yarn 1.16.0
npm 6.9.0

MacBook Pro (Retina, 13-inch, Early 2015)
Processor 2.7 GHz Intel Core i5
Memory 16 GB 1867 MHz DDR3

yarn install v1.16.0
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 1552.45s.

I actually can't run yarn install without losing 25+ minutes of productivity. It's absurd. I don't buy that this is a Windows issue. It seems very likely to me that there is some problem when running in a virtualised environment. Perhaps to do with synced folders / checking file state on the guest OS?

[thiefjim]

yarn v1.17.3
node v10.16.3
npm 6.9.0

Windows

I have tried to put my App project folder location at the same disk section as yarn cache dir.
yarn cache dir -> C:\Users<YOUR USERNAME>\AppData\Local\Yarn\Cache\v4

The result:
old location -> D:\myApp Done in 747.17s.
new location -> C:\myApp Done in 488.97s.

C and D are the same physical disk.

Mac

However Mac is faster than Windows Done in 121.37s

I think the bottleneck is the disk read/write speed?

[timotgl]

OS X 10.15
yarn v1.22.4
node v12.13.0
npm v6.12.0

I am still experiencing this. Project is located in a mounted encrypted disk image. Installing a single package takes several minutes with a relatively small package.json. Haven't benchmarked it but npm feels much faster.

Edit: Turns out that changing yarn's default cache folder to be on the same encrypted volume fixed this for me. This was really the root of the issue.

[nicholaschiang]

Just got hit by this too and I'm running:

OS: Ubuntu 18.04.2
Yarn: 1.22.4
Node: 14.7.0
NPM: 6.14.7

[emilyjablonski]

Just to yarn after a yarn install:

Done in 530.63s.
macOS Big Sur 11.2
yarn: 1.23.0
node: 12.18.4
npm: 6.14.6

It can take up to 25 minutes for me to install dependencies for a small project. I've looked at every GitHub issue and found nothing to resolve the issue. Any ideas?