Native packages rebuilt every time #932

jiripospisil · 2016-10-12T21:03:46Z

Do you want to request a feature or report a bug?

Bug.

What is the current behavior?

It seems that all native packages are rebuilt every time yarn is asked to either add a new package or just install the currently locked.

If the current behavior is a bug, please provide the steps to reproduce.

Add some native packages.

yarn add leveldown bcrypt

Run yarn again and observe that both of the packages will be rebuilt for no reason.

yarn

The same happens when adding a completely unrelated packages which, as far as I can tell, cannot affect the native packages in any way.

yarn add co

What is the expected behavior?

Native packages should not be rebuilt if there's no reason to do that. Note that #248 seems pretty similar.

Please mention your node.js, yarn and operating system version.

Node.js 6.7.0
Yarn 0.15.1
Ubuntu 12.04

The text was updated successfully, but these errors were encountered:

jkimbo · 2016-10-12T21:44:44Z

Can't reproduce this in Mac OSX (10.11.6) so might be a Ubuntu specific issue?

Daniel15 · 2016-10-13T03:58:30Z

I could repro on Windows 10, but only once. The second time I ran "yarn", it didn't rebuild the native libraries. Strange.

jiripospisil · 2016-10-15T15:52:43Z

I was playing with it some more and came up with a few more details:

If I run yarn add leveldown bcrypt, leveldown will be compiled as the first item in the list and the hash in node_modules/.yarn-integrity will equal to 595306... when it's finished (cut for brevity; I assume that's a checksum which determines whether something needs to be done). Now if I run just yarn again, both of the packages will be rebuilt with bcrypt being compiled as the first in the list (the order is different). The resulting checksum will equal to cbe480.... Subsequent invocation of yarn will not rebuilt the packages and the checksum will stay the same. This contradicts my original report (I probably made a mistake somewhere) but aligns with what @Daniel15 was seeing.
When I reverse the order of the packages right from the start (yarn add bcrypt leveldown), bcrypt will be first in the list and the resulting checksum will equal to cbe480... immediately. Subsequent invocation of yarn will not rebuilt the packages.
The bcrypt package will always finish first (as expected since there's simply not that much to compile) regardless of the position in the list.

I'm not at all familiar with the internals but it seems to me that the order in which the packages are compiled matter and they are simply not being sorted when first installed (and they are sorted when later invoking yarn) which affects the checksum in some way.

Daniel15 · 2016-10-15T19:30:59Z

Thank you for investigating! That sounds like a good lead. The hash is written here:

yarn/src/cli/commands/install.js

Lines 581 to 583 in f04b157

    
           const loc = await this.getIntegrityHashLocation(); 
        
           invariant(loc, 'expected integrity hash location'); 
        
           await fs.writeFile(loc, this.generateIntegrityHash(lockSource, patterns));

. It might be worth debugging that code and seeing what's different in the lockfile, as the hash in .yarn-integrity is based off the lockfile.

jiripospisil · 2016-10-15T20:10:59Z

It might be worth debugging that code and seeing what's different in the lockfile, as the hash in .yarn-integrity is based off the lockfile.

I suspected that but what threw me off was the fact that the lock file doesn't change, it's always the same. It's just the hash in .yarn-integrity that changes.

$ yarn add leveldown bcrypt
$ cat node_modules/.yarn-integrity
59530680cc0a4ee12feb373980b5abf2edf2fe2aefa16d556bfb531af8299e71
$ cp yarn.lock leveldown_bcrypt_initial.lock
$ yarn
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock leveldown_bcrypt_afterwards.lock
$ diff -s leveldown_bcrypt_initial.lock leveldown_bcrypt_afterwards.lock
Files leveldown_bcrypt_initial.lock and leveldown_bcrypt_afterwards.lock are identical

$ yarn add bcrypt leveldown
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock bcrypt_leveldown_initial.lock
$ yarn
$ cat node_modules/.yarn-integrity
cbe48058288527f95dfe5643976b0735ee784860663e93ed782b98477c824ec1
$ cp yarn.lock bcrypt_leveldown_afterwards.lock
$ diff -s bcrypt_leveldown_initial.lock bcrypt_leveldown_afterwards.lock
Files bcrypt_leveldown_initial.lock and bcrypt_leveldown_afterwards.lock are identical

$ diff -s leveldown_bcrypt_initial.lock bcrypt_leveldown_initial.lock
Files leveldown_bcrypt_initial.lock and bcrypt_leveldown_initial.lock are identical

hustcer · 2016-10-29T23:45:11Z

I have the same issue: I use bcrypt too. Every time when I install some new modules or upgrade exist ones, I have to run npm rebuild to make my app runnable.

macOS 10.12 && node v7.0.0 && yarn v0.16.1

jiripospisil · 2016-11-17T18:33:33Z

I can no longer reproduce the original issue. It seems to have been fixed 👍. @Daniel15 Can you confirm?

@hustcer I don't think that's the same issue. You might want to test with the latest version and file a new bug if it's still not working for you.

hustcer · 2016-11-18T15:51:27Z

@jiripospisil Thanks, It's okay now after upgrade to yarn v0.17.4.

seansfkelley · 2016-12-25T06:48:05Z

I'm still seeing this, or something very similar, on 0.18.1. Incidentally, it's also leveldown that keeps getting rebuilt repeatedly. Using leftpad as a package with no dependencies (and that is not depended on by leveldown) for demonstrative purposes, repro steps are as follows:

mkdir test && cd test
echo '{ "name": "test", "version": "1.0.0" }' > package.json
yarn add leveldown 
yarn add leftpad
yarn remove leftpad

My output when I run this follows. Note that removing leftpad takes almost 40 seconds, the majority of which is rebuilding leveldown. This happens consistently, with and without leveldown or leftpad in the Yarn cache, though only during remove and never add.

$ mkdir test && cd test
$ echo '{ "name": "test", "version": "1.0.0" }' > package.json
$ yarn add leveldown 
yarn add v0.18.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 145 new dependencies.
<... package list omitted for brevity ...>
✨  Done in 49.93s.
$ yarn add leftpad
yarn add v0.18.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─ [email protected]
✨  Done in 2.18s.
$ yarn remove leftpad
yarn remove v0.18.1
[1/2] Removing module leftpad...
[2/2] Regenerating lockfile and installing missing dependencies...
success Uninstalled packages.
✨  Done in 38.76s.
$ yarn why leftpad
<... just to confirm that leftpad and leveldown are entirely unrelated ...>
yarn why v0.18.1
[1/4] 🤔  Why do we have the module "leftpad"...?
[2/4] 🚚  Initialising dependency graph...
warning [email protected]: No license field
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 0.30s.

Versions:

Node v7.3.0
Yarn 0.18.1
OS X El Capitan (10.11.6)

ndraiman · 2017-02-22T07:21:26Z

Please reopen as this still happens.
Just did yarn add redux and it rebuilt bcrypt, node-sass and several others.

Node v6.9.4
Yarn v0.20.3
Windows 10

jiripospisil · 2017-02-22T09:23:31Z

@seansfkelley I followed your steps with the latest version and it worked. Could you try again?

/tmp/tp-20170222100611/test ∴ echo '{ "name": "test", "version": "1.0.0" }' > package.json
/tmp/tp-20170222100611/test ∴ yarn add leveldown
yarn add v0.20.3
info No lockfile found.
warning [email protected]: No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 54 new dependencies.
<cut>
warning [email protected]: No license field
✨  Done in 1.64s.
/tmp/tp-20170222100611/test ∴ yarn add leftpad
yarn add v0.20.3
warning [email protected]: No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─ [email protected]
warning [email protected]: No license field
✨  Done in 0.69s.
/tmp/tp-20170222100611/test ∴ yarn remove leftpad
yarn remove v0.20.3
[1/2] Removing module leftpad...
[2/2] Regenerating lockfile and installing missing dependencies...
warning [email protected]: No license field
success Uninstalled packages.
✨  Done in 0.66s.
/tmp/tp-20170222100611/test ∴ yarn why leftpad
yarn why v0.20.3
[1/4] 🤔  Why do we have the module "leftpad"...?
[2/4] 🚚  Initialising dependency graph...
warning [email protected]: No license field
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 0.20s.

@nexxado Could you please add a few reproduction steps? I tried a few combinations but it worked.

Node 7.3.0
Yarn 0.20.3
MacOS 10.12.3

ndraiman · 2017-02-23T07:36:12Z

@jiripospisil I have no reproduction steps to provide, simply installing an additional package causes yarn to link and rebuild everything.

Here's the output of adding one package (lock file already exists):

Linking dependencies:

Rebuilding:

seansfkelley · 2017-02-23T16:22:41Z

@jiripospisil I am also still seeing this, but during my repro I got tripped up because it looks like leveldown (or a dependency thereof) may have started shipping an OS X-compatible binary, so the install times dropped suspiciously from 50s to 3s. If you're on OS X and you specifically yarn add [email protected] instead of just yarn add leveldown, you should see the same behavior as before.

Jessidhia · 2017-03-02T09:12:52Z

I have an indirect dependency on ttf2woff2, which also rebuilds every time.

However, it only rebuilds every time when there is a change to yarn.lock. That is, yarn with a new yarn.lock, yarn upgrade, yarn upgrade-interactive. In the yarn upgrade-interactive case, if both devdeps and regular deps were updated, ttf2woff2 gets rebuilt twice(!).

jefftimlin · 2017-03-13T09:06:08Z

I'm also seeing this issue, though I could not reproduce it with the steps above. However I can reproduce it with these steps:

yarn install pouchdb-node

which builds leveldown. Then if I add another package:

yarn add co

then again it builds leveldown.

It doesn't seem to matter what package I add, it always rebuilds leveldown.

I'm using Yarn v0.21.3, Windows 10, and Node v7.7.1

jackalcooper · 2017-03-13T13:36:42Z

I'm seeing this as well. I'm using BuckleScript (bs-platform)....

tkesgar · 2017-03-30T02:07:33Z

I'm encountering this issue, too, with sharp. Every time I run yarn add or yarn remove, sharp would get rebuilt, even with non-native packages.

Tested in yarn v0.21.3, Node 7.0.0, under Windows 10 and Ubuntu Linux 16.04.

Here is package.json dependencies, if it helps:

{
  "devDependencies": {
    "auto-reload-brunch": "^2.7.1",
    "babel-brunch": "^6.1.1",
    "babel-preset-env": "^1.2.1",
    "brunch": "^2.10.8",
    "chai": "^3.5.0",
    "clean-css-brunch": "^2.10.0",
    "css-brunch": "^2.10.0",
    "express-mysql-session": "^1.2.0",
    "javascript-brunch": "^2.10.0",
    "jquery": "^3.1.1",
    "less-brunch": "^2.10.0",
    "mocha": "^3.2.0",
    "nodemon": "^1.11.0",
    "npm-run-all": "^4.0.2",
    "postcss-brunch": "^2.0.5",
    "postcss-cssnext": "^2.9.0",
    "postcss-font-magician": "^1.6.1",
    "uglify-js-brunch": "^2.10.0"
  },
  "dependencies": {
    "body-parser": "^1.17.1",
    "connect-redis": "^3.2.0",
    "cookie-parser": "^1.4.3",
    "debug": "^2.6.1",
    "express": "^4.15.2",
    "express-session": "^1.15.1",
    "jstransformer-marked": "^1.0.2",
    "md5": "^2.2.1",
    "morgan": "^1.8.1",
    "multer": "^1.3.0",
    "node-mysql": "^0.4.2",
    "passport": "^0.3.2",
    "passport-local": "^1.0.0",
    "pug": "^2.0.0-beta11",
    "serve-favicon": "^2.4.1",
    "sharp": "^0.17.2"
  }
}

svoynow · 2017-04-18T22:53:59Z

also seeing this with bs-platform

patrickhulce · 2017-04-20T02:57:42Z

Also experiencing this with ttf2woff2 every call to yarn add rebuilds ttf2woff2 even though it hasn't been published in over a year.

technicallyfeasible · 2017-04-21T12:30:45Z

I have this with couchbase as well

edit: seems to be fixed in 0.23.2

raleksandar · 2017-04-21T14:32:59Z

It still happens to me on 0.23.2 (argon2 and node-sass get rebuilt every time I add/remove some unrelated package like moment which has no dependencies)

OS: Windows 10
Node: 7.9.0
Yarn: 0.23.2

patrickhulce · 2017-04-24T23:35:26Z

To add some more color, my perception of this happening on yarn add <some-package> was much greater than reality as many cases for me were actually triggered by combining with yarn remove <unrelated-package> immediately before due to the force: true on this line.

Certainly convenient to reuse the install logic in remove to generate the lockfile, but it'd be nice if it didn't come with all the baggage of a forced install :)

kriscarle · 2017-04-29T00:27:05Z

For me this started happening again when I upgraded to 0.23.x. I reverted to 0.21.3 and it no longer builds each time. Also this led to this issue where my IP was blocked by unicode.org after upgrading a few packages in a row dodo/node-unicodetable#16

zhangjyr · 2017-05-03T04:52:55Z

While 0.21.3 does not rebuild all the packages on add a new package, it rebuilds all the packages on remove. And it seems yarn doesn't regard it a failure if rebuilding fails.

akxcv · 2017-05-06T10:22:24Z

For me, downgrading to 0.21.3 did not help. bs-platform gets rebuilt on every add/remove.

EthanMichaels · 2017-05-16T07:41:23Z

Seeing this on macOS with Yarn 0.23.4. Rebuilds sqlite3 every time I run yarn add.

$ yarn add gulp-rename gulp-notify gulp-sass -D                                                                                         1 ↵
yarn add v0.23.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
warning [email protected]: The platform "darwin" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
warning [email protected]: The platform "darwin" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 42 new dependencies.
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
├─ [email protected]
└─ [email protected]
$ install-app-deps
Rebuilding native production dependencies for darwin:x64
Rebuilding native dependency sqlite3
✨  Done in 56.61s.

theandrewlane · 2018-03-29T00:30:40Z

@snowyu did you delete yarn.lock, node_modules and yarn cache clean ? Does ./yarn-offline-mirror/prebuilt get populated post install?

snowyu · 2018-03-29T03:46:55Z

It's a new project in temp. Yes I can see the node-sass-4.8.3.tgz file in the cache folder.
Now, I run yarn cache clean. BUT STILL download binary repeatly*.

> yarn init -y
> yarn add node-sass
yarn add v1.6.0-20180327.1507
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...Downloading binary from https://github.com/sass/node-sass/releases/download/v4.8.3/linux-x64-57_binding.node
success Saved lockfile.
success Saved 152 new dependencies.
Done in 11.98s.

> yarn add node-sass
yarn add v1.6.0-20180327.1507
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...Downloading binary from https://github.com/sass/node-sass/releases/download/v4.8.3/linux-x64-57_binding.node
success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ [email protected]
info All dependencies
└─ [email protected]
Done in 13.45s.
``

vlmonk · 2018-03-29T03:50:25Z

OK, i did simple git repo to reproduce this bug.

https://github.com/vlmonk/yarn-bug-test

yarn perform unnecessary rebuild ttf2woff2 when I try to add zero-dependency left-pad

git clone https://github.com/vlmonk/yarn-bug-test
cd yarn-bug-test
yarn
[ ... building binary package here ... ]
yarn add left-pad
[ ... rebuilding binary packages here ... ]

I can reproduce this in both host OSX system and in docker container with latest node image

NPM works correctly in this case:

git clone https://github.com/vlmonk/yarn-bug-test
cd yarn-bug-test
npm i 
[ ... building binary package here ... ]
npm i left-pad
[ ... don't rebuild binary packages here ... ]

My yarn version 1.5.1

karlhorky · 2018-03-29T07:44:55Z

@vlmonk does this still happen if you clone https://github.com/rally25rs/yarn from @rally25rs and use the code in #5470 (branch fix-linking-rebuilding-uws-932)?

vlmonk · 2018-03-29T08:02:44Z

@karlhorky yes, yarn still rebuild ttf2woff2 after adding left-pad

# which yarn
yarn: aliased to node /Users/monk/work/yarn/lib/cli/index.js
# yarn --version
1.6.0-0

sth · 2018-03-30T01:59:52Z

The ttf2woff2 package comes with files that are changed in the build step. On the next run yarn sees those files changed and reinstalls the package.

Yarn should handle this situation better: It should see that those files changed during the build step and it should accept those changed files as the "correct" files, not treat them as a reason for a reinstall.

I checked this with additional logging (https://github.com/sth/yarn/tree/trace-rebuild). On first install it shows:

build artifacts for ttf2woff2
  modified file: build
  modified file: build/Makefile
  modified file: build/Release
  modified file: build/Release/.deps
  modified file: build/Release/.deps/Release
  modified file: build/Release/.deps/Release/addon.node.d
  modified file: build/Release/.deps/Release/obj.target
  modified file: build/Release/.deps/Release/obj.target/addon
  modified file: build/Release/.deps/Release/obj.target/addon/csrc
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/addon.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/backward_references.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/block_splitter.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/brotli_bit_stream.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/encode.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/encode_parallel.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/entropy_encode.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/histogram.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/literal_cost.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/metablock.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/enc/streams.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/font.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/glyph.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/normalize.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/table_tags.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/transform.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/variable_length.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/woff2_common.o.d
  modified file: build/Release/.deps/Release/obj.target/addon/csrc/woff2/woff2_enc.o.d
  modified file: build/Release/.deps/Release/obj.target/addon.node.d
  modified file: build/Release/addon.node
  modified file: build/Release/obj.target
  modified file: build/Release/obj.target/addon
  modified file: build/Release/obj.target/addon/csrc
  modified file: build/Release/obj.target/addon/csrc/addon.o
  modified file: build/Release/obj.target/addon/csrc/enc
  modified file: build/Release/obj.target/addon/csrc/enc/backward_references.o
  modified file: build/Release/obj.target/addon/csrc/enc/block_splitter.o
  modified file: build/Release/obj.target/addon/csrc/enc/brotli_bit_stream.o
  modified file: build/Release/obj.target/addon/csrc/enc/encode.o
  modified file: build/Release/obj.target/addon/csrc/enc/encode_parallel.o
  modified file: build/Release/obj.target/addon/csrc/enc/entropy_encode.o
  modified file: build/Release/obj.target/addon/csrc/enc/histogram.o
  modified file: build/Release/obj.target/addon/csrc/enc/literal_cost.o
  modified file: build/Release/obj.target/addon/csrc/enc/metablock.o
  modified file: build/Release/obj.target/addon/csrc/enc/streams.o
  modified file: build/Release/obj.target/addon/csrc/woff2
  modified file: build/Release/obj.target/addon/csrc/woff2/font.o
  modified file: build/Release/obj.target/addon/csrc/woff2/glyph.o
  modified file: build/Release/obj.target/addon/csrc/woff2/normalize.o
  modified file: build/Release/obj.target/addon/csrc/woff2/table_tags.o
  modified file: build/Release/obj.target/addon/csrc/woff2/transform.o
  modified file: build/Release/obj.target/addon/csrc/woff2/variable_length.o
  modified file: build/Release/obj.target/addon/csrc/woff2/woff2_common.o
  modified file: build/Release/obj.target/addon/csrc/woff2/woff2_enc.o
  modified file: build/Release/obj.target/addon.node

The package file https://registry.npmjs.org/ttf2woff2/-/ttf2woff2-2.0.3.tgz indeed contains those files.

* fix(linking): Dont integrity check build artifacts If an install script modified a file that existed in the package, then yarn would view the file as stale on every command, mark the package as "fresh", and re-run the install scripts. With this change, known build artifacts from the integrity file are checked, and if this package file is a build atrifact, it's size and timestamp are not checked. #932 * WIP log message for debugging on CI * WIP adding logging for debugging CI * WIP adding logging for debugging CI * remove debugging console statements. * fix file timestamps for node 8 copyFile. fixes issue on linux. * change file mode to what works on windows * add file timestamp comparison and skip if correction not needed * fix lint errors * add error handling to fixTimes for some edge cases * pass undefined instead of 0 as a file id * refactor fs code, move OS specific edge cases to fs-normalized.js * empty commit to rerun CI * fix logic error when closing file, and fix Flow error

cisaacson · 2018-04-15T03:07:02Z

We see this with OS X as well, adding any package with yarn add triggers a recompile of any dependent packages. We have a node-gyp package with native code, it is taking over 1 minute each time another package is added, and there isn't much code in the native module yet (it will get much worse). This is with yarn 1.5.1.

We are using yarn add ../a with relative paths if that makes a difference.

Please let me know if there are any workarounds, or when it will be fixed.

BYK · 2018-04-16T09:57:08Z

This is with yarn 1.5.1.

This issue was fixed in 1.6.0 which is out quite recently.

grantila · 2018-04-16T10:39:16Z

I still see this with 1.6. Since moving from npm to yarn a long time ago uws as always rebuilt (or at least yarn has been stuck on uws for roughly 5-10 seconds).

Steps to reproduce:

$ yarn outdated
pick an outdated package
$ yarn upgrade [package]

karlhorky · 2018-04-16T10:51:15Z

@grantila can you provide a complete package.json or repo with steps that reproduce this with Yarn 1.6.0?

sibelius · 2018-04-25T15:10:58Z

this is still happening on 1.6.0

you can repro this using this #932 (comment)

brenc · 2018-04-29T17:56:26Z

I have a simple project and I'm seeing this too. Adding or removing a package seems to trigger a complete rebuild of at least one package every time.

"dependencies": {
    "bufferutil": "3.0.4",
    "chance": "1.0.16",
    "discord.js": "11.3.2",
    "dog-facts": "1.0.3",
    "erlpack": "discordapp/erlpack",
    "flickr-sdk": "3.7.0",
    "html-entities": "1.2.1",
    "node-opus": "0.2.7",
    "snekfetch": "4.0.0",
    "sodium": "2.0.3",
    "utf-8-validate": "4.0.1"
  }

After these were installed, I added the unescape package, which triggered a rebuild of sodium. Then I removed it which triggered a rebuild of what seemed like every packaged that needed to be compiled. Adding this simple package took 36s and removing it took 100s!

EDIT: using Node 8.11.1 and yarn 1.6.0 on Debian Stretch.

stereokai · 2018-04-30T11:12:55Z

@arcanis @rally25rs pleaase reopen the issue, multiple reports of this still happening, even with the merged fix.

BYK · 2018-04-30T19:07:18Z

I think this is more of a @rally25rs issue :)

rally25rs · 2018-04-30T19:56:43Z

@grantila an upgrade will always rebuild all. This is intentional. npm does the same thing (I mention this in a comment somewhere in this long thread.) though we could potentially try to stop doing that. I'm not sure what repercussions it may have.

Everyone else:

In #5680 I point out that this still happens if a package deletes it's own files (why oh why do they do these things 😿) and yarn doesn't track that anywhere (we track what files are created or modified), so it just thinks the package is missing files and rebuilds it.

I suppose we can reopen this, but this has been fixed for most packages. If anyone wants to add "me too" to this, please either provide your package.json, or mention specifically which package is continually rebuilding, since you may have some dependencies that do rebuild and some that do not.

Anyone is welcome to make a PR for this! (see my debugging comments in #5680 )

james-rabbit · 2018-04-30T20:16:31Z

Sorry for adding more noise, but I'd like to suggest locking this issue and pointing to a new one with this latest information at the top.

I think the issue here has shifted quite a bit and is at least partially resolved. With all the posts here it's difficult for someone new to get up-to-date with what has been fixed and what is still an issue.

stereokai · 2018-05-01T11:47:28Z

I agree with @james-rabbit

arcanis · 2018-05-01T12:31:21Z

Yep, you're right. I'm going to lock this one so that @rally25rs' answer stays visible.

If you have an issue with native packages:

If it happens with every native dependency, please open a generic issue. This problem should have been resolved, so I don't expect to see one any time soon - still, if you can provide reproduction steps, feel free to open a new issue (and you can link to this one if you like).
If it happens with one specific native dependency, please also open an issue but don't forget to specify the name of the dependency in the title (as explained, different packages might be rebuilt for different reasons - keeping one issue for each of those will make it easier for everyone to share information).

Daniel15 added the cat-bug label Oct 13, 2016

jiripospisil closed this as completed Nov 19, 2016

jiripospisil reopened this Feb 23, 2017

Daniel15 mentioned this issue Mar 14, 2017

Why yarn build node-gyp deps at each install? #2916

Closed

This was referenced Mar 29, 2018

Save/restore yarn cache heroku/heroku-buildpack-ruby#654

Open

Loosen up yarn and support npm rails/webpacker#475

Closed

rally25rs mentioned this issue Apr 3, 2018

WSL: yarn always links all dependencies #5591

Closed

arcanis closed this as completed in #5470 Apr 5, 2018

brandonmp mentioned this issue Apr 7, 2018

Stop cypress from downloading & unpacking every time cypress-io/cypress#1363

Closed

lydell mentioned this issue Apr 16, 2018

1.6.0: [email protected] rebuilt every time anything is installed #5680

Closed

rally25rs reopened this Apr 30, 2018

rally25rs removed good first issue triaged labels Apr 30, 2018

arcanis closed this as completed May 1, 2018

yarnpkg locked as resolved and limited conversation to collaborators May 1, 2018

Native packages rebuilt every time #932

Native packages rebuilt every time #932

Comments

jiripospisil commented Oct 12, 2016

jkimbo commented Oct 12, 2016

Daniel15 commented Oct 13, 2016

jiripospisil commented Oct 15, 2016

Daniel15 commented Oct 15, 2016

jiripospisil commented Oct 15, 2016 • edited Loading

hustcer commented Oct 29, 2016 • edited Loading

jiripospisil commented Nov 17, 2016

hustcer commented Nov 18, 2016

seansfkelley commented Dec 25, 2016

ndraiman commented Feb 22, 2017

jiripospisil commented Feb 22, 2017

ndraiman commented Feb 23, 2017 • edited Loading

seansfkelley commented Feb 23, 2017

Jessidhia commented Mar 2, 2017

jefftimlin commented Mar 13, 2017 • edited Loading

jackalcooper commented Mar 13, 2017

tkesgar commented Mar 30, 2017

svoynow commented Apr 18, 2017

patrickhulce commented Apr 20, 2017 • edited Loading

technicallyfeasible commented Apr 21, 2017 • edited Loading

raleksandar commented Apr 21, 2017 • edited Loading

patrickhulce commented Apr 24, 2017

kriscarle commented Apr 29, 2017

zhangjyr commented May 3, 2017

akxcv commented May 6, 2017

EthanMichaels commented May 16, 2017

theandrewlane commented Mar 29, 2018 • edited Loading

snowyu commented Mar 29, 2018

vlmonk commented Mar 29, 2018 • edited Loading

karlhorky commented Mar 29, 2018

vlmonk commented Mar 29, 2018

sth commented Mar 30, 2018

cisaacson commented Apr 15, 2018

BYK commented Apr 16, 2018

grantila commented Apr 16, 2018

karlhorky commented Apr 16, 2018

sibelius commented Apr 25, 2018

brenc commented Apr 29, 2018 • edited Loading

stereokai commented Apr 30, 2018

BYK commented Apr 30, 2018

rally25rs commented Apr 30, 2018

james-rabbit commented Apr 30, 2018

stereokai commented May 1, 2018

arcanis commented May 1, 2018 • edited Loading

jiripospisil commented Oct 15, 2016 •

edited

Loading

hustcer commented Oct 29, 2016 •

edited

Loading

ndraiman commented Feb 23, 2017 •

edited

Loading

jefftimlin commented Mar 13, 2017 •

edited

Loading

patrickhulce commented Apr 20, 2017 •

edited

Loading

technicallyfeasible commented Apr 21, 2017 •

edited

Loading

raleksandar commented Apr 21, 2017 •

edited

Loading

theandrewlane commented Mar 29, 2018 •

edited

Loading

vlmonk commented Mar 29, 2018 •

edited

Loading

brenc commented Apr 29, 2018 •

edited

Loading

arcanis commented May 1, 2018 •

edited

Loading