Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.6.0: [email protected] rebuilt every time anything is installed #5680

Closed
lydell opened this issue Apr 16, 2018 · 8 comments
Closed

1.6.0: [email protected] rebuilt every time anything is installed #5680

lydell opened this issue Apr 16, 2018 · 8 comments
Assignees
Labels
cat-bug fixed-in-modern This issue has been fixed / implemented in Yarn 2+. help wanted

Comments

@lydell
Copy link

lydell commented Apr 16, 2018

Do you want to request a feature or report a bug?
bug

What is the current behavior?
[email protected] is rebuilt if installing five.

If the current behavior is a bug, please provide the steps to reproduce.
See below.

What is the expected behavior?
Installing five does not cause a rebuild of node-rdkafka.

Please mention your node.js, yarn and operating system version.

  • Node.js 8.9.4
  • yarn 1.6.0
  • Ubuntu 17.10

Description
This is a follow-up issue to #932. yarn 1.6.0 seem like a big step in the right direction! Thanks! I have one really odd case where the "native modules always rebuilt" problem still persists, though.

In short, if I install [email protected] (that exact version) and then five (which depends on nothing and nothing depends on), node-rdkafka is rebuilt unnecessarily.

The strange thing is that if I install any later version of node-rdkafka, such as 2.3.2, everything works as expected (no rebuild)!

Steps to reproduce:

~/stuff  
❯ yarn cache clean -f
yarn cache v1.6.0
success Cleared cache.
Done in 0.03s.

~/stuff  
❯ mkdir tmp

~/stuff  
❯ cd tmp

~/stuff/tmp  
❯ echo '{"private":true}' > package.json

~/stuff/tmp  
❯ yarn add [email protected]
yarn add v1.6.0
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 3 new dependencies.
info Direct dependencies
└─ [email protected]
info All dependencies
├─ [email protected]
├─ [email protected]
└─ [email protected]
Done in 40.62s.

~/stuff/tmp 40s  
❯ yarn add five
yarn add v1.6.0
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ [email protected]
info All dependencies
└─ [email protected]
Done in 39.06s.

~/stuff/tmp 39s  
❯ echo Kafka was rebuilt!

I've tried this on two different computers, one of which I had never installed node-rdkafka on before.

I apologize if this turns out not to be a problem with yarn after all. But considering the number of people interested in #932 I think it can be good to have an answer in this issue tracker that people might find. Thanks!

@ghost ghost assigned arcanis Apr 16, 2018
@ghost ghost added the triaged label Apr 16, 2018
@rally25rs
Copy link
Contributor

rally25rs commented Apr 16, 2018

This happens for me on OSX as well 😿
I'll try to figure out why...


Update 1

My previous fix for #932 ensured that differences in build artifacts tracked in the node_modules/.yarn-integrity file wouldn't trigger a rebuild.

In this case, it seems like a bunch of build artifacts for node-rdkafka don't get added to that integrity file.

Many files do get added, but a bunch named build/Release/obj.target/librdkafka/deps/librdkafka/src/lz4* don't get added.


Update 2

OK It looks like the build process for that package deletes the files I mentioned above. They exist and are shipped with the package .tgz (probably incorrectly. why would you ship .o files that need to be built?)
Then the C/C++ build process deletes those files (they must not be rebuilt for whatever reason.)
We track changed and new files in .yarn-integrity but we don't track removed files.

When yarn finds files that are in the package's .tgz that are missing from node_modules it decides it needs to reinstall / rebuild it.

@rally25rs rally25rs assigned rally25rs and unassigned arcanis Apr 16, 2018
@felixrabe
Copy link

felixrabe commented Apr 20, 2018

I've been reading all the way from #932 via other related issues to this one, and I'm really looking forward to the speedups related to compiling native modules, especially when used together with electron. Hopefully #5314 (comment) just works for me already :) (EDIT: it does!) Will try that and if I don't come back with specific issues, consider this comment a big Thank You for working on this! 🎉

PS: Not related to Yarn, but to my Electron-plus-native-modules issue: Together with Yarn's experimental caching, I ended up wrapping electron-rebuild in a custom script and running that in scripts.postinstall. The wrapper keeps ./node_modules as-is so its native packages still work with native Node processes, whereas it creates the directory ./electron-rebuild/<electronVersion>/node_modules with only the native modules (modules containing binding.gyp) for module.paths.unshift()-ing for Electron processes. As intended, native modules in that directory only get built if they have changed.

PS 2: Wrapper (in minimal example project) published at https://github.com/felixrabe/e-2018-065-yarn-better-sqlite3-electron.

@rally25rs
Copy link
Contributor

So I guess this "fix" for this case will have to be tracking deleted files in the yarn integrity artifacts list. If anyone is willing to work up a PR, we would appreciate it.

@elpupi
Copy link

elpupi commented Aug 9, 2019

node-sass is always recompiling after any yarn add package command. And it is really long. Also the package sharp postinstall is not getting called and I have to do it by hand every time.

Here you find my package.json for node-sass

{
    "name": "webpack",
    "version": "1.0.0",
    "main": "dist",
    "license": "MIT",
    "scripts": {
        "clean": "shx rm -r dist",
        "prebuild": "yarn clean",
        "postbuild": "yarn all-test",
        "build": "rm -R dist && tsc",
        "all-test": "yarn test && yarn e2e",
        "test": "jest --config jest-ut.config.js",
        "e2e": "yarn e2e-plain-webpack && yarn e2e-angular",
        "e2e-plain-webpack": "jest --config jest-e2e-plainwebpack.config.js",
        "e2e-angular": "jest --config jest-e2e-angular.config.js",
        "set-follow-symlinks-watcher": "sed -i 's/followSymlinks: false/followSymlinks: true/g' node_modules/watchpack/lib/DirectoryWatcher.js",
        "push": "npm version patch && git push && git push --tags",
        "publish": "tsc && npm publish --access public"
    },
    "devDependencies": {
        "@angular-devkit/build-angular": "^0.13.3",
        "@angular/core": "^7.2.7",
        "@types/globby": "^8.0.0",
        "@types/jest": "^24.0.9",
        "@types/node": "^11.9.5",
        "@types/parse5": "^5.0.0",
        "@types/webpack": "^4.4.25",
        "clean-webpack-plugin": "^1.0.1",
        "jest": "^24.1.0",
        "jest-expect-message": "^1.0.2",
        "micromatch": "^3.1.10",
        "p-each-series": "^1.0.0",
        "parse5": "^5.1.0",
        "shx": "^0.3.2",
        "ts-jest": "^24.0.0",
        "ts-node": "^8.0.2",
        "ts-util-is": "^1.1.3",
        "typescript": "^3.3.3333",
        "webpack": "^4.29.5",
        "webpack-cli": "^3.2.3"
    },
    "dependencies": {
        "@angular-devkit/core": "^7.3.3",
        "@ud-angular-builders/custom-webpack-7": "^7.5.1",
        "@upradata/browser-util": "^1.0.11",
        "@upradata/node-util": "^2.0.2"
    },
    "majestic": {
        "jestConfig": "jest-ut.config.js"
    }
}

@rimmartin
Copy link

openssl needs to be rebuilt after every yarn install
with

    "openssl" : "openssl/openssl#OpenSSL_1_1_1g",

btw I'm finding c++ projects can be fairly organized with yarn and package json's. Independent of IDE's

@jedwards1211
Copy link

jedwards1211 commented Oct 27, 2020

this can get especially frustrating when working with global packages like now, which must be rebuilt anytime you add or remove any other global.

Also I got so fed up with fsevents getting rebuilt every time I installed or updated local deps in my packages that I replaced @babel/cli, which transitively depends on it, with my own script. It really wears on you that much.

@jedwards1211
Copy link

jedwards1211 commented Oct 27, 2020

@rally25rs I don't guess it's possible to turn off integrity checks for cases like this? Personally I would rather yarn never ever touch a package directory within node_modules after installation, unless a different version of the package needs to be installed.

It's kind of a classic case of how solutions usually create other problems and everything is a tradeoff...

@merceyz
Copy link
Member

merceyz commented Jan 3, 2021

Closing as fixed in v2 where the tracking of what needs to be rebuilt has been greatly improved

https://yarnpkg.com/getting-started/migration

@merceyz merceyz closed this as completed Jan 3, 2021
@merceyz merceyz added the fixed-in-modern This issue has been fixed / implemented in Yarn 2+. label Jan 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cat-bug fixed-in-modern This issue has been fixed / implemented in Yarn 2+. help wanted
Projects
None yet
Development

No branches or pull requests

8 participants