Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't install correct dependencies for 1.3.2 #5152

Closed
adamreisnz opened this issue Jan 3, 2018 · 24 comments · Fixed by #5059
Closed

Doesn't install correct dependencies for 1.3.2 #5152

adamreisnz opened this issue Jan 3, 2018 · 24 comments · Fixed by #5059

Comments

@adamreisnz
Copy link

adamreisnz commented Jan 3, 2018

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Installation of packages fails when installing in production, Yarn 1.3.2 errors with:

Error: Cannot find module 'are-we-there-yet'

This does not happen with Yarn 0.18.1, where the installs complete successfully.

If the current behavior is a bug, please provide the steps to reproduce.
These are the relevant dependencies for a particular package that fails:

"dependencies": {
    "@meanie/angular-analytics": "^2.0.0",
    "@meanie/angular-api": "^3.0.1",
    "@meanie/angular-convert": "^2.0.1",
    "@meanie/angular-duplicate-requests-filter": "^2.0.0",
    "@meanie/angular-focus": "^2.0.0",
    "@meanie/angular-key-codes": "^2.0.0",
    "@meanie/angular-log": "^2.0.0",
    "@meanie/angular-modal": "^3.0.0",
    "@meanie/angular-storage": "^2.0.1",
    "@meanie/angular-store": "^2.0.1",
    "@meanie/angular-url": "^2.0.1",
    "angular": "1.6.8",
    "angular-animate": "1.6.8",
    "angular-cookies": "1.6.8",
    "angular-messages": "1.6.8",
    "angular-sanitize": "1.6.8",
    "angular-touch": "1.6.8",
    "angular-ui-router": "^0.4.2",
    "fastclick": "^1.0.6",
    "marked": "^0.3.9",
    "meanie-angular-form-controls": "^1.4.5",
    "moment": "^2.20.1",
    "ng-infinite-scroll": "^1.3.0",
    "raven-js": "^3.21.0"
  },
  "devDependencies": {
    "babel-polyfill": "^6.23.0",
    "babel-preset-es2015": "^6.24.0",
    "browser-sync": "^2.18.6",
    "chalk": "^1.1.3",
    "del": "^2.2.2",
    "git-rev-sync": "^1.8.0",
    "gulp": "git+https://github.com/gulpjs/gulp#4.0",
    "gulp-angular-templatecache": "^2.0.0",
    "gulp-autoprefixer": "^3.1.1",
    "gulp-babel": "^6.1.2",
    "gulp-concat": "^2.6.1",
    "gulp-csso": "^3.0.0",
    "gulp-file": "^0.4.0",
    "gulp-htmlclean": "^2.7.16",
    "gulp-inject": "^4.3.0",
    "gulp-ng-annotate": "^2.1.0",
    "gulp-ng-constant": "^1.1.0",
    "gulp-plumber": "^1.1.0",
    "gulp-preprocess": "^2.0.0",
    "gulp-remove-empty-lines": "^0.0.8",
    "gulp-remove-html-comments": "^1.0.1",
    "gulp-rename": "^1.2.2",
    "gulp-sass": "^3.1.0",
    "gulp-sourcemaps": "^2.4.1",
    "gulp-uglify": "^2.1.2",
    "gulp-wrapper": "^1.0.0",
    "http-proxy-middleware": "^0.17.4",
    "merge-stream": "^1.0.1",
    "npm-run-all": "^4.0.2",
    "mime": "^1.3.4",
    "netlify-cli": "^1.2.2",
    "replace-in-file": "^2.5.0",
    "rimraf": "^2.6.1",
    "s3-cli": "^0.13.0",
    "serve-static": "^1.12.1",
    "yargs": "^10.0.3"
  }

What is the expected behavior?
For the Yarn install to work.

Please mention your node.js, yarn and operating system version.
Node.JS 8.9.3, Yarn 1.3.2, the build was failing on Netlify VM's.

Ref #761

@ghost ghost assigned BYK Jan 3, 2018
@ghost ghost added the triaged label Jan 3, 2018
@rally25rs
Copy link
Contributor

rally25rs commented Jan 7, 2018

I was unable to reproduce using the same dependencies:

~/Projects/yarn-test 🐒   cat package.json
{
  "name": "yarn-test",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
"dependencies": {
    "@meanie/angular-analytics": "^2.0.0",
    "@meanie/angular-api": "^3.0.1",
    "@meanie/angular-convert": "^2.0.1",
    "@meanie/angular-duplicate-requests-filter": "^2.0.0",
    "@meanie/angular-focus": "^2.0.0",
    "@meanie/angular-key-codes": "^2.0.0",
    "@meanie/angular-log": "^2.0.0",
    "@meanie/angular-modal": "^3.0.0",
    "@meanie/angular-storage": "^2.0.1",
    "@meanie/angular-store": "^2.0.1",
    "@meanie/angular-url": "^2.0.1",
    "angular": "1.6.8",
    "angular-animate": "1.6.8",
    "angular-cookies": "1.6.8",
    "angular-messages": "1.6.8",
    "angular-sanitize": "1.6.8",
    "angular-touch": "1.6.8",
    "angular-ui-router": "^0.4.2",
    "fastclick": "^1.0.6",
    "marked": "^0.3.9",
    "meanie-angular-form-controls": "^1.4.5",
    "moment": "^2.20.1",
    "ng-infinite-scroll": "^1.3.0",
    "raven-js": "^3.21.0"
  },
  "devDependencies": {
    "babel-polyfill": "^6.23.0",
    "babel-preset-es2015": "^6.24.0",
    "browser-sync": "^2.18.6",
    "chalk": "^1.1.3",
    "del": "^2.2.2",
    "git-rev-sync": "^1.8.0",
    "gulp": "git+https://github.com/gulpjs/gulp#4.0",
    "gulp-angular-templatecache": "^2.0.0",
    "gulp-autoprefixer": "^3.1.1",
    "gulp-babel": "^6.1.2",
    "gulp-concat": "^2.6.1",
    "gulp-csso": "^3.0.0",
    "gulp-file": "^0.4.0",
    "gulp-htmlclean": "^2.7.16",
    "gulp-inject": "^4.3.0",
    "gulp-ng-annotate": "^2.1.0",
    "gulp-ng-constant": "^1.1.0",
    "gulp-plumber": "^1.1.0",
    "gulp-preprocess": "^2.0.0",
    "gulp-remove-empty-lines": "^0.0.8",
    "gulp-remove-html-comments": "^1.0.1",
    "gulp-rename": "^1.2.2",
    "gulp-sass": "^3.1.0",
    "gulp-sourcemaps": "^2.4.1",
    "gulp-uglify": "^2.1.2",
    "gulp-wrapper": "^1.0.0",
    "http-proxy-middleware": "^0.17.4",
    "merge-stream": "^1.0.1",
    "npm-run-all": "^4.0.2",
    "mime": "^1.3.4",
    "netlify-cli": "^1.2.2",
    "replace-in-file": "^2.5.0",
    "rimraf": "^2.6.1",
    "s3-cli": "^0.13.0",
    "serve-static": "^1.12.1",
    "yargs": "^10.0.3"
  }
}

~/Projects/yarn-test 🐒   yarn --production
yarn install v1.3.2
info No lockfile found.
[1/4] 🔍  Resolving packages...
warning [email protected]: This npm package 'angular-ui-router' has been renamed to '@uirouter/angularjs'. Please update your package.json. See https://ui-router.github.io/blog/uirouter-scoped-packages/
warning [email protected]: Package moved to @meanie/angular-form-controls
warning [email protected]: 🙌  Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
warning gulp-angular-templatecache > gulp-header > [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
warning gulp-autoprefixer > [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
warning gulp-file > through2 > xtend > [email protected]:
warning gulp-htmlclean > [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
warning gulp-remove-empty-lines > [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
warning gulp-remove-html-comments > [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
warning gulp-sass > [email protected]: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
warning netlify-cli > netlify > [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
warning s3-cli > s3 > [email protected]: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 18.16s.

Could you try running with --verbose? It might help point to the problem...

@adamreisnz
Copy link
Author

adamreisnz commented Jan 7, 2018

Unfortunately, I don't have control over what flags yarn is launched with when Netlify runs the build afaik.

This is the full relevant log I can produce at this stage;

9:53:26 AM: Now using node v8.9.3 (npm v5.5.1)
9:53:26 AM: Using version v8.9.3 of node
9:53:26 AM: Using /opt/buildhome/.rvm/gems/ruby-2.1.2
9:53:26 AM: Installing yarn at version 1.3.2
9:53:27 AM: Installing Yarn!
9:53:27 AM: > Downloading tarball...
...
9:53:40 AM: > Verifying integrity...
9:53:40 AM: gpg:
9:53:40 AM: Signature made Thu 02 Nov 2017 04:44:10 PM UTC using RSA key ID FD2497F5
9:53:40 AM: gpg: Good signature from "Yarn Packaging <[email protected]>"
9:53:40 AM: gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
9:53:40 AM: Primary key fingerprint: 72EC F46A 56B4 AD39 C907  BBB7 1646 B01B 86E5 0310
     Subkey fingerprint: 6A01 0C51 6600 6599 AA17  F081 46C2 130D FD24 97F5
9:53:40 AM: > GPG signature looks good
9:53:40 AM: > Extracting to ~/.yarn...
9:53:41 AM: > Adding to $PATH...
9:53:41 AM: > We've added the following to your /opt/buildhome/.profile
> If this isn't the profile of your current shell then please add the following to your correct profile:
   
export PATH="$HOME/.yarn/bin:$PATH"
9:53:41 AM: > Successfully installed Yarn 1.3.2! Please open another terminal where the `yarn` command will now be available.
9:53:41 AM: Installing NPM modules using Yarn version 1.3.2
9:53:41 AM: yarn install v1.3.2
9:53:41 AM: [1/5] Validating package.json...
9:53:41 AM: [2/5] Resolving packages...
9:53:41 AM: [3/5] Fetching packages...
9:53:44 AM: info [email protected]: The platform "linux" is incompatible with this module.
9:53:44 AM: info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
9:53:44 AM: [4/5] Linking dependencies...
9:53:50 AM: [5/5] Building fresh packages...
9:53:50 AM: error /opt/build/repo/node_modules/node-sass: Command failed.
Exit code: 1
Command: node scripts/install.js
Arguments: 
Directory: /opt/build/repo/node_modules/node-sass
Output:
module.js:538
    throw err;
    ^

Error: Cannot find module 'are-we-there-yet'
    at Function.Module._resolveFilename (module.js:536:15)
    at Function.Module._load (module.js:466:25)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/build/repo/node_modules/npmlog/log.js:2:16)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
9:53:50 AM: info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
9:53:50 AM: Error during Yarn install
9:53:53 AM: Build complete: exit code: 1

I made sure to clear the build cache etc. before deploying.

@rally25rs
Copy link
Contributor

You know what's weird about that... it's running the install script for node-sass but that is a transitive devDependency and shouldn't even be installed with --production. 🤔

devDependencies -> gulp-sass -> node-sass

@adamreisnz
Copy link
Author

adamreisnz commented Jan 7, 2018

Sorry, my bad! This of course is not run with the --production flag because it needs the dev deps to build the actual application. I was confused with the Heroku (backend) builds for a second...

I have updated the title and description of the issue, and will ask what other (if any) flags and parameters they use over at Netlify when kicking off the build, to see if that clarifies what's happening.

@adamreisnz adamreisnz changed the title yarn install --production doesn't install correct dependencies for 1.3.2 Doesn't install correct dependencies for 1.3.2 Jan 7, 2018
@bestander
Copy link
Member

cc @BYK. We discussed recently a known problem with transitive optional dependencies, could be related

@adamreisnz
Copy link
Author

I think it might be. This is the build script Netlify uses, and from what I can see it uses the --ignore-optional flag.

@kimjoar
Copy link
Contributor

kimjoar commented Jan 15, 2018

Btw, #5059 is the PR to fix --ignore-optional. I haven't had time to fix the latest review feedback just yet.

@BYK
Copy link
Member

BYK commented Jan 15, 2018

@adamreisnz yeah, @kjbekkelund's patch should fix this then.

@kjbekkelund I'm not sure if we need another pass on that patch. I think we'll just merge it as it is and then try to address the larger problem separately.

@kimjoar
Copy link
Contributor

kimjoar commented Jan 15, 2018

@BYK That works for me. Been busy on a release here, so was planning to get back to it in a couple days.

@BYK
Copy link
Member

BYK commented Jan 15, 2018

@kjbekkelund my latest comment was about simpler tests anyways. I'll try to see if I can do it myself and then merge the patch.

@adamreisnz
Copy link
Author

adamreisnz commented Jan 15, 2018

Awesome thanks guys, let me know once it's released and I'll run some tests on Netlify again.

@BYK
Copy link
Member

BYK commented Jan 15, 2018

@adamreisnz well, if you can check in https://8470-49970642-gh.circle-artifacts.com/0/yarnpkg/yarn-1.3.2.js and then add yarn-path "path/to/checked-in/file" to your .yarnrc file, you should be able to test the version with the patch applied.

@adamreisnz
Copy link
Author

adamreisnz commented Jan 15, 2018 via email

@adamreisnz
Copy link
Author

Is there a new version going to be released soon? I feel really sad for still having to use yarn 0.18.2 on Netlify :(

@lourd
Copy link

lourd commented Feb 6, 2018

Hey folks. I'm also having this problem, also ran into it with Netlify.

@BYK I just tried your suggestion and it worked! 👏Thank you. This was the end of quite a frustrating day.

@lourd
Copy link

lourd commented Feb 6, 2018

@adamreisnz, you can use the release candidate on Netlify, which has this fix, by making a netlify.toml file in your repo with at least these contents:

[build.environment]
  YARN_VERSION = "1.4.0"

@adamreisnz
Copy link
Author

Thanks I'll give it a go. Didn't think specifying the version flag would work if the version hasn't officially been released et.

@adamreisnz
Copy link
Author

adamreisnz commented Feb 6, 2018

1.4.0 doesn't seem to have fixed it.
Still getting errors for packages that have failed to install:

12:45:19 PM: yarn install v1.4.0
12:45:19 PM: [1/5] Validating package.json...
12:45:19 PM: [2/5] Resolving packages...
12:45:20 PM: [3/5] Fetching packages...
12:45:29 PM: info [email protected]: The platform "linux" is incompatible with this module.
12:45:29 PM: info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
12:45:29 PM: [4/5] Linking dependencies...
12:45:32 PM: [5/5] Building fresh packages...
12:45:32 PM: error An unexpected error occurred: "/opt/build/repo/node_modules/node-sass: Command failed.\nExit code: 1\nCommand: sh\nArguments: -c node scripts/install.js\nDirectory: /opt/build/repo/node_modules/node-sass\nOutput:\nmodule.js:540\n    throw err;\n    ^\n\nError: Cannot find module 'wrappy'\n    at Function.Module._resolveFilename (module.js:538:15)\n    at Function.Module._load (module.js:468:25)\n    at Module.require (module.js:587:17)\n    at require (internal/module.js:11:18)\n    at Object.<anonymous> (/opt/build/repo/node_modules/inflight/inflight.js:1:76)\n    at Module._compile (module.js:643:30)\n    at Object.Module._extensions..js (module.js:654:10)\n    at Module.load (module.js:556:32)\n    at tryModuleLoad (module.js:499:12)\n    at Function.Module._load (module.js:491:3)".
12:45:32 PM: info If you think this is a bug, please open a bug report with the information provided in "/opt/build/repo/yarn-error.log".
12:45:32 PM: info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
12:45:32 PM: Error during Yarn install

These are the dependencies I'm trying to install (all packages are up to date):

"dependencies": {
    "@meanie/angular-api": "^3.0.1",
    "@meanie/angular-convert": "^2.0.1",
    "@meanie/angular-duplicate-requests-filter": "^2.0.0",
    "@meanie/angular-focus": "^2.0.0",
    "@meanie/angular-form-controls": "^3.2.1",
    "@meanie/angular-google-maps": "^2.0.0",
    "@meanie/angular-key-codes": "^2.0.0",
    "@meanie/angular-log": "^2.0.0",
    "@meanie/angular-modal": "^3.1.0",
    "@meanie/angular-storage": "^2.0.1",
    "@meanie/angular-store": "^2.0.1",
    "@meanie/angular-url": "^2.0.1",
    "@uirouter/angularjs": "^1.0.13",
    "@uirouter/core": "^5.0.14",
    "angular": "^1.6.9",
    "angular-animate": "^1.6.9",
    "angular-cookies": "^1.6.9",
    "angular-messages": "^1.6.9",
    "angular-sanitize": "^1.6.9",
    "angular-touch": "^1.6.9",
    "moment": "^2.20.1",
    "raven-js": "^3.22.1"
  },
  "devDependencies": {
    "babel-core": "^6.26.0",
    "babel-plugin-angularjs-annotate": "^0.8.2",
    "babel-plugin-syntax-async-functions": "^6.13.0",
    "babel-plugin-transform-regenerator": "^6.26.0",
    "babel-polyfill": "^6.26.0",
    "babel-preset-env": "^1.6.1",
    "babel-preset-es2015": "^6.24.1",
    "browser-sync": "^2.23.6",
    "chalk": "^2.3.0",
    "deepmerge": "^2.0.1",
    "del": "^3.0.0",
    "git-rev-sync": "^1.9.1",
    "gulp": "next",
    "gulp-angular-templatecache": "^2.2.0",
    "gulp-autoprefixer": "^4.1.0",
    "gulp-babel": "^7.0.0",
    "gulp-cached": "^1.1.1",
    "gulp-concat": "^2.6.1",
    "gulp-csso": "^3.0.1",
    "gulp-file": "^0.4.0",
    "gulp-htmlclean": "^2.7.16",
    "gulp-inject": "^4.3.0",
    "gulp-ng-constant": "^1.2.0",
    "gulp-plumber": "^1.2.0",
    "gulp-preprocess": "^2.0.0",
    "gulp-remove-empty-lines": "^0.1.0",
    "gulp-rename": "^1.2.2",
    "gulp-sass": "^3.1.0",
    "gulp-sourcemaps": "^2.6.4",
    "gulp-strip-comments": "^2.5.2",
    "gulp-uglify": "^3.0.0",
    "http-proxy-middleware": "^0.17.4",
    "merge-stream": "^1.0.1",
    "npm-run-all": "^4.1.2",
    "replace-in-file": "^3.1.0",
    "serve-static": "^1.13.1",
    "yargs": "^11.0.0"
  }

And to clarify, this works fine with 0.18.2 or when installing without the --ignore-optional flag on my dev machine.

@lourd
Copy link

lourd commented Feb 7, 2018

Aw bummer ☹️not sure why that is.

I also just heard back from one of the Netlify engineers who recommended this:

The setting we set is "--ignore-optional" (which is mentioned in that issue you linked to) and to override that, set a YARN_FLAGS Build Environment Variable (most easily on your build & deploy settings page, second configuration card - also possible in netlify.toml if you use that).

You should set it to "--no-ignore-optional" and I suspect your build will work better. We'll change the default shortly, but for the meantime, that should help your build succeed.

@adamreisnz
Copy link
Author

Hmm, so is Netlify going to change the default flag?
Regardless, this still seems like a Yarn bug with resolving (optional) dependencies incorrectly.

@fool
Copy link

fool commented Feb 7, 2018

FYI: Netlify is going to change the default flag - I work there, and we are working on rolling that fix out in the next couple of days.

@peduarte
Copy link

peduarte commented Feb 7, 2018

Setting the flags did the trick for me 👍

image

@BYK BYK closed this as completed in #5059 Mar 7, 2018
BYK pushed a commit that referenced this issue Mar 7, 2018
)

**Summary**

Closes #5054, closes #4876, closes #5152.

Currently when running `yarn --ignore-optional` required dependencies can be marked as optional because they exist in the tree of `optionalDependencies` of one of the dependencies (but it's required by some other non-optional dependency).

For example: `once` depends on `wrappy`, but `wrappy` is also in the chain from `fsevents` (`glob` -> `inflight` -> `wrappy`) which is an optional dependency of `chokidar`. So if the first `wrappy` comes from an "optional chain" when yarn processes it, it ends up being marked as `_reference.optional = true`, and the package hoister therefore doesn't mark it as required.

When running the released version of yarn with https://github.com/spalger/reproduce-issues/blob/master/yarn-ignores-non-optional-dependencies you'll see

```
~/dev/reproduce-issues/yarn-ignores-non-optional-dependencies
$ yarn --ignore-optional
yarn install v1.3.2-20171204.1856
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 0.17s.

~/dev/reproduce-issues/yarn-ignores-non-optional-dependencies
$ yarn check --verify-tree
yarn check v1.3.2-20171204.1856
error "once#wrappy" not installed
error An unexpected error occurred: "Found 1 errors.".
```

This fix checks whether or not the parent is marked as required, and if so, marking the dependency as required _unless_ it's listed in the parent's `optionalDependencies`.

**Test plan**

I tested this implementation with both https://github.com/spalger/reproduce-issues/blob/master/yarn-ignores-non-optional-dependencies and #4876, and in both cases `yarn check --verify-tree` succeeded after running `yarn --ignore-optional`.

Also added automated tests.
@alancwoo
Copy link

alancwoo commented Aug 8, 2018

I'm unfortunately running into this problem still, even setting the YARN_FLAGS to --no-ignore-optional and various specific versions of yarn, 1.4.0, 1.3.2 etc, everything fails on netlify as it seems to be missing the semver module:

3:11:30 PM: > node build/build.js
3:11:30 PM: module.js:549
3:11:30 PM: throw err;
3:11:30 PM: ^
3:11:30 PM: Error: Cannot find module 'semver'

This build works perfectly locally, so I'm not sure what can be done if all the build flags are being set properly according to this thread.

I'm even having this problem with a completely empty vue-init webpack project.

@fool
Copy link

fool commented Aug 8, 2018 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants