Yarn --production tries to resolve devDeps as well

[Diokuz]

Do you want to request a feature or report a bug?
This is a bug, or unexpected behaviour.

What is the current behavior?

1. Have a package.json which have devDependencies and one of them are private
2. Run yarn --production
3. Yarn tries to resolve all packages and fails, because have no access to private packages

To fix that I need additional workaround with authToken (the problem arises in docker build process).

What is the expected behavior?
Yarn should only resolve production packages when --production flag exposed.

Please mention your node.js, yarn and operating system version.

node -v
v8.1.0
yarn --version
0.24.6

[rally25rs]

This happens because Yarn needs to consider the devDependencies so that it can produce a deterministic build between prod and dev builds (packages will be hoisted to the same locations).

[Diokuz]

Not sure I got it... lets consider two situations:

1. (rm -rf ./node_modules) We have package.json with deps and devDeps, and we run yarn --production
2. (rm -rf ./node_modules) We remove all devDeps from that package.json and run yarn

Will there be the difference in node_modules? If yes, why?

[Diokuz]

Actually, I think this is related to my another problem #1379

[rally25rs]

Possibly. Consider something like:

dependencies {
  [email protected]
}
devDependencies {
  [email protected]
}

And A also has a dependency:

dependencies {
  [email protected]
}

---

With devDependencies, the dependency tree is:

[email protected]
|- [email protected]
[email protected]

so A@1 can not longer be hoisted to top level, resulting in:

node_modules/
|- A/ (1.0.0)
|  |- node_modules/
|     |- B/ (1.0.0)
|- B/ (2.0.0)

---

Without devDependencies, the dependency tree is:

[email protected]
|- [email protected]

But B can be hoisted to the top level, so the result is:

node_modules/
|- A/ (1.0.0)
|- B/ (1.0.0)

(This is non-deterministic because B@1 could end up in 2 different paths; either in node_modules/B or node_modules/A/node_modules/B depending on the type of install)

---

What Yarn wants to do for a deterministic build is to make sure that [email protected] is installed to the same location every time, so a --production build should notice that [email protected] will be hoisted to top level if the devDependencies were there, so the resulting install should really be:

node_modules/
|- A/ (1.0.0)
   |- node_modules/
      |- B/ (1.0.0)

---

I hope that makes sense :) Basically all the devDependencies are built into the dependency tree when figuring out where to hoist packages to, and calculating that dependency tree needs the metadata for all those packages.

[rally25rs]

@Diokuz I haven't tried it myself, but you might see if NPM5 has the same issue. If I understand the NPM5 lockfile correctly, I think it saves the resolved / hoisted position for each package in their lockfile (Yarn does not), so it might not need to query the metadata for your private repos. As much as I hate to steer anyone toward NPM, this might be a case where NPM5 works better. It's worth a test anyway!

You might also be able to solve this using the Yarn offline mirror, since the metadata and package would be saved there and shouldn't have to query the actual server. However that might have other security concerns for you, since a copy of your private repo code would be in that mirror directory.

[Diokuz]

@rally25rs thank you so much for explanation!

So, I think I get my answer. The only question now: will that be done in near future? I mean, saving positions in lock file.

[arcanis]

It's not only a question of position. Let's say you have this package.json:

{
    "dependencies": {
        "a": "^1.0.0"
    },
    "devDependencies": {
        "b": "^1.0.0"
    }
}

And that the following are true:

• [email protected] and [email protected] both exist
• [email protected] exists and depends on [email protected]

If we don't resolve devDependencies, then in prod we'll have the following:

/node_modules/
/node_modules/a (@1.1.0)

And in dev, because of the deduplication optimization, we'll have the following:

/node_modules/
/node_modules/a (@1.0.0)
/node_modules/b (@1.0.0)

Note that we end up using a different version of a in prod than in dev - that might not be good.

[arcanis]

Now that I think about this, maybe in practice it could work, since the lockfile would still pin the version - at least assuming that this lockfile has been generated in dev mode.

Still, I'm not convinced this is an issue worth fixing right now. I feel like this is something that could probably be better solved through your build environment.

[Diokuz]

Well, we have solved our problem via NPM_TOKEN env var. But if npm is better here, why dont you want to make yarn even better in near (or far) future?)

Anyway, thanks for answers)

[kentor]

@arcanis We don't control the build environment. For example I am using Netlify which does not have access to private dependencies through github (not npm private registry, so I can't do something like NPM_TOKEN). I thought it would be ok to move the private deps to devDependencies but yarn in production mode still tries to resolve it. The lock file should be enough to determine the positions of the packages.

Edit: Just tested npm5. It does not need to resolve devDependencies

[saikojosh]

I love Yarn but this has repeatedly been raised as an issue since last year and is always closed. Surely this is a common use case to have access to different packages in production and development?

My own situation is that I'm using a private Git dependency in development and in CI/production I am using a private module hosted on NPM. I don't want to give my CI system or production systems access to GitHub - Yarn should not require access to that dependency. If NPM can figure out a solution that works, so can Yarn.

For now I am going to have to switch back to NPM.

[adrienharnay]

Hey, bumping into this issue at well.

yarn --version
0.27.5

How to reproduce:

• Create an empty folder, cd into it
• Create a package.json:

{
  "devDependencies": {
    "doesnt-exist": "file:./nope"
  }
}

• Run: yarn --production

npm i --production does work.

[BYK]

Have any of you tried using --prefer-offline or --offline? That should work IMO.

[adrienharnay]

Just tried, doesn't seem to work.

[olingern]

@BYK Opened a PR for this fix: #4210

[thedaniel]

That PR has been closed without merging, any idea when a new one might make it onto the schedule?

[olingern]

The issue is slightly complex. I believe the solution could be attaching metadata of sorts to the lockfile, but I defer to @arcanis @BYK

[BYK]

The thing is, we still need those package.json files to have a consistent image of the resolved file tree, even if those dependencies won't be installed.

An alternative would be to use the offline mirror feature that would include the dev dependency packages or, their stripped-down versions which only contains the package.json files.

@olingern's suggestion also sounds good but I don't know what the implications for that are. @kaylieEB, @arcanis, @bestander - any thoughts?

[xPaw]

This is still an issue, and can even stop the installation when a dev dependency fails to resolve (e.g. flatmap-stream).

Repro:

git clone https://github.com/thelounge/thelounge
git checkout v3.0.0-rc.5
yarn --production

Which ends up with:

yarn install v1.12.3
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
error https://registry.yarnpkg.com/flatmap-stream/-/flatmap-stream-0.1.1.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "Unexpected end of data"

flatmap-stream along with event-stream are only required in a devDependency, so there is absolutely no reason as to why Yarn should try to resolve them.

[mkopinsky]

I ran into this in a similar scenario. My build server has an old version of node which isn't compatible with eslint 5 (which is a devDependency), but yarn install --production still fails with

error [email protected]: The engine "node" is incompatible with this module. Expected version "^6.14.0 || ^8.10.0 || >=9.10.0".

[olingern]

@mkopinsky This doesn't have anything to do with production. You need to give your package.json a range for valid node versions if your build server is running a different version of node.

I would recommend being consistent, though.

https://yarnpkg.com/lang/en/docs/package-json/#toc-engines

From the docs:

{
  "engines": {
    "node": ">=4.4.7 <7.0.0"
  }
}

[mkopinsky]

I'm not sure we're understanding each other.

eslint's engine constraint is totally fine. My issue is that I wouldn't expect the engine constraint of a dev dependency to block a production install.

[arcanis]

I think this is purely subjective and doesn't take the overall architecture in mind. I'm tempted to close as wontfix.

[arcanis]

Giving it a second thought the problem you mention is a bug, @mkopinsky, but it's a different topic than what is discussed in this thread. You should open a new issue.

[akshetpandey]

I ran into an issue related to this, since yarn tries to resolve devDependencies when installing in production it fails when the devDependency is not resolvable. This happens if the devDeps are part of a unpublished mono-repo but aren't needed anymore after the compilation is done and a package published.

I second a vote for embedding this information in the yarn.lock file so devDeps are required to be resolvable.

[billouboq]

This is a really big problem when deploying to Heroku because bundle size is capped at 500mo max and some devDependencies are really big.. Can we get rid of the devDependencies installation ?

[BYK]

@billouboq

This is a really big problem when deploying to Heroku because bundle size is capped at 500mo max and some devDependencies are really big.. Can we get rid of the devDependencies installation ?

The issue is not installing devDependencies in production mode, it is merely resolving them. If your Heroku instance is trying to install devDependencies, make sure your NODE_ENV is set to production.

@akshetpandey

I second a vote for embedding this information in the yarn.lock file so devDeps are required to be resolvable.

This should already be the case for the initial creation of the lock file. After that, you can run Yarn with --prefer-offline and --frozen-lockfile options to skip another resolution.

[Akryum]

@BYK --prefer-offline --frozen-lockfile doesn't work :(

[armordog]

@BYK By "it is merely resolving them" do you mean it is "adding them to node_modules"?
Because for most packages that indistinguishable from installing them.

If you don't mean that, then this flag doesn't work at all (v1.21.1)

[imagine10255]

Very puzzled, the packages appearing in dependencies and devDependencies should be unique.
How can there be the same package?

[BYK]

@BYK By "it is merely resolving them" do you mean it is "adding them to node_modules"?
Because for most packages that indistinguishable from installing them.

Resolving means reading their package.json to determine the whole install tree. It does not put them under node_modules. That step is the linking step.

If you don't mean that, then this flag doesn't work at all (v1.21.1)

It does since we have tests for this behavior. It maybe getting overridden by your environment variables or something else.

[NotWearingPants]

Temporary fixes (deleting all the dev dependencies from package.json):

jq 'del(.devDependencies)' package.json > package.json.tmp && mv package.json.tmp package.json
    && yarn install --prod --frozen-lockfile

(requires installing jq)

or

awk '/},/ { p = 0 } { if (!p) { print $0 } } /"devDependencies":/ { p = 1 }' package.json > package.json.tmp && mv package.json.tmp package.json
    && yarn install --prod --frozen-lockfile

[dvdotsenko]

Use of --frozen-lockfile (as in yarn install --prod --frozen-lockfile ) AVOIDS pulling (downloading, "installing") dev dependencies. You just need to resolve yarn.lock elsewhere before.

Tested with yarn 1.17.3

[IlyaSemenov]

@dvdotsenko At least not if a dev dependency points to an external Git repo... Yarn wants to fetch the remote repo when I run yarn install --production --frozen-lockfile in a CI/CD pipeline.

[mattnathan]

I know this issue is old, but as this isn't resolved and my specific case isn't mentioned I thought I'd chip in.

The project I work on is a js monorepo using Firebase (Firestore, functions, hosting, etc). We've defined .d.ts definitions for the data we store in Firestore and keep those in a types directory in the project root. The functions located in functions/ reference these types as devDependencies using file:../types.

During deploy yarn install --production is run on the firebase server using the functions package.json - this is completely out of our control to configure. This doesn't work because the file:../types fails to resolve even though it's not needed for production, only the functions/ directory is actually uploaded.

Obviously we could use one of the work arounds mentioned here (copy/link/embed the files, use NPM5, etc) but they are all work arounds involving extra work and are less than ideal.

[kivi]

This is not fixed for yarn 1.22.5

I am using node:14-buster-slim in my Dockerfile
yarn install --production --frozen-lockfile also installs devDependencies and that is resulting a huge image.

For now I have a workaround, thanks to @NotWearingPants #3630 (comment)

[avshyz-gs]

This is also not fixed in 1.22.17.
An certain eslint plugin requires a different node version from our production image's node version.
I'd follow NotWearingPants's comment.

[felsspat]

Really, no solution after 6 years? This is a real problem :(

[K-Mistele]

I'm running into this too, running yarn install --production --frozen-lockfile still results in a docker image that contains all of my dev dependencies in my node_modules directory, resulting in the image being 10x larger than with npm