make --pure-lockfile default for install

[bestander]

Do you want to request a feature or report a bug?

feature

What is the current behavior?

Not passing --pure-lockfile for install command confuses me because it modifies the lock file while installing node_modules.
We agreed on semantics that add/upgrade/remove are to change dependencies and install is to consistently rebuild node_modules from lockfile.

Consistency gets lost when lockfile is modified depending on environment (version of yarn currently installed).

What is the expected behavior?

Not write yarn.lock or package.json when doing yarn install.
To update yarn.lock use yarn upgrade

Please mention your node.js, yarn and operating system version.

yarn 0.14

[eliihen]

I agree. There should be a discussion on why yarn install writes a lockfile by default on the first place, as it seems to be at odds with the entire lockfile concept. Why have a lockfile if it is not locking versions by default?

There is a case for yarn install creating a lockfile if none is present, i.e. when someone is converting a project to use yarn, but the rationale for always writing it is not clear. I agree with @bestander's opinion that only mutating actions should update the lockfile by default, i.e. add/upgrade/remove.

[ide]

Should there be a way to modify the lockfile without add/remove/upgrade ex: in the scenario when you upgrade Yarn and it uses a new lockfile version?

[bestander]

I suppose the option could be inveresed

yarn install --save-lockfile

On 17 October 2016 at 18:53, James Ide [email protected] wrote:

Should there be a way to modify the lockfile without add/remove/upgrade
ex: in the scenario when you upgrade Yarn and it uses a new lockfile
version?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#570 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACBdWJLpdvqiwcBwqE4KB3x3f4oCn_nVks5q07YYgaJpZM4KSlSw
.

[fingermark]

I am also confused by this. What is the reasoning for the current default behavior?

[bestander]

Afaik, there was no strong reasons for the default behavior.
The idea, I suppose, is to keep people's lockfiles "evergreen".

[bestander]

BTW PR is welcome

[jamiebuilds]

I think the reason for it was that yarn was originally designed with a single install command which was split into install/add/upgrade

[Guuz]

So, as to check if i understand this correctly:

yarn installs all dependencies and also modifies the lockfile. On a CI server you should use yarn install --pure-lockfile?
Why does the lockfile get modified during an install? Since you are not upgrading anything... Yarn should just install the packages as described in the lockfile, right?

Thanks for the explanation!

[sebmck]

The problem is that if the lockfile is pure by default then people are going to forget to update it since it'd be a separate command.

[idris]

@kittens Shouldn't the lock file only be updated upon add/remove/upgrade of any packages? Those should always upgrade the lock file, as well as an initial install.

[eliihen]

The problem is that if the lockfile is pure by default then people are going to forget to update it since it'd be a separate command

That being a problem depends on what you consider to be the main objective of a package manager.

In my opinion, one of the roles a package manager fills is make it as easy as possible to get started with developing on a project. A simple yarn install should get all the packages you need to start developing, without any confusion involved.

With npm I've had many instances of developers join a project, only to find a project does not work on their machine. These instances have occurred due to transient dependencies bumping versions to versions with breaking changes or simply not following semver. I had hoped yarn would solve these issues, but if the takeaway is that all developers on a project should run yarn install --pure-lockfile to be 100% sure that the project is going to build, then that is not the case.

Another role of a package manager is giving projects control of their dependencies. If it is made pure by default, developers are able to have a look at yarn outdated to see the outdated versions and then review the change notes, avoiding any breaking changes. This would give developers full control to only bump versions in a given release timeframe instead of banning developers from doing git commit -a to avoid accidental lockfile commits.

[fredva]

I agree with everything @esphen says. I am surprised the pure behavior is not the default in Yarn – I thought this kind of consistency was the major benefit Yarn had over NPM. This should be the most compelling reason for switching from NPM the way I see it.

[adamchainz]

Totally surprised us by breaking build after we started using yarn for a few days. I honestly thought --pure-lockfile was the default behaviour after reading much of the documentation and about how it's better than npm with shrinkwrap. Please make default :)

[sebmck]

@ide Imagine a scenario where someone is just using npm and updates package.json, how is yarn.lock going to be updated?

[sebmck]

Can someone please write up the scenarios that lead to the lockfile being modified unexpectedly? This change is a serious one and does make the lockfile a second class citizen by requiring updates to it to be explicit which means a lot of overhead in remembering what operations result in it being updated etc.

[adamchainz]

More info on the above: our build has coffeescript from Github as a subdependency. coffeescript pushed some commits and we got a modified yarn.lock in our build process from running just yarn install:

diff --git a/foo/yarn.lock b/foo/yarn.lock
index ec667fa..bb1f6ae 100644
--- a/foo/yarn.lock
+++ b/foo/yarn.lock
@@ -930,9 +930,9 @@ code-point-at@^1.0.0:
   version "1.6.3"
   resolved "https://registry.yarnpkg.com/coffee-script/-/coffee-script-1.6.3.tgz#6355d32cf1b04cdff6b484e5e711782b2f0c39be"

-"coffee-script@github:jashkenas/coffeescript":
+coffee-script@jashkenas/coffeescript:
   version "1.11.1"
-  resolved "https://codeload.github.com/jashkenas/coffeescript/tar.gz/887052de079b2f0af9f080031a00bb7544eaca08"
+  resolved "https://codeload.github.com/jashkenas/coffeescript/tar.gz/0d132318ce8f7116a436d97db1f2a5c8b1dedf28"

 [email protected]:
   version "0.3.0"

[bestander]

Can someone please write up the scenarios that lead to the lockfile being modified unexpectedly? This change is a serious one and does make the lockfile a second class citizen by requiring updates to it to be explicit which means a lot of overhead in remembering what operations result in it being updated etc.

I perceive yarn install as a command that builds node_modules for me.
It is opposite to yarn add and yarn remove that modify package.json, yarn.lock and cleanup node_modules.
Opposed to add and remove I run install 100 times more often especially in CI where I never expect to have side effects.

Examples when I don't expect things changing:

1. I am on Yarn 0.15.0, my team mates are on Yarn 0.16.0.
   Because 0.16.0 added spaces between entries in yarn.lock every time I run yarn install against yarn.lock that was generated by my team mates I get a modified yarn.lock file that I need to remember not to commit.
   And vice versa.
2. My other build tools depend on yarn.lock as "the source of truth" of node_modules state. If it changes unexpectedly I will get non determinism in my builds

[101100]

@kittens

The problem is that if the lockfile is pure by default then people are going to forget to update it since it'd be a separate command.

Imagine a scenario where someone is just using npm and updates package.json, how is yarn.lock going to be updated?

If we assume that yarn install should not update yarn.lock, then it should also fail if yarn.lock is out of sync with package.json to highlight the fact that a yarn install --save-lockfile is needed to bring everything back in sync.

[jasonLaster]

+1 yarn install should not mutate the yarn.lock

1. The debugger is an oss app. We want contributors to be able to yarn install and get the good versions. We've had people npm install and say it's breaking because of transitive properties . With yarn install, contributors yarn install and don't know what to do with the yarn lock changes.

I'm not worried about updating the lock file. Ideally greenkeeper would do this when deps change and we could merge the lock file change then.

[jamiebuilds]

I want to update this issue with the current thoughts about it. @kittens and I both think that --pure-lockfile should not be the default for a couple of reasons.

It starts with how people add/remove/update dependencies. While there are commands for it, it is common practice to manually update the package.json either by hand or by another tool like Lerna.

Once you have manually modified the package.json the expectation in both Yarn and npm is that when you run another install it syncs with the package.json. In that sense yarn install could almost be renamed yarn sync.

On the topic of syncing, when you run an install with new dependencies you expect the node_modules directory to reflect those changes. Since yarn.lock acts as an assistant to node_modules you should expect it to stay in sync the same way.

Your package.json is the ultimate source of truth, that is your interface to yarn, it's your configuration and it's the only thing you should ever be concerned with. In an ideal world you simply commit your yarn.lock and then never have to think about it again.

---

On a side note I believe many people who are voicing support for this issue are confused about what's actually being discussed here.

Using --pure-lockfile by default does not mean that Yarn does not produce consistent and reliable results. The same package.json will result in the same yarn.lock which will result in the same node_modules 100% of the time.

When you update your package.json your yarn.lock file updates and then your node_modules updates. That is a very natural order to things and we should keep it that way

---

In regards to CI being able to get different dependencies when you have updated your package.json but have not run yarn install to sync everything which I'm sure someone will bring up (although I do not see as an issue)-- myself and others have been speaking to various CI tools about integrating Yarn, we can easily push for them to use --pure-lockfile by default if people see it as a big issue.

---

If we were to make this change it would have a negative impact far more often when changing dependencies. For the reasons I've listed I say we should close this issue.

[Pauan]

@thejameskyle I would appreciate it if you could clarify something:

1. A developer has a package.json which contains a dependency "foo": "^1.0.0"
2. The developer runs yarn install. The foo package is currently version 1.0.0, so it creates a yarn.lock file which locks in [email protected]
3. The developer adds yarn.lock to Git.
4. The developer runs unit tests on their local copy of the repo, everything works fine.
5. The developer pushes their repo to CI (e.g. Travis).
6. CI runs yarn install, but foo has now updated to version 1.1.0, so Yarn installs [email protected] and overwrites yarn.lock with the new version of foo
7. The CI breaks because foo had a breaking change in version 1.1.0

---

Here's a similar situation:

1. A developer has a package.json which contains a dependency "foo": "^1.0.0", which is locked in as [email protected], and yarn.lock is saved in Git.
2. Unit tests work fine on the developer's local copy of the repo.
3. A contributor clones the repo with the intent of making a modification + pull request.
4. When the contributor runs yarn install they get version [email protected] which causes yarn.lock to be updated.
5. Now the contributor's build is broken because foo had a breaking change in version 1.1.0

---

I think those are the kind of situations that most people are worried about.

So if you could clarify that the current behavior of yarn install does not have the above problems, I think that would remove most of our fears. 👍

[bestander]

@jspiro, thanks for writing this up.
There are a few issues raised here.
It would be better to open each issue separately otherwise they will be lost in the comments.

Are you on latest version of Yarn?
As of 0.18-0.19 we don't see modifications to yarn.lock files between machines.

Questions:

Is it being said that despite the lockfile being changed, the contents of node_modules will always be identical to when it was generated? I don't believe this is the case but if it is, then I understand the confusion in this thread -- it would mean that yarn does the right thing despite the appearance that it does not.

Dev and optional dependencies can be left out for the same lockfile.
But the ones that are bing installed, except for platform specific packages, node_modules should have identical packages in identical places.

When package.json changes, the lockfile is regenerated. Couldn't that unintentionally change a lot of dependencies depending on the state of that particular programmer's node_modules? Yarn should determine a delta and try to preserve existing locks as best as it can (if it doesn't already).

That is a nice feature request, would love to see a PR for that.

Why does yarn add specify versions in package.json with a ^? Again, I understood yarn's promise was to freeze dependencies.

That reflects npm's behavior.
You can do yarn add [email protected] or yarn add is-array --exact for exact version.
Maybe at some point we should make exact versions default, this can be a discussion in an RFC.

When a random package is deleted in node_modules, yarn install says success without reinstalling it. When a lot of them are gone, it reinstalls them. npm is a bit more thorough in this regard.

Yarn runs a quick shallow check by default.
Doing a deeper check will be slower but we are working on it, I have an idea how we could do a quick deep check.
You are not supposed to touch files in node_modules though, verifying each file for modification would result in a very slow install experience.
If you want to skip shallow check then remove node_modules/.yarn-integrity file before installation. This is not official and subject to change.
An official way is to run yarn install --force, it would force full install but it would rewrite yarn.lock as a side effect.

The lockfile tends to get regenerated if you delete node_modules and do a clean install (which is literally the opposite of what you would expect -- I expect it to install exactly what's in the lockfile and do absolutely nothing else)

Haven't seen this for a while.
Open an issue and cc me if this can be reproduced.

If you delete the lockfile without touching package or node_modules after a clean install, yarn regenerates it and its usually very different than the previous version. This is like a compiler producing different code each time you run it despite changing nothing.

After some time new versions of transitive dependencies might have been released.
Because of that the structure of node_modules can change significantly because of the hoisting logic.
That works as designed.
There is import command coming #2580.
That would allow you generating a lockfile from existing node_modules.

@jspiro, Yarn is a young community driven project, your PRs to make it better work for you are welcome.

[FezVrasta]

Any chance to get at least an option to set the desired default behavior?

[bestander]

Currently we are fixing this issue #3490, sometimes yarn install may cause lockfile to be optimized which is not expected behavior and we will fix it.
That might be the reason why you are asking for this change, otherwise yarn.lock file should change only if you make changes to package.json manually.

You can set --pure-lockfile/--frozen-lockfile to true in .yarnrc and it will be appended to the install command by default:

--install.pure-lockfile true

[FezVrasta]

My problem is that if I don't use pure-lockfile I get the wrong version of the dependencies installed. It's not related the unwanted yarn.lock changes

[bestander]

Can you submit an issue with repro steps?
We'll sort it out

[k0pernikus]

I was bitten by this as well when a package.json and yarn.lock got out of sync due to a developer mistakenly adding a dependency via npm install --save instead of yarn add.

I disagree though that pure-lockfile should be the default and argue that rather frozen-lockfile should be the default for yarn install.

As frozen-lockfile yields an error message if the yarn.lock and package.json are out of sync. The frozen-lockfile is therefore very helpful on a build machine (i.e. jenkins) as it will mark those build, as should be expected, as a failure.

It's then up to the developer to decide which version to add in the package.json / yarn.lock.

The unfortunate default of yarn install will just fetch the most current version of the not-yet locked in dependencies, and write an updated version yarn.lock, which will never be part of the project. Therefore allowing future breaks of the build due to a unexpected version bump. That's the very reason we have a lockfile to begin with.

The gist should be though:

Only commands like add, remove, and upgrade should mutate the yarn.lock.

install should just do that, namely either install the dependencies in their locked in version or fail if it detects a mismatch between the package.json and yarn.lock. (The only exception being if there's no yarn.lock in the first place. Then, and only then, it may create one, yet it never, ever should touch it again.)

[BYK]

The frozen-lockfile is therefore very helpful on build machine (i.e. jenkins) as those build will fail.

I think we can enable this automatically when we detect we are in CI mode?

[k0pernikus]

@BYK I didn't realize this issue was closed before adding in here. Should I maybe open a new one or can this be reopened?

[BYK]

I'd say open a new one ☺️

[trusktr]

I agree with @thejameskyle and @kittens that yarn.lock should be kept in sync with package.json automatically

Not sure if this has been said, but just in case: you don't have to invalidate the entire yarn.lock when anything in package.json changes. You can invalidate only the dependencies of only packages that were modified inside of package.json. F.e. if you updated only TypeScript, on the dependencies of TypeScript would need to be modified (with considerations respecting other unchanged packages).