Yarn handling of unpublished packages in lock files

[ahmadnassri]

I'm the maintainer of har-vlidator which is a direct dependency of request (and hence exists as an upstream dependency to many many packages, including yarn itself and npm cli!)

I un-published two versions from the npm registry, namely: 5.1.1, 5.1.2 to remedy the backward support for Node <= 6 and browsers.

The two versions were released back-to-back

Details of what lead to un-publishing

ℹ️ har-validator is really a simple package that wraps AJV to provide HTTP Archive Schema validation.

Last week, during a maintenance and clean up cycle, I updated a bunch of packages I maintain, to remove security warnings through bumping upstream dependencies.

The relevant change was [email protected] to [email protected]. which was published under [email protected]

🔍 ahmadnassri/node-har-validator@v5.1.0...v5.1.1#diff-b9cfc7f2cdf78a7f4b91a753d10865a2L43

according to the AJV docs, the major version bump was mainly an API change, and their docs still indicated support for "browsers and Node.js 0.10-8.x" (at the time of writing, this is still the case)_ and changelogs didn't indicate dropping support for Node <= 6 which is where the problems started.

dropping support lead to issues for downstream libraries including request and (specifically in compiling for browsers)

so, I made the decision to un-publish the released versions, namely: 5.1.1, 5.1.2 to remedy the backward support for Node <= 6 and browsers.

---

Reports started coming in, people seem to be having problems with their install step failing due to yarn looking for a 404 package version.

From my understanding: This means the yarn.lock for downstream users must have been updated to point at new version, which must be a user action! How else would downstream apps seem to be looking for 5.1.2 specifically?

so users must have manually executed yarn upgrade?

however, the indication from the community is that they didn't touch their lock files and installs "suddenly" stopped working.

I don't think there is a bug here, but I do think there is potentially mis-alignment and mis-communication on the workflows and documentation as it relates to lock files. so I'm concerned with both the install / add behavior and documentation relating to lock files and dependency pinning.

also, I'm not certain if there is some workflow path that lock files are somehow updated without the user specifically taking action?

What's more puzzling: while har-validator is a dependency of many packages, only request is the connection to all the problems reported and all the downstream issues discussed.

and request is still pointing to ~5.1.0 so how come people's install steps are looking for newer versions of har-validator?

To clarify:
existing projects: they would already have a lock file with 5.1.0
new projects: they would get 5.1.3 when adding request for the first time

so the problem must be only limited to people who initiated a new project between the publishing and and un-publishing events however, that doesn't seem to be the case from the conversations and issues surfaced.

I'm tempted to just release a 5.1.2 and 5.1.1 duplicates of 5.1.0 to stop the noise. but honestly, I'm baffled to how broken the yarn process is here and how badly lock files are misunderstood! and most importantly this doesn't address the vuln / security warnings people are getting from deep nested dependencies.

The Question:

I'm looking for guidance here on what / why and how un-publishing a package and communicating back the community needs to look like.

as dependency management tooling, Yarn could provide a more seamless path to deal with 404 on a deep dependency buried in yarn.lock (vs. a direct dependency in your package.json)

References:

• v5.1.2 backwards compatibility, and unpublished version. ahmadnassri/node-har-validator#112
• Getting a 404 on an npm install for har-validator-5.1.2.tgz ahmadnassri/node-har-validator#113

Affected Packages (that I've been linked to)

• fix har-validator dependency request/request#3054
• chore(deps): move codecov to devDeps with resolution prettier/tslint-plugin-prettier#283
• https://github.com/serverless/components/pull/348
• har-validator not found #2900
• Har-Validator cannot be found 5.1.2 - sporadic  #6690

on un-publishing packages

several people have pointed out best practices / ideas / opinions on why un-publishing packages should not be allowed, but it is, and it is part of the registry workflow and therefor a valid and legitimate step to take for package maintainers as they see fit.

we can debate opinions about best practices, but I think that's beside the point.

I do not think I've done anything technically wrong in un-publishing, and if I have, I would love to learn "why" without waving into opinions on versioning and best practices.

not to devalue a discussion on best practices, I very much welcome such a discussion, but wanted to keep the context here to any technical challenge to yarn.

yarn vs. npm

this topic can equally be opened with npm cli, however, most of the reporters and people with issues are yarn users, so I decided to post it here.

thank you all for your help!

[arcanis]

also, I'm not certain if there is some workflow path that lock files are somehow updated without the user specifically taking action?

Not that I'm aware of. I believe there are simply a lot of people using har-validator as a N-level dependency (usually through request), and as a result you unpublishing those versions affected everyone that has run yarn add request or any yarn add pkg command where pkg depends on request in some way.

[arcanis]

I do not think I've done anything technically wrong in un-publishing, and if I have, I would love to learn "why" without waving into opinions on versioning and best practices.

Anyone running yarn add something with something depending on your package will see it locked to the specific version locked when they first ran the tool. If you cause this package to become unavailable, they suddenly won't be able to install it, which can break their deployment workflow if they didn't use our offline mirror feature.

Unpublishing is imo a last-resort weapon that should never be used for simply patching bugs (which fixing the Node compatibility is), especially if your package has 35M downloads per month. It's almost guaranteed to cause pain to your users.

[Macil]

Is yarn doing anything different than npm here? In npm, removing the package-lock.json file and re-running npm install fixes the issue. In yarn, removing the yarn.lock file and re-running yarn fixes the issue. Yarn and npm only update the package-lock.json/yarn.lock file when the user runs a command to add/remove/upgrade dependencies (or when yarn/npm install is run when the lock file isn't present).

I think the reason that more yarn users complained in that thread is because only instructions to fix the issue in npm were posted (no one posted "delete yarn.lock and run yarn"), not because yarn is doing anything fundamentally different than npm.

Users complaining that the issue only suddenly popped up days/weeks after the har-validator versions were unpublished probably installed har-validator 5.1.1/5.1.2 back while they were still published, had the lockfile generated with one of those versions, had those versions cached on their machine, and only ran into the issue when trying to install the dependencies on a new machine (ie. many of the users in the thread talk about CI systems or heroku) or locally after their local cache was wiped. I'm not sure there's any evidence that har-validator 5.1.1/5.1.2 was added into anyone's lockfiles after they were unpublished. (I just tested publishing a new version of a package, unpublishing the latest version, and then checking that yarn add mypackagename picked the most recent still-published version.)