Moving repositories without recreating yarn.lock?

[imsnif]

Do you want to request a feature or report a bug?

Bug (I think!)

What is the current behavior?
We're trying to move to a new internal npm repository (specifically, moving from Sinopia to Nexus). We've been using yarn extensively for all of our packages. So when moving, we have to do a search/replace of the repository-url in the yarn.lock file.
While not ideal, this is not a problem.

The problem is with the sha checksum of each package. The checksum includes (from the generated package.json file of each package in node_modules) the repository url. Meaning it is by definition different and yarn fails with:
error <package_in_new_repository>.tgz: Bad hash. Expected "<old_checksum>" but got "<new_checksum>"

Looking through the documentation and the issues I could not find any mention of a solution for this. Am I missing something? Is there a better solution than updating all the checksums in addition to the repository urls in yarn.lock?

If the current behavior is a bug, please provide the steps to reproduce.

1. install a private package through one private repository (Sinopia/Verdaccio/Nexus/etc.) and save it to package.json and yarn.lock
2. delete node_modules and yarn cache, change the repository url in yarn.lock to a different private repository
3. try to install the package again

What is the expected behavior?
If indeed I did not miss an existing option to do so, I think a yarn --update-checksum flag would be nice. I'd be willing to work out a PR.

[csvan]

There is an ongoing RFC that may resolve your issue: yarnpkg/rfcs#84 (comment)

[bestander]

I don't see tarball fetcher using URL when checking hash

yarn/src/fetchers/tarball-fetcher.js

Line 82 in 5386c41

const expectHash = this.hash;

.
Only tarball contents should matter.

Are you sure that when you switch to another private repository the tarball content does not change?

[bestander]

On the other hand I like the idea --update-checksum, a few times when dealing with local packages I needed to update the hash of a dependency in yarn.lock file.
Would you mind sending a short RFC/PR for this change?
It will totally be useful.

[imsnif]

Hey @bestander - so this is apparently a bit of an edge case:
The checksums can potentially differ because of this field in package.json:
https://docs.npmjs.com/all#resolved
eg:

// package.json
// ...
"_resolved": "https://old-repo:1337/package-name/-/package-name-1.1.1.tgz",
// ...

In such cases the checksum can be affected (I think this happens mainly in setups such as Sinopia/Verdaccio where some packages are private and some are public, but I'll admit that I have not checked the use-case here thoroughly yet!)

I'd love to do a PR for --update-checksum. I'm not familiar with the process though, do I need to write an RFC first or can I jump straight in?

[bestander]

RFC is a chance to get comments from other contributors, sometimes people notice edge cases or the opposite - suggest a more generic solution that solves other problems.

This change is quite small, non breaking and quite focused though, I think a good PR with a decent description and unit tests would be enough.

[bestander]

Back you your problem, afaik _resolved field is not necessary, is it possible to strip it before you publish?

[imsnif]

It is possible, but it would complicate the repository migration. I thought an --update-checksum flag would solve this and benefit others in similar situations as well. (Currently I'm scripting the checksum update).

I'll start working on the PR and hopefully have something soon. Thanks!

[imsnif]

Come to think of it, if we strip it the checksum would still be changed.

[bestander]

Yeah, it would require a 4 step change:

1. strip checksum
2. republish to old repo
3. update yarn.lock
4. switch registry

Whatever works best for you :)

[imsnif]

Hey @bestander - I created a PR for this: #4860
It's my first one to this repo, so I hope I didn't make any faux pas :)

[imsnif]

If I may make an additional related suggestion (sorry, it just came to mind): right now, if there is no hash string at all in a package in yarn.lock (rather than a corrupted one) I think yarn leaves things as they are (i.e. with no hash string). Perhaps if --update-checksums is on, it could update it with the new checksum? I'd be happy to make another PR to that effect (I'm thinking it's out of scope for this one?)

[bestander]

Yeah, go ahead please

…

On Mon, Nov 6, 2017 at 12:00 PM Aram Drevekenin ***@***.***> wrote: If I may make an additional related suggestion (sorry, it just came to mind): right now, if there is no hash string at all in a package in yarn.lock (rather than a corrupted one) I think yarn leaves things as they are (i.e. with no hash string). Perhaps if --update-checksums is on, it could update it with the new checksum? I'd be happy to make another PR to that effect (I'm thinking it's out of scope for this one?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4817 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWCB5HJQskVFzpdyjCvp2Pxmf0Eaxks5sz2VagaJpZM4QOD8i> .

[imsnif]

I'll wait for the current one to be merged though in order to decrease the amount of chaos. :)

[kopax-polyconseil]

I don't see tarball fetcher using URL when checking hash

yarn/src/fetchers/tarball-fetcher.js

Line 82 in 5386c41

const expectHash = this.hash;

.
Only tarball contents should matter.
Are you sure that when you switch to another private repository the tarball content does not change?

The checksum of the yarn lock change, that's already a problem, another problem is the git repo not being up to date after change. A workaround would be to be able to specifiy the lock file, and create a 2nd one with the first one. But that's also not possible.