yarn probably shouldn't cache packages resolved with a file path

[donaldpipowitch]

Do you want to request a feature or report a bug?

I guess a bug.

What is the current behavior?
If the current behavior is a bug, please provide the steps to reproduce.

Say you have the following project structure:

component-foo/
└── package.json
└── index.js

yarn-test/
└── package.json

With the following files:

component-foo/package.json:

{
  "name": "component-foo",
  "version": "1.0.0",
  "private": true,
  "main": "index.js"
}

component-foo/index.js:

console.log('foo');

yarn-test/package.json:

{
  "name": "yarn-test",
  "version": "1.0.0",
  "private": true,
  "dependencies": {
    "component-foo": "file:../component-foo"
  }
}

Now you run $ yarn install inside yarn-test/ and yarn-test/node_modules/component-foo/index.js is:

console.log('foo');

Now remove yarn-test/node_modules/ and yarn-test/yarn.lock and change component-foo/index.js to this:

console.log('bar');

Now you run $ yarn install inside yarn-test/ again and yarn-test/node_modules/component-foo/index.js will be:

console.log('foo');

It uses the cached version of component-foo, but component-foo/index.js has changed.

What is the expected behavior?

I'd expect that yarn-test/node_modules/component-foo/index.js should be this at the end:

console.log('bar');

Maybe packages installed with local paths like file:../ shouldn't be cached at all, if we can't figure out, if they have changed.

(FYI: It looks like npm doesn't cache them.)

Please mention your node.js, yarn and operating system version.

$ node -v
v6.9.1

$ yarn -V
0.18.0

macOS 10.12.1

[hantuzun]

This fooled me as well. There must be a way to get around this without clearing all the cache.

I think there could the three ways to make yarn usable for this case:

1. @donaldpipowitch 's suggestion of ignoring all local dependencies.
2. Reinstalling local dependencies if a file in there is modified later than when it's cached. This would require us to keep this metadata. For local dependencies we may insert lines like this to "Resolved" column: file://<path>@<cache_timestamp>
3. Ignoring packages by package names with commands such as yarn cache rm <package> and yarn cache add <package>. For all dependencies.

I'd like to see the second suggestion to be implemented. Unless, the third option could be useful for other cases as well. For instance, yarn cache add <package> can be used to refresh the cache for already cached dependencies in case something could go wrong while downloading a dependency.

[satya164]

@hantuzun why cache local dependencies at all? they are local anyways, so it will be fast regardless cached or not.

[hantuzun]

@satya164 you're right. Although, I'm more inclined to the third approach would help if a dependency from network is modified intentionally.

[satya164]

Something like yarn cache ignore <package> will be useful. But I think they don't have to be mutually exclusive. Ignoring a package is useful, but it requires manual work. File dependencies could be made to work without any additional effort if they were ignored by default.

[donaldpipowitch]

Can someone explain me the internal logic?

My understanding:
When a dependency with file: is encountered the file-resolver.js kicks in. It says the dependency should be copied and is not hashed. Doesn't no hash mean it should not be cached already...? But the copy-fetcher.js seems to set the hash to an empty string instead of keeping null... Is this problem?

[donaldpipowitch]

@bestander or @kittens Maybe you could explain this a little bit more...? Would love to get a little help to create a PR ♥

[bestander]

Hash means the md5 hash that is used for tarball-fetcher, for example.
This hash is then added to yarn.lock file for future verifications and also appended to folder name when saving the unzipped folder in cache.
You are digging in the right direction, thanks for looking into this, a PR is very much welcome.
You can probably start with a PR that adds a broken unit tests

[donaldpipowitch]

You can probably start with a PR that adds a broken unit tests

Okay, thank you for your response. Should I ping you as a reviewer in the PR or should I ping someone else (or no one)...?

[bestander]

Yeah, ping me

[satya164]

@bestander probably this issue shouldn't be closed since it's not fixed yet?

[donaldpipowitch]

Yes, it should be re-opened. It was closed because I wrote " fixes #2165" in the title of my PR. In the beginning I thought it would be an ongoing PR, but to fix this bug we want two PRs: the first one added a unit test (the assertion which would fail is not really active, so CI doesn't blow up) and the second one will actually fix it.

[bestander]

Sorry, github closes issues when PR is merged

[moimikey]

so obviously this is still a problem? it's quite annoying to develop with, to be honest. it's causing a bit of a disruption for me on a personal level at work, where i'm making use of file: to create a modular development environment. the sucky part is that every local package that I edit (using the file: path in package.json), requires doing this, in order to pull down the refreshed contents:

editing the contents of my eslint-config-base-eslint package

yarn cache clean && rm -rf node_modules/eslint-config-base-eslint && yarn install --force && yarn lint

[bestander]

Everyone is welcome to contribute to the project.
It could be anything - submitting a broken integration test for this case, a fix or convincing someone to work on the fix.
If you want to chat about the best way to approach that, you can find help on the discord channel.

[bestander]

I actually think that the fix should be 10-15 lines of code, it will probably save a lot of time fixing it sooner

[ghost]

Can we reopen this issue? We are facing the same problem. Two projects referencing another package via file:. After some builds in our CI/CD server, the cache folder is starting to occupy a lot of space.

[alxtz]

I think the link protocol is the best fix for this, file:// doesn't work since it still requires manually clean caches & force installing

#2165 (comment)

just make your dependency something like this

"<package>": "link:./libs/<package>"

[wKich]

@alxtz does links protocol works with packages in tgz?

[silasdavis]

I think the link protocol is the best fix for this, file:// doesn't work since it still requires manually clean caches & force installing

#2165 (comment)

just make your dependency something like this

"<package>": "link:./libs/<package>"

Thanks for this this replicates the behaviour of NPM which will symlink file:.. references in node_modules. It does not appear to be documented, at least not here where I would expect to find it: https://yarnpkg.com/lang/en/docs/package-json/

[ojboj]

Sadly, link can't be used in all cases.

For example, I need a shared/peer dependency from my SDK package, which when I make changes, I would link for local dev work.
With link, yarn does not realise that the dependency is a shared/peer dependency and uses the local package where the symlink is, which is incorrect.

I've been using yarn pack alongside yarn add file:<path_to_packed_tgz> to get around this.
Whilst I can continue to rename the package every time I pack it and paste it into my repo, so as not to generate the same hash, as per @wKich fantastic find, it's damn annoying.

I forked the repo and put an extra clause in the if statement to stop yarn from ever loading a local .tgz package from its cache if it's specified using yarn add file:<path>.

const dest = this.config.getTemp(crypto.hash(url));
// If specified using file: never load from cache
if (!url.match(/file:/) && (await this.config.isValidModuleDest(dest))) {
  // load from local cache
} else {
  // continue as if it's a new package
}

I can make a PR if people want it, though I've never done that before and it's a pretty hacky fix.
Could someone confirm if this approach would continue to add the local packages to yarn's cache or not, please?

[souporserious]

@ojboj for now, yalc might help until this gets a proper fix. It has been working really well for me to test packages locally before publishing.

[ojboj]

@souporserious That is exactly what I needed/wanted, thank you so much!

[bobber205]

Insane this is still an issue after so many years.

[wKich]

Is it fixed in [email protected]?

[souporserious]

@wKich I believe so! I haven't personally tested it, but it looks like they solve local development through the new portal protocol.

[garyo]

I still get "unpack in same destination" error using the link: protocol, and it's referencing my cache dir, so it seems to me yarn is still caching link: dependencies. I'm referencing 2 copies of the same local package, copied into the .deps/ folder of each app that uses it -- front-end and renderer in the error below. (Can't link to the original for unrelated reasons)

warning Pattern ["@horizon/common@link:packages/front-end/.deps/@horizon/common"] is trying to unpack in the same destination "/home/garyo/.cache/yarn/v6/[email protected]/node_modules/@horizon/common" as pattern ["@horizon/common@link:packages/renderer/.deps/@horizon/common"]. This could result in non-deterministic behavior, skipping.

[thejohnfreeman]

This issue is not fixed by the link: and portal: protocols for those of us like @ojboj and me who install packages from a tarball instead of a directory, e.g. file:path/to/dependency.tgz. One annoying workaround is to use a throwaway cache directory like yarn install --cache-folder $(mktemp -d). My preferred workaround is to just bust the cache for every offline dependency: rm -rf $(yarn cache dir)/.tmp. Thanks to @wKich for the detective work.

[geoidesic]

Why is this closed?
I still get this issue in Yarn 1.22.19... Yarn caches local packages, which is not helpful for development.

[justsaul]

Believe it or not, it's still an issue with yarn berry (3+). I wish there were better alternatives to deal with devDeps.

[hoegge]

It is actually 4+ years #6037

[sickdyd]

When developing a package with React 18 I had various issues using yarn link, so I just rely on building a local package and installing in my React app, this issue is pretty annoying... there should be some way to forcefully reinstalling the single local package ignoring the cache instead of using yarn cache clean.