lstat error when use yarn to install nodejs package

[lingxuan630]

• Your Windows build number: windows 10 1803

• What you're doing and what's happening:
  When i use yarn to install nodejs package, it will cause error with lstat.

• What's wrong / what should be happening instead:

Error Logs:

Yarn version: 
  1.6.0

Node version: 
  8.9.1

Platform: 
  linux x64

Trace: 
  Error: ENOENT: no such file or directory, lstat '/mnt/***/node_modules/asynckit/index.js'

[logaretm]

I'm having the same issue since the April update, it happens for one project though. The rest of my projects use yarn and no issues there, other than random nodemon restarts.

I tried the same project on other machine and it also fails, it fails specifically when running jest tests. Happens on both node 9.8 and 10. Yarn version is 1.6.0

[Fumler]

Same issue here after the april update. Tried with yarn 1.3, 1.5 and 1.6, and in two different projects. Also tried with yarn installed as a windows binary as well as installed inside wsl.

Error from project 1:
error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/mnt/c/Users/<username>/.cache/yarn/v1/npm-lodash-4.17.10-1b7793cf7259ea38fb3661d4d38b3260af8ae4e7/fp/assocPath.js'".

Error from project 2:
error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/mnt/c/Users/<username>/.cache/yarn/v1/npm-debug-2.6.9-5d128515df134ff327e90a4c93f4e077a536341f/src/debug.js'".

[Slotos]

@therealkenc repro steps on build 1803 are as following

git clone https://github.com/babel/babel
cd babel
yarn

[Slotos]

The problem is almost 100% reproducible with watchman git hook running. Visit a few large git repositories, run git status in each one, npm install breaks.

[onomatopellan]

Yet another victim of #2937
Since version 1803 WSL will always enable casesensitive flag in every created folder in /mnt/*.
Adding case=off to wsl.conf should solve the problem until the bug in mixed case path is fixed.

I recommend using LxFs folders (/home/*) when possible since is the most Linux-like file system.

[therealkenc]

Thanks. That makes sense.

[Slotos]

I was struggling with this problem in /home. Should I open a new issue or build a from-the-scratch reproducible case for this one?

[therealkenc]

Go with a new one. There was no repro in the OP so we can't even be sure @onomatopellan's theory is correct.

[Fumler]

case=off did not fix the problem for me :(

❯ cat /etc/wsl.conf 
[automount]
enabled = true
root = /
options = "metadata,umask=22,fmask=11,case=off"

❯ mount -l
rootfs on / type lxfs (rw,noatime)
root on /root type lxfs (rw,noatime)
home on /home type lxfs (rw,noatime)
data on /data type lxfs (rw,noatime)
cache on /cache type lxfs (rw,noatime)
mnt on /mnt type lxfs (rw,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
none on /dev type tmpfs (rw,noatime,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,noatime,gid=5,mode=620)
none on /run type tmpfs (rw,nosuid,noexec,noatime,mode=755)
none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,noatime)
none on /run/shm type tmpfs (rw,nosuid,nodev,noatime)
none on /run/user type tmpfs (rw,nosuid,nodev,noexec,noatime,mode=755)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noatime)
C: on /mnt/c type drvfs (rw,noatime,uid=1000,gid=1000,umask=22,fmask=11,metadata,case=off)
D: on /d type drvfs (rw,noatime,uid=1000,gid=1000,umask=22,fmask=11,metadata,case=off)
F: on /f type drvfs (rw,noatime,uid=1000,gid=1000,umask=22,fmask=11,metadata,case=off)
G: on /g type drvfs (rw,noatime,uid=1000,gid=1000,umask=22,fmask=11,metadata,case=off)
I: on /i type drvfs (rw,noatime,uid=1000,gid=1000,umask=22,fmask=11,metadata,case=off)

Still getting errors when using yarn:
error An unexpected error occurred: "ENOENT: no such file or directory, copyfile '/mnt/c/Users/<username>/.cache/yarn/v1/npm-@storybook/ui-3.4.3-8f3f43da32290abab2e4293d2567316b2d2ce91a/dist/modules/ui/components/stories_panel/stories_tree/tree_header.js' -> '/mnt/c/Users/<username>/code/wolverine/node_modules/@storybook/react/node_modules/@storybook/ui/dist/modules/ui/components/stories_panel/stories_tree/tree_header.js'".

[onomatopellan]

@Fumler I can't repo that. My yarn cache is on a Lxfs folder /usr/local/share/.cache/yarn/v1. You can try calling yarn with something like yarn --cache-folder /home/yarn when using case=off option.

Also I delete any project folder and create it again after using case=off option.

edit:fixed misunderstanding

[Fumler]

Thank you @onomatopellan !

Putting yarn cache location on a Lxfs folder fixed the issue.

[yshterev]

Still get errors reported by @Fumler
Sometimes it works sometimes it does not. Different files reported missing every time it does not work.
New install - npm and yarn installed after case=off in wls.conf

OS Name Microsoft Windows 10 Pro
Version 10.0.17134 Build 17134
Ubuntu 18.04

Video:
https://drive.google.com/file/d/1QWwVblsVIxnopNWu7CeuoYavyFELKv8q/view?usp=sharing

[onomatopellan]

@yshterev Are you sure your D:\workspace is a brand new folder created after enabling case=off?
You can try on cmd with
fsutil file queryCaseSensitiveInfo D:\workspace
fsutil file queryCaseSensitiveInfo D:\workspace\go
fsutil file queryCaseSensitiveInfo D:\workspace\go\src and so on...
If some of them say is Enabled then you will have this issue. Otherwise it could be a race condition like #2712

[yshterev]

@onomatopellan All folders say disabled so I guess it could be #2712
I never got that kind of errors before the update to Build 17134 like 2 weeks ago. 😢
Thanks!

[therealkenc]

It won't be case related if "Different files reported missing every time it does not work". The problem would be more deterministic. Will hold open for a bit until the #2712 hypothesis is confirmed. Which sounds plausible, but I already jumped the gun once on this one. Try 17655 or better and if the problem goes away we'll put it to bed.

[vinz243]

Not working, my wsl.conf is correct:

$ cat /etc/wsl.conf
[automount]
options = case=off

Yet I get random error with jest so WSL is unusable for me...

[nieltg]

Hi, I think I found something similar to this problem.

I was dabbling with a piece of code which list files in a specified directory recursively using fs and rxjs and I discovered that it was likely to fail if I don't wait for lstat call to be finished before calling another lstat.

I've made a simple NPM package, async-dirtree-test, to test this problem and I found that it is likely to fail while handling a directory which has many child directories, like node_modules, even with a low number of concurrency.

% async-dirtree-test -n 32 > /dev/null
/home/nieltg/.config/yarn/global/node_modules/async-dirtree-test/node_modules/rxjs/internal/util/hostReportError.js:4
    setTimeout(function () { throw err; });
                             ^

Error: ENOENT: no such file or directory, lstat 'React/React-16-Demo/node_modules/merge/LICENSE'
async-dirtree-test -n 32 > /dev/null  3.56s user 8.66s system 267% cpu 4.561 total

[therealkenc]

Hi, I think I found something similar to this problem.

If you can get your test case to fail on 17760, then your repro is probably worthy of rebooting into a new issue. Your fail looks to be on LxFS as well, while this issue wandered into the case sensitivity flag on DrvFS. You'll need (a) repro steps from a clean install (most of which you will personally find "obvious") that can be cut and pasted and (b) link to an strace log.

found that it is likely to fail while handling a directory which has many child directories, like node_modules, even with a low number of concurrency.

As well, those repro steps are going need to include steps to populate "the directory" that has many child directories not "a directory" that has many child directories (or build the tree in the test code). Also some indications on what "likely" means. "Will fail" is what you are shooting for here, because if someone cuts-and-pastes your repro steps and it doesn't fail, the new issue will go dark pretty quick. Loop the test if you have to.

It's a good effort, so thanks for that. It unfortunately makes for a less than ideal test case, because while the code is short there are a lot of big moving pieces (nodejs, V8, libuv). But it's more likely than not you're onto something.

[nieltg]

@therealkenc,
I'm running on Microsoft Windows [Version 10.0.17134.286] and still waiting for 1809 to be released. I'm sorry right now I'm not running on 17760. I think I need someone else to help me testing this.

Preparing Sample Directory for Testing

As well, those repro steps are going need to include steps to populate "the directory" that has many child directories not "a directory" that has many child directories (or build the tree in the test code). Also some indications on what "likely" means. "Will fail" is what you are shooting for here, because if someone cuts-and-pastes your repro steps and it doesn't fail, the new issue will go dark pretty quick. Loop the test if you have to.

Thank you very much for your tips. At the moment, I don't know how big exactly the directory tree should be to trigger the failure. I'm trying this on a directory, /mnt/c/Users/nieltg/Projects, which is actually an SSD and contains all of my active projects, which consists of, approximately, 12 Node projects, 3 Django projects, some Go projects, etc.

10 out of 10 invocations of the sample program on /mnt/c/Users/nieltg/Projects have failed. Here is the snippet of command I used for testing in zsh:

ok=0
fail=0
for ((i=0; i<10; i++)); do
  if ( async-dirtree-test > /dev/null 2>&1 ); then
    ((ok++))
  else
    ((fail++))
  fi
done
echo ok: $ok
echo fail: $fail

Here is the result of /mnt/c/Users/nieltg/Projects:

( async-dirtree-test > /dev/null 2>&1; )  1.83s user 5.17s system 181% cpu 3.865 total
( async-dirtree-test > /dev/null 2>&1; )  15.34s user 45.44s system 188% cpu 32.309 total
( async-dirtree-test > /dev/null 2>&1; )  12.94s user 25.73s system 316% cpu 12.201 total
( async-dirtree-test > /dev/null 2>&1; )  13.67s user 28.77s system 315% cpu 13.440 total
( async-dirtree-test > /dev/null 2>&1; )  24.11s user 52.55s system 309% cpu 24.778 total
( async-dirtree-test > /dev/null 2>&1; )  27.45s user 56.59s system 316% cpu 26.594 total
( async-dirtree-test > /dev/null 2>&1; )  11.83s user 23.14s system 267% cpu 13.063 total
( async-dirtree-test > /dev/null 2>&1; )  2.44s user 5.70s system 261% cpu 3.108 total
( async-dirtree-test > /dev/null 2>&1; )  14.91s user 30.75s system 302% cpu 15.078 total
( async-dirtree-test > /dev/null 2>&1; )  3.50s user 5.55s system 222% cpu 4.074 total
ok: 0
fail: 10

I've also tried this on AvaKathrynShaw/React-16-Demo which can be prepared with these commands:

$ git clone https://github.com/AvaKathrynShaw/React-16-Demo
$ cd React-16-Demo && yarn

5 out of 5 invocations of the sample program on it have failed on my machine. Here is the result:

( async-dirtree-test > /dev/null 2>&1; )  6.50s user 10.72s system 279% cpu 6.164 total
( async-dirtree-test > /dev/null 2>&1; )  3.05s user 5.36s system 262% cpu 3.204 total
( async-dirtree-test > /dev/null 2>&1; )  4.48s user 7.77s system 285% cpu 4.289 total
( async-dirtree-test > /dev/null 2>&1; )  2.11s user 3.31s system 228% cpu 2.368 total
( async-dirtree-test > /dev/null 2>&1; )  4.73s user 8.47s system 282% cpu 4.668 total
( async-dirtree-test > /dev/null 2>&1; )  8.75s user 14.81s system 265% cpu 8.860 total
( async-dirtree-test > /dev/null 2>&1; )  9.56s user 15.27s system 272% cpu 9.109 total
( async-dirtree-test > /dev/null 2>&1; )  7.52s user 11.69s system 214% cpu 8.939 total
( async-dirtree-test > /dev/null 2>&1; )  10.38s user 15.47s system 230% cpu 11.208 total
( async-dirtree-test > /dev/null 2>&1; )  9.14s user 15.28s system 231% cpu 10.536 total
ok: 5
fail: 5

More Things and strace Log

After conducting those and more experiments, I found more things:

• The sample program itself is unlikely to be failed if it does other things other than calling fs.stats(), like logging, so it does make me harder to debug.
• The sample program itself is unlikely to be failed if I run it on LxFs. It needs ~3 minutes to fail while running on WSL's home directory, /home/nieltg, but it only needs ~10 seconds to fail while running on /mnt/c/Users/nieltg/Projects which is on DrvFs.
• Tracing the sample program using strace make it needs a longer time then without strace before it failed.
• The fastest way to make it fail is running it without any tracing and logging mechanism, for example, async-dirtree-test > /dev/null which discards all of the output it produces so the lstat calls are made more frequent.
• The failure stills appear with or without case=force in DrvFs mount options on my machine.

From this part of article and from the strace logs that I collected from the sample program, I learnt that lstat syscall isn't actually asynchronous and is being called on Node's worker threads. Here is a portion of the strace log of a single worker thread:

...
lstat("compfest/2/composite", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
lstat("compfest/2/observer", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
lstat("compfest/2/strategy", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
lstat("compfest/4/go-academy-game01", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
...

It's a good effort, so thanks for that. It unfortunately makes for a less than ideal test case, because while the code is short there are a lot of big moving pieces (nodejs, V8, libuv). But it's more likely than not you're onto something.

You're welcome. 😃
And with all of those facts, I wonder whether the C/C++ version will show the same behavior as the Node version. I'm thinking of a simple program which just calls lstat() and readdir() recursively until all files in the directory and its descendants are visited and maybe with some threading.

The complete strace logs are available on this archive and the command I used to harvest them is available below:

% strace -ff -o strace.txt async-dirtree-test > /dev/null
/home/nieltg/.config/yarn/global/node_modules/async-dirtree-test/node_modules/rxjs/internal/util/hostReportError.js:4
    setTimeout(function () { throw err; });
                             ^

Error: ENOENT: no such file or directory, lstat 'lazy-transform/lazy-transform/node_modules/date-fns/is_this_hour'
strace -ff -o strace.txt async-dirtree-test > /dev/null  23.98s user 64.06s system 59% cpu 2:27.00 total

Summary

Well, it feels that I was just writing an article. Anyway, I think this can be tested by following this guide:

1. Turn off Windows Defender to speed up things.
2. Install the test program.

   $ yarn global add async-dirtree-test
   

3. Go to any directory in DrvFs, for example, My Documents:

   $ cd /mnt/c/Users/[ Place your Windows username here ]/Documents
   

4. Prepare a sample directory.

   $ git clone https://github.com/AvaKathrynShaw/React-16-Demo
   $ cd React-16-Demo && yarn
   

5. Run the test program with the output discarded. If it doesn't get failed, repeat it 5-10 times.

   $ async-dirtree-test > /dev/null
   

Thank you very much. I'm looking forward to more hearing about this.

[artem2302]

fsutil behavior set disablelastaccess 1 in PowerShell worked for me

[artem2302]

UPD: seems to be fixed in 1809

[therealkenc]

This one lacks any actionable repro let alone declaration of fixeedness. Closing on the basis that we'll probably never know the cause of the OP Error: ENOENT: no such file or directory, lstat '/mnt/***/node_modules/asynckit/index.js'. Case problem? Some async thread moving a file out of the way? Who knows. Whatever the problem, it would need to be spawned into a new issue with a test case. Nod to Daniel who did make a good effort; props for that.