[BUG] yarn runs node-gyp operations in parallel which can cause file lock error on ~/.node-gyp resources and fail

[zumwald]

Do you want to request a feature or report a bug?
bug

What is the current behavior?
yarn installation with certain packages causes EBUSY error in node-gyp step. Perhaps due to Yarn trying to install all native modules simultaneously while npm i installs them synchronously.

If the current behavior is a bug, please provide the steps to reproduce.
run yarn in a directory with the following package.json:

{
  "name": "testing",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "diskusage": "0.1.5",
    "electron-spellchecker": "0.5.8"
  }
}

What is the expected behavior?
yarn installation of native packages is not impacted by how many/which other native packages are compiling at the same time.

Please mention your node.js, yarn and operating system version.
node: 6.2.2
OS: Windows x64 10.0.14393
Yarn: 0.18.1 (installed via npm i -g yarn)

detailed output
output from yarn:

error N:\repo\DesktopClient\app\node_modules\keyboard-layout: Command failed.
Exit code: 1
Command: C:\windows\system32\cmd.exe
Arguments: /d /s /c node-gyp rebuild
Directory: N:\repo\DesktopClient\app\node_modules\keyboard-layout
Output:
N:\repo\DesktopClient\app\node_modules\keyboard-layout>if not defined npm_config_node_gyp (node "C:\Users\username\AppD
ata\Roaming\npm\node_modules\yarn\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node  rebuild )
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | win32 | x64
gyp info spawn C:\Users\username\.windows-build-tools\python27\python.exe
gyp info spawn args [ 'C:\\Users\\username\\AppData\\Roaming\\npm\\node_modules\\yarn\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'msvs',
gyp info spawn args   '-G',
gyp info spawn args   'msvs_version=2015',
gyp info spawn args   '-I',
gyp info spawn args   'N:\\repo\\DesktopClient\\app\\node_modules\\keyboard-layout\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\username\\AppData\\Roaming\\npm\\node_modules\\yarn\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\username\\.node-gyp\\6.2.2\\include\\node\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=C:\\Users\\username\\.node-gyp\\6.2.2',
gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\username\\AppData\\Roaming\\npm\\node_modules\\yarn\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=node.lib',
gyp info spawn args   '-Dmodule_root_dir=N:\\repo\\DesktopClient\\app\\node_modules\\keyboard-layout',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'N:\\repo\\DesktopClient\\app\\node_modules\\keyboard-layout\\build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp ERR! build error
gyp ERR! stack Error: EBUSY: resource busy or locked, open 'C:\Users\username\.node-gyp\6.2.2\Release\node.lib'
gyp ERR! stack     at Error (native)
gyp ERR! System Windows_NT 10.0.14393
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\username\\AppData\\Roaming\\npm\\node_modules\\yarn\\node_modules\\
node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd N:\repo\DesktopClient\app\node_modules\keyboard-layout
gyp ERR! node -v v6.2.2
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

note that npm i in the same directory seems to run without error:

PS N:\repo\DesktopClient\app> npm i

> @paulcbetts/[email protected] install N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\@pau
lcbetts\spellchecker
> node-gyp rebuild


N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\@paulcbetts\spellchecker>if not defined npm_c
onfig_node_gyp (node "C:\Users\username\AppData\Roaming\npm\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gy
p.js" rebuild )  else (node "" rebuild )
Warning: Missing input files:
N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\@paulcbetts\spellchecker\build\..\vendor\huns
pell\src\build\build.h
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  md5.cc
  string_piece.cc
  bdict_reader.cc
  bdict.cc
  affentry.cxx
  affixmgr.cxx
  csutil.cxx
  dictmgr.cxx
  filemgr.cxx
  hashmgr.cxx
  hunspell.cxx
  hunzip.cxx
  phonet.cxx
  replist.cxx
  suggestmgr.cxx
  textparser.cxx
  win_delay_load_hook.cc
  hunspell.vcxproj -> N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\@paulcbetts\spellcheck
  er\build\Release\\hunspell.lib
  main.cc
  spellchecker_hunspell.cc
  spellchecker_win.cc
  transcoder_win.cc
  win_delay_load_hook.cc
     Creating library N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\@paulcbetts\spellcheck
  er\build\Release\spellchecker.lib and object N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modul
  es\@paulcbetts\spellchecker\build\Release\spellchecker.exp
  Generating code
  Finished generating code
  spellchecker.vcxproj -> N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\@paulcbetts\spellc
  hecker\build\Release\\spellchecker.node
  spellchecker.vcxproj -> N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\@paulcbetts\spellc
  hecker\build\Release\spellchecker.pdb (Full PDB)

> [email protected] install N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\keyboard-layo
ut
> node-gyp rebuild


N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\keyboard-layout>if not defined npm_config_nod
e_gyp (node "C:\Users\username\AppData\Roaming\npm\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" reb
uild )  else (node "" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  keyboard-layout-manager-windows.cc
  win_delay_load_hook.cc
     Creating library N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\keyboard-layout\build\
  Release\keyboard-layout-manager.lib and object N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_mod
  ules\keyboard-layout\build\Release\keyboard-layout-manager.exp
  Generating code
  Finished generating code
  keyboard-layout-manager.vcxproj -> N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\keyboar
  d-layout\build\Release\\keyboard-layout-manager.node
  keyboard-layout-manager.vcxproj -> N:\repo\DesktopClient\app\node_modules\electron-spellchecker\node_modules\keyboar
  d-layout\build\Release\keyboard-layout-manager.pdb (Full PDB)

[juturu]

@kittens @bestander this issue is blocking on using yarn in our project. Can you please help with this issue? Is there a way to run node-gyp on packages sequentially to avoid this issue temporarily?

[bestander]

Now that we have a workaround with CHILD_CONCURRENCY.
How about sending a PR to node-gyp or forking it for Yarn so that it would work concurrently?

[timfish]

Is there a node-gyp issue for this?

[bestander]

Here #1874 and here nodejs/node-gyp#1054

[timfish]

Ah, so no consensus as to where this should be fixed?

[bestander]

Help from the community is needed.

[timfish]

How about disabling parallel builds on Windows until we have an option. I'm sure most would prefer slightly slower installs over intermittent install failures without using an environment variable!

[bestander]

This is configured via child-concurrency setting via .yarnrc or env variable.
If we set it to 1 by default then there will be no pressure on package authors to make their packages behave well.

[BYK]

Closing as a duplicate of #1874.