npm install corrupts large entries in package.tgz content (node 6)

[simonbuchan]

As in title, npm cache contains valid package.tgz, but (it seems) the extract step in npm install generates causes larger files to be garbled (smaller files are good).
Same steps in git bash (ie MSYS2) with the same node, npm works fine. node installed with curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -

root@QOF:~# npm version
{ npm: '3.8.6',
  ares: '1.10.1-DEV',
  http_parser: '2.7.0',
  icu: '56.1',
  modules: '48',
  node: '6.0.0',
  openssl: '1.0.2g',
  uv: '1.9.0',
  v8: '5.0.71.35',
  zlib: '1.2.8' }
root@QOF:~# mkdir xxx
root@QOF:~# cd xxx
root@QOF:~/xxx# npm i bower
/root/xxx
└── [email protected]

npm WARN enoent ENOENT: no such file or directory, open '/root/xxx/package.json'
npm WARN xxx No description
npm WARN xxx No repository field.
npm WARN xxx No README data
npm WARN xxx No license field.
root@QOF:~/xxx# $(npm bin)/bower
/root/xxx/node_modules/bower/lib/node_modules/q/q.js:446
};
^
SyntaxError: Unexpected token }
    at exports.runInThisContext (vm.js:53:16)
    at Module._compile (module.js:511:25)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)
    at Module.require (module.js:466:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/root/xxx/node_modules/bower/lib/bin/bower.js:3:9)
    at Module._compile (module.js:541:32)
root@QOF:~/xxx# tar --extract --file /root/.npm/bower/1.7.9/package.tgz
root@QOF:~/xxx# ls package/
bin  CHANGELOG.md  lib  LICENSE  package.json  README.md
root@QOF:~/xxx# diff -u {package,node_modules/bower}/lib/node_modules/minimatch/minimatch.js | head -30
--- package/lib/node_modules/minimatch/minimatch.js     2016-04-05 11:51:43.000000000 +0000
+++ node_modules/bower/lib/node_modules/minimatch/minimatch.js  2016-04-05 11:51:43.000000000 +0000
@@ -17,316 +17,264 @@
 var star = qmark + '*?'

 // ** when dots are allowed.  Anything goes, except .. and .
-// not (^ or / followed by one or two dots followed by $ or /),
-// followed by anything, any number of times.
-var twoStarDot = '(?:(?!(?:\\\/|^)(?:\\.{1,2})($|\\\/)).)*?'
-
-// not a ^ or / followed by a dot,
-// followed by anything, any number of times.
-var twoStarNoDot = '(?:(?!(?:\\\/|^)\\.).)*?'
-
-// characters that need to be escaped in RegExp.
-var reSpecials = charSet('().*{}+?[]^$\\!')
-
-// "abc" -> { a:true, b:true, c:true }
-function charSet (s) {
-  return s.split('').reduce(function (set, c) {
-    set[c] = true
-    return set
-  }, {})
-}
+// not (^ or / followed by one or two dots followed by $ egExpEscape(p)
+      : p._src
+    }).join('\\\/')
+  }).join('|')

-// normalizes slashes.

q.js diff is very boring for a long time.

Very clear split: package/lib/node_modules/boom/test/index.js (17399 bytes) and every file smaller is OK, package/lib/node_modules/cli-width/coverage/lcov-report/prettify.js (17569 bytes) and every file bigger is corrupt:

root@QOF:~/xxx# find package -size +17500c -printf %s\ %p\\n | sort -n > bigfiles
root@QOF:~/xxx# diff -qr package/ node_modules/bower/ | cut -d\  -f 2 | xargs -d\\n stat -c%s\ %n | sort -n > difffiles
root@QOF:~/xxx# diff -u bigfiles difffiles
--- bigfiles    2016-05-02 08:22:57.094789400 +0000
+++ difffiles   2016-05-02 08:23:24.173012000 +0000
@@ -1,3 +1,4 @@
+353 package/package.json
 17569 package/lib/node_modules/cli-width/coverage/lcov-report/prettify.js
 18117 package/lib/node_modules/handlebars/dist/cjs/handlebars/compiler/printer.js
 18137 package/lib/node_modules/github/api/v3.0.0/gitdataTest.js

package.json diff is due to npm munging.

[benhillis]

Looks like the fix that fixed rbenv install also fixed this issue:

root@BENHILL-VM7:~# npm version
{ npm: '3.8.6',
  ares: '1.10.1-DEV',
  http_parser: '2.7.0',
  icu: '56.1',
  modules: '48',
  node: '6.0.0',
  openssl: '1.0.2g',
  uv: '1.9.0',
  v8: '5.0.71.35',
  zlib: '1.2.8' }
root@BENHILL-VM7:~# mkdir xxx
root@BENHILL-VM7:~# cd xxx
root@BENHILL-VM7:~/xxx# npm i bower
/root/xxx
└── [email protected]

npm WARN enoent ENOENT: no such file or directory, open '/root/xxx/package.json'
npm WARN xxx No description
npm WARN xxx No repository field.
npm WARN xxx No README data
npm WARN xxx No license field.
root@BENHILL-VM7:~/xxx# echo $(npm bin)
/root/xxx/node_modules/.bin
root@BENHILL-VM7:~/xxx# $(npm bin)/bower
bower ESUDO         Cannot be run with sudo

Additional error details:
Since bower is a user command, there is no need to execute it with superuser permissions.
If you're having permission errors when using bower without sudo, please spend a few minutes learning more about how your system should work and make any necessary repairs.

http://www.joyent.com/blog/installing-node-and-npm
https://gist.github.com/isaacs/579814

You can however run a command with sudo using --allow-root option
root@BENHILL-VM7:~/xxx# $(npm bin)/bower --allow-root

Usage:

    bower <command> [<args>] [<options>]
Commands:

    cache                   Manage bower cache
    help                    Display help information about Bower
    home                    Opens a package homepage into your favorite browser
    info                    Info of a particular package
    init                    Interactively create a bower.json file
    install                 Install a package locally
    link                    Symlink a package folder
    list                    List local packages - and possible updates
    login                   Authenticate with GitHub and store credentials
    lookup                  Look up a package URL by name
    prune                   Removes local extraneous packages
    register                Register a package
    search                  Search for a package by name
    update                  Update a local package
    uninstall               Remove a local package
    unregister              Remove a package from the registry
    version                 Bump a package version
Options:

    -f, --force             Makes various commands more forceful
    -j, --json              Output consumable JSON
    -l, --loglevel          What level of logs to report
    -o, --offline           Do not hit the network
    -q, --quiet             Only output important information
    -s, --silent            Do not output anything, besides errors
    -V, --verbose           Makes output more verbose
    --allow-root            Allows running commands as root
    -v, --version           Output Bower version
    --no-color              Disable colors
See 'bower help <command>' for more information on a specific command.
root@BENHILL-VM7:~/xxx# tar --extract --file /root/.npm/bower/1.7.9/package.tgz
root@BENHILL-VM7:~/xxx#
root@BENHILL-VM7:~/xxx# ls package/
bin  CHANGELOG.md  lib  LICENSE  package.json  README.md
root@BENHILL-VM7:~/xxx# diff -u {package,node_modules/bower}/lib/node_modules/minimatch/minimatch.js | head -30
root@BENHILL-VM7:~/xxx#

[simonbuchan]

@benhillis Sweet!

[ballPointPenguin]

related npm/npm#12436

[russalex]

Toot cause is fixed in build 14352 which just went out.

Marking this one as fixed.

[simonbuchan]

Confirming fixed on 14352