published package is corrupted if "files" is used in package.json

[tkloht]

Do you want to request a feature or report a bug?
bug
What is the current behavior?
when using the "files" array to list files included in the package and publishing with yarn, the publish seems to work (there is no error message). However installing the package fails with the following error:

error https://registry.npmjs.org/yarntest2/-/yarntest2-1.0.0.tgz: invalid tar file
    at Extract.Parse._startEntry (/Users/tkloht/.nvm/versions/node/v6.2.1/lib/node_modules/yarnpkg/node_modules/tar/lib/parse.js:149:13)
    at Extract.Parse._process (/Users/tkloht/.nvm/versions/node/v6.2.1/lib/node_modules/yarnpkg/node_modules/tar/lib/parse.js:131:12)
    at BlockStream.<anonymous> (/Users/tkloht/.nvm/versions/node/v6.2.1/lib/node_modules/yarnpkg/node_modules/tar/lib/parse.js:47:8)
    at emitOne (events.js:96:13)
    at BlockStream.emit (events.js:188:7)
    at BlockStream._emitChunk (/Users/tkloht/.nvm/versions/node/v6.2.1/lib/node_modules/yarnpkg/node_modules/block-stream/block-stream.js:145:10)
    at BlockStream.resume (/Users/tkloht/.nvm/versions/node/v6.2.1/lib/node_modules/yarnpkg/node_modules/block-stream/block-stream.js:58:15)
    at Extract.Reader.resume (/Users/tkloht/.nvm/versions/node/v6.2.1/lib/node_modules/yarnpkg/node_modules/fstream/lib/reader.js:249:34)
    at Entry.<anonymous> (/Users/tkloht/.nvm/versions/node/v6.2.1/lib/node_modules/yarnpkg/node_modules/tar/lib/parse.js:261:8)
    at emitNone (events.js:86:13)

I have downloaded the tarball and it seems to be corrupted.
To double check I have:

• published without "files" with yarn - this works, can be installed correctly
• published with "files, but with npm - this works too, can be installed correctly

If the current behavior is a bug, please provide the steps to reproduce.

• create a simple package and publish with yarn
• add files array and include your index.js, for example. publish again
• try to install, or download the tarball. this should produce the error

Small repo where I tried this: https://github.com/t-obi/yarntest
and the corresponding npm package: https://www.npmjs.com/package/yarntest2

• 1.0.0 published with yarn and files, this is broken
• 1.0.1 published with yarn without files, this works
• 1.0.2 published with npm with files, this works

What is the expected behavior?
It should always work ;)

Please mention your node.js, yarn and operating system version.
node v6.2.1, yarn 0.15.1, OSX 10.12

[vjpr]

I have a similar error but it was caused by the tar file http request returning {error: 'not found'} when using a private scoped module.

It must be passing bad data to the tar module, unfortunately the trace is not long enought to see the exact origin.

[tkloht]

@vjpr Yes I think there might be multiple problems here - I published a public scoped module with yarn, then couldn't install because the package.json could not be found in the published package if i read the error correctly(@tkloht/[email protected], in case anyone wants to have a look at it). I couldn't reproduce this though, and I have no idea if this is related or a completely different error.
Actually I found the error described in this issue when trying to reproduce this one.

[tkloht]

I have started to look a little deeper into this and just wanted to share what I found so far:

First of all, this happens not only with yarn publish but also with yarn pack.
What I mean is: take the [email protected] from the reproduce section of the original bugreport.
Run yarn pack -> no error, try to open the archive -> not possible because the file is corrupted
run npm pack -> everything works fine

I have then tried to recover the corrupted archive (using this tool: http://www.urbanophile.com/arenn/coding/gzrt/gzrt.html) to see where it actually goes wrong.

This is the content of the archive produced by npm pack, so this should be the correct result:

[tkloht@Tobias-MacBook-Air ~/Code/gzrt/gzrt-0.8:master*] tree -h ../npm
../npm
`-- [ 136]  package
    |-- [  28]  index.js
    `-- [ 145]  package.json

This is the content of the recovered archive produced by yarn pack:

[tkloht@Tobias-MacBook-Air ~/Code/gzrt:master*] tree -h yarn
yarn
`-- [ 204]  package
    `-- [  28]  index.js

So basically package.json is missing which seems very weird to me.
I guess the next step is taking a closer look at the pack script, at least I would not know how to get more information from the produced archives.

[tkloht]

Okay I think I have found the problem.
There is a list of default ignores in the pack.js script that lists files which should never be ignored (https://github.com/yarnpkg/yarn/blob/master/src/cli/commands/pack.js#L43)
These default ignores are not used if 'files' is defined in package.json, which is ok as far as I can see. But those files which should never be ignored should still be included in the archive, and that is not the case. I don't actually see why this would create a corrupted archive instead of a valid one where the files are simply missing, but in my quick testing it seems to work correctly.

I have a draft for a PR but it's not quite done, think it will be ready tomorrow or in the next days.

[devoto13]

"files" seems to be very broken. I'll add my issue here instead of creating new one, since it's probably related. See example project https://github.com/devoto13/yarn-git with explanations.