Cannot parse file header with uncompressed size, compressed size, or local file header offset of 0xffffffff, if not in Zip64 format

[AxbB36]

This issue is a companion to #108, but affects a different part of the parser. yauzl 2.10.0 cannot parse zip files whose uncompressedSize, compressedSize, or relativeOffsetOfLocalHeader is 0xffffffff, unless the file is in Zip64 format. The relevant code is here:

yauzl/index.js

Lines 333 to 337 in 02a5ca6

	if (entry.uncompressedSize === 0xffffffff ||
	entry.compressedSize === 0xffffffff ||
	entry.relativeOffsetOfLocalHeader === 0xffffffff) {
	// ZIP64 format
	// find the Zip64 Extended Information Extra Field

When yauzl sees a 0xffffffff value in one of these fields, it assumes that the zip file must be in Zip64 format. But APPNOTE.TXT 4.4.8, 4.4.9, and 4.4.16 say (emphasis mine):

If an archive is in ZIP64 format and the value in this field is 0xFFFFFFFF, the size will be in the corresponding 8 byte ZIP64 extended information extra field.

The way I interpret this statement, the logic should not be what yauzl does now:

if a value is 0xffffffff:
    parse the Zip64 extended information extra field

but should instead be:

if a Zip64 extended information extra field is found:
    replace only the values that are 0xffffffff

This issue is basically the same as golang/go#31692. The difference is that Go archive/zip has a special-case workaround for the compressed size, but not the other two fields.

The test cases which follow are able to be parsed by Info-ZIP UnZip (unzip) and Python zipfile (python3 -m zipfile -e). Go archive/zip can parse the uncompressed size test but not the other two.

Uncompressed size test case

ffffffff-uncompressedSize.zip.gz (remove 1 layer of gzip before testing)

# 65535 * 65537 = 0xffffffff
dd if=/dev/zero bs=65535 count=65537 of=pad
touch -d '2019-05-01 00:00:00 UTC' pad
rm -f ffffffff-uncompressedSize.zip
TZ=UTC zip -9 -X ffffffff-uncompressedSize.zip pad
gzip -9 -k ffffffff-uncompressedSize.zip

zipinfo -v says:

  uncompressed size:                              4294967295 bytes

yauzl does not parse it:

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: expected zip64 extended information extra field
    at node_modules/yauzl/index.js:347:46
    at node_modules/yauzl/index.js:631:5
    at node_modules/fd-slicer/index.js:32:7
    at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

Compressed size test case

It's difficult to construct a test case that has compressedSize = 0xffffffff and uncompressedSize < 0xffffffff. It could plausibly happen if the compressor implements "store" mode not as compressionMethod 0, but as compressionMethod 8 (DEFLATE) and non-compressed blocks, which increase the compressed size slightly. The example here instead has compressedSize = uncompressedSize = 0xffffffff.

ffffffff-compressedSize.zip.gz.gz (remove 2 layers of gzip before testing)

# 65535 * 65537 = 0xffffffff
dd if=/dev/zero bs=65535 count=65537 of=pad
touch -d '2019-05-01 00:00:00 UTC' pad
rm -f ffffffff-compressedSize.zip
TZ=UTC zip -0 -X ffffffff-compressedSize.zip pad

zipinfo -v says:

  compressed size:                                4294967295 bytes
  uncompressed size:                              4294967295 bytes

yauzl does not parse it:

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: expected zip64 extended information extra field
    at node_modules/yauzl/index.js:347:46
    at node_modules/yauzl/index.js:631:5
    at node_modules/fd-slicer/index.js:32:7
    at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

Local file header offset test case

ffffffff-relativeOffsetOfLocalHeader.zip.gz.gz (remove 2 layers of gzip before testing)

# 216186 * 19867 = 0xffffffff - len("pad") - 30
dd if=/dev/zero bs=216186 count=19867 of=pad
echo test > test.txt
touch -d '2019-05-01 00:00:00 UTC' pad test.txt
rm -f ffffffff-relativeOffsetOfLocalHeader.zip
TZ=UTC zip -0 -X ffffffff-relativeOffsetOfLocalHeader.zip pad test.txt

zipinfo -v says:

  offset of local header from start of archive:   4294967295
                                                  (00000000FFFFFFFFh) bytes

yauzl does not parse it:

events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: expected zip64 extended information extra field
    at node_modules/yauzl/index.js:347:46
    at node_modules/yauzl/index.js:631:5
    at node_modules/fd-slicer/index.js:32:7
    at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

[thejoshwolfe]

Thanks again for the thorough writeup! This is fixed in yauzl 3.1.2.

josh@nixos:~/dev/yauzl$ nix-run --pure -p nodejs -- node examples/compareCentralAndLocalHeaders.js ~/tmp/asdf/ffffffff-compressedSize.zip
pad
┌───────────────────────────┬──────────┬──────────┬────┐
│field                      │   central│     local│diff│
├───────────────────────────┼──────────┼──────────┼────┤
│versionMadeBy              │     0x31e│         -│    │
│versionNeededToExtract     │       0xa│       0xa│    │
│generalPurposeBitFlag      │       0x0│       0x0│    │
│compressionMethod          │       0x0│       0x0│    │
│lastModFileTime            │       0x0│       0x0│    │
│lastModFileDate            │    0x4ea1│    0x4ea1│    │
│crc32                      │       0x0│       0x0│    │
│compressedSize             │0xffffffff│0xffffffff│    │
│uncompressedSize           │0xffffffff│0xffffffff│    │
│fileNameLength             │       0x3│       0x3│    │
│extraFieldLength           │       0x0│       0x0│    │
│fileCommentLength          │       0x0│         -│    │
│internalFileAttributes     │       0x0│         -│    │
│externalFileAttributes     │0x81a40000│         -│    │
│relativeOffsetOfLocalHeader│       0x0│         -│    │
└───────────────────────────┴──────────┴──────────┴────┘
central.fileName: 706164
(local matches)
central.extraField:
(local matches)
central.comment:

josh@nixos:~/dev/yauzl$ nix-run --pure -p nodejs -- node examples/compareCentralAndLocalHeaders.js ~/tmp/asdf/ffffffff-relativeOffsetOfLocalHeader.zip
pad
┌───────────────────────────┬──────────┬──────────┬────┐
│field                      │   central│     local│diff│
├───────────────────────────┼──────────┼──────────┼────┤
│versionMadeBy              │     0x31e│         -│    │
│versionNeededToExtract     │       0xa│       0xa│    │
│generalPurposeBitFlag      │       0x0│       0x0│    │
│compressionMethod          │       0x0│       0x0│    │
│lastModFileTime            │       0x0│       0x0│    │
│lastModFileDate            │    0x4ea1│    0x4ea1│    │
│crc32                      │0xf86bd088│0xf86bd088│    │
│compressedSize             │0xffffffde│0xffffffde│    │
│uncompressedSize           │0xffffffde│0xffffffde│    │
│fileNameLength             │       0x3│       0x3│    │
│extraFieldLength           │       0x0│       0x0│    │
│fileCommentLength          │       0x0│         -│    │
│internalFileAttributes     │       0x0│         -│    │
│externalFileAttributes     │0x81a40000│         -│    │
│relativeOffsetOfLocalHeader│       0x0│         -│    │
└───────────────────────────┴──────────┴──────────┴────┘
central.fileName: 706164
(local matches)
central.extraField:
(local matches)
central.comment:

test.txt
┌───────────────────────────┬──────────┬──────────┬────┐
│field                      │   central│     local│diff│
├───────────────────────────┼──────────┼──────────┼────┤
│versionMadeBy              │     0x31e│         -│    │
│versionNeededToExtract     │       0xa│       0xa│    │
│generalPurposeBitFlag      │       0x0│       0x0│    │
│compressionMethod          │       0x0│       0x0│    │
│lastModFileTime            │       0x0│       0x0│    │
│lastModFileDate            │    0x4ea1│    0x4ea1│    │
│crc32                      │0x3bb935c6│0x3bb935c6│    │
│compressedSize             │       0x5│       0x5│    │
│uncompressedSize           │       0x5│       0x5│    │
│fileNameLength             │       0x8│       0x8│    │
│extraFieldLength           │       0x0│       0x0│    │
│fileCommentLength          │       0x0│         -│    │
│internalFileAttributes     │       0x0│         -│    │
│externalFileAttributes     │0x81a40000│         -│    │
│relativeOffsetOfLocalHeader│0xffffffff│         -│    │
└───────────────────────────┴──────────┴──────────┴────┘
central.fileName: 746573742e747874
(local matches)
central.extraField:
(local matches)
central.comment:

These were excellent test cases! 💯