Errors - silently fails upon extracting

[gooor]

It doesn't throw errors, works unexpectedly, better to just install zip and do child_process. Come on, I just wasted 3 hours of my life trying to use it and it's complete mess

[ZJONSSON]

Sorry that you wasted 3 hours of your life. This is an open source project where anyone can contribute fixes and solutions to make it better. If you wanted to be constructive you could have included detail of what you had problems with and suggestions for fixes. It might be related to #92 which will land soon.

[gooor]

Okay excuse me for my tone - it was wrong!

Anyway I think if it's known issue it should be stated in README that this might cause problems while extracting full zip.

  const zipper = fs.createReadStream(filePath).pipe(unzipper.Extract({ path: path.resolve(path.dirname(require.main.filename), 'dist') }));
  zipper.on('close', () => {
    console.log('done');
  });
  zipper.on('error', (error) => {
    console.log(error);
  });

1. zip is not fully extracted (sometimes it's only 2 files sometimes its 3 or 4 while there are 10 files inside
2. neither close or error is called so it's looks like Reject from pullstream errors #92
3. Most common usage of it would be to unzip file uploaded from browser - if there's something wrong with it (ie. file corrupted) then it will just silently fail and there's no way of knowing that
   But if so I cannot rely on it at all :(

[ZJONSSON]

Thank you for this report. I just extracted a bunch of zip files and was unable to recreate this issue (i.e. the extract matches the content of manual uncompress). This is also confirmed by unit tests that are tested on multiple node versions before every release.92

Please note that the fix to #92 was released yesterday and the most current version of unzipper is 0.9.8

To help me debug your particular situation, please upload a failing zip file (or send to me at [email protected]) and/or give me more information about your setup, i.e. OS, node version and unzipper version.

[ZJONSSON]

@gooor - following up on ^^^

[gooor]

@ZJONSSON original file itself was not corrupted. Only one that was downloaded from url and extracted after that.
Full code is

    https.get(url, (response) => {
      response.pipe(file);
      file.on('finish', () => {
        file.close();
        const zipper = fs.createReadStream(filePath).pipe(unzipper.Extract({ path: '.\\' }));
    
        zipper.on('close', () => {
          console.log(`ZIP deployed`);
        });
        zipper.on('error', (error) => {
          console.error(`Erroe while uploading`, error);
        }); 
      });
    }).on('error', () => {
      fs.unlink(filePath, () => {});
    });

For some reason sometimes file that was sent was "incomplete" (size was smaller than original one). No close nor error events were triggered. I didn't try #92 yet. I'll try to check it today

[ZJONSSON]

Thanks @gooor. Obviously I can only be helpful if you share the full detail (sample file, your os etc).

Please note that in your example you do not explicitly need to write the file to fs unless you need to retain the zip file. You should be able to:

http.get(url, async response => async new Promise( (resolve, reject) => {
  response.pipe(unzipper.Extract({path}))
    .on('close', resolve)
    .on('error', reject);
}));

and furthermore, if you are only interested in particular files inside the zip (not all of them), you can download/unzip only the files of interest from inside the zipfile (not the other files) directly (see README and also webunzipper (a cli tool based on unzipper). Here is an example of fetching smaller files out of 500mb within a zip file that exists only remotely:

Nevertheless, if you ware having issues with extract it's important to get to the bottom of this, so I appreciate you help

[gooor]

@ZJONSSON ok found a steps to reproduce. I tested this only on OSX. Create any ZIP file with any contents (put more than one file to see results). Use ie. split to take some bytes off this zip fie ie.:

• my zip wile was 2,5M -> split -b 2400k seg
• this will create file segaa and segab. Add .zip to segaa file. It will be corrupted when trying to extract in OS

const zipper = fs.createReadStream(filePath).pipe(unzipper.Extract({ path: __dirname }));
zipper.on('close', () => {
  console.log(`ZIP deployed`);
});
zipper.on('error', (error) => {
  console.error(`Erroe while extracting`, error);
});  

When unzipping correct file console states: ZIP deployed but when extracting corrupted one nothing is thrown. As expected all files except one are extracted (the one that part probably left in second segment).

[ZJONSSON]

Thanks @goor that is very helpful. Will take a look!

[willpox]

I am also experiencing this problem.

I am missing several files using unzipper. However, if I unzip the zip file manually, all the files are in there.

[mmajcica]

I'm experiencing a similar problem. In case of the mixed slashes in my destination path, (e.g. on win "C:\temp/anotherDir") the extract method silently fails. A simple path.normalize over the path makes extract succeed.

[ZJONSSON]

Thanks @mmajcica, can you submit a PR?

[dangear]

@ZJONSSON Hi! I have the same error when use it with node ftp client.

const fs = require('fs');
const path = require('path');
const unzipper = require('unzipper');
const async = require('async');
const Client = require('ftp');

var c = new Client(); //init ftp client
var connectionProperties = { 
    host: "PM Please if you need this for test",
    user: "PM Please if you need this for test",
    password: "PM Please if you need this for test"
};

var year='2019';
var folders=['purchaseNotice','purchaseNoticeAE','purchaseNoticeAE94','purchaseNoticeEP','purchaseNoticeOA','purchaseNoticeOK','purchaseNoticeZK'];

var listFile = function(ftp_files_folder, region){ //Get folder file list
  return new Promise(function(resolve, reject){
    c.list(ftp_files_folder,function (err, list) {
      if (err) reject(err)
      var mask=region+'_'+year; //mask for search
      var paths=list.filter(path=>{return path.name.split(mask).length>1}).map(path=>{return path.name});
      resolve(paths) //send files list
    })
  })
}

var listRegions = function(ftp_main_folder){ //Get ftp subfolders
  return new Promise(function(resolve, reject){
    c.list(ftp_main_folder,function (err, list) {
      if (err) reject(err)
      var paths=list.filter(path=>{
        if(path.type=='d' && path.name!='archive') return path;
      }).map(path=>{return path.name});
      resolve(paths) //send files list
    })
  })
}

c.connect(connectionProperties); //connect ftp client

listRegions('/out/published/').then((all_regions)=>{ //get folder for all regions from ftp
  async.mapLimit(all_regions, 1, function(region, callback){ //get every region and check for exist files in folders from folders[array]
    console.log('\x1b[35m','[1]: Get folders for '+region,'\x1b[0m');
    async.mapLimit(folders, 1, function(folder, callback){ //get every folder from folder[array]
      var ftp_region_folders='/out/published/'+region+'/'+folder+'/daily';
      listFile(ftp_region_folders, region).then(ftp_region_folder_filelist=>{ //get file list from ftp folder like folders[array]
        console.log('\x1b[36m','[2]: Get files form '+region+'/'+folder+' that exist: '+ftp_region_folder_filelist.length+' files','\x1b[0m');
        async.mapLimit(ftp_region_folder_filelist, 1, function(file, callback){ //get every file from file list
          var ftp_region_folder_filepath=ftp_region_folders+'/'+file; //ftp file path
          var local_filepath=__dirname+'/alldata/'+region+'/'+folder+'/'+file.split('.xml').shift(); //local folder for save
          if(!fs.existsSync(local_filepath)) { //if subfolder not exist
            console.log('[3.1]: Try to download: '+ftp_region_folder_filepath);
            c.get(ftp_region_folder_filepath, function (err, stream) { //get file from ftp file path
              if(err) callback(err);			    
              stream.pipe(unzipper.Extract({path: path.normalize(local_filepath)})).on('entry', (entry)=>{ //get and unzip to subfolder
                console.log(entry.path); //show every file in archive
              }).on('finish', ()=>{
                console.log('[3.2]: File: '+file+' unzipped');
                console.log('--------------------------------------------------');
                callback();
              })
            })
          } else {
            callback();
          }
        }, function(err, res){
          if(err) console.log(err)
          console.log('\x1b[36m','[!!!] Updates for '+region+'/'+folder+' complete','\x1b[0m');
          console.log('=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=');
          callback();
        })
      })
    }, function(err, res){
      if(err) console.log(err)
      console.log('\x1b[35m','[!!!]: All folders and files for '+region+' downloaded','\x1b[0m');
      console.log('=============================================================');
      callback();
    })
  }, function(err, res){
    if(err) console.log(err);
    console.log('\x1b[35m','===== ALL DONE!!! =====','\x1b[0m');
    c.end();
  })
})

But sometimes process freeze and nothing happend (see picture). path.normalizer don't help with it. I try to catch errors with .on('error') but nothing. I use MacOS 10.14.4, Node.js v11.10.1.

[MRayermannMSFT]

@ZJONSSON , I opened a PR to do the normalize fix: #121

[Shailaputri]

I am still facing this issue. Some files and folders went missing upon extraction on web hosting platform (00webhost). Has anyone found a work around?

Edit : This is a free webhosting site .My extraction was failing due to quota limit exceed.

[jabidof]

Experiencing the same. Any fix found for this?
Thanks for the nice library!