Releases: fcorbelli/zpaqfranz
Windows 32/64 binary
In this release there are several changes, it is not stable yet
Minor fixes in source code
Different zpaq auto-test file (not pseudorandom anymore, but Iliad), for Debian-awareness
list command with orderby, datefrom and size
Now it is possible to sort the list, filter on datefrom dateto, and maxsize, minsize
List the 10 greatest file l z:\1.zpaq -all -orderby size -desc -n 10
List files >1G l z:\1.zpaq -minsize 1g
List files >100M of 2017 l z:\1.zpaq -minsize 100m -datefrom 2017 -dateto 2017
New command e (extract)
Extract on current folder
New command find
Search for files with filtering
*BEWARE OF DOUBLE QUOTES ON NIX !!
'Double filter' find c:\dropbox *.txt -minsize 1000000 -find franco
Search in multiple folders find c:\uno z:\kb -only *franco*
Search file ending with 01.cpp find c:\zpaqfranz *01.cpp
Search all .cpp find c:\zpaqfranz *.cpp -verbose
Search *francia* BEWARE DOUBLE QUOTE! find /root "*francia*"
Search filtered on date find c:\zpaqfranz *.cpp -datefrom 20220920 -dateto 2022-10-01
Search every file of 2017 find . -datefrom 2017 -dateto 2017
Search every EXE file of 2017 find . *.exe -datefrom 2017 -dateto 2017
10 biggest cpp find . *.cpp -orderby size -desc -limit 10 -verbose
Command cp with -append (resumable-append-only copy)
ZPAQ archives are append only, therefore it is possible to update a copy writing only the differential data. Can be resumed if interrupted (ex. control-c)
Resumable copy (append only) cp k:\*.zpaq -to z:\pluto -append
Command x (extract) with -range
Extract the data inside a range of version. The interval can be selected from-to, from-until-end, fromfirst-upto, or a single value
zpaqfranz x versioni.zpaq -to z:\destination -all -range 2-4
zpaqfranz x versioni.zpaq -to z:\destination -all -range 2-
zpaqfranz x versioni.zpaq -to z:\destination -all -range -3
zpaqfranz x versioni.zpaq -to z:\destination -all -range 2
It can be used to extract, for example, all the version of a specific source code
zpaqfranz x copia_zarc.zpaq -only *compagnie.pas -to z:\tuttecompagnie -all -range 100-1000
On Windows
Command a (add) with switch -open
Fail early if the output archive is already opened. It is used for collisions on local network copies
zpaqfranz a z:\pippo.zpaq *.cpp -open
Command a (add) with switch -windate
This will enable a special mode (cannot be used with different hasher) that retaing the CREATION datetime of the files
zpaqfranz a z:\test *.txt -windate
Command l (list) with switch -windate
A new column appears, with the (C)reation date-time
zpaqfranz l z:\test.zpaq -windate
zpaqfranz v55.16h-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (03 Oct 2022)
franz:winhash64 (-windate)
z:/test.zpaq:
1 versions, 11 files, 11 fragments, 3 blocks, 20.593 bytes (20.11 KB)
- 2022-10-03 18:39:37 (C) 2022-08-12 18:59:02 0 A 1.txt
- 2022-09-17 16:56:17 (C) 2022-09-15 16:06:52 570 A 2.txt
- 2022-09-18 15:12:54 (C) 2022-08-25 16:49:36 874 A 3.txt
- 2022-09-18 15:12:54 (C) 2022-08-25 16:50:23 426 A 4.txt
- 2022-09-08 15:54:43 (C) 2022-08-12 16:24:16 13.050 A cpuz.txt
- 2022-09-08 15:54:43 (C) 2022-08-12 19:56:11 153 A lavoretti.txt
- 2022-09-18 19:25:43 (C) 2022-09-18 19:25:41 190 A news.txt
- 2022-09-08 15:54:43 (C) 2022-09-07 10:12:57 21.966 A nz.txt
- 2022-09-08 15:54:43 (C) 2022-09-07 10:36:54 1.398 A nz2.txt
- 2022-09-08 15:54:43 (C) 2022-08-26 13:52:12 14.338 A provapc.txt
- 2022-09-08 15:54:43 (C) 2022-08-14 16:45:52 434 A thelog.txt
53.399 (52.15 KB) of 53.399 (52.15 KB) in 11 files shown
20.593 compressed
0.032 seconds (00:00:00) (all OK)
Command x (extract) with switch -windate
Set file's creation date file, if any, after extraction
C:\zpaqfranz>zpaqfranz a z:\uno *.cpp -windate
zpaqfranz v55.16h-experimental-JIT-L archiver, SFX64 v55.1, (03 Oct 2022)
franz:winhash64 (-windate)
Creating z:/uno.zpaq at offset 0 + 0
Adding 41.299.225 (39.39 MB) in 21 files (0 dirs), 32 threads @ 2022-10-04 18:17:15
21 +added, 0 -removed.
0 + (41.299.225 -> 18.970.637 -> 2.543.696) = 2.543.696 @ 40.65 MB/s
0.969 seconds (00:00:00) (all OK)
C:\zpaqfranz>zpaqfranz x z:\uno -to z:\estratto -windate
zpaqfranz v55.16h-experimental-JIT-L archiver, SFX64 v55.1, (03 Oct 2022)
franz:winhash64 (-windate)
z:/uno.zpaq:
1 versions, 21 files, 255 fragments, 4 blocks, 2.543.696 bytes (2.43 MB)
Extracting 41.299.225 bytes (39.39 MB) in 21 files (0 folders) with 32 threads
12.80% 00:00:00 ( 5.04 MB) of ( 39.39 MB) 5.04 MB/sec
Files to be worked 21 => founded 21 => OK 21
0.203 seconds (00:00:00) (all OK)
-maxsize, -minsize now can be used with K,M,G,T and KB,MB,GB,TB
Examples: -maxsize 1gb, -maxsize 200mb, -minsize 10k, -minsize 300000
KMGT 1.000,1.000.000 etc. KB MB GB TB 1.024, 1.048.576 etc
-datefrom, -dateto are now more developed
Examples OK, please use leading zeros (03 for March, not 3)
- 2022 (year 2022)
- 202210 (year 2002, month 10)
- 2022-12-25 (year 2022, month 12, day 25)
- 20221225 (year 2022, month 12, day 25)
- 2002-12-25_03:04:05 (year 2002, month 12, day 25, hour 3, min 4, sec 5)
- 20121225030405 (year 2012, month 12, day 25, hour 3, min 4, sec 5)
Windows 32/64 binary, 64 bit-HW accelerated, ESXi
Append command (aka: ransomware-protected archives)
First public release (strongly in development) for the command append(), that work just like add()
On some systems (BSD/Mac) there is the chflags command, with sappend/uappend flags, enforcing append-only files. Using the append command, or the -append switch in add(), try to write data in append-only way
Those are BSD 4.4 security levels, not very common on Linux
C:\zpaqfranz>zpaqfranz append z:\onlyappend *.txt
zpaqfranz v55.15q-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 18 2022)
franz:append (-append)
Append 2-pass *** ALPHA STAGE *** ANTI-RANSOMWARE (FOR chflags sappend)
Processing 53.399 (52.15 KB) in 11 files (0 dirs), 32 threads @ 2022-09-19 14:40:15
(53.399 -> 53.399) @ 3.18 MB/s
z:/onlyappend.zpaq:
0 versions, 0 files, 0 fragments, 0 blocks, 0 bytes (0.00 B)
Updating z:/onlyappend.zpaq at offset 0 + 0
Adding 53.399 (52.15 KB) in 11 files (0 dirs), 32 threads @ 2022-09-19 14:40:15
11 +added, 0 -removed.
0 + (53.399 -> 53.399 -> 20.512) = 20.512 @ 1.59 MB/s
0.047 seconds (00:00:00) (all OK)
After extract (with -verbose or -stat) shows infos on files
C:\zpaqfranz>zpaqfranz x longrelative.zpaq -to z:\proka -longpath -stat
zpaqfranz v55.15q-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 18 2022)
franz:Long path (on Windows) -stat
31876: INFO: setting Windows' long filenames
longrelative.zpaq:
1 versions, 12 files, 3 fragments, 3 blocks, 1.667 bytes (1.63 KB)
Extracting 27 bytes (27.00 B) in 4 files (8 folders) with 32 threads
Long filenames 1
Windows path 12
0.016 seconds (00:00:00) (all OK)
-longpath on Windows should work adding relative-path based folders
Please use always local-full-path with -longpath on Windows
Improved ESXi compatibility
Monothread works without pthread (fake on ESXi). More work needed to reduce RAM occupation during extraction
[root@localhost:/vmfs/volumes/bbbb7ffc-82ceffff] uname -a
VMkernel localhost.station 6.5.0 #1 SMP Release build-4564106 Oct 26 2016 22:24:57 x86_64 x86_64 x86_64 ESXi
[root@localhost:/vmfs/volumes/bbbb7ffc-82ceffff] ./zpaqfranz a test1 zpaqfranz* -verbose
zpaqfranz v55.15q-experimental-ESX archiver, (Sep 19 2022)
franz:-verbose
Integrity check type: XXHASH64+CRC-32
Creating test1.zpaq at offset 0 + 0
Adding 26.085.852 (24.88 MB) in 10 files (0 dirs), 1 threads -method 14 @ 2022-09-19 12:49:25
39234: monothread compress
10 +added, 0 -removed.
0 starting size
26.085.852 data to be added
22.875.056 after deduplication
+ 5.669.205 after compression
5.669.205 total size
Total speed 8.34 MB/s
2.983 seconds (000:00:02) (all OK)
-DBIG switch
Enforce BIG-endian compatibility
Debian-compatible first release. Debian package under development
Work in progress...
Minor fixes (enhanced compilers compatibility)
Windows 32/64 binary
Nix stuff
I am attempting inclusion in both FreeBSD and Debian. The first seems to have been successful, the second is in progress
Support for BIG ENDIAN CPUs with -DBIG (at compile time)
CPUs are sometimes "weird", now zpaqfranz compatibility is extended (example: PowerPC Mac)
New -DANCIENT switch for running on "old" systems
Compile on gcc 4.0, working for 3.x. Reduce used memory too
xxhash changed. Needing BIG ENDIAN support the used source is different
The implementation I used was very elegant, but I had to take it off
a small... man
Yes, a man zpaqfranz that ... say... "Luke, use the embedded help!". Debian loves man
Windows stuff
The sfx modules are now compressed with... zpaqfranz (!)
I used to compress them with lz4, which is faster, but less efficient. It is now slower to write sfx modules, but both the executable and the source are smaller
findzpaq switch
Look for an existing .zpaq file on all letters. If it finds it, update that
zpaqfranz a z:\prova\ci.zpaq c:\nz\* -findzpaq
WHAAATT?
Suppose you have a USB device used for backups by writing the "ci.zpaq" file in the "prova" folder.
When you put inside USB port, the Windows letter associated with the USB drive may change.
Maybe today the USB stick is called D:\ and tomorrow E:\ (for a thousand reasons).
Using the -findzpaq switch, zpaqfranz will search within the units the .zpaq file. If found, write on this one.
Therefore a batch file (written on the desktop PC) will be able to update backups on USB devices, even if they change letter.
Why do not run from USB device? Because of -key, the password.There is no point in scripting the password in clear text and saving it on the same device
All
Reworked help
Various minor fix (less compiler warnings)
No "init stages" in benchmark
The embedded test zpaq-mime64 encoded is splitted in 50K chunks (ancient compiler)
New command pause, to pause script. For a key, or X seconds.
In *nix there is not something like Windows's pause-batch command
Wait for a key zpaqfranz pause
Wait for 5 seconds (or key) zpaqfranz pause -n 5
Wait 5 seconds (or key) small output zpaqfranz pause -n 5 -pakka
Wait 5 seconds (or key) NO output zpaqfranz pause -n 5 -pakka -silent
New -rename in summa()
Rename files by hashes, with -force to... enforce
Rename all files in z: with -xxh3 hash (do not ask -force)
First run
zpaqfranz sum z:\* -rename -xxh3 -force
Now check if everything match (second run)
zpaqfranz sum z:\* -rename -xxh3 -force
(...)
Creating 32 hashing thread(s) with XXH3
Renamed files 0
NOT renamed files 40.491
The main thing: autotest
There are now three levels of verification to verify operation on different systems
First: hasher quick check (the output must be this one, you can use -n X to extend)
C:\zpaqfranz\release\55_14>zpaqfranz autotest
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 5 2022)
Self-test for correct internal functioning
Quick check chunsize 5: ABCDE
This seems a LITTLE ENDIAN CPU (aka:'normal')
-----------------------------------------------------------------------------------------------------
CALC BLAKE3: 61274278289E9F6233DF34ABB392AAFE03EE7CE9D77167F3A8D9CDE1AD9861C0
CALC CRC-32: 72D31AD5
CALC CRC-32C: B6AFE183
CALC MD5: 2ECDDE3959051D913F61B14579EA136D
CALC SHA-256: F0393FEBE8BAAA55E32F7BE2A7CC180BF34E52137D99E056C817A9C07B8F239A
CALC SHA-3: 034AF02F68F8874B6668CCBEE49143A64BE435610E1282D93BF35FD80ACCE1FB
CALC SHA1-PUT: 7BE07AAF460D593A323D0DB33DA05B64BFDCB3A5
CALC SHA1-WRITE: 7BE07AAF460D593A323D0DB33DA05B64BFDCB3A5
CALC WHIRLPOOL: C73D8F890F181CE6EB9FF431E08828D6BADA50AC2427546BA10A8F8226F527850FB61E638F798CE86028248262DF17D77D9D00FA5FE5E6CE6C94267E1DC2E99C
CALC XXH3: 1C8288B6013152D97B4A5D7E6C7893D4
CALC XXHASH64: E47599E7C7CEF609
CALC XXHASH64Y: E47599E7C7CEF609
-----------------------------------------------------------------------------------------------------
Time 0.00 seconds for bytes 0
Second: more thorough verification, -all
C:\zpaqfranz\release\55_14>zpaqfranz autotest -all
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 5 2022)
Self-test for correct internal functioning
Iteration 0/9 chunksize 1.000.000
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
Iteration 3/9 chunksize 37.037
Iteration 4/9 chunksize 12.345
Iteration 5/9 chunksize 4.115
Iteration 6/9 chunksize 1.371
Iteration 7/9 chunksize 457
Iteration 8/9 chunksize 152
Iteration 9/9 chunksize 50
This seems a LITTLE ENDIAN CPU (aka:'normal')
BLAKE3 : OK
CRC-32 : OK
CRC-32C : OK
MD5 : OK
SHA-256 : OK
SHA-3 : OK
SHA1-PUT : OK
SHA1-WRITE : OK
WHIRLPOOL : OK
XXH3 : OK
XXHASH64 : OK
XXHASH64Y : OK
Time 25.58 seconds for bytes 1.748.003.691
Third: the paranoid, -all -to
This will write, into -to, a special folder, with a .bat/.sh file
C:\zpaqfranz\release\55_14>zpaqfranz autotest -all -to z:\checkme
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 5 2022)
Self-test for correct internal functioning
Creating autotest folder in <<z:/checkme/>>
Iteration 0/9 chunksize 1.000.000
(...)
Time 31.33 seconds for bytes 1.748.003.691
The test batchfile is: z:\checkme\dotest.bat
31.360 seconds (000:00:31) (all OK)
Getting something like that
Z:\checkme>dir
Il volume nell'unità Z è RamDisk
Numero di serie del volume: 843A-4C36
Directory di Z:\checkme
05/09/2022 11:14 <DIR> .
05/09/2022 11:14 <DIR> ..
05/09/2022 11:13 <DIR> 00
05/09/2022 11:13 <DIR> 01
05/09/2022 11:14 <DIR> 02
05/09/2022 11:14 <DIR> 03
05/09/2022 11:14 <DIR> 04
05/09/2022 11:14 <DIR> 05
05/09/2022 11:14 <DIR> 06
05/09/2022 11:14 <DIR> 07
05/09/2022 11:14 <DIR> 08
05/09/2022 11:14 <DIR> 09
05/09/2022 11:14 2.961 dotest.bat
05/09/2022 11:13 149.066 sha256.zpaq
Now, run the batchfile inside the testfolder (needs ~4GB free space)
You get a lot of things
First test
(...)
Found (9.04 MB) => 9.481.472 bytes (9.04 MB) / 256 files in 0.000000
Creating 1 hashing thread(s) with SHA-256
Renamed files 0
NOT renamed files 256
0.063 seconds (00:00:00) (all OK)
(1) You should read NOT renamed files 256
**** Hit a key to continue ****
NOT renamed files 256 means: yes, I can extract an "alien" .zpaq in this system
After pressing a key you should get here
zpaqfranz v55.14b-experimental-JIT-L (HW BLAKE3), SFX64 v55.1, (Sep 5 2022)
franz:-sha256
Getting SHA-256 ignoring .zfs and :$DATA
Found (976.56 KB) => 1.000.000 bytes (976.56 KB) / 1 files in 0.015000
Creating 1 hashing thread(s) with SHA-256
SHA-256: 67708591460BCE3BC45AE086A342F9F390AD2913A22639EC7AF3646B7D2AEA78 [ 1.000.000] z:/checkme/verifica
0.031 seconds (00:00:00) (all OK)
(2) You should read SHA-256: 67708591460BCE3BC45AE086A342F9F390AD2913A22639EC7AF3646B7D2AEA78
**** Hit a key to continue ****
The situation is: yes, I can make an archive, then extract a single file, and this match
Now a lot of other things and you came here
SUMMARY : 15.58 (total time)
Virtual : 10 (skipped, does not exists in 7.15)
Total : 42.727
GOOD : 42.727 of 42.727 (stored=decompressed)
SURE : 42.727 of 42.727 (stored=decompressed=file on disk)
All OK (paranoid test with check against filesystem)
15.625 seconds (000:00:15) (all OK)
(3) You should read SURE : 42.727 of 42.727 (stored=decompressed=file on disk)
**** Hit a key to continue ****
This means: I made an archive, and this is good
Now the last: can zpaqfranz extract files (on the filesystem)?
Worked on 971.003.691 bytes avg speed (hashtime) 862.347.860 B/s
GLOBAL SHA256: 7B32DB3F6F0180062773C5A046A2C99D17BE7668202379BA136E7A4D134FFC26
2.313 seconds (000:00:02) (all OK)
(4) You should read GLOBAL SHA256: 7B32DB3F6F0180062773C5A046A2C99D17BE7668202379BA136E7A4D134FFC26
**** Hit a key to continue ****
Windows 32/64 binary
New switch -recursion (on a add command)
Do not recurse into subdirs
zpaqfranz a z:\1.zpaq f:\zarc\*.* -norecursion
When add() with comment, check for not-unique comment
C:\zpaqfranz>zpaqfranz a z:\1.zpaq *.txt -comment "onlytext"
zpaqfranz v55.10i-experimental archiver, SFX64 v55.1, compiled Aug 14 2022
(...)
0.032 seconds (00:00:00) (all OK)
Now the same comment
C:\zpaqfranz>zpaqfranz a z:\1.zpaq *.txt -comment "onlytext"
zpaqfranz v55.10i-experimental archiver, SFX64 v55.1, compiled Aug 14 2022
(...)
38579: version comment must be unique, aborting <<onlytext>>
0.016 seconds (00:00:00) (with errors)
Improved -comment handling on l (list)
C:\zpaqfranz>zpaqfranz l z:\1.zpaq -comment
zpaqfranz v55.10i-experimental archiver, SFX64 v55.1, compiled Aug 14 2022
franz:-comment <<>>
z:/1.zpaq:
2 versions, 22 files, 171 fragments, 6 blocks, 951.004 bytes (928.71 KB)
Version(s) enumerator
00000001 <<onlytext>>
00000002 <<thecpp>>
0.000 seconds (00:00:00) (all OK)
C:\zpaqfranz>zpaqfranz l z:\1.zpaq -comment "thecpp"
zpaqfranz v55.10i-experimental archiver, SFX64 v55.1, compiled Aug 14 2022
franz:-comment <<thecpp>>
z:/1.zpaq:
2 versions, 22 files, 171 fragments, 6 blocks, 951.004 bytes (928.71 KB)
Version(s) enumerator
00000002 <<thecpp>>
Found version -until 2 scanning again...
z:/1.zpaq -until 2:
2 versions, 22 files, 171 fragments, 6 blocks, 951.004 bytes (928.71 KB)
(...)
Improved comment on i (info command)
C:\zpaqfranz>zpaqfranz i z:\1.zpaq -comment
zpaqfranz v55.10i-experimental archiver, SFX64 v55.1, compiled Aug 14 2022
franz:-comment <<>>
z:/1.zpaq:
2 versions, 22 files, 171 fragments, 6 blocks, 951.004 bytes (928.71 KB)
-------------------------------------------------------------------------
< Ver > < date > < time > < added > <removed> < bytes added >
-------------------------------------------------------------------------
00000001 2022-08-14 16:26:16 +00000004 -00000001 -> 12.497 <<onlytext>>
00000002 2022-08-14 16:27:33 +00000018 -00000001 -> 938.507 <<thecpp>>
0.015 seconds (00:00:00) (all OK)
utf command reworked
Check/sanitize utf-8 charsets in folder / filenames
Check (dry run by default)
C:\zpaqfranz>zpaqfranz utf z:\kekke
zpaqfranz v55.10i-experimental archiver, SFX64 v55.1, compiled Aug 14 2022
Fix UTF-8 dry run (because no -kill)
UTF-8 dirs: 157
UTF-8 files: 6.564
LONG files: 1
*** WINDOWS WARNING *** found long path. Suggestion: use -longpath switch
4.828 seconds (000:00:04) (all OK)
Fix (look at -longpath on Windows)
zpaqfranz utf z:\kekke -kill -longpath
b (benchmark) -debug
Quick-and-dirty test for checking compatibility with "non Intel" (aka arm, PowerPC etc) CPU. Not reliable at 100% (BIG ENDIANs are hard to find)
C:\zpaqfranz>release\55_10\zpaqfranzhw b -debug
zpaqfranz v55.10-experimental (HW BLAKE3,SHA1), SFX64 v55.1, compiled Aug 14 2022
franz:DEBUG very verbose (-debug)
Autotest for main hash functions
Iteration 0/9 chunksize 1.000.000
CPU feature 001F
Iteration 1/9 chunksize 333.333
Iteration 2/9 chunksize 111.111
(...)
Iteration 9/9 chunksize 50
This seems a LITTLE ENDIAN CPU (aka:'normal')
BLAKE3 : OK
(...)
XXHASH64 : OK
Time 25.89 seconds for bytes 1.748.003.691
25.921 seconds (000:00:25) (all OK)
e (extract) filenames only
Recreating the file tree, without unpacking the archive. It serves - essentially - to check that non-Latin files that are too long /forbidden can be restored on a regular basis
C:\zpaqfranz>zpaqfranz x stazz.zpaq -to z:\testdir\ -debug -kill -zero -longpath
zpaqfranz v55.10i-experimental archiver, SFX64 v55.1, compiled Aug 14 2022
(...)
****** WARNING: -zero switch. Create 0 bytes files on filesystem
****** Full-scale extraction test (UTF-8 strange filenames, path too long...)
****** Highly suggested output on RAMDISK
stazz.zpaq:
1 versions, 515.646 files, 1 fragments, 6.446 blocks, 13.255.059 bytes (12.64 MB)
32152: -longpath and ONE to: deal with rename()
Extracting 0 bytes (0.00 B) in 480.594 files (35.052 folders) with 32 threads
**** CREATING FILES AND FOLDER STRUCTURE *****
**** EXTRACTIBILITY TEST (ON FILESYSTEM) *****
(...)
Start counting... done in 0.23s
Folders 35.052 (long 6)
Files 480.594 (long 4.069)
Done 515K / 515.646
Time 42.69s
Total 515.646
Folders 35.052
Files 480.594
Errors 0
49.953 seconds (000:00:49) (all OK)
33040: call xcommand with a different errorcode (not 1, not 2) 0
switch -out works (almost) everywhere
It is possible to redirect output with -out
zpaqfranz a z:\1.zpaq *.txt -out thelog.txt
switch -silent to "cut off" output
Fix space check on Linux
A long-standing "bug" that reports 0 bytes free
Switch -to into command q (Windows' C: copy)
For example with
-to c:\snap_$PCNAME
it is possible to archive multiple different PCs on a single backup (same VM templates etc)
Switch -frugal cut off more folders
During q/g commands (Windows C) -frugal cuts more folders
- /ProgramData/Microsoft/Windows Defender/;
- /ProgramData/Microsoft/Search/;
- /ProgramData/Package Cache/;
A bit of source refactoring (newer Linux compiler), some comments deleted
Improved non arm64 and x86+SSE2 (aka: "Intel) support by -NOJIT
Windows 32/64 binary
Only a very minor fix in source code for a "lost commit" (github, I love you too!)
Identical to 55.6
Source forge link to 55.6
I hope this will compile on non-x86 CPUs (almost never tested, sorry)
Windows 32/64 binary
Support for HW accelerated SHA1 instructions (Windows 64)
Thanks to Igor Pavlov (7-Zip's author) I introduce a new EXE: zpaqfranzhw.exe with the new switch -hw to use (if any) CPU's HW SHA1 instructions.
Not very "Intel friendly" (only a few Intel models), but more common in the AMD field: the Ryzen processors implement them (AFAIK) since 2017
The difference in overall performance is modest.
However, if you have a Ryzen CPU (like me) ... just try and DO NOT forget the -hw :)
More advanced errors (Windows)
Filesystems error (access denied etc) are now kept (on Windows) and written briefly (-verbose) or in detail (-debug) after add()
In case of error zpaqfranz try to get filesystem's attributes, very helpful for debug (aka: send me!)
In this example it's very clear "why"
-verbose
File errors report
Error 00000032 # 1 |sharing violation |
-debug
File errors report
Error 00000032 # 1 |sharing violation |
c:/pagefile.sys>>
00000026 ARCHIVE;HIDDEN;SYSTEM;
Refactored help
The "no command" is now this (a bit tighter). Of course Windows and Non-Windows are different
The "help on help" (can be asked by zpaqfranz /? zpaqfranz h zpaqfranz -h etc) is more concise, but more readable
Commands "kind of C: backup" g/q refined a bit
Default exclusions are:
- c:\windows
- RECYCLE BIN
- %TEMP%
- ADS
- .zfs
- pagefile.sys
- swapfile.sys
- System Volume Information
- WindowsApps (maybe)
Switches
-frugal
Exclude program files and program files x86
-forcewindows
Takes C:\WINDOWS, $RECYCLE.BIN, %TEMP%, ADS and .zfs
-all
All EXCEPT pagefile, swapfile, system volume information, WindowsApps
Starting of reparsepoint support
Windows have a LOT of strange files (for example in WindowsApps folder), work in progress...
Windows 32/64 binary
20-07-2022: 55.5
Command q (paQQa) on Windows: archive ("fake backup") of drive C
The new command archive (a large part) of the C: drive, excluding, by default, the swap files, system volume information, and the Windows folder
Admin rights are required, and the C:\FRANZSNAP FOLDER MUST NOT EXISTS
EVERY KIND OF PRE-EXISTING VSS WILL BE DELETED
Usage is quite easy
zpaqfranz q z:\pippo.zpaq
the -forcewindows add the c:\windows folder, the %TEMP% and ADS
Almost all add() switches are enabled, with the exception of -to
zpaqfranz q z:\pippo.zpaq -only *.cpp -key migthypassword
This is NOT a bare-metal restorable backup but a (blazingly fast) snapshotted/versioned archiving of precious data
On my PC updating the "backup" (aka: new version) takes less than 2 minutes for ~200GB C: drive
Note: some special folders cannot be accessed/archived at all (Windows' Defender, CSC etc.), because Windows does not like. I am working on it
This is a "rebranded" 55.4 release. Github does not like multiple release with the same number
Windows 32/64 binary
16-07-2022: 55.2
Command r (robocopy) MAYBE works with -longpath on Windows (caution using r)
Some minor fixes in help
Source code in "Unix" text format (no CR/LF) for easy ancient make compatibility
Windows 32/64 binary
Build 55.2
Some fixes on ETA computations
"Smarter" -longpath getfreediskspace()
A little bit better handling of verify on legacy 7.15 archives
-verbose during a (add) shows individual % of files bigger than 100.000.000 (10%) or 1.000.000.000 (1%)
One of the problems with zpaq is that, while updating a large, little changed file, there is no response: the program seems to hang
This is evident, in particular, for virtual machine disks, hundreds of GBs each, in which perhaps only a few MB need to be added to the archive.
In this case you get zero feedback maybe for an hour (when zpaq is reading and deduplicating data), then 5 seconds of ETA (compressing delta data), then all is done.
So now, using the -verbose switch, you get a progress indicator on big files (nothing on smaller one to reduce console output time).The indication is not accurate, but it is better than nothing
-touch switch on command a add()
Converting a legacy 7.15 archive to zpaqfranz (aka: saving CRC-32s and hashes for every file) IN PLACE (=without extracting/repack etc) is not trivial and potentially, it is a dangerous thing to do.
As a workaround I implemented the -touch switch that, during the add phase, "fakely" change the timestamp of the files.
Just about as a -force on steroids.
Therefore IF you own the original files, you can switch from 7.15 to zpaqfranz in two steps
Suppose you have a z:\1.zpaq file, created by zpaq 7.15, containing years of backups WITHOUT CRC-32. Now you want to use zpaqfranz to get better testing features. The data folder is c:\nz and is available
zpaqfranz a z:\1.zpaq c:\nz\ -touch
This will force to add ANYTHING from c:\nz into z:\1.zpaq. Lenghty, but litte space needed. Then
zpaqfranz a z:\1.zpaq c:\nz\
This time without -touch. Add, again, everything, BUT now with "true" timestamps.
Yes, I know, it is rather weird, but the complexity (on the source code) is minimal.
It is actually possible to carry out this operation WITHOUT having the data archive available, and in a much faster way. However, as it is normally a one-off operation, it is enough for me
Windows 32/64 binary
This new release introduce... new features
As all new code should be deeply tested, before production use
The more feedback and issues open on github, the more I will perfect the program
Compile on OpenBSD 6.6+ and OmniOS (open Solaris Unix) r151042
Not very common, but I like Unix more than Linux
(Windows) sfx module updated
Now, if no -to is specified, it asks from console where to extract the data
General behaviour
Show less infos during execution. It is a working in progress
Switches to REDUCE output are -noeta, -pakka and -summary
Switches to INCREASE output are -verbose and -debug
Advancing by 1s in progress infos
The "dir" command, by default, show dates in European format
Instead of 2022-12-25 => 25/12/2022
It does NOT use "local" translator, because it's way too hard to support many platforms
-utc turn off local time (just like 7.15)
-flagflat use mime64 encoded filenames
In some cases there were problems with reserved words on NTFS filesystems
On Windows more extensive support for -longpath (filenames longer than 255 chars)
On Windows -fixcase and -fixreserved
Handle case collisions (ex. pippo.txt and PIPPO.txt) and reserved filenames (ex lpt1)
Disabled some computation. Use -stat to re-activate
Some checks can be slow for huge archives (reserved filenames, collisions etc)
The -all switch, for turning on multi-thread computation, is now -ssd
In zpaqfranz 55- -all can means "all versions" or "all core". On 55+ (zpaqfranz's extensions) for multithread read/write use -ssd
command d
It is now possible to use different hashes
zpaqfranz d c:\dropbox\ -ssd -blake3
command dir
It is possible now to use different hashes to find duplicates
zpaqfranz dir c:\dropbox\ /s -checksum -blake3
Main changes command a (add)
-debug -zero Add files but zero-filled (debugging)
If you want to send me some kind of strange archive, for debug, without privacy issues
-debug -zero -kill Add 0-byte long file (debugging)
Store empty files, very small archive, to test and debug filesystem issues
-verify Verify hashes against filesystem
-verify -ssd Verify hashes against filesystem MULTITHREAD (do NOT use on spinning drives)
Command i (information)
Is now about 30% faster
-stat to calc statistics on files, like case collisions (slow)
On Windows: new command rd()
Delete hard-to-erase folders on Windows
-force Remove folder if not-zero files present
-kill wet-run
-space do not check writeability (example delete folder from a 0 bytes free drive)
And now... the main thing!
command w Chunked-extraction
Extract/test in chunks, on disk or 'ramdisk' (RAM)
The output -to folder MUST (should) BE EMPTY
The w command essentially works like the x (extract) command but in chunks
It can extract the data into RAM, and to simply check (hash) it, or even write it in order
PRELIMINARY NOTE AGAIN: the -to folder MUST BE EMPTY (no advanced checks-and-balance as zpaq)
There are various scenarios
-
Extracting on spinning drive (HDD)
zpaq's extraction method is not very fit for spinning drive, because can make a lot of seeks while writing data to disk. This can slow down the process. On media with lower latency (SSD, NVMe, ramdisk) the slowdown is much less noticeable. Basically zpaq "loves" SSD and isn't very HDD friendly.
If the largest file in the archive is smaller than the free RAM on your computer, you can use the -ramdisk switch
Extract to a spinning drive (Windows)
zpaqfranz w z:\1.zpaq -to p:\muz7\ -ramdisk -longpath
In this example the archive 1.zpaq will be extracted in RAM, then written on p:\muz7 folder (suppose an HDD drive) at the max speed possible (no seeks at all) -
Checking the hashes without write on disk AND MULTITHREAD AND MULTIPART and whatever
The p (paranoid) command can verify the hash checksum of the files into the archive WITHOUT writing to disk, but it is limited to a SINGLE CORE computation, without multipart archive support
The w command (if the largest file in the archive is smaller than free RAM) does not have such limitations
zpaqfranz w z:\1.zpaq -ramdisk -test -checksum -ssd -frugal -verbose
will test (-test, no write on drive) the archive 1.zpaq, into RAM (-ramdisk), testing hashes (-checksum), in multithread way (-ssd), using as little memory as possible (-frugal) and in -verbose mode -
Paranoid no-limit check, for huge archives (where the largest uncompressed file is bigger than free RAM)
zpaqfranz w z:\1.zpaq -to z:\muz7\ -paranoid -verify -verbose -longpath
will extract everything from 1.zpaq into z:\muz7, with longpath support (example for Windows), then do a -paranoid -verify
At the end into z:\muz7\zfranz the BAD files will be present. If everything is ok this folder should be empty -
Paranoid test-everything, when the biggest uncompressed file is smaller then RAM and using a SSD/NVMe/ramdisk as output (z:)
zpaqfranz w z:\1.zpaq -to z:\kajo -ramdisk -paranoid -verify -checksum -longpath -ssd
Recap of switches
- : -maxsize X Limit chunks to X bytes
- : -ramdisk Use 'RAMDISK', only if uncompressed size of the biggest file (+10%) is smaller than current free RAM (-25%)
- : -frugal Consume as litte RAM as possible (default: get 75% of free RAM)
- : -ssd Multithread writing from ramdisk to disk / Multithread hash computation (ex -all)
- : -test Do not write on media
- : -verbose Show useful infos (default: rather brief)
- : -checksum Do CRC-32 / hashes test in w command (by default: NO)
- : -verify Do a 'check-against-filesystem'
- : -paranoid Extract "real" to filesystem, then delete if hash OK (need -verify)
Yes, I understand, the w command can seems incomprehensible (but this is a zpaq fork, afterall :)
In fact it is developed to avoid the limitations of zpaq in the management of very large archives with huge files (for example virtual machine disks) kept on spinning HDD or handling archives containing a very large number (millions) of relatively small files (such as for example the versioned backup of a file server full of DOC, EML, JPG etc), to be checked on a high-powered machine (with many cores, lots of RAM and SSD), without wearing the media. As is known, writing large amounts of data reduces the lifespan of SSDs and NVMes (HDDs too, but to a lesser extent). And remember: the p command is monothread AND cannot handle archive bigger than RAM.
To make a "quick check" of the "spiegone" try yourself: compare the execution time on a "small" archive (=uncompressed size smaller than your RAM, say 5/10GB for example)
zpaqfranz p p:\1.zpaq
against
zpaqfranz w p:\1.zpaq -test -checksum -ramdisk -ssd -verbose