Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding --long support for --patch-from #1959

Merged
merged 34 commits into from
Apr 17, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
73edb1b
adding long support for patch-from
Apr 7, 2020
77aa272
adding refPrefix to dictionary_decompress
Apr 7, 2020
81549bc
adding refPrefix to dictionary_loader
Apr 7, 2020
69cbfc1
conversion nit
Apr 7, 2020
4c9fc6d
triggering log mode on chainLog < fileLog and removing old threshold
Apr 7, 2020
f9335a6
adding refPrefix to dictionary_round_trip
Apr 7, 2020
1f339cc
adding docs
Apr 7, 2020
510e08b
adding enableldm + forceWindow test for dict
Apr 8, 2020
9df3e9e
separate patch-from logic into FIO_adjustParamsForPatchFromMode
Apr 8, 2020
07a204a
moving memLimit adjustment to outside ifdefs (need for decomp)
Apr 8, 2020
c949f2e
removing refPrefix gate on dictionary_round_trip
Apr 8, 2020
fe332df
rebase on top of dev refPrefix change
Apr 8, 2020
7eaafaa
making sure refPrefx + ldm is < 1% of srcSize
Apr 8, 2020
ec5c4c6
combining notes for patch-from
Apr 8, 2020
abda0ae
moving memlimit logic inside fileio.c
Apr 8, 2020
f6d66bd
adding display for optimal parser and long mode trigger
Apr 8, 2020
d67bf90
conversion nit
Apr 8, 2020
36d50ec
fuzzer found heap-overflow fix
Apr 9, 2020
80cd488
another conversion nit
Apr 9, 2020
d054841
moving FIO_adjustMemLimitForPatchFromMode outside ifndef
Apr 9, 2020
025b44a
making params immutable
Apr 9, 2020
6e9baf8
moving memLimit update before createDictBuffer call
Apr 9, 2020
ffd1a8a
making maxSrcSize unsigned long long
Apr 9, 2020
e60e7a6
making dictSize and maxSrcSize params unsigned long long
Apr 9, 2020
16c826d
error on files larger than 4gb
Apr 9, 2020
0320dbf
extend refPrefix test to include round trip
Apr 9, 2020
14e075d
conversion to size_t
Apr 9, 2020
c6e9d7f
making sure ldm is at least 10x better
Apr 15, 2020
2177a0d
removing break
Apr 15, 2020
aa17451
including zstd_compress_internal and removing redundant macros
Apr 16, 2020
92a9775
exposing ZSTD_cycleLog()
Apr 16, 2020
fc4844a
using cycleLog instead of chainLog
Apr 16, 2020
7b113a1
add some more docs about user optimizations
Apr 17, 2020
a18409c
formatting
Apr 17, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions programs/fileio.c
Original file line number Diff line number Diff line change
Expand Up @@ -814,6 +814,7 @@ static void FIO_adjustParamsForPatchFromMode(FIO_prefs_t* const prefs,
}
if (cParams.strategy >= ZSTD_btopt) {
DISPLAYLEVEL(1, "[Optimal parser notes] Consider the following to improve patch size at the cost of speed:\n");
DISPLAYLEVEL(1, "- Use --single-thread mode in the zstd cli");
bimbashrestha marked this conversation as resolved.
Show resolved Hide resolved
DISPLAYLEVEL(1, "- Set a larger targetLength (eg. --zstd=targetLength=4096)\n");
DISPLAYLEVEL(1, "- Set a larger chainLog (eg. --zstd=chainLog=31)\n");
DISPLAYLEVEL(1, "Also consdier playing around with searchLog and hashLog\n");
Expand Down
4 changes: 3 additions & 1 deletion programs/zstd.1.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,9 +131,11 @@ the last one takes effect.
Note: `--long` mode will be automatically activated if chainLog < fileLog
(fileLog being the windowLog requried to cover the whole file). You
can also manually force it.
Node: for all levels, you can use --patch-from in --single-thread mode
to improve compression ratio at the cost of speed
Note: for level 19, you can get increased compression ratio at the cost
of speed by specifying `--zstd=targetLength=` to be something large
(i.e 4096)
(i.e 4096), and by setting a large `--zstd=chainLog=`
* `-M#`, `--memory=#`:
Set a memory usage limit. By default, Zstandard uses 128 MB for decompression
as the maximum amount of memory the decompressor is allowed to use, but you can
Expand Down