-T LZVN doesn't work

[lucianmarin]

afsctool -T LZVN -c /path/to/random/file
/path/to/random/file: unsupported compression type 2 bytes
Unable to compress file (already compressed).

I think LZVN support is enabled (cmake LZVN repo then cmake afsctool), but it doesn't seem to work on High Sierra.

[RJVB]

afsctool -T LZVN -c /path/to/random/file

Two things seem to be going on here:

/path/to/random/file: unsupported compression type 2 bytes

This message indicates that LZVN support has in fact not been enabled.

Unable to compress file (already compressed).

And this suggests that your file was already compressed. But it can be an artefact because of LZVN support not having been enabled. What does `/usr/bin/ls -lO /path/to/random/file` print?

I think LZVN support is enabled (cmake LZVN repo then cmake afsctool), but it doesn't seem to work on High Sierra.

Did you install LZVN? Just building it isn't enough, the headers and library must be available in a location where cmake can find them. Cmake should report whether LZVN support is being enabled. I've made a few small changes to afsctool so that it gives clearer error messages when LZVN support has not been enabled. Please pull afsctool, rebuild and try to compress your random file again. Then, install LZVN and re-cmake and re-build afsctool .

[lucianmarin]

make install LZVN

Install the project...
/usr/local/Cellar/cmake/3.13.2/bin/cmake -P cmake_install.cmake
-- Install configuration: ""
-- Up-to-date: /usr/local/lib/libFastCompression.1.0.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.1.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.a
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: /usr/local/lib/libFastCompression.a(liblzvn.c.o) has no symbols
-- Up-to-date: /usr/local/bin/lzvn
-- Up-to-date: /usr/local/include/lzvn.h
-- Up-to-date: /usr/local/include/FastCompression.h

cmake -Wno-dev .. afsctool (after last git pull)

-- LZVN_HEADER,LIB: /usr/local/include/FastCompression.h /usr/local/lib/libFastCompression.dylib
-- Enabling LZVN support
-- Setting afsctool version to: 1.6.9. (1.6.8.6-127-g731aeaa)

Later edit: seems to work when I do cmake .. instead of cmake -Wno-dev .. and make install instead of make install/fast for both projects.

afsctool -v afsctool
File is HFS+/APFS compressed.
Compression type: LZVN in resource fork (8)
Compression savings: 71.0%

[lucianmarin]

Can you make LZVN the default compression format if support is enabled?

afsctool -c afsctool
afsctool -v afsctool
File is HFS+/APFS compressed.
Compression type: ZLIB in resource fork (4)
Compression savings: 73.6%

I also get lzvn compression failed on chunk #0 on some files if they were previously ZLIB compressed then decompressed then compressed with LZVN. ZLIB works on those files, but LZVN doesn't. They are too tiny for LZVN compression. lzvn infile outfile produces a 0 bytes outfile.

[RJVB]

I could but I won’t.

[gingerbeardman]

LZVN works for me on High Sierra, so you'll need to check that you are compiling afsctool correctly.

I use:

afsctool -c -T LZVN "$@"

I'm using afsctool 1.6.9. (1.6.8.6-125-ge269dbf) as I've not built a new version in a while.

[RJVB]

I also get lzvn compression failed on chunk #0 on some files if they were previously ZLIB compressed then decompressed then compressed with LZVN. ZLIB works on those files, but LZVN doesn't. They are too tiny for LZVN compression. lzvn infile outfile produces a 0 bytes outfile.

I didn't notice this part. I'd need to have one of those files to do some testing, but if lzvn fails itself there probably isn't much I can do about that.

I don't really get why you'd want to recompress files with LZVN though. In every test I did it compresses less well than (higher levels of) ZLIB. It may decompress faster, but do you really notice that IRL? OS X has a rather agressive file cache, and to the best of my knowledge this comes after the file system (i.e. it caches data after "HFS decompression").

[gingerbeardman]

FWIW I use LZVN because macOS App Store uses LZVN, and I mainly compress apps so I want to keep things the same as those apps.

[RJVB]

Whatever floats your boat :) I tend to recompress apps to make them smaller, not to do things the way Apple do them. Apple also makes money from selling disk space so they have a vested interest in not minimising the space occupied by the software they sell ;)

[gingerbeardman]

I read that here the other day, and I countered by saying that Apple ship their OS and apps from the Mac App Store with HFS+ compression turned on, so I don't really see the reasoning?

My reason is not to "do it like Apple" but so that my scripts and workflow do not undo what Apple have done and then redo with the same. Also, LZVN is more efficient in terms of speed and CPU usage, so I don't notice when it's working in the background (compared to ZLIB that makes my fans go crazy). But, yes, whatever works for each of us.

Also, try FSNotes for an app that gives this:

/Applications/FSNotes.app/Contents/Resources/de.lproj/InfoPlist.strings: lzvn compression failed on chunk #0

[lucianmarin]

Also, try FSNotes for an app that gives this:

/Applications/FSNotes.app/Contents/Resources/de.lproj/InfoPlist.strings: lzvn compression failed on chunk #0

I can confirm this on multiple apps. LZVN utility produces 0 bytes output on the same files.

[RJVB]

Apple ship their OS and apps from the Mac App Store with HFS+ compression turned on

But not with the most efficient kind of compression.

lzvn compression failed on chunk #0

I can reproduce that. It's not overly surprising given that the input file is only 2 bytes large... It looks like there is an undocumented 8 byte lower limit to LZVN compression, under which it simply fails. I pushed updates to lzvn and afsctool in order to ignore files < 8 bytes (and to improve verbose output a bit, the error I mentioned elsewhere that I wanted to fix).

[gingerbeardman]

Thanks René!

[gingerbeardman]

define for LZVN_MINIMUM_COMPRESSABLE_SIZE is missing in latest commit
make sure you pull, build and install the latest LZVN

[malthe]

Now we just need ARM-support for LZVN :-)

[RJVB]

Now we just need ARM-support for LZVN :-)

In the library I presume, or is there (also) an endianness issue in my own code?

[malthe]

In the library yeah – there's lots of assembly code in there.

[RJVB]

I'm going to compare that implementation with the generic-C implementation in liblzfse. Turns out there's one in there... If performance is close enough to the one from the assembly code I'm just going to use a single dependency.

[RJVB]

@malthe - Can you check the lzfse branch please (be sure to initialise the submodule!)? A priori it should provide LZVN and LZFSE support on both Apple's architectures now.

[malthe]

@RJVB works perfectly.

[RJVB]

@RJVB works perfectly.

That was fast! :) Both LZVN and LZFSE compression?

[malthe]

It takes quite a while, but this is an 8GB run:

Number of HFS+/APFS compressed files: 140960
Total number of files: 145251
Total number of folders: 20516
Total number of items (number of files + number of folders): 165767
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 7840604172 bytes / 8.2 GB (gigabytes, base-10)
Folder size (compressed): 3696919434 bytes / 3.45 GiB
Compression savings: 52.8% over 140960 of 145251 files
Approximate total folder size (files + file overhead + folder overhead): 3704770889 bytes / 3.45 GiB

(Both LZVN and LZFSE tested.)

[RJVB]

You could have tested with a single file ;) For reference, LZVN compression of a 128Mb file takes about 3.5s on my 2nd gen. i7 using the lzfse library. That's maybe 10% slower than with the assembly implementation. That surprises me a little bit, the C implementation almost looks like assembly, with lots of cases and gotos so it wouldn't surprise me if the assembly implementation actually came from an older version of that same code. We'll see if I get flamed for a performance regression from users on Intel hardware!

[gingerbeardman]

@malthe off-topic, slightly, but regarding your (old) blog post: https://maltheborch.com/#using-built-in-transparent-compression-on-macos ditto is in-built tool that can do this (HFS+ in-place compression) to a degree, and Apple do it for all system files in an install and all apps installed/updated using the Mac App Store. Just an FYI

[malthe]

@gingerbeardman thanks – I was planning to update that post once this is all merged in. Will try and see if I can come up with a command-line for ditto.

[gingerbeardman]

@malthe from man it's ditto --hfsCompression src dest

[malthe]

@gingerbeardman doesn't work if src is the same as dest though:

<src> and <dest> are identical (not copied).

[gingerbeardman]

ah yes, you have to do a dance through a temporary file

[RJVB]

FWIW, it's always been a pet theory of mine that Apple don't provide a tool for in-place compression (and even less so, transparent automatic FS-level compression) because they also sell storage space. You may remember that it was long possible (and somewhat common usage) to clean off all the useless translation data (which often comes with copies of image resources) and indeed even the OS X installer allowed you to select to install only the languages you had use for (which would then be "completed" during subsequent updates which didn't have that option). App bundle signing has made that impossible.

[gingerbeardman]

I've heard that before, but my counter is that they apply this to the system and Mac App Store so they're doing more than nothing.

I do think they should enable it by default but there are so many issues that would raise.

Apple storage is pretty reasonable price these days.

[gingerbeardman]

Oh, and I use CleanMyMac to trim language files which still works well.

[RJVB]

My counter would be that they needed compression to bring down the OS and size of most apps to something reasonable enough to be able to sell their entry-level computers with fixed storage back when those barely had enough space. Once you start deploying compression it probably becomes cheaper just to apply it unconditionally to everything.

Apple storage is pretty reasonable price these days.

Yeah right, as in just buy a new computer because you can't increase your internal storage? ;) I presume "CleanMyMac" only removes language resources that are not inside app bundles, and possibly from apps that aren't signed (or are known not to be crippled when they lose their signing)?

[gingerbeardman]

I presume "CleanMyMac" only removes language resources that are not inside app bundles, and possibly from apps that aren't signed (or are known not to be crippled when they lose their signing)?

I'd say: you presume too much :)

Well, I just installed OmniGraffle from the Mac App Store and it stripped the localisation out of that with no effect on the running of the app.

before

after

[RJVB]

Good to know but I wonder how they pull it off. Maybe Apple changed something in the way apps are signed, taking into account the fact that removing a translation resource cannot possibly be the cause of a security issue?

[Dr-Emann]

BTW, According to libfsapfs (scroll past the table):

If the LZVN compressed data starts with 0x06 (end of stream oppcode) the data is stored uncompressed after the first compressed data byte.

Similar to how an 0xFF prefix can be used for zlib compression to store uncompressed data with only 1 byte overhead.

I think this could be used to handle cases where the last block is too small for LZVN (or if LZVN compression would ever increase the size of the block)

[RJVB]

I think this could be used to handle cases where the last block is too small for LZVN

If memory serves me well that case is already handled (and not by rejecting the entire file).

(or if LZVN compression would ever increase the size of the block)

Both options would need to be tested to see if the kernel supports them!

[Dr-Emann]

It looks like it's currently handled by rejecting the whole file:

afsctool/src/afsctool.c

Lines 239 to 246 in cadc2a3

	if (comptype == LZVN){
	int lastChunkSize = inFileInfo->st_size % 0x10000;
	if (lastChunkSize > 0 && lastChunkSize < LZVN_ENCODE_MIN_SRC_SIZE) {
	if (printVerbose >= 2) {
	fprintf( stderr, "\"%s\": file too small or will contain a too small compression chunk (try ZLIB compression)\n",
	inFile);
	}
	return false;

$ truncate -s $((0x10000 + 7)) /tmp/bad_block

$ ./afsctool -c -vvvv -T LZVN /tmp/bad_block
"/tmp/bad_block": file too small or will contain a too small compression chunk (try ZLIB compression)
"/tmp/bad_block": file too small or will contain a too small compression chunk (try ZLIB compression)
/tmp/bad_block:
Unable to compress file.
File content type: public.data
File data fork size (reported size by Mac OS X Finder): 65543 bytes / 70 KB (kilobytes, base-10)
Number of extended attributes: 0
Total size of extended attribute data: 0 bytes
Approximate total file size (data fork + resource fork + EA + EA overhead + file overhead): 69632 bytes / 68 KiB

[RJVB]

It looks like it's currently handled by rejecting the whole file:

Right, I remember now being p*d to have to reject a file for a single chunk. Would you like to have a go at implementing your idea?

[RJVB]

I have a hunch though that the original author might have thought of (and tested) the ZLIB equivalent for increasing chunks - instead he implemented an option to allow compressing them.