Container format

[Cyan4973]

Because LZ4 doesn't define a container format, the python bindings will insert the original data size as an integer at the start of the compressed payload, like most bindings do anyway (Java...)

No pressure here, just willing to note that this sentence is no longer true : there is now a container format defined for file or stream compression : http://fastcompression.blogspot.fr/2013/04/lz4-streaming-format-final.html
(It's also present into root directory of SVN directory).

All LZ4 versions are invited to support a common format in order to improve compatibility between versions.

Best Regards

[steeve]

Hey Yann,
Thanks for the heads up.

Is the format spec implemented in lz4.c? In which case I'll find some time to update it.

[Cyan4973]

The format is implemented into lz4cli.c.
Since this file is in charge of both the command line parsing and i/o management, it may not be obvious.

[steeve]

Where is this file? I don't see it in https://code.google.com/p/lz4/source/browse/#svn%2Ftrunk

[Cyan4973]

https://code.google.com/p/lz4/source/browse/#svn%2Ftrunk%2Fprograms

[Cyan4973]

Would that help if functions to create compressed files/stream using latest published specification where given their own source file, separated from command line management ?

[steeve]

Definitely. The simpler it is to wrap, the better :)

On Mon, Jan 13, 2014 at 11:55 PM, Cyan4973 [email protected] wrote:

Would that help if functions to create compressed files/stream using
latest published specification where given their own source file, separated
from command line management ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/13#issuecomment-32221227
.

Steeve Morin
twitter.com/steeve
github.com/steeve
linkd.in/smorin

[Cyan4973]

OK, I've tried to find a way to attach files, but failed. So copying directly from the .h,
what do you think of this interface ?
Is it understandable ? Is it usable ? Is it what you expected ?

/* ************************************************** /
/ Special input/output values /
/ ************************************************** */
#define NULL_OUTPUT "null"
static char stdinmark[] = "stdin";
static char stdoutmark[] = "stdout";
#ifdef _WIN32
static char nulmark[] = "nul";
#else
static char nulmark[] = "/dev/null";
#endif

/* ************************************************** /
/ ****************** Functions ********************* /
/ ************************************************** */

int LZ4IO_compressFilename (char* input_filename, char* output_filename, int compressionlevel);
int LZ4IO_decompressFilename(char* input_filename, char* output_filename);

[steeve]

The issue is, what about stream?
Basically python's lz4 takes a string as input, and outputs a string with the data.

Is there a way for it to use char* as input and output, while maitaining the header structure?

[Cyan4973]

Could you describe what you think would be the best API for your use case ?
This would help much.

[doktorstick]

Howdy. Are there plans to resolve the incompatibility between python-lz4 and native-lz4, soon?

[steeve]

I'm stuck on other projects, but I'll surely review any PR implementing the container format!

[darkdragn]

Hey, steeve! I implemented the lz4io methods in the following fork: https://github.com/darkdragn/python-lz4

I have a tag (r119) for the pure branch from you, with lz4io implemented, and the master has incorporated lz4 r121. Right now the methods are compressFileDefault and decompressFileDefault, but tonight I'm going to make a compressFile and decompressFile that allows the user to specify the output file. Right now, the defaults assume the compressed extension is '.lz4'. Please let me know what you think. This is my first time really working on github, or with either c++ or python.

[dreamflasher]

@steeve @darkdragn are there any plans to merge this PR? :)

[jonathanunderwood]

This is fully implemented in the new project here: https://github.com/python-lz4/python-lz4