BLOSC_MAX_BUFFERSIZE disappeared

[mrocklin]

It appears that blosc.BLOSC_MAX_BUFFERSIZE has turned into blosc.MAX_BUFFERSIZE, which I believe causes issues within bloscpack.

[mrocklin]

cc @esc @FrancescAlted

[FrancescAlted]

Yes, that's correct. Probably I should keep maintaining the old symbols for backward compatibility. Only BLOSC_MAX_BUFFERSIZE is causing issues?

[mrocklin]

So far that's all I've run into. https://travis-ci.org/dask/dask/jobs/119893085

[FrancescAlted]

I have had a look at bloscpack, and it seems like it is pinned at python-blosc 1.2.7:

https://github.com/Blosc/bloscpack/blob/master/requirements.txt#L1

so it should not be using python-blosc 1.3.0. But anyway, I would be open to put BLOSC_MAX_BUFFERSIZE back into 1.3 series.

Thinking aloud, this should be a good moment to produce a new bloscpack based on python-blosc 1.3.0 that would implement support for the new BITSHUFFLE filter:

http://python-blosc.blosc.org/tutorial.html#using-different-filters

[esc]

So, the last time I checked, Bloscpack was pretty incompatible with anything except python-blosc 1.2.7:

#45
https://travis-ci.org/Blosc/bloscpack/jobs/76175299

This is why I decided to pin it to that specific version. Anyway, I suggest using Blosc 1.2.7 explicitly in the build-jobs, if you need to keep those going.

As for an upgrade, I'm afraid I won't have time to take care of this within any reasonable time-frame. In fact, I think I'd favor deprecating and orphaning the package in favor of things like: https://github.com/wesm/feather which is likely to be a much more maintained/developed solution. Of course you are free to do as you like with the source-code and should you decided to continue development, I can of course give you access to the PyPi project.

[FrancescAlted]

I'd be happy to adopt bloscpack and make a quick release supporting the new features introduced in python-blosc 1.3.0. So @esc please add me to the maintainers in PyPI. Thanks!

[esc]

I am not convinced that it will be "quick"... Anyway, I just checked and you already had the owner role for the project on PyPi so feel free to release as you please, there are notes about doing a release here: https://github.com/Blosc/bloscpack#maintainers-notes-on-cutting-a-release. Furthermore, you probably want to start here: https://github.com/Blosc/bloscpack/blob/master/bloscpack/args.py specifically: https://github.com/Blosc/bloscpack/blob/master/bloscpack/args.py#L296

Also note, that I'll be silently watching. 🙈 🙉 🙊 .

[FrancescAlted]

I have spent quite a bit of quality time on this but I am getting sporadic segfaults with bloscpack command line tests (via cram). More investigation reveals that python-blosc (1.3.0) also gets some segfaults. I have been able to consistently reproduce these segfaults only on Ubuntu 15.10 either with gcc 4.9.3 or gcc 5.2.1. On my MacOSX box I am not getting any issue (clang 3.7), so I tried with clang 3.6 and 3.7 on Ubuntu 15.10, and everything is fine there too, so it seems something triggered only by gcc.

Valgrind does not offer a clear sign on what's going on with the gcc compilers, so I am a bit at a loss here. Help from someone offering more insights would be highly appreciated.

[esc]

Any hunches or example code that can be used to reproduce?

[FrancescAlted]

Unfortunately very little. Yesterday I tried a lot of things like enabling/disabling different compressors, re-implemented the compression function using several approaches, looking into the refcounts and so on and so forth with no evidence of a nice hint. Anyway, I filed an issue in python-blosc about this: Blosc/python-blosc#110.

Also, yesterday I fixed several things in python-blosc, and in particular the disappeared symbols are back again: Blosc/python-blosc@a40bb58. So I think this ticket can be closed now.