Speed considerations

[shirish93]

Hey cloudpipe team!

I'm doing an exploratory analysis for the gensim library to potentially use cloudpickle (here's the discussion), and noticed that 'regular' cloudpickle is consistently ~8x slower than python's pickle module for pretty much all the data structures I threw at it.

Is this the expected normal behavior, or am I doing something wrong in my tests? I'm using python2.7/3.4 on windows, without C-compilers (not using the optimized versions if there are any),

Would you guys have any ideas if we could modify the module selectively for certain tasks to improve performance on the most-used features?

[rgbkrk]

If you have fixes for optimizations, send patches right along!

This code base largely comes from the original cloud module, broken out, relicensed, and patched by both pyspark contributors and the folks who have contributed directly here.

[ogrisel]

It's going to be slower than the pickle implementation of Python 3 or the cPickle of Python 2 as they are implemented in C and can be 10x faster on large Python objects composed of many sub-objects (e.g. a Python dict or list with millions of entries).

On small Python objects with few subobjects (e.g. a tuple with a couple of big strings or large numpy arrays) you should not see a significant difference in speed.

[ogrisel]

If you have specific speed improvements please feel free to open PRs but let's close this issue as their is no easy general resolution.