cloudpickle 1.5.0 breaks on python 3.6+7 without pickle5

[yannikschaelte]

cloudpickle >=1.5.0 fails to dump a complex object using python 3.6+7. Works with 1.4.1. After installing pickle5, it works (related to #370 ?). All works just fine on python 3.8.

Unfortunately, I currently cannot create a minimum test case, as I am not sure what the problem is (tests in an own project fail here https://github.com/ICB-DCM/pyABC/runs/1463556418?check_suite_focus=true, in particular pytest test/base/test_samplers.py). For simple things with local variables etc. I could not reproduce the error. Any hints are welcome, but for the moment installing pickle5 or downgrading to 1.4.0 (or upgrading to python 3.8 ;)) solves the problem for me.

Error message:

pyabc/sampler/multicore_evaluation_parallel.py:104: in sample_until_n_accepted
    simulate_one = pickle.dumps(simulate_one)
../anaconda3/envs/python37/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py:102: in dumps
    cp.dump(obj)
../anaconda3/envs/python37/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py:563: in dump
    return Pickler.dump(self, obj)
../anaconda3/envs/python37/lib/python3.7/pickle.py:437: in dump
    self.save(obj)
../anaconda3/envs/python37/lib/python3.7/pickle.py:504: in save
    f(self, obj) # Call unbound method with explicit self
../anaconda3/envs/python37/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py:745: in save_function
    *self._dynamic_function_reduce(obj), obj=obj
../anaconda3/envs/python37/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py:682: in _save_reduce_pickle5
    dictitems=dictitems, obj=obj
../anaconda3/envs/python37/lib/python3.7/pickle.py:638: in save_reduce
    save(args)
../anaconda3/envs/python37/lib/python3.7/pickle.py:504: in save
    f(self, obj) # Call unbound method with explicit self
../anaconda3/envs/python37/lib/python3.7/pickle.py:789: in save_tuple
    save(element)
../anaconda3/envs/python37/lib/python3.7/pickle.py:504: in save
    f(self, obj) # Call unbound method with explicit self
../anaconda3/envs/python37/lib/python3.7/pickle.py:789: in save_tuple
    save(element)
../anaconda3/envs/python37/lib/python3.7/pickle.py:504: in save
    f(self, obj) # Call unbound method with explicit self
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

pickler = <cloudpickle.cloudpickle_fast.CloudPickler object at 0x7f96ee0095d0>
obj = <cell at 0x7f96ee009510: empty>

    @register(CellType)
    def save_cell(pickler, obj):
        log.info("Ce: %s" % obj)
>       f = obj.cell_contents
E       ValueError: Cell is empty

[pierreglaser]

Hi,

The error does originate from cloudpickle. The lines responsible for raising the ValueError (as shown on your traceback) are

@register(CellType)
def save_cell(pickler, obj):
    log.info("Ce: %s" % obj)
    f = obj.cell_contents
    pickler.save_reduce(_create_cell, (f,), obj=obj)
    log.info("# Ce")
    return

Which are present in dill, another Python serialization library (see https://github.com/uqfoundation/dill/blob/615eee9560be64a451bc9ea276f4706d287f8d38/dill/_dill.py#L1174-L1180).

It is known that when being imported, dill generates some global side effects which will interfere with cloudpickle, see
#393. So I suspect that somewhere either in your code, or in your imported dependencies, dill is imported. Depending on what is possible/best for you, you can fix your original problem by either not importing dill,or using the workaround suggested in
#393 (comment)

[yannikschaelte]

Hi @pierreglaser , thanks for the fast response! Indeed dill is imported in another (here unused) part of the package. Delaying its import indeed fixed the error and will suffice here as both won't be used at the same time, thanks!