Deprecate all backends except text

[fbnrst]

In https://pymc-devs.github.io/pymc3/api/backends.html#selecting-a-backend there is some pseudo code on how to select backends:

import pymc3 as pm
db = pm.backends.Text('test')
trace = pm.sample(..., trace=db)

I tried to get a minimal example running, but I did run into some issues. First, I think db = pm.backends.Text('test') needs a context. Furthermore, for the sampling we need at least one random variable

model = pm.Model()

with model:
    db = pm.backends.Text('test')
    a = pm.Normal('a', mu=0, sd=1)
    trace = pm.sample(1000, n_init=1000, trace=db)

However, I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-195-1b336b9db146> in <module>()
      6     db = pm.backends.Text('test')
      7     a = pm.Normal('a', mu=0, sd=1)
----> 8     trace = pm.sample(1000, n_init=1000, trace=db)

/home/fabian/anaconda2/envs/py3/lib/python3.5/site-packages/pymc3/sampling.py in sample(draws, step, init, n_init, start, trace, chain, njobs, tune, nuts_kwargs, step_kwargs, progressbar, model, random_seed, live_plot, **kwargs)
    258         sample_func = _sample
    259 
--> 260     return sample_func(**sample_args)
    261 
    262 

/home/fabian/anaconda2/envs/py3/lib/python3.5/site-packages/pymc3/sampling.py in _sample(draws, step, start, trace, chain, tune, progressbar, model, random_seed, live_plot, **kwargs)
    273     try:
    274         strace = None
--> 275         for it, strace in enumerate(sampling):
    276             if live_plot:
    277                 if it >= skip_first:

/home/fabian/anaconda2/envs/py3/lib/python3.5/site-packages/tqdm/_tqdm.py in __iter__(self)
    814 """, fp_write=getattr(self.fp, 'write', sys.stderr.write))
    815 
--> 816             for obj in iterable:
    817                 yield obj
    818                 # Update and print the progressbar.

/home/fabian/anaconda2/envs/py3/lib/python3.5/site-packages/pymc3/sampling.py in _iter_sample(draws, step, start, trace, chain, tune, model, random_seed)
    373                     strace.record(point, states)
    374                 else:
--> 375                     strace.record(point)
    376             else:
    377                 point = step.step(point)

/home/fabian/anaconda2/envs/py3/lib/python3.5/site-packages/pymc3/backends/text.py in record(self, point)
     92         """
     93         vals = {}
---> 94         for varname, value in zip(self.varnames, self.fn(point)):
     95             vals[varname] = value.ravel()
     96         columns = [str(val) for var in self.varnames for val in vals[var]]

/home/fabian/anaconda2/envs/py3/lib/python3.5/site-packages/pymc3/model.py in __call__(self, state)
    755 
    756     def __call__(self, state):
--> 757         return self.f(**state)
    758 
    759 

/home/fabian/anaconda2/envs/py3/lib/python3.5/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    777 
    778             if len(args) + len(kwargs) > len(self.input_storage):
--> 779                 raise TypeError("Too many parameter passed to theano function")
    780 
    781             # Set positional arguments

TypeError: Too many parameter passed to theano function

Interestingly, using the SQLite backend gives the same error, but the example without a trace backend works just fine:

import pymc3 as pm

model = pm.Model()

with model:
    db = pm.backends.Text('test')
    a = pm.Normal('a', mu=0, sd=1)
    trace = pm.sample(1000, n_init=1000)

[twiecki]

Thanks @fabianrost84. What do you need the backend for? Have you tried the hdf5 backend?

We are considering deprecating the backends.

[fbnrst]

I thought of backends as a convenient way to store traces that result from long sampling, such that I can work with them later. Would there be another way to do that?

hdf5 backend gives the same error.

[twiecki]

Why do you need long sampling runs? Do you have poor convergence?

[fbnrst]

Haha ;) I should have been more specific. No, rather many datasets which to which I fit the same model. For one dataset the sampling is reasonably fast (~1 minute). But when I fit 20 datasets, it takes 20 minutes, and so I just wanted to store those traces in case I want to do further analysis.

[twiecki]

I see, so if you were able to store traces after sampling, that would be sufficient?

[fbnrst]

Exactly. I also tried to pickle them:

pt = pickle.dumps(trace)

However, pickle.loads(pt) then results in a recursion error.

[junpenglao]

you can also try to do pm.trace_to_dataframe and save the trace as pandas dataframe

[fbnrst]

Thanks for the suggestion @junpenglao. That would work for saving some important information. But I would loose the sampler stats, right? And, with a dataframe I loose some convenient functionalities, for instance I cannot create a traceplot from the dataframe.

Maybe to conclude from the discussion above:

1. There is an issue with the documentation of backends (because I could not get it working from what was written there). One reason might be, because it is considered to deprecate the backends.

2. There are two use-cases of backends:
   One is to safe memory, but this should not be necessary any more, because "good" models should converge fast enough and we only need short traces.
   The other one is to store the trace for later use. But this seems not to be the originally intended use-case.

So maybe we should split this issue into three, something like:

1. Depreciaton of backends
2. Update backend documentation
3. Storing traces after sampling

\2) and 3) will have to wait until 1) is decided. 3) Is a feature request which might have to wait.

[twiecki]

Can't you just pickle the trace object?

[ColCarroll]

Yes! This would all be great! For now I think a good hack would be something like:

with open('model_{}.pkl'.format(model_id), 'wb') as buff:
    pickle.dump({'model': model, 'trace': trace}, buff)

where you specify a model id you can use. (You might not even need to keep the model, but you could use it to keep sampling)

[ColCarroll]

Also, I call this a "hack" because you shouldn't trust pickle objects! They can run arbitrary python code, but it should be fine for local work.

[madanh]

In my case I have the same errors as @fabianrost84, while I need long runs because there is some obvious random walk behaviour in my (admittedly non-trivial) model. Anyway being able to save intermediate traces is a legitimate need, so some mechanism to achieve this must be present.

[junpenglao]

Maybe we can refactor some part of the SMC code to make the saving and loading intermediate trace a general feature.
@ColCarroll @hvasbath

[madanh]

I also wouldn't mind doing short runs interleaved with saving, but this means that we must be able to restart sampling from the previous state (possibly loaded from a file). Is there a clean way to do it now?

[hvasbath]

SMC does it already, yes.
But it seems then very model specific that the other backends stop working?!
Maybe the SMC trace wouldnt work either. I also dont understand why this error occurs?
Seems like your the point that is supposed to be recorded has more RVs to be stored than your model function takes as input.
These backends (hdf5, sqlite) reevaluate the model during point recording, which in my oppinion should be changed anyways as it is a huge performance drain to evaluate again what you know already ...

You do not get the error not specifying the trace, because it uses the numpy array then as backend...

Yes the SMC trace (doesnt reevaluate the model again) uses a list of arrays as record input compared to the other backends that need a point dict. Once we decided on how to best refactor we can do that.
Now that we have the list to dict bijection since the last SMC update we could refactor the SMC trace to take points as well, so the smc trace should work for any sampler then...

[ColCarroll]

@madanh How would you intend to use the file-backed traces? If it is truly too large to fit in memory, then it seems like you would need bespoke machinery to do any analysis on it anyways.

It seems like one of dask, blaze, pytables, or numpy.mmap might be appropriate. In my beautiful future, there would be a small external project implementing a O(1) memory backend using whichever of those is most appropriate that could be merged back into pymc3. At the same time, we would deprecate the other backends, and provide a performant save and load function for the NDArray for data persistence.

[twiecki]

@madanh What sampler do you use? What's your effective sample size? Is the model continuous?

[madanh]

@twiecki I'm in the process of figuring out what to use. I tried NUTS first, but the model is pretty finicky - it has a plateau where some components of the gradient are exactly zero for a certain region of parameter space (It;s a feature), so NUTS is slowly random walking (which is expected, in the hindsight). Metropolis is fast, but for some starting conditions rarely accepts (also expected). Slice hangs randomly - the reason seems to lie in the implementation - it tries to sample uniformly from a hypercube for non-scalar variables, rather than working them out component-by-component. And when those variables have large sizes and some covariance, it practically never succeeds. But that is a separate issue, which I will raise once I make sure that that is indeed the case.

Anyway, above stuff has nothing to do with saving traces. And as it is now I understand that, I don't need saving traces, but rather try fix the Slice sampler.

@ColCarroll For me it's not about the memory. One thing why having access to traces is nice is that it aids diagnosing/debugging. Imagine your sampler hangs after reasonable amount of work (which happened to me with Slice) - if you interrupt it - all is lost and you can't diagnose it. So you restart in debug mode and wait twice as long, or make a shorter run, trying to figure out what will happen, before it happens, or whatever. But if your traces are safe on disk, you can begin investigating straight away. Doubly useful if you have njobs>1.

[hvasbath]

Why dont you give the SMC a try- would be also good for us to further improve it and make it more smooth in terms of API?! How to start it is shown here:
https://github.com/pymc-devs/pymc3/blob/master/docs/source/notebooks/SMC2_gaussians.ipynb

[madanh]

@hvasbath I took a quick look, I will try that if hacking the Slice sampler will not help.

[madanh]

OK, fixed the Slice sampler in a quick and dirty fashion and now it at least does not hang. It's suboptimal though, as it returns only 1 sample per group of variables.
Here's a gist
https://gist.github.com/madanh/ca9dcf193d610042225118d2ba252e59

Also found a bug in my model, now NUTS is the best.

[gaow]

I have a model that currently only works well with large number of samples used, due to poor convergence. The in-memory backend cannot be used due to memory limits (32GB is not enough). At least before I can rethink and able to implement alternative model, I would like to see the possibility of at least using large sample to work out my problem. I think backends is a useful feature that should not be deprecated.

[michaelosthege]

@gaow thank you for this information.

Can you be more specific which backend you're using?

Because we might still be able to deprecate some of the others. We had deprecation warnings about those for several months now.

#3903 #3904 #3905 #3907

[canyon289]

Per conversation drop all backends except text backend

[hvasbath]

Just wanted to add my five cents to underline the importance of keeping at least one backend. For the parts of the community that come from the physical modelling side sampling often takes days to weeks, because one likelihood evaluation may take several tens of seconds. Few hours would be fast!

[fonnesbeck]

Yes, the plan is to keep the plain text backend.

[gaow]

Can you be more specific which backend you're using?

I was using in-memory backend. But I'm happy now that I see the text backend will be preserved.

[twiecki]

This is done.