name 'HTML' is not define with pm.sample_smc()

[VViCaiM]

Description of your problem

I try the the code, gauss_abc, from the recommended book Bayesian Modeling and Computation in Python. But it can't run with NameError: name 'HTML' is not defined.

A minimal, self-contained, and reproducible example.

import pymc as pm
import numpy as np
data = np.random.normal(loc=0, scale=1, size=1000)

def normal_simulator(mu, sigma):
    return np.random.normal(mu, sigma, 1000)
with pm.Model() as gauss:
    μ = pm.Normal("μ", mu=0, sigma=1)
    σ = pm.HalfNormal("σ", sigma=1)
    s = pm.Simulator("s", normal_simulator, params=[μ, σ],
                     distance="gaussian",
                     sum_stat="sort",  
                     epsilon=1,
                     observed=data)
    trace_g = pm.sample_smc()

The full traceback.

Complete error traceback

RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "d:\miniforge3\envs\pymc\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "d:\miniforge3\envs\pymc\lib\multiprocessing\pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "d:\miniforge3\envs\pymc\lib\site-packages\pymc\smc\sample_smc.py", line 432, in _apply_args_and_kwargs
    return fn(*args, **kwargs)
  File "d:\miniforge3\envs\pymc\lib\site-packages\pymc\smc\sample_smc.py", line 354, in _sample_smc_int
    progressbar.update_bar(getattr(progressbar, "offset", 0) + 0)
  File "d:\miniforge3\envs\pymc\lib\site-packages\fastprogress\fastprogress.py", line 76, in update_bar
    else: self.on_update(val, f'{100 * val/self.total:.2f}% [{val}/{self.total} {elapsed_t}<{remaining_t}{end}]')
  File "d:\miniforge3\envs\pymc\lib\site-packages\fastprogress\fastprogress.py", line 127, in on_update
    if self.display: self.out.update(HTML(self.progress))
NameError: name 'HTML' is not defined
"""

The above exception was the direct cause of the following exception:

NameError                                 Traceback (most recent call last)
c:\Users\AMD Ryzen\Documents\Python Learning Materials\BookCode_Edition1-main\notebooks\chp_08.ipynb Cell 11 in <cell line: 1>()
      [3](vscode-notebook-cell:/c%3A/Users/AMD%20Ryzen/Documents/Python%20Learning%20Materials/BookCode_Edition1-main/notebooks/chp_08.ipynb#ch0000010?line=2) σ = pm.HalfNormal("σ", sigma=1)
...
    769     return self._value
    770 else:
--> 771     raise self._value

NameError: name 'HTML' is not defined

Additional information.
I have notice a previous and similar issue. I try to set cores=1 or other value, but it doesn't work. I have also install ipywidgets 7.7.1. When I set cores=1, the error changes into:

Error traceback when cores=1

TypeError                                 Traceback (most recent call last)
File d:\miniforge3\envs\pymc\lib\site-packages\aesara\compile\function\types.py:975, in Function.__call__(self, *args, **kwargs)
    973 try:
    974     outputs = (
--> 975         self.vm()
    976         if output_subset is None
    977         else self.vm(output_subset=output_subset)
    978     )
    979 except Exception:

File d:\miniforge3\envs\pymc\lib\site-packages\aesara\graph\op.py:543, in Op.make_py_thunk.<locals>.rval(p, i, o, n, params)
    539 @is_thunk_type
    540 def rval(
    541     p=p, i=node_input_storage, o=node_output_storage, n=node, params=None
    542 ):
--> 543     r = p(n, [x[0] for x in i], o)
    544     for o in node.outputs:

File d:\miniforge3\envs\pymc\lib\site-packages\aesara\tensor\random\op.py:368, in RandomVariable.perform(self, node, inputs, outputs)
    366 rng_var_out[0] = rng
--> 368 smpl_val = self.rng_fn(rng, *(args + [size]))
    370 if (
    371     not isinstance(smpl_val, np.ndarray)
    372     or str(smpl_val.dtype) != out_var.type.dtype
...
Inputs values: [Generator(PCG64) at 0x1D632DDADC0, array([1000], dtype=int64), array(11, dtype=int64), array(1.6595701), array(2.74061535)]
Outputs clients: [['output'], [SortOp{quicksort, None}(sim_value, TensorConstant{-1})]]

HINT: Re-running with most Aesara optimizations disabled could provide a back-trace showing when this node was created. This can be done by setting the Aesara flag 'optimizer=fast_compile'. If that does not work, Aesara optimizations can be disabled with 'optimizer=None'.
HINT: Use the Aesara flag `exception_verbosity=high` for a debug print-out and storage map footprint of this Apply node.

I encountered the same issue in my two PC, one with Intel Core 12700K, the other with AMD Ryzen 5800H. But normal pm.sample() works well.

Versions and main components

• PyMC Version: 4.1.3
• Aesara Version: 2.7.7
• Python Version: 3.10.5
• Operating system: Windows 11 21H2
• How did you install PyMC/PyMC3: (conda/pip): conda.

[ColCarroll]

Hey -- thanks for the nice repro. It does look like this is a fastprogress issue (I guess you're running a notebook in VSCode?). I'll leave open for a bit in case someone has more experience with this sort of thing.

I got this snippet running on a colab here, but never was able to get the same error as you.

Note @aloctavodia that the API has changed a bit from the linked book example (I'll try to make a PR soon!)

[VViCaiM]

Hey -- thanks for the nice repro. It does look like this is a fastprogress issue (I guess you're running a notebook in VSCode?). I'll leave open for a bit in case someone has more experience with this sort of thing.

I got this snippet running on a colab here, but never was able to get the same error as you.

Note @aloctavodia that the API has changed a bit from the linked book example (I'll try to make a PR soon!)

Thank you so much! I do use VSCode. However, switching to Jupyter Notebook or Jupyter Lab couldn't make it better. I think I'll have to use colab while employing smc for now. Thank you.

[michaelosthege]

The first issue is definitely this bug (AnswerDotAI/fastprogress#32) in fastprogress, which you run into because this import fails: https://github.com/fastai/fastprogress/blob/3f68f4bda3a227401aff7ad360f4ecb5f9de1f6b/fastprogress/fastprogress.py#L97-L103

@ColCarroll could you reproduce the second traceback? If not I would take that as a sign that @VViCaiM 's environment is just broken.

[aloctavodia]

@VViCaiM did you try installing creating an environment as provided with the book?
https://github.com/BayesianModelingandComputationInPython/BookCode_Edition1#conda

[VViCaiM]

@VViCaiM did you try installing creating an environment as provided with the book? https://github.com/BayesianModelingandComputationInPython/BookCode_Edition1#conda

I used conda create -c conda-forge -n pymc_env "pymc==4.1.3", which is the method in pymc documentation, or conda install pymc. Moreover I have tried both Miniforge3 and Anaconda but it is the same.

I also test the code in newly installed ubuntu, but the error changed slightly:

Complete error traceback

RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/aesara/compile/function/types.py", line 975, in __call__
    self.vm()
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/aesara/graph/op.py", line 543, in rval
    r = p(n, [x[0] for x in i], o)
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/aesara/tensor/random/op.py", line 368, in perform
    smpl_val = self.rng_fn(rng, *(args + [size]))
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/distributions/simulator.py", line 55, in rng_fn
    return cls.fn(*args, **kwargs)
TypeError: normal_simulator() takes 2 positional arguments but 4 were given

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniforge3/envs/pymc/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/root/miniforge3/envs/pymc/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/smc/sample_smc.py", line 432, in _apply_args_and_kwargs
    return fn(*args, **kwargs)
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/smc/sample_smc.py", line 356, in _sample_smc_int
    smc._initialize_kernel()
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/smc/smc.py", line 229, in _initialize_kernel
    likelihoods = [self.likelihood_logp_func(sample) for sample in self.tempered_posterior]
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/smc/smc.py", line 229, in <listcomp>
    likelihoods = [self.likelihood_logp_func(sample) for sample in self.tempered_posterior]
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/aesara/compile/function/types.py", line 988, in __call__
    raise_with_op(
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/aesara/link/utils.py", line 534, in raise_with_op
    raise exc_value.with_traceback(exc_trace)
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/aesara/compile/function/types.py", line 975, in __call__
    self.vm()
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/aesara/graph/op.py", line 543, in rval
    r = p(n, [x[0] for x in i], o)
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/aesara/tensor/random/op.py", line 368, in perform
    smpl_val = self.rng_fn(rng, *(args + [size]))
  File "/root/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/distributions/simulator.py", line 55, in rng_fn
    return cls.fn(*args, **kwargs)
TypeError: normal_simulator() takes 2 positional arguments but 4 were given
Apply node that caused the error: Simulator_rv{0, (0, 0), floatX, True}(RandomGeneratorSharedVariable(<Generator(PCG64) at 0x7F01C48312A0>), TensorConstant{(1,) of 1000}, TensorConstant{11}, Subtensor{int64}.0, σ_log___log)
Toposort index: 9
Inputs types: [RandomGeneratorType, TensorType(int64, (1,)), TensorType(int64, ()), TensorType(float64, ()), TensorType(float64, ())]
Inputs shapes: ['No shapes', (1,), (), (), ()]
Inputs strides: ['No strides', (8,), (), (), ()]
Inputs values: [Generator(PCG64) at 0x7F01C48312A0, array([1000]), array(11), array(-2.45004673), array(1.44735956)]
Outputs clients: [['output'], [SortOp{quicksort, None}(sim_value, TensorConstant{-1})]]

HINT: Re-running with most Aesara optimizations disabled could provide a back-trace showing when this node was created. This can be done by setting the Aesara flag 'optimizer=fast_compile'. If that does not work, Aesara optimizations can be disabled with 'optimizer=None'.
HINT: Use the Aesara flag `exception_verbosity=high` for a debug print-out and storage map footprint of this Apply node.
"""

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Input In [10], in <cell line: 5>()
      7 σ = pm.HalfNormal("σ", sigma=1)
      8 s = pm.Simulator("s", normal_simulator, params=[μ, σ],
      9                  distance="gaussian",
     10                  sum_stat="sort",  
     11                  epsilon=1,
     12                  observed=data)
---> 13 trace_g = pm.sample_smc(chains=4)

File ~/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/smc/sample_smc.py:220, in sample_smc(draws, kernel, start, model, random_seed, chains, cores, compute_convergence_checks, return_inferencedata, idata_kwargs, progressbar, **kernel_kwargs)
    217 t1 = time.time()
    219 if cores > 1:
--> 220     results = run_chains_parallel(
    221         chains, progressbar, _sample_smc_int, params, random_seed, kernel_kwargs, cores
    222     )
    223 else:
    224     results = run_chains_sequential(
    225         chains, progressbar, _sample_smc_int, params, random_seed, kernel_kwargs
    226     )

File ~/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/smc/sample_smc.py:401, in run_chains_parallel(chains, progressbar, to_run, params, random_seed, kernel_kwargs, cores)
    399 params = tuple(cloudpickle.dumps(p) for p in params)
    400 kernel_kwargs = {key: cloudpickle.dumps(value) for key, value in kernel_kwargs.items()}
--> 401 results = _starmap_with_kwargs(
    402     pool,
    403     to_run,
    404     [(*params, random_seed[chain], chain, pbars[chain]) for chain in range(chains)],
    405     repeat(kernel_kwargs),
    406 )
    407 results = tuple(cloudpickle.loads(r) for r in results)
    408 pool.close()

File ~/miniforge3/envs/pymc/lib/python3.10/site-packages/pymc/smc/sample_smc.py:428, in _starmap_with_kwargs(pool, fn, args_iter, kwargs_iter)
    424 def _starmap_with_kwargs(pool, fn, args_iter, kwargs_iter):
    425     # Helper function to allow kwargs with Pool.starmap
    426     # Copied from https://stackoverflow.com/a/53173433/13311693
    427     args_for_starmap = zip(repeat(fn), args_iter, kwargs_iter)
--> 428     return pool.starmap(_apply_args_and_kwargs, args_for_starmap)

File ~/miniforge3/envs/pymc/lib/python3.10/multiprocessing/pool.py:372, in Pool.starmap(self, func, iterable, chunksize)
    366 def starmap(self, func, iterable, chunksize=None):
    367     '''
    368     Like `map()` method but the elements of the `iterable` are expected to
    369     be iterables as well and will be unpacked as arguments. Hence
    370     `func` and (a, b) becomes func(a, b).
    371     '''
--> 372     return self._map_async(func, iterable, starmapstar, chunksize).get()

File ~/miniforge3/envs/pymc/lib/python3.10/multiprocessing/pool.py:771, in ApplyResult.get(self, timeout)
    769     return self._value
    770 else:
--> 771     raise self._value

TypeError: normal_simulator() takes 2 positional arguments but 4 were given
Apply node that caused the error: Simulator_rv{0, (0, 0), floatX, True}(RandomGeneratorSharedVariable(<Generator(PCG64) at 0x7F01C48312A0>), TensorConstant{(1,) of 1000}, TensorConstant{11}, Subtensor{int64}.0, σ_log___log)
Toposort index: 9
Inputs types: [RandomGeneratorType, TensorType(int64, (1,)), TensorType(int64, ()), TensorType(float64, ()), TensorType(float64, ())]
Inputs shapes: ['No shapes', (1,), (), (), ()]
Inputs strides: ['No strides', (8,), (), (), ()]
Inputs values: [Generator(PCG64) at 0x7F01C48312A0, array([1000]), array(11), array(-2.45004673), array(1.44735956)]
Outputs clients: [['output'], [SortOp{quicksort, None}(sim_value, TensorConstant{-1})]]

HINT: Re-running with most Aesara optimizations disabled could provide a back-trace showing when this node was created. This can be done by setting the Aesara flag 'optimizer=fast_compile'. If that does not work, Aesara optimizations can be disabled with 'optimizer=None'.
HINT: Use the Aesara flag `exception_verbosity=high` for a debug print-out and storage map footprint of this Apply node.

A bit weird. Thank you for answering. I think that's my own problem since you can't reproduce it. Digging down seems a waste of developers precious time.

[aloctavodia]

Please use the environment provided in the book. Notice that the book was published prior to the release of PyMC 4.0, hence if you install PyMC 4.x you will need to make changes to some Code Blocks provided in the book, in particular the largest changes will be in for ABC and BART chapters.