No CmdStan installation found

[jolespin]

I installed v0.1.0 via the following:

conda install -c conda-forge biom-format patsy xarray arviz cmdstanpy
pip install birdman

When I try to run a negative binomial model, I get the following error:

INFO:cmdstanpy:No CmdStan installation found.
INFO:cmdstanpy:Cannot determine whether version is before 2.27.
INFO:cmdstanpy:No CmdStan installation found.
INFO:cmdstanpy:Cannot determine whether version is before 2.29.
INFO:cmdstanpy:compiling stan file /Users/jespinoz/anaconda3/envs/soothsayer_p3.9_env/lib/python3.9/site-packages/birdman/templates/negative_binomial.stan to exe file /Users/jespinoz/anaconda3/envs/soothsayer_p3.9_env/lib/python3.9/site-packages/birdman/templates/negative_binomial
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [8], in <module>
      3 biom_iris = biom.Table(X_iris.values.T,   X_iris.columns, X_iris.index)#, sample_metadata=y_iris.to_frame())
      5 nb = NegativeBinomial(
      6     table=biom_iris,
      7     formula="species",
      8     metadata=y_iris.to_frame("species"),
      9     num_iter=1000,
     10 )
---> 11 nb.compile_model()
     12 nb.fit_model()

File ~/anaconda3/envs/soothsayer_p3.9_env/lib/python3.9/site-packages/birdman/model_base.py:86, in BaseModel.compile_model(self)
     84 def compile_model(self):
     85     """Compile Stan model."""
---> 86     self.sm = CmdStanModel(stan_file=self.model_path)

File ~/anaconda3/envs/soothsayer_p3.9_env/lib/python3.9/site-packages/cmdstanpy/model.py:213, in CmdStanModel.__init__(self, model_name, stan_file, exe_file, compile, stanc_options, cpp_options, user_header)
    210         get_logger().debug("TBB already found in load path")
    212 if compile and self._exe_file is None:
--> 213     self.compile(force=str(compile).lower() == 'force')
    214     if self._exe_file is None:
    215         raise ValueError(
    216             'Unable to compile Stan model file: {}.'.format(
    217                 self._stan_file
    218             )
    219         )

File ~/anaconda3/envs/soothsayer_p3.9_env/lib/python3.9/site-packages/cmdstanpy/model.py:419, in CmdStanModel.compile(self, force, stanc_options, cpp_options, user_header, override_options)
    417 sout = io.StringIO()
    418 try:
--> 419     do_command(cmd=cmd, cwd=cmdstan_path(), fd_out=sout)
    420 except RuntimeError as e:
    421     sout.write(f'\n{str(e)}\n')

File ~/anaconda3/envs/soothsayer_p3.9_env/lib/python3.9/site-packages/cmdstanpy/utils.py:163, in cmdstan_path()
    161 cmdstan_dir = os.path.expanduser(os.path.join('~', _DOT_CMDSTAN))
    162 if not os.path.exists(cmdstan_dir):
--> 163     raise ValueError(
    164         'No CmdStan installation found, run "install_cmdstan" or'
    165         ' (re)activate your conda environment!'
    166     )
    167 latest_cmdstan = get_latest_cmdstan(cmdstan_dir)
    168 if latest_cmdstan is None:

ValueError: No CmdStan installation found, run "install_cmdstan" or (re)activate your conda environment!

I tried installing cmdstan via conda but it says that it's already installed.

Here's the executable I found:

(soothsayer_p3.9_env) jespinozlt2-osx:~ jespinoz$ which cmdstan_model
/Users/jespinoz/anaconda3/envs/soothsayer_p3.9_env/bin/cmdstan_model

I then tried running your install_cmdstan command but it needs an argument.

(soothsayer_p3.9_env) jespinozlt2-osx:~ jespinoz$ install_cmdstan
Traceback (most recent call last):
  File "/Users/jespinoz/anaconda3/envs/soothsayer_p3.9_env/bin/install_cmdstan", line 10, in <module>
    sys.exit(main())
TypeError: main() missing 1 required positional argument: 'args'

[PaulaEterovick]

Dear jolespin,
I had exactly the same problem, very similar model and error messages (the lines with the arrows and the last one are identical)... and tried the same things you did. Have you found out how to fix it?

At first my model was giving error messages with many repetitions of

Exception: neg_binomial_2_log_lpmf: Precision parameter is 0, but must be positive finite! (in '/Users/paulaeterovick/anaconda3/lib/python3.11/site-packages/birdman/templates/negative_binomial.stan', line 40, column 6 to column 94)

Then I tried to update the negative_binomial.stan file with the version available at https://github.com/biocore/BIRDMAn/blob/main/birdman/templates/negative_binomial.stan and got the error

13:33:16 - cmdstanpy - INFO - compiling stan file /Users/paulaeterovick/anaconda3/lib/python3.11/site-packages/birdman/templates/negative_binomial.stan to exe file /Users/paulaeterovick/anaconda3/lib/python3.11/site-packages/birdman/templates/negative_binomial 13:33:18 - cmdstanpy - WARNING - CmdStan's precompiled header (PCH) files may need to be rebuilt.Please run cmdstanpy.rebuild_cmdstan(). If the issue persists please open a bug report

And then I tried to install cmdstan like you did, I reinstalled BIRDMAn, updated Xcode, reestarted the computer, but nothing works anymore, I just get the messages you got.

I hope you found a solution. If not, I'll be happy to share if I get to solve this...

Best regards
Paula

[mortonjt]

Hi @PaulaEterovick I'd recommend trying to run this on linux (@gibsramen not sure your thoughts on this, but I never got this to run on mac). The precision parameter is 0 often occurs when there is a numerical issue -- which I think can be resolved with adjusting the step_size (see the cmdstanpy docs)

I'd also double check the cmdstan / cmdstanpy versions, since there have been major breaking changes in one of the recent releases.

[PaulaEterovick]

Hi, @mortonjt, thank you very much for the fast answer!
I think I messed something up when trying to fix the first error (about the 0 parameter) and now I'm having other problems related to model compilation by Stan, so I think I need to fix this first...

I tried to update both cmdstanpy and cmdstan; I also installed cmdstanjupyter because I am using a jupyter notebook.
After that I used the install_cmdstan command and I'm getting the following error:

(base) paulaeterovick@Paulas-MacBook-Pro ~ % install_cmdstan ls -F ~/.cmdstan Traceback (most recent call last): File "/Users/paulaeterovick/anaconda3/bin/install_cmdstan", line 6, in <module> from cmdstanpy.install_cmdstan import main ImportError: cannot import name 'main' from 'cmdstanpy.install_cmdstan' (/Users/paulaeterovick/anaconda3/lib/python3.11/site-packages/cmdstanpy/install_cmdstan.py) cmdstan-2.32.2/ (base) paulaeterovick@Paulas-MacBook-Pro ~ %
These are the installed versions:

(base) paulaeterovick@Paulas-MacBook-Pro ~ % conda list cmdstan packages in environment at /Users/paulaeterovick/anaconda3: Name Version Build Channel cmdstan 2.31.0 h33b0451_0 cmdstanjupyter 0.7.0 pypi_0 pypi cmdstanpy 1.1.0 pyhd8ed1ab_0 conda-forge

Do you know how to fix this?

[jolespin]

@mortonjt thats good to know! I was trying it on Mac but I'll install it on Expanse to try again (I've had to pivot to other projects in the meantime).

[PaulaEterovick]

Dear @mortonjt,
I built a vitual machine and I'm ready to start all over in Linux. Would you be so nice to please send the the exact version of all the needed programs you used when you got to run the model in BIRDMAn? Also, what should I install with "conda" or "pip" or other command? I've heard that using different installers may also cause incompatibility problems...
Thank you very much in advance

[mortonjt]

Hi, this is a @gibsramen question.

I don't think a virtual machine is a good idea : you won't have the compute required to run these models ...

[jolespin]

Apologies if this is in the docs and I missed it but is there a resource that has benchmarking (or resource requirements) for running this on different dataset sizes?

[mortonjt]

I dont have benchmarks — and it is highly dependent on the model you use. For our ASD meta analysis, it takes about 20 min per microbe (for ~15K microbes)

…

On Mon, Jul 31, 2023 at 5:14 PM Josh L. Espinoza ***@***.***> wrote: Apologies if this is in the docs and I missed it but is there a resource that has benchmarking (or resource requirements) for running this on different dataset sizes? — Reply to this email directly, view it on GitHub <#81 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA75VXLKVDGOHYKCCNPL72LXTAN2PANCNFSM6AAAAAAWBZ3YE4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[PaulaEterovick]

Dear all, I was able to overcome the problem in my MacOS (with help from my brother, who is a programmer) and I am sharing here what worked for me, hoping it may help @jolespin or others who need:

First of all, my PATH was leading to an older version of Python because I deleted some extra copies and one happened to be in priority (my mistake, as this is all very new to me). In order to avoid problems with several versions of programs installed in my attempts to solve the problem, we did everything directly from the basal directory (not from Anaconda). Thus, after fixing the problem with Python, we used the following commands to install the needed programs:

pip install birdman
pip install cmdstanpy
pip install --upgrade jupyter
pip install --upgrade ipywidgets
jupyter nbexternsion enable --py widgetsnbextension
pip install biom-format
jupyter lab #to open Jupyter

(p.s. it is important to prioritize pip for installations in order to garantee compatibility among programs and appropriate paths - I may have messed it up by using different installers)
My model is running now and it may take almost two days (55% after 23h)

I hope the explanation is clear and that it will help others :-)