-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Docker image to v4
#5323
Comments
The second issue looks wrong |
A typo. Thanks. Also forgot to mention why this issue is high-priority: We want to re-run a ton of |
Using @bwengals docker image the model graph to png saving is work. Its not minimal but at least shows it can work!
model.py
|
4.0.0b1
4.0.0b2
@canyon289 do you want to add this Dockerfile alongside a Github action? (https://github.com/marketplace/actions/build-and-push-docker-images#usage) Should we include the extra dependencies from pymc-examples? That would make it easier to re-run notebooks. |
I wont be able to do this anytime soon, my current focus is on time series. |
Hey guys, I would be glad to take over. I have some experience with pymc, docker and github actions. Let me know if you are ok with that. |
Sure, go ahead! |
Hi all, I tried building the above docker image. It failed for me with mamba not being able to find Even if I was to drop the |
Sorry I didn't respond sooner! @nvasilevv what sort of errors are you getting? I can build the image alright on my machine, also osx (but no M1 chip). Are you using the |
Hi @bwengals, indeed I am using the Dockerfile shared above. When installing the dependencies mamba cannot find the right package for P.S.: I am configuring a windows setup so that I do not face this issue. Will be able to work on the issue once the setup is in place. Should be ready tomorrow. |
4.0.0b2
v4
how's progress on this one? I had some luck: see https://github.com/alephinsights/docker-pymc-gpu I was hoping to get GPU acceleration working for both JAX/Aesara. Jax seems to work, but still having issues with Aesara. |
AFAIK there's currently nobody working on it. Any contributions are welcome! |
I'll tidy this up and PR idc
Exciting! |
Anyone wants to work on this one? |
Maybe @5hv5hvnk? |
Sure! let me get started on this |
While using the above |
@5hv5hvnk I don't think we do. |
Hi, let me try to build the docker image for PyMC v4 this week :) |
Currently we are getting an error while downloading the dependencies while creating the conda environment (on execution of start_container.sh
And here is the Dockerfile we are using
|
Hi. Can you also share the |
@5hv5hvnk Try going into your docker preferences and let it use more RAM. I think that error code is OOM |
@danhphan It is the same file quoted above but with name
|
@bwengals Increasing allocated memory of docker did help, now the Dockerfile runs smoothly but it still has the same issue I was facing earlier the interactive terminal of the docker container can execute the same file without errors however in the jupyter lab launched by docker itself (while using an .ipynb) the dependencies aren't installed and we have to use |
Hi @5hv5hvnk I have pushed a docker image on dockerhub https://hub.docker.com/r/danhphan/pymc/tags . Can you check if it work on your side. I put the dockerfile here: https://github.com/danhphan/pymc-docker You can either pull the docker image from dockerhub, or build it from dockerfile. 1. Install docker image1.1. Pull image from docker hub
1.2. Build from dockerfile
2. How to runNote, if you pull from dockerhub, the image name will need to be changed from 2.1. Run with Jupyter notebook
2.2 Run with command lines
Currently it only works with CPU. For the GPU support, I may need to do it later. |
A note that I build the docker image using |
On executing the
|
You have the wrong aeppl version installed. Make sure it's the same version as in the requirements.txt. |
Yes, but when I am executing the code on the docker container there shouldn't be any disparity because of difference in versions between host kernel and the docker container? |
Hi, I have updated the latest dependency versions (including Could you pls to pull the updated docker image and check again? The dockerfile is also updated in the github repos. Thanks. I run this demo notebook, it seems alright. Besides, the result when running the docker container using command lines as follow:
The versions from watermark:
|
@danhphan Any reason you didn't use conda? It would allow easy installation for MKL which would make the blas warning go away./ |
Looking in the current docker scripts, the only reason its broken is because its expecting a
|
The main use-case is for deployment I'd say and maybe for users who have a hard time installing. |
Hi, I may be biased here, but to me I have some difficult experience when using However, I have not thought about For docker use-case, I thought that it will mainly use for deployment, not development. Besides, if the users know how to use docker, then the installation should not be a problem for them? Should we separate into these docker use-case:
|
Yup it's very true, pip and pyenv are lighter weight, and are used more in Python overall. Conda's useful for the pydata stack because it's heavy weight. For computing stuff, you need all sorts of optimized C libraries, Fortran, CUDA. Conda can even do R, matlab, and all kinds of other stuff. Unlike pip it handles non-python packages so it's more of a general purpose package manager. Outside of scientific python I don't think conda is used much at all, but I don't know for sure. It makes sense then that flask stuff on conda may not get as much attention. PyMC (but really Aesara under the hood) can run just fine on pure python, but runs way better with a C compiler and optimized linear algebra libraries installed (MKL ideally). New Aesara can use Numba and Jax too (and all of their deps!). Conda usually handles all that better than pip. So pip is the lightest weight / fastest route to a PyMC install, but conda is the the quickest route to a full and more optimized install. Not that the pure python backend isn't useful -- its how @twiecki did this: https://www.pymc-labs.io/blog-posts/pymc-in-browser/ My conda/pip knowledge might be a few years old though, so keep that in mind! |
Heres my 2 cents.
So TLDR my vote is a minimal conda environment. All the other stuff can get added on later in containers that use this first container as a base image |
Agree with Ravin, although I think using mamba is just better in every way, but it doesn't matter too much. |
Hi, I have successfully installed a docker image with conda (mamba). The Can you check it out? If it is alright, I will incorporate it into the PyMC dockerfile. The current conda installation uses the PyMC environment-dev-py39.yml file, and added The image size is quite large though. It is 2.41 GB now, compared to 900MB for yesterday version with pip. I will try to tidy up and reduce the image size. If we just want to build a minimal conda environment, can I just remove any packages related to test ( Also, just want to ask why we need to keep different .yml files such as |
This is the outputs when run docker in bash
|
Did you run conda clean? You can also remove the base environment. You can also use https://github.com/mamba-org/mamba#micromamba.
Yes.
Yes for |
Hi, yes sure. After cleaning up, the docker image size now about 1.38GB, which I think is acceptable. This is the latest Dockerfile, I will try to tidy up and reduce the image size a little bit more. Also, I am checking related scripts in https://github.com/pymc-devs/pymc/tree/main/scripts and will open a PR to incorporate the dockerfile into these scripts. If you have any suggestions, please let me know. Will need to wait for this PR #5842 to be passed as well :) Thanks. |
Hi, docker images for PyMC version 4.0.0 on both CPUs and GPUs are ready now. The Dockerfile for CPUs and GPUs (and guideline) have been updated on https://github.com/danhphan/pymc-docker I'm going to integrate it into the PyMC repos this weekend. Thanks |
@danhphan Excellent news! |
Closing (mostly done), superseded by #5939 |
As discussed in the meeting earlier today, it is now time to update the Docker image: https://hub.docker.com/u/pymc
Short-term goals
Dockerfile
to a fresh base image (done in Update dockerfile to PyMC v4 #5881)NumPy C-API
warning... (done in Update dockerfile to PyMC v4 #5881)pm.model_to_graphviz()
doesn't raise errors. (done in Update dockerfile to PyMC v4 #5881)Simplifications/Constraints
3.9
for now?4.0.0b1
that'll be a manually built one, but for upcoming ones we can automate it?Tagging @canyon289 @AustinRochford @bwengals who have relevant insight.
I can also contribute a (stripped-down)
environment.yml
to start from 👇Known limitations:
NumPy C-API
warningpm.model_to_graphviz()
doesn't raise, but rendering to fixes crashes.When this issue is dealt with we can also close #3160 + #4386 as fixed and wontfix respectively.
The text was updated successfully, but these errors were encountered: