Newest `aiidateam/aiida-core:stable` docker image contains broken `pymatgen` dependency #4339

greschd · 2020-08-28T18:45:43Z

Short description

The newest aiidateam/aiida-core:stable docker image contains pymatgen, but that is missing its dependency ruamel.yaml.

Update: The 1.3.0.1 image is also broken, while 1.3.0 works. I suspect it's something about these being built more recently - the latest pymatgen release is from 2020-08-13.

Steps to reproduce

docker pull aiidateam/aiida-core:stable
docker run aiidateam/aiida-core:stable
docker exec --tty $DOCKERID wait-for-services
docker exec -it $DOCKERID bash
su aiida
verdi shell
import pymatgen

Result:

(base) aiida@3e0b99c6409f:~$ verdi shell
Python 3.7.4 (default, Aug 13 2019, 20:35:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pymatgen
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-a3a39e60e980> in <module>
----> 1 import pymatgen

/opt/conda/lib/python3.7/site-packages/pymatgen/__init__.py in <module>
     12 from fnmatch import fnmatch
     13 
---> 14 import ruamel.yaml as yaml
     15 
     16 from monty.json import MontyEncoder, MontyDecoder, MSONable

ModuleNotFoundError: No module named 'ruamel'

Expected behavior

The last command should import pymatgen.

Your environment

Docker version 19.03.12, build 48a66213fe
aiida-core version: 1.3.1
Python version: 3.7.4

Tagging @csadorf @yakutovicha

The text was updated successfully, but these errors were encountered:

csadorf · 2020-08-31T07:55:52Z

@greschd Thank you for reporting!
@yakutovicha Could you investigate, please?

yakutovicha · 2020-08-31T07:57:40Z

Thanks @greschd. I will have a look!

yakutovicha · 2020-09-02T10:31:22Z

I am investigating this issue, and what I can tell already - when downgrading the pymatgen version the problem remains.

yakutovicha · 2020-09-02T10:38:06Z

The content of /opt/conda/lib/python3.7/site-packages in the broken image:

ruamel_yaml/                   ruamel_yaml-0.15.80.dist-info/

Of the working (1.3.0) image:

ruamel/   ruamel_yaml/  ruamel_yaml-0.15.80.dist-info/  ruamel.yaml-0.16.10.dist-info/ 
ruamel.yaml-0.16.10-py3.8-nspkg.pth  ruamel.yaml.clib-0.2.0.dist-info/

So for some reason, the aiida-core==1.3.0 image installs the latest version of ruamel.yaml, while the newer version doesn't.

yakutovicha · 2020-09-02T10:51:41Z

To temporarily fix the problem it is enough to do:

pip install --upgrade ruamel.yaml

greschd · 2020-09-02T10:59:28Z

Yeah, it's weird that it needs to be pip installed though. AFAICT, the docker image uses conda in general, right?

yakutovicha · 2020-09-02T11:00:30Z

Yeah, it's weird that it needs to be pip installed though. AFAICT, the docker image uses conda in general, right?

We use it mostly for the environment, for the moment. Many plugins are not available through conda

greschd · 2020-09-02T11:02:20Z

Ah, maybe that's the source of the problem? If either pymatgen or ruamel.yaml are installed differently, maybe that could cause issues.

yakutovicha · 2020-09-02T11:02:41Z

So for some reason, in old image build the log looked as follows:

Collecting ruamel.yaml>=0.15.6
Downloading ruamel.yaml-0.16.10-py2.py3-none-any.whl (111 kB)

Recently, it has been changed:

Requirement already satisfied: ruamel.yaml>=0.15.6 in /opt/conda/lib/python3.7/site-packages (from pymatgen>=2019.7.2; extra == "atomic_tools"->aiida-core==1.3.1) (0.15.80)

greschd · 2020-09-02T11:03:39Z

Maybe some other dependency started requiring ruamel.yaml? I think that one is conda - installed, which may be incompatible with a pip - installed pymatgen for some reason?

greschd · 2020-09-02T11:04:03Z

I did try installing pymatgen onto a clean conda env (via conda), and that worked fine.

greschd · 2020-09-02T11:05:35Z

I'm really not a conda expert, but this link seems relevant: https://www.anaconda.com/blog/using-pip-in-a-conda-environment

yakutovicha · 2020-09-02T11:05:37Z

I did try installing pymatgen onto a clean conda env (via conda), and that worked fine.

yes, I've tried that as well, but via pip (being in a conda environment). Also worked fine.

yakutovicha · 2020-09-02T11:07:51Z

ah, I might have a clue. The base image (aiidateam/aiida-prerequisites:0.2.0) has ruamel pre-installed

greschd · 2020-09-02T11:12:50Z

Alright, here's a minimal repro (in a clean conda env, conda 4.8.4 on Ubuntu 18.04):

conda install ruamel_yaml
pip install pymatgen
python -> import pymatgen

It seems indeed the conda and pip installs are incompatible in this case.

yakutovicha · 2020-09-02T11:15:04Z

Hmmmm, the old image did also have ruamel pre-installed as the dependency of miniconda:

The following packages will be downloaded:
package | build
---------------------------|-----------------
_libgcc_mutex-0.1 | conda_forge 3 KB conda-forge
_openmp_mutex-4.5 | 0_gnu 435 KB conda-forge
cffi-1.13.2 | py37h8022711_0 220 KB conda-forge
chardet-3.0.4 | py37_1003 167 KB conda-forge
conda-package-handling-1.6.0| py37h516909a_1 942 KB conda-forge
cryptography-2.8 | py37h72c5cf5_1 625 KB conda-forge
idna-2.9 | py_1 52 KB conda-forge
libffi-3.2.1 | he1b5a44_1006 46 KB conda-forge
libgcc-ng-9.2.0 | h24d8f2e_2 8.2 MB conda-forge
libgomp-9.2.0 | h24d8f2e_2 816 KB conda-forge
libstdcxx-ng-9.2.0 | hdf63c60_2 4.5 MB conda-forge
ncurses-6.1 | hf484d3e_1002 1.3 MB conda-forge
pycosat-0.6.3 |py37h516909a_1002 105 KB conda-forge
pycparser-2.19 | py_2 88 KB conda-forge
pyopenssl-19.1.0 | py_1 47 KB conda-forge
pysocks-1.7.1 | py37_0 27 KB conda-forge
readline-7.0 | hf8c457e_1001 391 KB conda-forge
requests-2.23.0 | py37_0 85 KB conda-forge
ruamel_yaml-0.15.80 |py37h516909a_1000 268 KB conda-forge
setuptools-45.2.0 | py37_0 654 KB conda-forge
six-1.14.0 | py37_0 23 KB conda-forge
sqlite-3.31.1 | h7b6447c_0 1.1 MB defaults
tk-8.6.10 | hed695b0_0 3.2 MB conda-forge
tqdm-4.43.0 | py_0 47 KB conda-forge
urllib3-1.25.7 | py37_0 159 KB conda-forge
wheel-0.34.2 | py_1 24 KB conda-forge
xz-5.2.4 | h14c3975_1001 366 KB conda-forge
yaml-0.2.2 | h516909a_1 82 KB conda-forge
zlib-1.2.11 | h516909a_1006 105 KB conda-forge

greschd · 2020-09-02T11:15:43Z

Maybe related, even though it's old: materialsproject/pymatgen#758

yakutovicha · 2020-09-02T11:18:23Z

My second guess is: we have different pip versions, and new pip might handle dependencies differently

yakutovicha · 2020-09-02T11:19:12Z

Maybe related, even though it's old: materialsproject/pymatgen#758

I think this is not a problem. Ruaml has been put as a dependency quite some time ago, and we have working images build with it.

greschd · 2020-09-02T11:20:41Z

Alright, here's a minimal repro (in a clean conda env, conda 4.8.4 on Ubuntu 18.04):
* `conda install ruamel_yaml`

* `pip install pymatgen`

* `python` -> `import pymatgen`
It seems indeed the conda and pip installs are incompatible in this case.

Does this work on the old aiida-prerequisites container?

greschd · 2020-09-02T11:26:46Z

So.. conda has packages ruamel, ruamel_yaml and ruamel.yaml. Installing only ruamel_yaml does not work, but installing ruamel.yaml does.

greschd · 2020-09-02T11:28:10Z

When installing ruamel_yaml, ... import ruamel_yaml works, but not import ruamel.yaml. For some reason pip still considers that a valid version for ruamel.yaml.

yakutovicha · 2020-09-02T11:29:48Z

so looks like we should install ruamel.yaml==0.16.10 in the aiida-prerequisites image, what do you think?

greschd · 2020-09-02T11:31:40Z

Yeah, I think that makes sense. TBH, I have no idea at all why there are two different conda-force sources for that.

yakutovicha · 2020-09-02T11:34:22Z

I can now confirm that this is the pip problem. I downgraded pip and rebuild the image. It worked: ruamel.yaml got installed.

greschd · 2020-09-02T11:36:04Z

Even without the pip problem though, conda install ruamel_yaml doesn't work as advertised. I think pip installing a newer version on top is just masking the problem.

yakutovicha · 2020-09-02T11:37:15Z

Even without the pip problem though, conda install ruamel_yaml doesn't work as advertised. I think pip installing a newer version on top is just masking the problem.

I agree.

greschd · 2020-09-02T11:41:49Z

So, the two are actually different packages (ever so slightly): conda-forge/ruamel.yaml-feedstock#7 (comment)

Not sure who exactly is responsible for the fact that pip doesn't realize that, though..

yakutovicha · 2020-09-02T11:47:04Z

So, the two are actually different packages (ever so slightly): conda-forge/ruamel.yaml-feedstock#7 (comment)

Not sure who exactly is responsible for the fact that pip doesn't realize that, though..

ah, my understanding was that it is the same package that just changed its name. But if those two packages are the different ones, we come back to the pip problem.

greschd · 2020-09-02T11:50:00Z

Yeah, so newer pip considers all of ruamel_yaml, ruamel-yaml and ruamel.yaml to be the same package..

The first two are fine (this is normal canonicalization), not sure what the standards say about the latter.

yakutovicha · 2020-09-02T11:51:47Z

should we report this to pip?

greschd · 2020-09-02T11:52:14Z

Might be pypa/pip#8659

greschd · 2020-09-02T11:55:15Z

It seems to be on purpose: pypa/pip#8645 (comment)

greschd · 2020-09-02T11:58:46Z

According to this page, package names aren't even supposed to contain dots 🙈

greschd · 2020-09-02T12:05:33Z

That's only semi-official though.. it's the packaging guide, not the official Python doc. Tried to find a PEP that specifies this, but so far no luck.

greschd · 2020-09-02T12:18:10Z

Yeah, so that's definitely intentional on the side of pip and PyPI. When you upload a package with any of -, _, ., it ends up in the same page. The displayed name is actually chosen by whichever it was uploaded as last 😄

Obviously, conda appears to have a different policy here. But it might be worthwhile to check with them if that is intentional.

greschd · 2020-09-02T12:23:46Z

Ah, it was actually changed just recently: conda-forge/ruamel_yaml-feedstock#68

That's only the conda-forge version though, not the default.

greschd added type/bug dependencies Pull requests that update a dependency file labels Aug 28, 2020

yakutovicha self-assigned this Aug 31, 2020

csadorf added topic/dependencies and removed dependencies Pull requests that update a dependency file labels Aug 31, 2020

This was referenced Sep 2, 2020

Prepare release 0.2.1 aiidateam/aiida-prerequisites#22

Merged

Bump base docker image version. #4353

Merged

yakutovicha closed this as completed in #4353 Sep 4, 2020

ltalirz mentioned this issue Mar 8, 2021

docs: update compatibility guidance materialsproject/pymatgen#2086

Merged

5 tasks

yakutovicha mentioned this issue Jun 10, 2022

Add build for arm64 aiidateam/aiida-prerequisites#38

Merged

Newest aiidateam/aiida-core:stable docker image contains broken pymatgen dependency #4339

Newest aiidateam/aiida-core:stable docker image contains broken pymatgen dependency #4339

Comments

greschd commented Aug 28, 2020 • edited Loading

Short description

Steps to reproduce

Expected behavior

Your environment

csadorf commented Aug 31, 2020

yakutovicha commented Aug 31, 2020

yakutovicha commented Sep 2, 2020

yakutovicha commented Sep 2, 2020 • edited Loading

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020 • edited Loading

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020

greschd commented Sep 2, 2020

greschd commented Sep 2, 2020

yakutovicha commented Sep 2, 2020 • edited Loading

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020 • edited Loading

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020

yakutovicha commented Sep 2, 2020

yakutovicha commented Sep 2, 2020 • edited Loading

greschd commented Sep 2, 2020

greschd commented Sep 2, 2020

greschd commented Sep 2, 2020 • edited Loading

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020 • edited Loading

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020

yakutovicha commented Sep 2, 2020

greschd commented Sep 2, 2020

greschd commented Sep 2, 2020

greschd commented Sep 2, 2020

greschd commented Sep 2, 2020 • edited Loading

greschd commented Sep 2, 2020

greschd commented Sep 2, 2020 • edited Loading

Newest `aiidateam/aiida-core:stable` docker image contains broken `pymatgen` dependency #4339

Newest `aiidateam/aiida-core:stable` docker image contains broken `pymatgen` dependency #4339

greschd commented Aug 28, 2020 •

edited

Loading

yakutovicha commented Sep 2, 2020 •

edited

Loading

greschd commented Sep 2, 2020 •

edited

Loading

yakutovicha commented Sep 2, 2020 •

edited

Loading

greschd commented Sep 2, 2020 •

edited

Loading

yakutovicha commented Sep 2, 2020 •

edited

Loading

greschd commented Sep 2, 2020 •

edited

Loading

greschd commented Sep 2, 2020 •

edited

Loading

greschd commented Sep 2, 2020 •

edited

Loading

greschd commented Sep 2, 2020 •

edited

Loading