cartopy cannot find geos library #24805

husnoo · 2017-04-10T21:16:58Z

Issue description

Following this: http://nixos.org/nixpkgs/manual/#handling-dependencies, propagatedBuildInputs should link in run time dependencies.

Steps to reproduce

default.nix:

with import <nixpkgs> {};

(let cartopy = pkgs.pythonPackages.buildPythonPackage rec{
  name = "cartopy-${version}";
  version = "";

  src = pkgs.fetchurl{
  url = "mirror://pypi/c/cartopy/cartopy-0.11.0.tar.gz";
    sha256 = "85eb55d6ce16be5f7e745fcc2870b3fa7450db3a77899daade8b18711532be45";
  };
  doCheck = false;
  
  propagatedBuildInputs =  [ gcc pkgs.geos proj pythonPackages.numpy  pythonPackages.cffi pythonPackages.cython];
};
in python.buildEnv.override {
  extraLibs = [
    cartopy
    pkgs.geos
    proj
];
  ignoreCollisions = true;
}).env

Running it:

$ nix-shell
[nix-shell:~/]$ python
Python 2.7.13 (default, Dec 17 2016, 20:05:07) 
[GCC 5.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cartopy
Failed `CDLL(libgeos_c.so.1)`
>>> 
>>> 
[nix-shell:~/]$ LD_LIBRARY_PATH=/nix/store/6ln1621k5rmnvc92dmcyf18614py37va-geos-3.6.1/lib/ python
Python 2.7.13 (default, Dec 17 2016, 20:05:07) 
[GCC 5.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cartopy
>>>

Technical details

[nix-shell:~/]$ nixos-version
17.03.git.ac3cae0 (Gorilla)

[nix-shell:~/]$ nix-env --version
nix-env (Nix) 1.11.8

[nix-shell:~/]$ nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion
"17.03.git.ac3cae0631"

The text was updated successfully, but these errors were encountered:

husnoo · 2017-04-10T21:24:25Z

Related? #7307
I think ctypes is trying to load the library at runtime, so somehow I need to set LD_LIBRARY_PATH in my default.nix, preferably without the hash in the /nix/store/..............geos/lib/.

Any ideas?

thanks!

FRidh · 2017-04-11T06:42:31Z

With Python on Nix you cannot rely on ctypes and have to patch code to use an absolute path to a library.
Therefore, in this case, you will have to patch the cartopy code to use an absolute path to refer to libgeos_c.so.1.

FRidh · 2017-04-11T06:59:39Z

The issue here seems to be a bit different since ctypes or cffi doesn't seem to be used. The only references in cartopy to geos I can find are in _trace.h and trace.pyx. If Cython is used then cartopy should be able to find the library during runtime.

I took your Nix expression and built cartopy but got

$ python
Python 2.7.13 (default, Dec 17 2016, 20:05:07) 
[GCC 5.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cartopy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/h48zwwlpy3jf6idsql6svx5xzr32ayi7-python-2.7.13-env/lib/python2.7/site-packages/cartopy/__init__.py", line 23, in <module>
    import shapely.speedups
ImportError: No module named shapely.speedups

The NixOS version you mention is not a revision that is available via https://github.com/NixOS/nixpkgs/. What other changes did you make?

husnoo · 2017-04-11T17:08:32Z

husnoo@ac3cae0

I only removed the system images from android, as not everyone has a 1 TB hdd.

I did have a few more libraries installed, I'll check later.

so I need to patch cartopy at build time? it's that in the let.. clause above?

husnoo · 2017-04-11T20:52:52Z

Full default.nix

with import <nixpkgs> {};

(let cartopy = pkgs.pythonPackages.buildPythonPackage rec{
  name = "cartopy-${version}";
  version = "";

  src = pkgs.fetchurl{
  url = "mirror://pypi/c/cartopy/cartopy-0.11.0.tar.gz";
    sha256 = "85eb55d6ce16be5f7e745fcc2870b3fa7450db3a77899daade8b18711532be45";
  };
  doCheck = false;
  
  propagatedBuildInputs =  [ gcc pkgs.geos proj pythonPackages.numpy  pythonPackages.cffi pythonPackages.cython];
};
in python.buildEnv.override {
  extraLibs = [
    cartopy
    pkgs.geos
    proj
    pkgs.pythonPackages.pygments
    pkgs.pythonPackages.pyaml
    pkgs.pythonPackages.markdown
    pkgs.pythonPackages.numpy
    pkgs.pythonPackages.msgpack
    pkgs.pythonPackages.scipy
    pkgs.pythonPackages.tables
    pkgs.pythonPackages.sqlalchemy
    pkgs.pythonPackages.scikitlearn
    pkgs.pythonPackages.sympy
    pkgs.pythonPackages.pyqt4
    pkgs.pythonPackages.shapely
    pkgs.pythonPackages.pyshp
    pkgs.pythonPackages.tensorflowWithoutCuda
    pkgs.pythonPackages.gnureadline
    pkgs.pythonPackages.h5py
    pkgs.pythonPackages.pandas
    pkgs.pythonPackages.dask
    pkgs.pythonPackages.scrapy
    pkgs.pythonPackages.matplotlib
    pkgs.pythonPackages.statsmodels
    pkgs.pythonPackages.seaborn
    pkgs.pythonPackages.bokeh
    pkgs.pythonPackages.requests2
    pkgs.pythonPackages.jupyter
    pkgs.pythonPackages.ipython
    pkgs.pythonPackages.cffi
    pkgs.pythonPackages.imageio
    pkgs.theano
];
  ignoreCollisions = true;
}).env

husnoo · 2017-04-11T21:12:13Z

I think this issue was closed by accident - but I don't think I can re-open it. Should I submit a new one? thanks!

husnoo · 2017-04-11T21:21:37Z

I attempted to supply the -I and -L paths at build time, but it didn't help.

  setupPyBuildFlags = ["-I${geos}/include" "-L${geos}/lib"];

FRidh, what's the recommended way to patch the path to geos?

thanks!

FRidh · 2017-04-11T21:26:11Z

No, it wasn't closed by accident. You describe an issue with a package that is not part of Nixpkgs tree. Furthermore, I cannot reproduce your issue. I tried again, with nixos-17.03 @ 69d9061 and again I could not reproduce your issue.

I took the trace and made some fixes, but now another module is needed but missing (shapefile). I'm closing this again until you provide a minimal working example.

husnoo · 2017-04-11T21:54:09Z

Hi, sorry I misunderstood your earlier request. Having deleted most of the libraries above, and then tried it, I had the same problem you mention with shapely, shapefile and six. this example is the smallest I can make it and reproduce the missing geos business:

with import <nixpkgs> {};

(let cartopy = pkgs.pythonPackages.buildPythonPackage rec{
  name = "cartopy-${version}";
  version = "";

  src = pkgs.fetchurl{
  url = "mirror://pypi/c/cartopy/cartopy-0.11.0.tar.gz";
    sha256 = "85eb55d6ce16be5f7e745fcc2870b3fa7450db3a77899daade8b18711532be45";
  };
  doCheck = false;
  setupPyBuildFlags = ["-I${geos}/include" "-L${geos}/lib"];
  propagatedBuildInputs =  [ gcc pkgs.geos proj pythonPackages.numpy  pythonPackages.cffi pythonPackages.cython pythonPackages.shapely pythonPackages.six pythonPackages.pyshp];
};


in python.buildEnv.override {
  extraLibs = [
    cartopy
];
  ignoreCollisions = true;
}).env

[nix-shell:~/nixshellish]$ python
Python 2.7.13 (default, Dec 17 2016, 20:05:07) 
[GCC 5.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cartopy
Failed `CDLL(libgeos_c.so.1)`
>>>

husnoo · 2017-04-11T21:56:57Z

No such problem if I set the LD_LIBRARY_PATH:

[nix-shell:~/nixshellish]$ LD_LIBRARY_PATH=/nix/store/6ln1621k5rmnvc92dmcyf18614py37va-geos-3.6.1/lib/ python
Python 2.7.13 (default, Dec 17 2016, 20:05:07) 
[GCC 5.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cartopy
>>>

husnoo · 2017-04-12T11:47:55Z

is this enough information to reopen please?

FRidh · 2017-04-14T12:50:05Z

Now I can reproduce your issue, thank you.

FRidh · 2017-04-14T13:08:21Z

I think the issue is with Shapely, I seem to be getting errors there with e.g. from shapely.geometry import Point.

FRidh · 2017-04-14T13:22:38Z

Yep, shapely needs a bit more patching. Because it currently has no maintainers I think its up to you to fix it :)

husnoo · 2017-04-14T15:04:00Z

Good catch! Apparently it's already being handled:

[nix-shell:~/data/analysis]$ find /nix/store/ | grep shapely | grep ".py$" | xargs grep libgeos_c.so
/nix/store/644s1aciw6c5mnqsy4l8ls5sca79iin5-python2.7-Shapely-1.5.15/lib/python2.7/site-packages/shapely/geos.py:    _lgeos = load_dll('geos_c', fallbacks=['/nix/store/6ln1621k5rmnvc92dmcyf18614py37va-geos-3.6.1/lib/libgeos_c.so'])
/nix/store/644s1aciw6c5mnqsy4l8ls5sca79iin5-python2.7-Shapely-1.5.15/lib/python2.7/site-packages/shapely/geos.py:        _lgeos = load_dll('geos_c', fallbacks=['/nix/store/6ln1621k5rmnvc92dmcyf18614py37va-geos-3.6.1/lib/libgeos_c.so'])
/nix/store/644s1aciw6c5mnqsy4l8ls5sca79iin5-python2.7-Shapely-1.5.15/lib/python2.7/site-packages/shapely/geos.py:    _lgeos = load_dll('geos_c', fallbacks=['/nix/store/6ln1621k5rmnvc92dmcyf18614py37va-geos-3.6.1/lib/libgeos_c.so'])
/nix/store/644s1aciw6c5mnqsy4l8ls5sca79iin5-python2.7-Shapely-1.5.15/lib/python2.7/site-packages/shapely/geos.py:    _lgeos = load_dll('geos_c', fallbacks=['/nix/store/6ln1621k5rmnvc92dmcyf18614py37va-geos-3.6.1/lib/libgeos_c.so'])
/nix/store/644s1aciw6c5mnqsy4l8ls5sca79iin5-python2.7-Shapely-1.5.15/lib/python2.7/site-packages/shapely/_buildcfg.py:                         fallbacks=['libgeos_c.so.1', 'libgeos_c.so'])
/nix/store/644s1aciw6c5mnqsy4l8ls5sca79iin5-python2.7-Shapely-1.5.15/lib/python2.7/site-packages/shapely/_buildcfg.py:                         fallbacks=['libgeos_c.so.1', 'libgeos_c.so'])
/nix/store/644s1aciw6c5mnqsy4l8ls5sca79iin5-python2.7-Shapely-1.5.15/lib/python2.7/site-packages/shapely/_buildcfg.py:                         fallbacks=['libgeos_c.so.1', 'libgeos_c.so'])

Further testing shows the cartopy library is working (or subtly broken, depending on whether or not I'm exercising the correct path...). So I guess the "Failed" is just sort of a warning, as it's falling back on the correct path:

>>> import shapely
>>> shapely.geos._lgeos._name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'geos'
>>> import shapely.geometry
Failed `CDLL(libgeos_c.so.1)`
>>> shapely.geos._lgeos._name
'/nix/store/6ln1621k5rmnvc92dmcyf18614py37va-geos-3.6.1/lib/libgeos_c.so'
>>>

FRidh · 2017-04-14T17:54:58Z

Ah yes, now I see. Shapely has a function load_dll which loads a shared library. It first tries to find the library name, and then it checks the list of fallback names. Looking at the patch we have in Nixpkgs we pass the full library via the list of fallback names. That's why it fails initially but can be found afterall.

So, that means your issue is solved so now I can close this :)

husnoo · 2017-04-14T19:48:05Z

Yes, thank you :-)

…

On 14 April 2017 at 18:55, Frederik Rietdijk ***@***.***> wrote: Ah yes, now I see. Shapely has a function load_dll which loads a shared library. It first tries to find the library name, and then it checks the list of fallback names. Looking at the patch we have in Nixpkgs we pass the full library via the list of fallback names. That's why it fails initially but can be found afterall. So, that means your issue *is* solved so now I can close this :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24805 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACybSjzori7zGPzsLb8M9gnVUV2MhXtYks5rv7L1gaJpZM4M5VYU> .

FRidh closed this as completed Apr 11, 2017

FRidh added the 6.topic: python label Apr 11, 2017

FRidh changed the title ~~propagatedBuildInputs not respected for python runtime dependencies~~ cartopy cannot find geos library Apr 11, 2017

FRidh reopened this Apr 11, 2017

FRidh closed this as completed Apr 11, 2017

FRidh reopened this Apr 14, 2017

FRidh closed this as completed Apr 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cartopy cannot find geos library #24805

cartopy cannot find geos library #24805

husnoo commented Apr 10, 2017

husnoo commented Apr 10, 2017

FRidh commented Apr 11, 2017

FRidh commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 11, 2017

FRidh commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 12, 2017

FRidh commented Apr 14, 2017

FRidh commented Apr 14, 2017

FRidh commented Apr 14, 2017

husnoo commented Apr 14, 2017

FRidh commented Apr 14, 2017

husnoo commented Apr 14, 2017 via email

cartopy cannot find geos library #24805

cartopy cannot find geos library #24805

Comments

husnoo commented Apr 10, 2017

Issue description

Steps to reproduce

Technical details

husnoo commented Apr 10, 2017

FRidh commented Apr 11, 2017

FRidh commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 11, 2017

FRidh commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 11, 2017

husnoo commented Apr 12, 2017

FRidh commented Apr 14, 2017

FRidh commented Apr 14, 2017

FRidh commented Apr 14, 2017

husnoo commented Apr 14, 2017

FRidh commented Apr 14, 2017

husnoo commented Apr 14, 2017 via email