make pyzmq pip-installable without libzmq present

[minrk]

pyzmq-static is installable via pip or easy_install, even in the absence of libzmq, because it builds libzmq as a Python extension. pyzmq should do the same thing, with a few small changes:

1. don't bundle libzmq unless it's not found
2. don't bundle the libzmq source - only include the basic platform_x headers that would be necessary to build without running configure, and download the sources only as needed.
3. build libzmq properly (configure && make, etc.), rather than as a Python extension*.

*Possibly, but not necessarily [3]. The Python extension route would be much easier.

[onetalltree]

Minrk--

I am, once again, fighting my way through installing pyzmq onto a linux (centos) box. I had to refer back to an issue I raised two months ago and try to reconstruct the process that eventually led to a successful installation.

But perhaps you could explain why it's so difficult in the first place to install the python zmq binding onto a linux box. I would have thought a simple sudo easy_install pyzmq would do it, and I'm surprised that it doesn't work. Are you guys all using windows machines, or am I doing something stupid, or what?

Thanks!

--eric

[minrk]

ha, no, we don't use Windows on purpose ever. easy_install pyzmq has worked for me on every Linux box I have tried, and I do it almost daily on Ubuntu machines. I don't know what's weird about centos, but I certainly haven't used it.

[onetalltree]

Glad to hear you're not using windows. I've had to build pyzmq from source on both my Unbuntu and Centos machines. Both of them are 64-bit ... would that make a difference?

[guidog]

Using Debian on 32 and 64 bit architectures and installing pyzmq inside and outside of virtualenv. No problems so far.

Given the enormous amount of information available I'd say it's not a pyzmq issue and not an issue on most Linux distros.

But I've seen problems w/ Centos a lot on IRC. Most of the problems were related to the ancient python version used.

Cheers
Guido

[onetalltree]

Does Python 2.6.6 count as ancient?

[minrk]

No, 2.6.6 is not ancient, and my most often used system is Ubuntu 10.04 LTS with 2.6.5. The following script works to build&install zeromq and pyzmq in this environment:

wget http://download.zeromq.org/zeromq-2.1.11.tar.gz
tar -xzf zeromq-2.1.11.tar.gz
cd zeromq-2.1.11
./configure && make && sudo make install
sudo ldconfig

sudo easy_install pyzmq

If you have some constructive information about what errors you actually face, and what needs to be fixed to build reliably on CentOS would be much appreciated.

[onetalltree]

I don't know if this is constructive information, but I just spent a bunch of time making sure I could reliably replicate the problem. I can replicate this any time I want to by activating a virtualenv and then trying to run either easy_install or building from source. So, if I try to use easy_install I get the same error as when I do:

python setup.py configure

namely:

Error running version detection script:
detect/vers: error while loading shared libraries: libzmq.so.1: cannot open shared object file: No such file or directory

If I do this:

python setup.py configure --zmq=/usr/local

then it works. And once it works, I can just do:

python setup.py configure

which will then start working.

[minrk]

What if you run ldconfig prior to configuring libzmq?

[onetalltree]

Do you mean "run ldconfig prior to configuring pyzmq"?

I've tried it many many times! It still doesn't work. I have to go:

python setup.py configure --zmq=/usr/local

[minrk]

Do you have to use a virtualenv in order to cause this to fail? If so, then virtualenv is probably patching the link path, to explicitly prevent searching /usr/local by default. If that's the case, then presumably the 'right way' would seem to be to install libzmq into the virtualenv.

[onetalltree]

The error occurs regardless of whether or not I use virtualenv

[onetalltree]

I created a completely fresh Centos 6.1 install in a virtual machine. Here are the commands I gave.

sudo yum groupinstall 'Development Tools'
sudo yum install python-devel
sudo yum install libuuid-devel

wget http://download.zeromq.org/zeromq-2.1.11.tar.gz
gunzip zeromq-2.1.11.tar.gz
tar -xf zeromq-2.1.11.tar
cd zeromq-2.1.11
./configure
make
sudo make install
sudo ldconfig

wget https://github.com/zeromq/pyzmq/downloads/pyzmq-2.1.11.tar.gz
gunzip pyzmq-2.1.11.tar.gz
tar -xf pyzmq-2.1.11.tar
cd pyzmq-2.1.11
python setup.py configure --zmq=/usr/local
python setup.py build
sudo python setup.py install

The --zmq=/usr/local is required in order to get the

python setup.py configure --zmq=/usr/local

command to work.

[minrk]

That suggests to me that CentOS doesn't include /usr/local/lib in the link path by default. Is this true? If so, I can make the default behavior on centos be to use /usr/local instead of trusting the linker. The reason I don't do this in general is that if users have their env configured such that the linker will find libzmq without being specified a path (not uncommon), defaulting to /usr/local will fail.

What do you get trying to compile a trivial c program with -lzmq? Does it find libzmq?

[minrk]

Actually, what I will try to do first is to see if I can't fall back on /usr/local if the first check fails. That should cover both cases.

[minrk]

I just setup a CentOS 6.2 VM, and see exactly what you do. It appears that on CentOS /usr/local/lib is not on the default LD path, so files linked there will not be found unless you add export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH. This seems extremely weird to me, and a very bad choice, but it explains what we are seeing.

[onetalltree]

Cool. BTW, I just had to test it in Ubuntu 11.10 and after I've built zmq, easy_install pyzmq works fine.

[minrk]

@ejb11235 PR #184 introduces a fallback to /usr/local if the first build fails, which makes easy_install work for me without having to specify that libzmq is in /usr/local. It still seems extraordinarily stupid that CentOS has made this choice (their LD config is internally inconsistent, because the linker and runtime library paths are not the same).