zeromq should depend on libsodium

[minrk]

The zeromq package should be built against libsodium to provide encryption and authentication support. The version in conda-recipes does this, except on Windows.

[msarahan]

Agreed. I tried to get this working for the 2.4 release, but it didn't quite make it. We will have this shortly on all platforms - Windows included.

[minrk]

Awesome, thanks.

[msarahan]

@minrk how do you build/obtain libsodium for VS2008 (Py2.7 support?) It seems like they even omit VS2010 in their latest prebuilt distributions. Are they using C++11 stuff that requires VS2012+? If so, then we'll only be able to support libsodium for Py3.5+, or alternatively, pin an older version of libsodium. Thoughts on what might be most attractive?

[minrk]

Yikes. I don't know. When I build libsodium with pyzmq on Windows, I let Python compile it as an Extension rather than using VS to build it as a standalone library. My Windows-building expertise is lacking.

[msarahan]

Thanks - I think there are no issues with compiling with VS2008. The upstream guys probably just don't want to deal with supporting old build solutions. What's the state of pyzmq 15.0.0? Should I bundle that, or stick with 14.7? What does that major version bump mean, and does it break anything I should be wary of?

[minrk]

The main 'breaking' change in 15 is that it changes how it builds when old libzmq is found on the system, which won't affect conda. The Python API is stable.

[nehaljwani]

FWIW, I have a PR to add Win support to libsodium at conda-forge/libsodium-feedstock#7

[mingwandroid]

Do you mean "on Windows" here? On Linux it does at least depend on and install both zeromq and libsodium:

conda create -n pyzmq pyzmq
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/pyzmq

  added / updated specs:
    - pyzmq


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libsodium-1.0.15           |       hf101ebd_0         313 KB
    pyzmq-16.0.3               |   py36he2533c7_0         430 KB
    zeromq-4.2.2               |       hbedb6e5_2         662 KB
    ------------------------------------------------------------
                                           Total:         1.4 MB

The following NEW packages will be INSTALLED:

    ca-certificates: 2017.08.26-h1d4fec5_0
    certifi:         2018.1.18-py36_0
    libedit:         3.1-heed3624_0
    libffi:          3.2.1-hd88cf55_4
    libgcc-ng:       7.2.0-h7cc24e2_2
    libsodium:       1.0.15-hf101ebd_0
    libstdcxx-ng:    7.2.0-h7a57d05_2
    ncurses:         6.0-h9df7e31_2
    openssl:         1.0.2n-hb7f436b_0
    pip:             9.0.1-py36_5
    python:          3.6.4-hc3d631a_1
    pyzmq:           16.0.3-py36he2533c7_0
    readline:        7.0-ha6073c6_4
    setuptools:      38.5.1-py36_0
    sqlite:          3.22.0-h1bed415_0
    tk:              8.6.7-hc745277_3
    wheel:           0.30.0-py36hfd4bba0_1
    xz:              5.2.3-h55aa19d_2
    zeromq:          4.2.2-hbedb6e5_2
    zlib:            1.2.11-ha838bed_2

Proceed ([y]/n)?

I will update after some more tests.

[mingwandroid]

libzmq.so gets loaded, but libsodium.so does not. I believe there's still an issue with the build of zeromq here.

[chrisjbillington]

@migwandroid, I'll check again, maybe I accidentally used pip. Also, I deleted my comment since I thought I had posted it in the wrong project, but since you replied, here it is again for anyone else reading:

conda install pyzmq on linux doesn't pull in zeromq or libsodium, it appears to be using the version of zeromq bundled with pyzmq (which pyzmq uses only if it can't find libzmq at compile time), which pyzmq compiles without libsodium.

This results in slow encryption in pyzmq because the bundled tweetnacl library that comes with zmq is much slower than libsodium.

Encryption would be much faster if the conda pyzmq package was compiled against the conda zeromq package, which itself is compiled with the conda libsodium package, instead of using pyzmq's bundling method. Pyzmq docs recommend compiling against system zeromq, they only provide the bundling mechanism for convenience.

[mingwandroid]

@jjhelmus has recently uploaded new builds of zeromq, it would be good if you can test them. This is the correct place and issue at which to report this.

[jjhelmus]

The version 17.0.0 pyzmq packages do pull in libsodium and zeromq and the Python libraries link against libzeromq. Although installing, I'm not seeing libsodium being loaded which seems suspect.

[mingwandroid]

Link to the original report (with a testcase)

[chrisjbillington]

My mistake, looks like I got the dependencies wrong, pyzmq does depend on zeromq and libsodium.

But yes, as in the pyzmq bug report, the original observation was that conda pyzmq has observably slow crypto compared to the Ubuntu repo's pyzmq, indicating libsodium not being used, and then I leapt to conclusions about pyzmq bundling.

Confirmed with the testcase from the pyzmq report that crypto is slow with latest conda zeromq and pyzmq (6.5 seconds compared to about 0.5 seconds on my computer with system pyzmq):

Python Version: 3.6.3 |Anaconda custom (64-bit)| (default, Nov  3 2017, 19:19:16)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 17.0.0
zmq version: 4.2.3
time to encrypt and send 100MB: 6.546231508255005 s

[jjhelmus]

@mingwandroid and myself noticed an incorrect argument in the recipe we were using to build zeromq which resulted in zeromq not using libsodium. New build are on their way that should fix this.

[mingwandroid]

I believe we've fixed this in the recipe now. Package builds should be available soon, hopefully.

[mingwandroid]

Oops, seems our messages crossed each other!

[jjhelmus]

zeromq packages which use libsodium for crypto are now available. These are build _3 packages which should be installed if you run conda update zeromq or create a new environment. The timing on my system are:

Python Version: 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 17.0.0
zmq version: 4.2.3
time to encrypt and send 100MB: 0.7645316123962402 s

A significant improvements over the previous build of zeromq:

Python Version: 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 17.0.0
zmq version: 4.2.3
time to encrypt and send 100MB: 9.815795421600342 s

as well as the 4.2.2 version of the zeromq:

Python Version: 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19)  | [GCC 7.2.0]
Platform: linux
pyzmq version: 16.0.3
zmq version: 4.2.2
time to encrypt and send 100MB: 3.7638325691223145 s

[chrisjbillington]

This has regressed, I am once again seeing slow crypto in pyzmq as shipped by Anaconda in Windows. It looks like the zeromq package is no longer being built with libsodium on Windows as of 4.3.2. Here is a test script that prints dlls from the <prefix>\Library directory loaded into the process when pyzmq has been imported. It only prints libzmq.

import os
import psutil
import zmq
for dll in psutil.Process(os.getpid()).memory_maps():
    if 'Library' in dll.path:
        print(dll.path)

(base) C:\Users\bilbo\Documents\Scratch>python libcheck.py
pyzmq version: 19.0.1
zmq version: 4.3.2
C:\Users\bilbo\miniconda3\Library\bin\libzmq-mt-4_3_2.dll

But after downgrading to zeromq 4.3.1 libsodium is listed as well:

click to see output of downgrade command

(base) C:\Users\bilbo\Documents\Scratch>conda install zeromq==4.3.1
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: C:\Users\bilbo\miniconda3

  added / updated specs:
    - zeromq==4.3.1


The following packages will be DOWNGRADED:

  libsodium                               1.0.18-h62dcd97_0 --> 1.0.16-h9d3ae62_0
  pyzmq                               19.0.1-py38ha925a31_1 --> 18.1.1-py38ha925a31_0
  zeromq                                   4.3.2-ha925a31_2 --> 4.3.1-h33f27b4_3


Proceed ([y]/n)?

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
## Package Plan ##

  environment location: C:\Users\bilbo\miniconda3

  added / updated specs:
    - zeromq==4.3.1


The following packages will be DOWNGRADED:

  libsodium                               1.0.18-h62dcd97_0 --> 1.0.16-h9d3ae62_0
  pyzmq                               19.0.1-py38ha925a31_1 --> 18.1.1-py38ha925a31_0
  zeromq                                   4.3.2-ha925a31_2 --> 4.3.1-h33f27b4_3


Proceed ([y]/n)?

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

(base) C:\Users\bilbo\Documents\Scratch>python zmq_crypto.py
Python Version: 3.8.3 (default, May 19 2020, 06:50:17) [MSC v.1916 64 bit (AMD64)]
Platform: win32
pyzmq version: 18.1.1
zmq version: 4.3.1
time to encrypt and send 100MB: 0.8667018413543701 s

(base) C:\Users\bilbo\Documents\Scratch>python 18.py
pyzmq version: 18.1.1
zmq version: 4.3.1
C:\Users\bilbo\miniconda3\Library\bin\libsodium.dll
C:\Users\bilbo\miniconda3\Library\bin\libzmq-mt-4_3_1.dll

And this lines up with the observed crypto speed, being slower when lisodium is not loaded.