use of tweetnacl is considerably slower than libsodium

[chrisjbillington]

Tweetnacl seems to result in considerably slower encryption than libsodium (about a factor of 10 - see linked issue for an example with pyzmq). I've been prodding packagers to try and get libsodium linked as required in packages when used by pyzmq, but for both the pip wheels and Anaconda package this looks unlikely to happen on Windows, and I'm close to giving up on being able to get fast crypto for my Windows users.

One hope I have is that it's possible that tweetnacl might not be fundamentally slow, and that there might be something about the way zeromq uses it that makes things slow. Pieter said here that the change to tweetnacl would not change anything significant in terms of performance, but that appears not to be true, uness I'm missing something. It would be great if tweetnacl could be almost as fast as libsodium, as it's certainly easier for the packagers.

I've only been able to test that tweetnacl is slow with zeromq 4.1.6, as that is the version bundled with pyzmq, and I haven't managed to build pyzmq against other versions myself (or to make a test case in C/C++).

[sigiesec]

This mainly addresses tweetnacl rather than libzmq. Libzmq merely packages tweetnacl. Apart from that, tweetnacl is inherently slower than libsodium. Tweetnacl is small and portable, while libsodium contains many optimizations for different platforms.

I am not sure, but doesn't pyzmq use a dynamic build of libzmq, i.e. a DLL? Then you can exchange the packaged DLL by another built against libsodium.

[evoskuil]

The Visual Studio builds include the option to link libsodium.

[bluca]

Nobody should use tweetnacl in production. Speed is the last of its problems. It has not been audited, it's never updated. Libsodium on the other hand it's maintained, audited, gets security updates and is highly optimised with hardware acceleration support. Tell the packagers to switch to libsodium ASAP before they end up with a security nightmare on their hands.

I'm beginning to think it was not a good idea to embed tweetnacl...

Anyway, we are definitely not going to mess with that code, there is no D.J. Berstein-level of cryptographer among us afaik so it would be a disaster.

[chrisjbillington]

I can do whatever I can with my own packages, but my users are going to be installing pyzmq via either conda or pip, and it's not really within my ability to build and maintain binary packages for all platforms to ensure all my users have good crypto. I was not aware that tweetnacl was not considered production worthy (other than the speed thing), and would agree that something not production-worthy probably shouldn't be included by default.

It seems from the outside like a constant battle for the package maintainers to be able to build everything on Windows. Anaconda won't build pyzmq linked with anaconda's zeromq—built with libsodium—because completely unrelated R packages in the same package manager need zeromq to be built with gcc, which is presumably not ok with pyzmq, I'm not sure why, and pyzmq used to bundle libsodium (not sure if it built on Windows), but they stopped doing so, because "libsodium has build troubles on Windows", so I can see why tweetnacl was an attractive alternative.

I'll keep prodding package maintainers for now...

[bluca]

Sorry for the hassle. But as I said we don't have cryptographers working with us so we have to thread extremely carefully around that piece of code.

The production problem is that it's not audited and there's no maintenance support for it, unlike for libsodium, so it's extremely risky to blindly trust its crypto primitives.

If that can help, libzmq and pyzmq are built with GCC on all Linux distros - so not sure what the issue is there, but quite sure it's not to do with us.

[chrisjbillington]

No problem! Thanks for all your efforts. There's no remaining issue on linux for me (conda pyzmq packages use libsodium as of 2 weeks ago, next pip wheel for linux will be built with libsodium), but my users mostly use Windows.

Just for some context, I make open source physics lab control software, so Windows is pretty prevalent. The software has no security at the moment, and because the Python programs exchange pickled objects, is unsafe unless the network is trusted. So far it is only being run on trusted networks but we would like to make this not a requirement to use it.

For my use case, I would have been happy being able to plug in a different crypto library like the python cryptography library, but it is not trivial to do this at the application level with zeromq because some socket types are one-way so you can't do the required handshaking, and also can't get at some of the additional frames and other data that is sent on the wire by zeromq that isn't exposed to the application. I thought about proxying every connection through a RAW socket to do the handshaking and whatever, but that would require delving into some pretty low level stuff, so I decided to go with CurveZMQ.

If zeromq were to allow you to add callbacks for new connections (to do handshaking) and for sends and receives of wire-level data (to do encryption/decryption and message authentication), then I would be able to secure my software without libsodium. The new connection callback is already there as ZMQ_EVENT_CONNECTED I believe, so it might work to read and write from the raw file descriptor to do a handshake, but I'm not sure since zeromq threads might be reading from it at the same time. For my application I would have used symmetric crypto with a preshared key, since I don't care to distinguish between peers, and so I am already using CurveZMQ in a nonstandard way to achieve something like this. So pluggable crypto would make this a bit more flexible for me. Perhaps I am misunderstanding the zmq authentication system, but I understand there is pluggable authentication, but not pluggable crypto in general. I understand that the more moving parts crypto has the more likely a non expert is to get something wrong, and so pluggable crypto is a liability, but there are upsides too.

[bluca]

That sounds quite complicated - I would suggest to just focus on getting that Windows binary rebuilt with libsodium. It's supported and it works out of the box - not sure why that Windows package system doesn't use it. Drop them and find an alternative maybe?

[mingwandroid]

It's supported and it works out of the box

Drop them and find an alternative maybe?

@bluca, for the record, the history here is that zeromq has been in a very bad state (unusable due to race conditions) on Windows since at least last September when I first tried to use it with pyzmq and for that reason I was forced to not use it here, and not using zeromq means not using libsodium and instead having to use the bundled crypto implementation in pyzmq instead.

It's only now that 4.2.4 has been released that it works again thanks to the efforts of @sigiesec