Protect HTEX communication with CurveZMQ

[rjmello]

Description

The crux of the implementation revolves around two new classes: ServerContext and ClientContext. These replace the standard zmq.Context and share many commonly used methods, including term, destroy and, most importantly, socket. The latter applies the necessary certificates and options to each socket object.

A connection requires a ServerContext on one end (CurveZMQ server), which validates clients, and a ClientContext on the other (CurveZMQ client), which validates the server.

E.g.,

import zmq
from parsl import curvezmq

cert_dir = curvezmq.create_certificates(base_dir=".")

client_ctx = curvezmq.ClientContext(cert_dir)
client_socket = client_ctx.socket(zmq.PUSH)

server_ctx = curvezmq.ServerContext(cert_dir)
server_socket = server_ctx.socket(zmq.PULL)

port = server_socket.bind_to_random_port("tcp://127.0.0.1")
client_socket.connect(f"tcp://127.0.0.1:{port}")

client_socket.send(b"hola")
msg = server_socket.recv()
print(msg)

The interchange serves as a CurveZMQ server, while the executor and various managers serve as CurveZMQ clients. Thus, when enabled, all communication channels between these entities are encrypted.

The HTEX start method creates new certificates for each run in a private certificates/ directory. We generate a single shared client certificate because all clients will have access to this directory.

Users can enable encryption by setting the encrypted initialization argument for the HighThroughputExecutor to True. We disable encryption by default because, depending on the pyzmq installation path, it can cause a significant impact on throughput performance. I've included some recommended installation paths to address these issues in the docs (e.g., install via conda).

Fixes #2199
Partially addresses #952

Type of change

• New feature

[khk-globus]

Some minor comments from a static analysis so far, but I'm still poking at the functionality locally. I'll tentatively say "looking good" at the moment.

[rjmello]

We've decided to disable encryption by default because, depending on the pyzmq installation path, it can cause a significant impact on throughput performance. I've included some recommended installation paths to address these issues in the docs (e.g., install via conda).