Automate port forwarding

[annawoodard]

Idea from @benclifford, some facilities do not have outbound connectivity on the compute nodes, which makes running from an external location impossible. We could get around this by forwarding a network connection from the engine to the external controller through e.g. the login node.

Needed by @mgovoni-devel.

[mgovoni-devel]

The fact that executors cannot be set remotely on machines that don’t have network connectivity available from the computing notes weakens the overall scope of Parsl. Most of the big machines have compute nodes that don’t see the outside network, and so would fall in this category. I strongly suggest to prioritize the solution of this issue (if possible). Thanks.

[annawoodard]

Here's an example script for doing this we could use for inspiration: https://github.com/Twinklebear/theta-tunnel

[yadudoc]

I've done a quick manual test to confirm that this issue can be handled with ssh tunnels. Here's what I've done specifically for theta.alcf which restricts external network access from the compute nodes:

1. Start ipcontroller on laptop with static ip.
2. Log onto thetalogin6
3. Start a series of 7 tunnels for each of the ports used by the engines to connect to the controller
   laptop:port <======> thetalogin6:port
4. Start an interactive session
5. Now edit the ipcontroller-engine.json file, replacing the static ip of the laptop with the address of
   thetalogin6.
6. Start the engine.
   laptop:port <===TUNNEL====> thetalogin6:port <---0mq---- worker:port

To automate this process, we'll need to add a new operation at the init phase to start the tunnels and add an override to specify location to point to executor side end of the tunnel.

[WardLT]

Reviving this old topic. I just went through an exercise in SSH tunnels that seemed very automatable.

Will document the effort with a simple repo to at least show people how such a system could be built.

[benclifford]

pleases see issue #3515 about removing channels from Parsl