Adding support for IPV6 addresses

[yadudoc]

Description

This PR adds a new option: HighThoughputExecutor(loopback_address: str = "127.0.0.1") which can be used to specify the internal address used by HTEX for communication between the executor and the interchange. In addition, all ZMQ sockets are now are set to having IPv6 enabled.

The test config htex_local has been updated to use loopback_address="::1" for testing.

Changed Behaviour

• IPv6 support is enabled on all HTEX ZMQ components.
• HTEX now supports a loopback_address which allows configuring the address used for internal communication

Fixes # (issue)

Type of change

Choose which options apply, and delete the ones which do not apply.

• New feature
• Update to human readable text: Documentation/error messages/comments

[benclifford]

is this always exactly the string "127.0.0.1" or the string "::ffff:127.0.0.1"?

[yadudoc]

In all my testing it is.

[benclifford]

in my ipv6 only environment (without even a 127.0.0.1 loopback address), the default address detection detects no addresses and gives this error:

        Job is marked as MISSING since the workers failed to register to the executor. Check the stdout/stderr logs in the submit_scripts directory for more debug information
        EXIT CODE: 2
        STDERR: /home/benc/parsl/virtualenv-3.12/bin/process_worker_pool.py:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('parsl==2024.11.11+desc.2024.11.14a')
usage: process_worker_pool.py [-h] [-d] [-a ADDRESSES] --cert_dir CERT_DIR
                              [-l LOGDIR] [-u UID] [-b BLOCK_ID]
                              [-c CORES_PER_WORKER] [-m MEM_PER_WORKER] -t
                              TASK_PORT
                              [--max_workers_per_node MAX_WORKERS_PER_NODE]
                              [-p PREFETCH_CAPACITY] [--hb_period HB_PERIOD]
                              [--hb_threshold HB_THRESHOLD]
                              [--drain_period DRAIN_PERIOD]
                              [--address_probe_timeout ADDRESS_PROBE_TIMEOUT]
                              [--poll POLL] -r RESULT_PORT --cpu-affinity
                              CPU_AFFINITY
                              [--available-accelerators [AVAILABLE_ACCELERATORS ...]]
                              [--enable_mpi_mode]
                              [--mpi-launcher {srun,aprun,mpiexec}]
process_worker_pool.py: error: argument -a/--addresses: expected one argument

(I wanted to force that the workers would not be connecting over 127.0.0.1) -- i'll try this test in a different way but I think the address detection will still fail to detect the IPv6 addresses in address auto-detection.

[benclifford]

I didn't try very hard to make it work, but I couldn't get scoped addresses (https://en.wikipedia.org/wiki/IPv6_address#zone_index) to work. I think this probably isn't a big deal, but reporting it here anyway in case you want to document/investigate it more.

[LeiGlobus]

I didn't try very hard to make it work, but I couldn't get scoped addresses (https://en.wikipedia.org/wiki/IPv6_address#zone_index) to work. I think this probably isn't a big deal, but reporting it here anyway in case you want to document/investigate it more.

I think our goal is to get some basic variation of ipv6 addressing to work, not '100% support of ipv6', so that should be ok.

In other news, looking at socket.py in zmq, it just does "%s:*" and f'{addr}:{port}', which isn't confidence boosting, making me think I should bracket ipv6 addr to be safe. However, I'll test both with and without brackets (from the compute side), to possibly eliminate one of the two options (bracket or not to bracket).

Assuming both work, I'll go with bracketing to be safe, as that's the 'standard' and should be future-proof.