FIX: Race conditions in test_cli.py when spawning test server processes

[reneme]

Pull Request Dependencies

• FIX: potential use-after-free in ./botan tls_proxy #4177

Description

The test_cli.py script handles and orchestrates a few long-running processes (namely ./botan tls_server, ./botan tls_client, ./botan tls_proxy). For those tests to work properly, we have to ensure that the server processes are ready before trying to interact with them. Especially on the CI where runtime behavior is notoriously unpredictable. Until now, this was done by invoking time.sleep(1) after launching a server process. This worked most of the time, but introduced unnecessary waiting time while still being potentially racey.

This now uses Python's asyncio module to handle such long-running processes and interact with them. Each CLI server now reports a "Listening for new connections..." string on stdout once its ready to receive connections. The asyncio-based wrapper can wait for this string and thus ensure that the server is ready before the test continues. Also, the wrapper makes sure to timeout on any of those waits (hard-coded 15 seconds) and kill the process if necessary.

Result

Such tests can now rely on their server processes being in a defined state, without sleeping some arbitrary time and hoping for the best. As a side effect, running ./test_cli.py --threads=4 now takes 2.5 seconds on my laptop, instead of the more than 20 seconds it used to take due to the time.sleep() workarounds.

Fixes #4112

[coveralls]

coverage: 91.729% (-0.002%) from 91.731%
when pulling 3294602 on Rohde-Schwarz:fix/race_conditions_in_test_cli
into f4c26e4 on randombit:master.

[randombit]

Very nice work!

The pylint failures in CI look relevant. If there is anything it's warning about that isn't sensible to change, feel free to locally disable the warning.

[reneme]

In the first run macOS 14 (on Apple Silicon) failed obscurely:

     [...]
     INFO: Ran cli_zfec_tests in 0.06 sec
     INFO: Ran cli_trust_root_tests in 0.54 sec
     INFO: Ran cli_tls_proxy_tests in 5.14 sec
  Ran 234 tests with 0 failures in 6.40 seconds
  Command 'python3 ./src/scripts/test_cli.py --threads=3 ./botan' failed with error code -11

.... looks like the interpreter crashed after the script had finished. 😨
This didn't show up in a second CI run, but it is certainly worrisome. Obviously I cannot reproduce on my MacBook Air...

I'd suggest to just merge and see whether that comes back. Perhaps we'll end up having to update the python interpreter in the build job setup or disable those tests on macOS 14 for some time. 😒

[coveralls]

coverage: 91.723% (+0.001%) from 91.722%
when pulling 934f3fe on Rohde-Schwarz:fix/race_conditions_in_test_cli
into a75bb25 on randombit:master.

[randombit]

I think the macOS failure is #3991 so unrelated to this change.

[reneme]

Let's hope so. In it goes!

[randombit]

Couple of Windows builds in the merge are failing, maybe we need more than 15 seconds in practice on CI?