System tests: fix hang

[Tom-Willemsen]

Where?

As a developer I would like the system_tests and system_tests_debug builds on the build server not to hang and therefore hit timeouts.

How to Test

verbose instructions for reviewer to test changes
(Add before making a PR)

[Tom-Willemsen]

Hang is in python interpreter exit, specifically here: https://github.com/python/cpython/blob/3.10/Python/pylifecycle.c#L2017. This makes it quite tricky to debug using standard python tooling, as if you connect a debugger the bug will move to debugger exit rather than program exit, and the hang is not in python code anyway. I had more success connecting the VS debugger to python (with appropriate symbols loaded) to get to the bottom of this.

We have a deadlock due to a subtle behaviour change in python 3.9+; before python 3.9. a ThreadPoolExecutor would join all of it's threads using an atexit hook. Because atexit hooks are run in LIFO order, as long as we added an exit hook for stopping infinite processes after importing threading (which was the case), our hook would run first, and this meant we could set a flag to stop the thread. Now in python 3.9+, there is a separate threading._register_atexit hook that is run before all atexit hooks, and this is now where ThreadPoolExecutor attempts to join it's worker thread(s). So we now have a deadlock because python won't run our atexit hook until all threads are stopped, but the thread won't stop until we run our atexit hook.