HICAT-994 Device servers using Python shared memory managers

[jamienoss]

Upstream PR for https://github.com/spacetelescope/hicat-package/pull/607.

See catkit/testbed/README.md

Changes to existing key features:

• Infrastructure code has been factored out of Experiment into catkit.testbed.experiment.Testbed. This is mainly just the safety tests, however, the hicat derivation will also include ownership of all of the remote servers, i.e., the exception-handler, all device servers, the testbed state server, and the sim server.
• Experiment.start() (that on the parent process) doesn't run the safety checks, these are now run on a child process by Experiment.Testbed.
• Synchronization between processes, especially for halting upon a safety exception, is now implemented using events.
• For concurrency to work, nothing should ideally be run on the parent process such that a call to Experiment.start() doesn't block the start of another experiment.
• Experiment.start() doesn't wait and block until the experiment completes, it returns just as Process.start() does.
• Experiment.join() has been added and mimics Process.join().
• Experiment.run_experiment() (that run on the child process) monitors synchronization events from a dedicated monitor thread. This monitor thread will interrupt the main thread (that running Experiment.experiment()) upon certain events.
• The outer script pattern is now:

with Testbed():
  experiment.start()
  experiment.join()

• For concurrent usage, the out script pattern is now:

with Testbed():
  with experiment1:
    with experiment2:
      pass  # experment2.join() will then be called.

• In the case that the desired start and wait/join order prevents the use of nested context managers, multiple try-finally blocks must be implemented such that all joins are executed in the correct finally statement.

[jamienoss]

Kept as an example.

[ehpor]

Aside from the (trivial) merge conflict during the merge of #240, I don't see much wrong with this PR. I approve this PR for merging, and will not do detailed notes since that's totally uninteresting for everyone.

Some general comments.

• A lot of the code in this PR is very advanced, to the point that maintenance will likely (read: certainly) become an issue in the future. However, I feel that this is to some degree unavoidable with the current state of catkit.
• The mutex patterns are still unclear to me. This is the main concern for me for future unexpected bugs (from a user perspective at least).
• I really like the new Testbed class, making it do the safety checks rather than an experiment. Great separation of concerns.
• Thanks for the slight refactor of the data log into a context manager.
• Documentation is a bit sparse. Most internal comments are not accessible to a non-expert, and I presume even an expert that is reading it for the first time. However, catkit/testbed/README.md is a good start for people wanting to learn about the internals of catkit multiprocessing. Especially the example in "Hardware access full stack flow" is a good first step.
• The signal/events for experiment lifetime and safety are very well structured. See also the arguments of Experiment.__init__() for a further description.

[ehpor]

No enum? No need to change this; just curious.

[jamienoss]

🤣 Touché

[jamienoss]

I'll punt this to another PR.

[jamienoss]

@ehpor Just for the sake of it, would you mind re-approving this PR, please? Note that it's just the upstream hicat CI that has failed, as expected.

[ehpor]

Reapproved.